🎩🐪📡 Mediabot v3: We Held On to the Sorting Hat — and the Fresh Install Found Three Hidden Passages (MB380–MB382)

in Mediabot · started by TeuK · 1mo ago

TeuK · 1mo ago

The previous chapter ended with a warning:

Hold on to your Sorting Hat.

So we did.

The rebuilt configure system was taken out of its test chamber and placed on a genuinely fresh Debian 13 installation.

That real installation did exactly what a serious fresh-install test is supposed to do: it validated the large design, then exposed the remaining assumptions that unit tests had not yet forced into daylight.

The result was not a retreat from the new configurator.

It was the final integration work required to make the whole journey real:

MB380 made the CPAN installer independent of the directory from which it is launched;
MB381 separated secret-safe permissions from runtime-readable Perl module permissions;
MB382 prevented status from flooding IRC and repaired message argument handling across the IRC callbacks.

No Perl modules were replaced with Debian packages.

No database schema was changed.

The fresh installation remained a CPAN-only Mediabot installation.

🧭🐪 MB380 — The CPAN camel finally knows where its own tent is

The first fresh attempt reached the dependency stage successfully.

The database had been created.

The application account had been created.

The complete configuration had been generated and audited.

Then the first missing Perl module appeared:

IO::Async::Loop

The installer failed immediately.

At first glance, this looked like a CPAN or Debian 13 problem.

It was neither.

The hidden dependency on the current directory

Before the configure revolution, the root wizard entered the installation directory before launching the CPAN script:

cd install
sudo ./cpan_install.sh

Inside cpan_install.sh, the module helper was called using a relative path:

./install_perl_module.sh

That worked only because the caller had already moved into install/.

MB378 modernised the root workflow and launched the script through its real path from the repository root:

sudo /home/mediabot/mediabot_v3/install/cpan_install.sh

But the old helper lookup still depended on the caller’s directory.

The first modules already present appeared healthy.

The first missing module activated the broken path.

CPAN had not failed.

The installer had simply looked for its camel in the wrong tent.

MB380 anchors every internal path

The installer now resolves its own directory:

SCRIPT_DIR="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)"

The helper and both logs are derived from that directory:

INSTALL_HELPER="$SCRIPT_DIR/install_perl_module.sh"
SCRIPT_LOGFILE="$SCRIPT_DIR/cpan_install.log"
CPAN_LOGFILE="$SCRIPT_DIR/cpan_install_details.log"

It also enters its own directory before continuing.

The installer now behaves identically when launched:

from install/
from the repository root
through an absolute path
through sudo

Still CPAN, and only CPAN

The module installation path remains:

cpan -T -i Module::Name

No apt, dpkg, dnf, yum or system Perl package fallback was added.

The fresh wizard also checks the real CPAN toolchain before touching the database:

perl
cpan
make
gcc
mysql
sudo
mktemp
find
curl or wget

If the build environment is incomplete, the installation now stops before creating half of a system.

The camel knows its tent, its logs and its helper — regardless of where the wizard is standing.

🔐📚 MB381 — The Restricted Section was locked so well that Mediabot could not read it

With MB380 applied, the next fresh Debian 13 attempt went much further.

CPAN installed the missing modules correctly:

IO::Async
Net::Async::IRC
Config::Simple
Switch
Memory::Usage
String::IRC
DateTime
Moose
Hailo
JSON::MaybeXS
JSON
...

The root verification passed.

Then the installer moved on to the IRC configuration step as the mediabot user and Perl failed on one of its most basic modules:

Can't locate strict.pm:
Permission denied

The module existed.

The runtime user was not allowed to read it.

One safe umask used for two incompatible jobs

The new wizard begins with:

umask 077

That is exactly right for:

mediabot.conf;
files containing passwords;
temporary secret overlays;
configuration backups.

But the same mask was inherited by the CPAN process launched through sudo.

As a result, parts of the Perl site library under /usr/local could be created as:

directories : 0700
files       : 0600

Root could install and verify every module.

The mediabot runtime user could not traverse the directories afterwards.

The Restricted Section had been secured so thoroughly that the student who needed the book was also locked out.

MB381 separates secret permissions from library permissions

The configuration wizard keeps its protective:

umask 077

The CPAN installer now sets its own installation mask:

umask 022

This gives Perl libraries their expected runtime permissions:

directories : 0755
modules     : 0644

The CPAN logs remain private in 0600.

When the installer is launched through sudo, those logs are returned to the invoking user using SUDO_UID and SUDO_GID.

Root verification is no longer enough

MB381 adds:

install/cpan_install.sh --verify-only

This mode:

installs nothing;
does not require root;
loads strict and warnings;
verifies every required Mediabot module;
never calls the installation helper.

After the root CPAN phase, configure automatically runs this verification as the actual mediabot runtime user.

The fresh-install carriage cannot move on merely because root can read the library.

The user who will run the bot must prove it too.

🌱✅ What the real Debian 13 fresh install validated

After the CPAN path and permission fixes, the fresh installation demonstrated that the large MB378 redesign was fundamentally sound.

The wizard successfully:

generated a complete configuration from mediabot.sample.conf;
produced all 119 active settings;
reported zero missing defaults;
reported zero duplicate keys;
reported zero safety warnings;
created the fresh database;
loaded the full database structure;
created and verified the application database account;
updated [mysql] atomically;
created a protected configuration backup;
installed missing Perl modules through CPAN;
verified those modules;
continued into IRC setup.

The generated configuration was not merely technically complete.

It was also operationally satisfactory during the real test.

That is the important result: the Room of Requirement design survived contact with an actual empty room.

📡🧯 MB382 — `status` sent the entire timetable by owl, all at once

Once the fresh bot connected, registration and login worked.

m uptime worked.

Then came:

m status

The command sent:

4 bot / memory / server lines
1 Scheduler summary
1 line for every Scheduler task

The fresh installation had twelve scheduled tasks.

The result was:

17 consecutive NOTICE messages

The IRC server responded exactly as an IRC server should:

Excess Flood

The connection was closed.

The status command had attempted to deliver the entire Hogwarts timetable through the owlery in a single second.

Compact status by default

MB382 changes the normal status into a maximum of three messages:

1. version, bot uptime and memory
2. operating system and server uptime
3. Scheduler summary

Example:

Mediabot v3.2dev-... | bot up 47s | RAM RSS 109.83MB, VM 117.32MB
Server: Linux ... x86_64 | uptime up 0 days, 00:43
Scheduler: 12 total | 12 running | 0 stopped | details: status full

The separators are plain ASCII, avoiding the mojibake previously visible in the Scheduler summary.

Detailed status remains available

The operator can explicitly request:

m status full
m status scheduler
m status tasks

Even the detailed form remains bounded:

normal status : maximum 3 NOTICE messages
full status   : maximum 6 NOTICE messages

Scheduler tasks are packed into compact lines such as:

auth_session_cleanup=run/3600s/t0/last:never

If every task cannot fit, the final line reports:

+N more

There is no longer one botNotice() call per task.

🧨👻 The flood revealed a second bug inside the error callback

The server closed the connection for Excess Flood.

Mediabot then entered on_message_ERROR and crashed with:

Can't use string ("1") as an ARRAY ref

The callback contained:

@{ $message->args // [] }

The problem is Perl context.

args() was evaluated in scalar context before the dereference.

With the current CPAN Net::Async::IRC, scalar context returned the number of arguments:

"1"

The code then attempted the equivalent of:

@{ "1" }

The flood had opened the trapdoor, and the error handler fell through it.

MB382 adds one safe IRC argument reader

A shared helper now handles the contract:

_irc_message_args($message)

It:

invokes args() in list context;
preserves every scalar argument;
supports older test doubles returning one array reference;
returns an empty list if the accessor fails;
never treats a scalar count as an array reference.

The fix was applied across the IRC callbacks, not only to ERROR.

It now covers paths including:

ERROR
KILL
SERVER
NOTICE
MODE
WHO and WHOIS
LIST
TOPIC
MYINFO
ISUPPORT
INVITE
ERR_NICKNAMEINUSE
ERR_NEEDMOREPARAMS

If an IRC ERROR message contains no usable text, the callback falls back to:

IRC connection closed

The emergency staircase now works even when the main corridor is on fire.

🧪🧹 Validation ledger

The corrected installation chain and runtime fixes were tested through their focused and historical regressions.

MB378–MB381 installation/configuration chain : 79/79
MB361–MB381 recent regression                : 511/511

Status and Scheduler historical regression   : 93/93
MB361–MB382 recent regression                : 543/543

MB382 was also validated through two installer applications and an idempotent re-execution.

The key source files compile successfully:

configure
install/cpan_install.sh
Mediabot/AdminCommands.pm
mediabot.pl

The regression groups overlap and should not be added into an artificial total.

🧱 Database impact

None.

0 new tables
0 altered columns
0 migrations
0 schema changes

The fresh installation created the normal Mediabot schema.

These rounds did not modify that schema.

🗺️ What MB380–MB382 changed

Together, these three rounds ensure that:

the CPAN installer finds its helper from any working directory;
CPAN logs always land under install/;
fresh-install prerequisites are checked before database creation;
Perl dependencies remain installed through CPAN only;
CPAN libraries are readable by the runtime user;
sensitive configuration files still use restrictive permissions;
root installation is followed by non-root module verification;
normal status output cannot flood the IRC connection;
detailed Scheduler output is explicit and bounded;
modern list-returning IRC message objects are handled correctly;
older arrayref-based test doubles remain compatible;
an IRC disconnect cannot crash the error callback through a phantom array reference.

🎩✨ The Sorting Hat stayed on

The real fresh installation did not prove that every first draft was perfect.

It proved something more useful.

It proved that the new configuration architecture was strong enough to be tested honestly, fail visibly, be corrected precisely and continue without abandoning its design.

The first hidden passage was a relative path left behind by an older working directory.

The second was a security mask crossing the boundary between secret files and shared runtime libraries.

The third was a status command that confused “complete” with “send everything immediately”.

And behind that third passage waited an error callback that had never met the current CPAN argument contract.

All four assumptions are now explicit.

The installer knows where it lives.

The library knows who must read it.

The status command knows how many owls it may release.

And the error handler knows that one argument is not an array reference merely because it looks lonely.