Forum teuk.org

🕯️🛡️🗺️ Mediabot v3: The Castle Learns to Keep Secrets, Remember Its Name, and Survive a Second Storm — MB383 to MB393

in Mediabot · started by TeuK · 1mo ago

TeuK · 1mo ago

The previous chapter ended with a successful Debian 13 fresh installation, a repaired CPAN path, readable Perl libraries and a status command that had finally stopped releasing the entire owlery at once.

That was not the end of the work.

It was the moment Mediabot became stable enough for a different kind of review: not “does it start?”, but “does it behave like something we could eventually trust as 3.3?”

The answer was encouraging, but the castle still had several hidden passages:

  • private credentials could appear in deep debug logs;
  • blocked messages could be recorded as if they had really been sent;
  • the bot could forget its own version depending on its working directory;
  • a second IRC disconnection could fail to trigger another reconnect;
  • two processes could race for the same PID file;
  • help output could still behave like a flock of unsupervised owls;
  • and the supposedly CPAN-only installation still relied on a Perl database driver supplied by Debian.

MB383 through MB393 closed those gaps.

This article deliberately covers only the public Mediabot changes since MB382.

The project is still on the 3.2dev line.

The stable 3.3 release will happen only after broader live testing and an explicit decision that the bot is ready.

As always:

No database schema was changed.


📜🧹 MB383 — The old prophecies finally match the code they guard

MB383 did not modify production code.

It repaired three tests that still described older implementations:

CPAN log path       : relative → absolute and anchored
CPAN helper path    : ./helper → $SCRIPT_DIR/helper
IRC action payload  : raw Perl text → UTF-8 wire text

The tested production behaviours were already correct.

The tests simply needed to stop asking the castle to behave as it had before the renovations.

The focused result was:

42/42

The recent MB361–MB383 regression selection reached:

585/585

This was an important housekeeping round: a green test must mean that the current contract is protected, not that an obsolete line of code has been preserved in amber.


🛡️🕵️ MB384–MB387 — The Invisibility Cloak reaches the logs

The next four rounds focused on a single principle:

A secret must not become public merely because the debug level increased.

They also made the logs tell the truth about what was actually sent to IRC.


🔒 MB384 — Incoming private messages stop leaving readable footprints

Private IRC commands can contain passwords:

login
register
pass
xlogin
identify

The normal authentication paths already avoided printing the password in their success messages.

But three lower-level diagnostics still exposed the original payload:

on_private() debug output
raw PRIVMSG argument logging
passive DCC payloads rendered as hexadecimal

Hexadecimal is not redaction.

It is only a different alphabet.

MB384 replaced private payloads with metadata such as:

[private payload redacted bytes=24]

Passive DCC diagnostics now retain only useful structure:

type=dcc_chat
mode=passive
token_present=1

The secret itself is gone.

The same round also aligned channel detection with standard IRC prefixes:

#
&
!
+

A local &channel, safe !channel or modeless +channel is no longer misclassified as a private conversation.


🦉 MB385 — Private NOTICE messages receive the same protection

botNotice() already masked credentials in its normal log line.

Unfortunately, a deeper debug message ran first and still wrote the original text:

botNotice() called with target='NickServ',
text='identify real_password'

A long private NOTICE had another problem.

Redaction happened after splitting, so a credential moved into a later chunk could lose the command prefix that identified it as sensitive.

MB385 now creates one redacted log copy before:

debug logging
normal logging
chunking

The actual IRC payload remains untouched and is sent as requested.

Only the logs receive the protected copy.

Channel NOTICE messages keep their normal visible text.


🧼 MB386 — CR, LF and NUL are removed before logs and history

The outbound helpers already flattened line breaks before writing to the IRC socket.

But they did so too late.

Before sanitisation, the original payload could already reach:

LIVE logs
private logs
badword checks
CHANNEL_LOG
blocking diagnostics

A crafted message such as:

hello\r\n[ERROR] forged\0tail\nnext

could therefore create fake log lines even though the final IRC command was later flattened.

MB386 introduced one shared sanitizer:

_sanitize_irc_text($text)

It normalises:

CR
LF
NUL

before redaction, filtering, logging, history storage, UTF-8 chunking and network output.

The same canonical text is now used everywhere.

The journal no longer records a different reality from the wire.


⚖️ MB387 — A blocked message is no longer logged as delivered

botPrivmsg() and botAction() used to emit their [LIVE] line before the badword guard completed.

A message could therefore be:

blocked by policy
never sent to IRC
still written as LIVE
still stored as normal channel history

MB387 reordered the path:

normalise
AntiFlood
badwords
accepted history
LIVE log
IRC send

Rejected output still produces an operational blocking diagnostic.

It no longer impersonates a real conversation.

The Marauder’s Map now shows where the message actually went — not where it intended to go.


🪞🪪 MB388–MB389 — Mediabot remembers its own name

A screenshot exposed one of the most visible release blockers:

Mediabot version: Undefined

The VERSION file existed and contained a correct build identifier.

Two bugs combined to erase it.


🏷️ MB388 — VERSION no longer depends on the current directory

The old reader opened:

open my $fh, '<', 'VERSION'

That works only when the process starts from the repository root.

A service, wrapper or administrator launching Mediabot from another directory could make the file disappear without moving it.

MB388 anchors the path to the real location of the source tree.

It also prevents an asynchronous GitHub worker from overwriting a valid local version with an internal error sentinel such as:

Undefined

The local identity survives:

remote timeout
worker failure
invalid JSON
missing loop
fork failure
unusable remote payload

m status also receives a defensive final fallback and can no longer display the internal sentinel.

The bot’s name is now written on its identity card, not on a loose note left in whichever room happened to be current.


🌐 MB389 — Starting the bot no longer requires GitHub

Even after the local version fix, startup still performed a synchronous remote version check.

That made the critical launch path depend on:

DNS
Internet access
GitHub availability
remote latency

MB389 split the two responsibilities.

At startup:

read local VERSION
start Mediabot

On explicit request:

m version
perform the bounded asynchronous remote check

The bot can now start normally in an isolated network or during a public outage.

Semantic comparison replaces string inequality

The previous logic treated every different remote string as an available update.

That is not version comparison.

MB389 understands Mediabot’s formats:

3.3
3.3dev-20260703_024855

It compares:

major
minor
stable versus development
development timestamp

Therefore:

local  : 3.3
remote : 3.2dev-20991231_235959

does not produce a false update warning.

A local build may be equal, older, newer or incomparable.

The message now says which one.


⚡🔐 MB390 — The second storm reconnects too, and only one bot owns the PID

The first successful reconnect revealed a lifecycle trap.

irc_reconnect_in_progress was set when reconnecting.

Failure cleared it.

Success did not.

After the first recovery, the bot remained convinced that a reconnect was still in progress forever.

A second disconnection could therefore never schedule another one.

MB390 introduced a common reset for:

irc_restart_in_progress
irc_reconnect_requested
irc_reconnect_in_progress

Both success and failure use the same cleanup.

One storm no longer consumes the bot’s only recovery spell.


🔏 An atomic process lock replaces the delayed PID write

The historical PID flow was vulnerable to a race:

read PID
test process
continue startup
write PID later

In foreground and systemd mode, the file was not written until the first periodic tick — potentially five seconds after startup.

Two processes could both observe an empty doorway and enter the castle.

MB390 added:

Mediabot/ProcessLock.pm

It uses an exclusive non-blocking flock.

The lock is:

  • acquired immediately in foreground/systemd mode;
  • acquired after daemonisation when the final child PID is known;
  • held for the entire process lifetime;
  • never rewritten by the periodic timer;
  • released during clean shutdown;
  • removed only by the process that still owns it.

A stale PID belonging to a dead process can be replaced.

A live historical instance that wrote a PID without holding the new lock is still respected.

The same round corrected the shutdown message so that SIGTERM is no longer described as Ctrl+C.


📣⏱️ MB391 — Core commands receive a release contract

The future stable release needs a small set of commands that can always be trusted:

m version
m uptime
m help
m commands
m status
m status full

MB391 made their behaviour explicit.


m version answers locally first

The local version is available immediately.

The command now reports it without waiting for GitHub:

Mediabot version: 3.2dev-...

The remote worker may later add one useful follow-up:

update available
local build newer than repository
remote format cannot be compared

If the versions match or the remote check fails, no second message is needed.

The user never waits several seconds to learn something already written on disk.


⏳ One process uptime, everywhere

Three paths used different clocks:

m uptime
m status
Partyline .uptime

Some measured the process lifetime.

Others measured the current IRC connection.

After a reconnect, the same bot could therefore claim two different ages.

MB391 added one shared process-start resolver.

Its priority is:

Metrics process start
internal process start
connection timestamp only as legacy fallback
current time as last resort

A reconnect no longer makes an old process young again.


🦉 Help output joins a controlled owl queue

status had already been bounded.

help could still release up to nine NOTICE messages immediately for a privileged user.

MB391 introduced a shared NOTICE queue:

one line every 1.5 seconds
concurrent requests serialised
maximum 16 lines per request
explicit truncation message
expired timers removed from tracking

The queue is used by:

m help
m commands
showcommands
category help

Detailed help remains available.

It simply arrives at a pace the IRC server can accept.

The same context fixes ensure that channels beginning with #, &, ! or + receive channel replies rather than accidental private NOTICE messages.

A dedicated live smoke case was also added for the release journey.


🐪🗄️ MB393 — CPAN-only finally includes the actual MariaDB driver

The installation was described as CPAN-only.

The runtime used:

DBD::MariaDB

But the CPAN installer checked only:

DBI

The documentation compensated by asking Debian to install:

libdbi-perl
libdbd-mariadb-perl
libdbd-mysql-perl

That meant the fresh install still depended on Debian-packaged Perl modules.

MB393 made the contract honest.

The CPAN phase now installs and verifies:

DBI
DBD::MariaDB

The verification runs:

as root after installation
as the mediabot runtime user

The principal installer and the legacy server assistant now agree on:

DBI:MariaDB

Native build dependency versus Perl module

A compiled CPAN driver still needs the MariaDB client headers and libraries.

On Debian, those are supplied by:

libmariadb-dev

That package is a native C build dependency.

It is not a replacement Perl driver.

The separation is now clear:

Perl modules       : CPAN
C headers/libraries: operating-system package manager
runtime driver     : DBD::MariaDB

🌱✅ A second real Debian 13 fresh installation

The corrected installation path was then tested again from a clean Debian 13 system.

This time, the bootstrap deliberately omitted:

libdbi-perl
libdbd-mariadb-perl
libdbd-mysql-perl

The installation completed with the Perl database stack supplied by CPAN.

The generated bot started as:

Mediabot v3.2dev-20260703_024855

It connected, joined #boulets, registered the owner and logged in successfully.

Sensitive private input appeared only as:

[private payload redacted bytes=24]
[private payload redacted bytes=21]

The credentials themselves never appeared.


🪪 Live version check

The fresh bot returned its real local identity immediately:

Mediabot version: 3.2dev-20260703_024855

No Undefined.

No startup dependency on GitHub.


⏱️ Live uptime and status

m uptime reported the process age, RAM and load:

mediabot: up 1m 1s (since 09:00)
RAM 106.6 MB
load 0.18 0.33 0.17

m status returned exactly the compact release contract:

Mediabot v3.2dev-20260703_024855
bot up 01mn 07s
RAM RSS 106.61MB, VM 115.18MB

Server: Linux mbdeb13 ... x86_64
uptime up 0 days, 16:13

Scheduler: 12 total
12 running
0 stopped
details: status full

No flood.

No crash.

No mojibake.


📋 Live detailed Scheduler status

m status full returned the same three core lines plus two packed Scheduler lines.

All twelve tasks were represented without one NOTICE per task:

auth_session_cleanup
birthday_check
channel_ban_expire
channel_cache_refresh
channel_log_purge
claude_history_purge
daily_channel_report
health_check
karma_log_purge
reminder_purge
user_seen_purge
weekly_channel_report

The server remained connected.

The bot then received SIGINT and entered clean shutdown:

Received SIGINT (Ctrl+C). Initiating clean shutdown.

The journey from empty Debian installation to working IRC runtime was clean.


🧪 Validation ledger

The focused validations for these rounds were:

MB383 : 42/42
         recent MB361–MB383 selection 585/585

MB384 : 64/64
         security/DCC selection 202/202

MB385 : 48/48
         PRIVMSG/ACTION/NOTICE selection 92/92

MB386 : 44/44
         logging/redaction selection 148/148

MB387 : 71/71

MB388 : 58/58
         version/status selection 112/112

MB389 : 70/70
         version/status selection 84/84

MB390 : 20/20
         lifecycle selection 67/67

MB391 : 232/232

MB393 : 101/101

These groups overlap and must not be added into one artificial total.

The most important additional result is the live one:

fresh Debian 13 install
CPAN-provided DBI and DBD::MariaDB
successful IRC connection
private credential redaction
correct local version
consistent process uptime
bounded status output
bounded full Scheduler output
clean shutdown

🧱 Database impact

None.

0 new tables
0 altered columns
0 migrations
0 schema changes

The fresh installer created the normal existing Mediabot schema.

These rounds did not change it.


🏰 What changed since MB382

Mediabot now:

  • keeps private IRC credentials out of normal, debug and hexadecimal logs;
  • redacts private NOTICE messages before chunking;
  • sanitises CR, LF and NUL before logs, history and network output;
  • records only messages that were actually accepted for sending;
  • reads its local version independently of the current directory;
  • never replaces a valid version with Undefined;
  • starts without DNS, GitHub or Internet access;
  • compares stable and development versions semantically;
  • reconnects after the second disconnection as well as the first;
  • prevents concurrent instances with an atomic PID lock;
  • removes its PID file during clean shutdown;
  • reports one consistent process uptime everywhere;
  • answers m version immediately;
  • queues long help output instead of flooding IRC;
  • recognises all standard IRC channel prefixes;
  • installs and verifies the real MariaDB Perl driver through CPAN;
  • completes a clean Debian 13 fresh installation without Debian-packaged Perl DBI drivers.

✨ The road to 3.3 is now a test journey, not a version number

Mediabot is still:

3.2dev

That is intentional.

The work described here does not declare the castle finished merely because the foundations are stronger.

The future 3.3 release will happen after broader functional testing:

administration
authentication
Partyline
timers and reminders
radio
AI integrations
URL parsers
reports
metrics
reconnection
upgrade paths

The important difference is that the release candidate now has a far more reliable base.

Secrets remain under the Invisibility Cloak.

The Marauder’s Map reflects what was really sent.

The castle remembers its own name.

The front gate admits only one process.

The owls leave in an orderly queue.

And a fresh Debian 13 room can now summon the whole system through CPAN without borrowing a hidden Perl driver from the Ministry.

The Sorting Hat is still on the table.

We are not calling the result 3.3 yet.

But the castle is becoming ready to earn it.

TeuK

You must be logged in to reply.