Forum teuk.org

🪄 Mediabot v3 — The Achievement Workers Leave the Great Hall

in Mediabot · started by TeuK · 4w ago

TeuK · 4w ago

Development chronicle — mb558 to mb560 · 3.4dev

For a long time, the achievement guardians performed their heaviest investigations right in the middle of the Great Hall. On a large IRC history, one unlucky message could therefore awaken an ancient SQL spell and freeze the room for far too long.

That chapter is now closed. 🏰

🧙 mb558 — The heavy spells leave the message path

The three expensive CHANNEL_LOG achievement checks no longer run synchronously while Mediabot handles a PRIVMSG.

The hot path now performs one small and predictable action: it places the work in a bounded queue.

  • Case-insensitive deduplication prevents duplicate jobs.
  • Original nickname casing is preserved.
  • The queue is capped at 200 entries.
  • Existing unlock thresholds and throttles remain unchanged.
  • Slow checks are timed and exposed through Prometheus metrics.

The Great Hall can keep talking while the archivists work elsewhere. ✨

🦉 mb559 — An isolated worker enters the tower

Queued achievement work is now handled by a forked asynchronous worker.

The child process opens its own isolated MariaDB connection and never reuses the database socket inherited from the parent. It performs the historical scans, builds a bounded JSON response, and returns only validated results.

The parent remains in control:

  • only one worker may run at a time;
  • returned checks must belong to a strict whitelist;
  • unlocks are limited and tied to the original nickname and channel;
  • a 75-second timeout triggers TERM, then KILL if necessary;
  • failures rotate through the queue with backoff;
  • jobs are dropped only after three failed attempts;
  • shutdown cleanup prevents abandoned workers.

Prometheus now reports queue depth, in-flight work, results, timeouts and bounded drops. Grafana can finally distinguish a quiet tower from one where a spell has genuinely gone wrong. 📊

🧹 mb560 — The guards learn the real rules

Two final details were corrected.

First, successful workers were accidentally reported to Prometheus as failed. Healthy completions now keep the truthful label:

result="ok"

Second, two static regression guards had become stricter than the contracts they were meant to protect:

  • the database guard expected exactly two connection sites, while the new isolated worker correctly introduced a third;
  • the structure guard treated methods with the same name in different Perl packages as illegal duplicates.

The guards now check the real rules:

  • every DBI->connect site must have bounded timeouts;
  • duplicate subroutines are rejected inside the same package, while legitimate worker overrides remain allowed.

This last mb560 guard round changes no runtime behaviour. It simply teaches the castle watchmen to inspect what truly exists instead of reciting an outdated map. 🗺️

🛡️ What this protects

These changes remove expensive achievement aggregation from Mediabot’s message-processing path without weakening the feature itself.

Users keep the same achievements and unlock conditions, while operators gain:

  • a responsive IRC event loop;
  • isolated database work;
  • bounded queues and payloads;
  • controlled retries and hard timeouts;
  • accurate Prometheus results;
  • useful Grafana visibility;
  • stronger regression tests.

🧪 Validation

The complete offline suite finished successfully:

10980 / 10980 PASSED

The GitHub CI also validates the project on each push to master, including:

  • runtime Perl modules;
  • Perl syntax;
  • shell syntax;
  • the complete offline test suite;
  • security invariants;
  • startup integrity.

📜 Final parchment

No database schema change was required.

The achievement system still follows the same rules, but its heavy work now happens behind guarded doors, with its own database connection, strict limits and honest observability.

The castle remains lively. The archivists remain busy. And no slow SQL spell is allowed to silence the Great Hall again. ⚡

You must be logged in to reply.