Forum teuk.org

⏳ Mediabot 3.4dev — Gringotts no longer makes the first wizard wait 🏦🪄

in Mediabot · started by TeuK · 4w ago

TeuK · 4w ago

Mediabot v3 continues its 3.4 development cycle with a focused reliability pass on database latency and first-command responsiveness.

The trigger was a very specific production symptom on one Undernet instance:

the first "m check" is slow by about 10 seconds
the following commands are immediate

That pattern pointed toward a stale or half-dead MariaDB connection, but this round did not stop at guessing. It added hard limits, precise timings and a fallback trace so the next slowdown can identify itself.

🏦 The Gringotts door is now timed

Both MariaDB connection paths now include explicit network bounds:

mariadb_connect_timeout
mariadb_read_timeout
mariadb_write_timeout

They are configurable in the [mysql] section and safely bounded.

The goal is simple: a dead socket must not be allowed to hold a user command hostage indefinitely.

⏱️ Slow pings and reconnects leave evidence

The database reconnect path now measures what actually happens.

A slow ping can produce:

DB ping slow: 0.31s

A successful reconnect reports its own duration and the time already lost waiting for the dead connection:

DB reconnect ok in 0.42s (ping wait was 9.87s)

A failed reconnect is also reported honestly.

The final hardening fixed an important edge case: an old dead database handle can no longer remain in memory and make a failed reconnect look successful.

Gringotts may still close a vault door, but the goblin cannot stamp “OPEN” on a lock that never moved. 🏦

🛡️ No false success, no stale handle

During review, one subtle failure path was found.

If DBI->connect() failed while an old dead handle was still present, the reconnect code could mistakenly report success because the stale object was still truthy.

That has been corrected.

Now:

  • the stale handle is not returned as if it were usable;
  • reconnect success reflects the real connection result;
  • failed reconnects remain visible as failures;
  • the logs can be trusted during the next incident.

🌙 One keeper of the vault

The original proposal added a second keepalive path, but Mediabot already had a periodic database health check in its main timer tick.

The final version keeps that existing mechanism as the single keeper of the connection and removes the redundant extra ping.

This avoids duplicate work and keeps one clear source of truth for database health.

📨 Slow PRIVMSG tracing

A lightweight end-to-end timer now wraps the IRC PRIVMSG handler.

Any command path taking more than one second can produce:

SLOW PRIVMSG: processing took 10.23s (from Te[u]K)

If the next slowdown is not caused by MariaDB, Mediabot will still record when it happened, how long it took and which user action triggered it.

The wrapper also preserves the caller’s scalar, list or void context instead of forcing every call through list context.

🧪 What the next incident should reveal

After deployment, one of two things should happen.

Most likely, the explicit network timeouts and the existing periodic health check prevent the first-command pause.

If the lag returns, debug level 3 should now show one of these paths:

DB ping slow
DB reconnect ok
DB reconnect FAILED
SLOW PRIVMSG

Instead of “the bot froze for ten seconds,” the operator gets a timestamped suspect and a measured duration.

✅ Validation

The completed pass was validated with:

10809/10809 tests passed
Security audit: GO — all 11 invariants hold
Startup integrity: OK
41/41 modules loaded
83/83 cross-module methods resolved
197/197 dispatch handlers resolved
GitHub Actions: green

The checks cover:

  • bounded MariaDB connection, read and write timeouts;
  • slow-ping and reconnect timing;
  • truthful reconnect failure reporting;
  • rejection of stale dead handles;
  • preservation of the existing periodic DB health check;
  • correct PRIVMSG context handling;
  • slow-command diagnostics;
  • configuration documentation.

No database schema change.
No mandatory configuration change.
Existing defaults remain usable.

🧙 Summary

This round does not pretend that every ten-second pause has already been explained.

It does something better:

  • database waits are bounded;
  • reconnects are timed;
  • failures cannot masquerade as success;
  • stale handles are not reused;
  • redundant keepalive work is removed;
  • slow IRC command paths leave evidence;
  • CI, security and startup integrity remain green.

The next time Gringotts closes its doors during the night, the first wizard at the counter should no longer pay the full price—and if anyone is still kept waiting, the register will say exactly where the time went. ⏳🏦🪄

You must be logged in to reply.