Mediabot 3.6dev has gained a memory for plugin failures — but not the kind that turns every exception into a public spectacle or lets an automated recovery spell take control of the castle.
MB751 adds a bounded, instance-local failure ledger to Plugin API v3. Command handlers, event subscribers, scheduled jobs and asynchronous HTTP callbacks can now leave a small, safe operational trace when they fail. Operators can inspect that evidence through Partyline without receiving raw exception text, private configuration or mutable runtime objects.
The milestone was published on the 3.6dev-20260920_142343 development line.
Its new operator command is deliberately simple:
.plugins failures <name>
Think of it as a cabinet of guarded memory vials: enough evidence to understand which spell misfired, where it happened and whether it keeps happening — never the whole private memory. 🧪🛡️
MB750 gave operators a reliable vocabulary for the current API v3 state:
doctor reports whether a plugin is inactive, limited or ready;permissions explains requested, granted, effective and missing capabilities;why explains the decision for one channel: blocked, shadow or active.Those commands answer an essential question: what is the runtime allowed to do right now?
They could not answer another one: what has recently failed inside the runtime?
A plugin may be enabled, correctly authorized and active on an approved channel while one command, job, event handler or HTTP completion path is failing. Before MB751, the immediate log could contain that information, but the API v3 operator model had no bounded, structured history of its own.
MB751 fills that gap without changing the lifecycle decision itself. A plugin with a recorded failure does not suddenly become disabled, quarantined or restarted. The ledger adds evidence; it does not seize authority.
The new Mediabot::Plugin::FailureLedgerV3 module records outcomes from the
four executable paths currently owned by API v3:
| Failure kind | Runtime path |
|---|---|
command |
A mounted API v3 command handler |
event |
A versioned event subscriber |
job |
A bounded scheduled job |
http_callback |
Completion of a shared HTTP request |
Each failure record contains only the operational coordinates needed for diagnosis:
The raw exception is normalized only long enough to create the fingerprint. It is never retained in the ledger and never returned to Partyline.
Successful executions are recorded too, but only as state transitions. A success resets the consecutive streak for the matching resource while leaving the bounded recent-failure history intact. Operators can therefore distinguish a persistent failure from an old incident followed by healthy executions.
Failure reporting must never become a new source of unbounded memory growth. MB751 therefore makes its limits part of the contract:
The resource-state limit includes four reserved overflow buckets, one for each failure kind. Once the exact-resource allowance is full, new identities are aggregated into their kind’s bounded bucket instead of extending the ledger. The global ceiling therefore remains 128 even under hostile or accidental high-cardinality input.
The memory belongs to the loaded plugin instance. It survives a disable/enable cycle because the instance is still the same, but it is discarded when the plugin is unregistered or unloaded. A newly loaded instance starts with a clean ledger.
This is an operational memory, not a database. MB751 adds no schema, writes no private configuration and creates no persistent failure archive.
Exception text can contain much more than a useful diagnosis: endpoints, file paths, user input, implementation details or fragments supplied by an external service.
MB751 does not display or store that text. It derives an instance-salted SHA-256 fingerprint and keeps only its first 16 hexadecimal characters in the report. The per-instance salt prevents the same low-entropy error from becoming a reusable global dictionary key across processes.
The fingerprint is not intended to reconstruct the error. Its purpose is to answer a safer question: are these recent failures probably the same?
Detailed investigation can still use Mediabot’s normal application log under the operator’s existing access controls. Partyline receives a correlation clue, not a secret-bearing stack trace. 🔐
The command:
.plugins failures <name>
returns one bounded summary and no more than the five newest records. Its header exposes the current limits, making the truncation explicit rather than invisible:
Plugin failures 'hello-v3': total=0 recent=0/16 resources=0/128 active_streaks=0.
latest: none
When records exist, each line is restricted to the kind, resource, channel, epoch, fingerprint and streak. No raw exception, plugin object, invocation context, HTTP response body, repository handle or configuration value crosses the boundary.
The existing doctor view also gains a compact failure summary:
failures: total=0 recent=0 resources=0 active_streaks=0 last=0
This keeps the high-level health inspection concise while reserving the bounded
record list for the dedicated failures command.
MB751 deliberately keeps diagnostics and lifecycle decisions separate.
The failure aggregate is attached to the detached diagnostic report, but it
does not rewrite the established doctor verdict. Lifecycle and capability
facts still determine whether the plugin is inactive, limited or ready.
The ledger does not:
There is intentionally no reset or clear method in this milestone. Unload is the explicit lifecycle boundary that discards the instance and its in-memory history.
This restraint matters. Before the platform can make supervised remediation decisions, it first needs trustworthy, bounded and non-secret evidence. MB751 builds that evidence layer and stops there.
The development pilot used hello-v3, loaded with exactly its declared
capabilities and left disabled:
.plugins loadv3 hello-v3 events.subscribe,irc.reply,scheduler.jobs
.plugins doctor hello-v3
.plugins failures hello-v3
.plugins permissions hello-v3
.plugins why hello-v3 #test
.plugins unload hello-v3
The doctor correctly reported an inactive plugin with all runtime resources mounted and no failures:
Plugin doctor 'hello-v3': inactive (plugin_disabled).
lifecycle: disabled
permissions: complete missing=none
policies: total=0 active=0 on=0 observe=0 off=0
runtime: commands=1/1 events=1/1 jobs=1/1 saved_handlers=0
failures: total=0 recent=0 resources=0 active_streaks=0 last=0
The channel decision remained blocked because the plugin was disabled, and the package unloaded cleanly afterward. No channel policy was created and no plugin was left active.
We deliberately did not manufacture a failure on the live IRC service merely to make the counter move. The four failure paths, streak transitions, bounds, overflow behavior and secrecy rules are exercised deterministically by the automated tests.
MB751 passed every development and publication gate:
| Gate | Result |
|---|---|
| Failure-history contract | 4 kinds / 16 recent / 128 resources |
| Targeted suite | 459 / 459 tests across 32 files |
| Fast lane | 7,537 / 7,537 tests across 412 files |
| Final full suite | 19,877 / 19,877 tests across 979 files |
| Final full duration | 142 seconds |
| Changed surface | 22 files, and no others |
| Repository after push | clean and synchronized with origin/master |
| Development service | active and unchanged during the final gate |
The exact staged content was frozen before and after the unique final full suite:
STAGED_SHA256=180a6c14d6f8c476bf5e11e8a0d6ea6b4865cf6572a5d8d36b8f21b62c66ab5c
No source byte, service PID or restart counter changed while all 19,877 tests were running.
MB751 adds three dedicated tests:
t/cases/1094_mb751_plugin_failure_ledger.t
t/cases/1095_mb751_plugin_failure_runtime.t
t/cases/1096_mb751_partyline_plugin_failures.t
Together they protect:
The earlier diagnostic, Partyline and machine-contract sentinels were updated at the same time, so the failure ledger is now part of the documented API v3 platform rather than an isolated implementation detail.
MB751 was committed and pushed to master as:
f9eccd167e078d3d90a5690a122d3a8382c91b72
MB751: bottle API v3 mishaps in guarded memory vials 🧪🛡️
View the MB751 commit on GitHub
The commit contains 785 insertions and 42 deletions across 22 files, including the new failure-ledger module and its three dedicated regression wards.
The API v3 platform can now explain both its present decision and its recent execution history:
That gives future supervised recovery work a sound foundation. A later milestone can define explicit quarantine, retry or circuit-breaking policy without guessing from raw log fragments and without mixing evidence collection with remediation.
For now, the castle remembers carefully, reveals sparingly and changes nothing without an operator’s spell. The memory vials are labelled, sealed and safely back on the shelf. 🏰🧪✨
— Teuk
You must be logged in to reply.