Forum teuk.org

🧠 Mediabot v3 — The Pensieve Gets Real Residents

in Mediabot · started by TeuK · 2w ago

TeuK · 2w ago

Development chronicle — mb603 to mb605 · Living storage examples, cron state and storage observability

The plugin v2 persistence layer now has something more valuable than a protocol: real inhabitants.

This arc adds two working examples built on top of the storage and cron contracts, exposes persistence health through Prometheus, and finishes with a pre-commit hardening pass that makes the documented limits and observability match the runtime exactly.

No database schema change was required.


🪄 mb603 — Storage and cron become visible through real examples

Two new sidecar plugins turn the previous infrastructure into something a plugin author can immediately understand.

karma.py — a small persistent notebook

The Python example demonstrates explicit read-modify-write storage:

thanks SlaY
→ SlaY now has 4 karma.

karma SlaY
→ SlaY has 4 karma.

karma
→ Top karma: bob (3), slay (2)

The script reads its current state from:

data.storage

and persists the replacement document through a store action.

It also demonstrates two important rules:

  • thanking yourself never writes anything;
  • plugin-side limits should stay below Mediabot’s global storage limits.

The example therefore keeps at most 200 tracked nicknames, below the bot-wide 256-key boundary.

daily.tcl — cron + config + storage

The Tcl example combines three v2 features:

[plugins]
daily.CHANNEL="#quebec"
daily.HOUR="9"
daily.TEXT="Good morning! The cider is cold."

The plugin listens to:

plugin_cron_observed

At the configured hour it emits one message, then stores the calendar day so another tick — or a restart — does not repeat the announcement.

A cron event has no channel context of its own, so the outgoing reply carries an explicit target from configuration.

That distinction is now documented in the cookbook.


🔭 mb604 — Persistence enters Grafana

The storage subsystem now exposes four Prometheus metrics:

mediabot_plugin_storage_bytes{plugin}
mediabot_plugin_store_total{plugin}
mediabot_plugin_store_rejected_total{plugin,reason}
mediabot_plugin_storage_read_invalid_total{plugin}

These answer practical operational questions.

Which plugins are growing?

topk(5, mediabot_plugin_storage_bytes)

Which plugins write frequently?

sum by (plugin) (
  rate(mediabot_plugin_store_total[5m])
)

Why are stores rejected?

sum by (plugin, reason) (
  increase(mediabot_plugin_store_rejected_total[1h])
)

The rejection label deliberately uses a closed vocabulary:

too_large
too_deep
too_many_keys
key_too_long
not_an_object
duplicate
no_sink
invalid_name
write_failed
other

Raw error strings never become metric labels, preventing accidental Prometheus cardinality explosions.

Both early planning failures and application-time failures are counted.


🛡️ mb605 — The final truthfulness pass

The final review found several places where the implementation could technically drift away from the guarantees described above.

They are now closed.

Exactly 200 karma entries

The pruning logic could briefly select 200 entries and then reinsert the current nickname, leaving 201.

The algorithm now guarantees the advertised ceiling after every update.

Calendar identity includes the year

The daily example originally remembered only month and day.

That meant an announcement on August 4 could incorrectly suppress August 4 of the following year.

The cron envelope now carries the year and the stored day identity is complete:

2026-8-4

Storage gauges reflect reality

mediabot_plugin_storage_bytes is now kept accurate when:

  • an existing storage file is discovered after startup;
  • a plugin writes new state;
  • .plugins cleardata removes the state;
  • a stored document becomes invalid.

A stale gauge no longer survives after the backing data disappears.

Invalid local storage is observable

Oversized files and unsafe symbolic-link cases now contribute to:

mediabot_plugin_storage_read_invalid_total

instead of being silently excluded from the metric path.

Metrics stay best-effort

Observability code no longer assumes every external or legacy runner returns perfectly shaped error arrays.

Malformed diagnostic structures cannot crash an otherwise valid dispatch merely because Prometheus accounting inspected them.

File permissions are part of the write contract

The temporary storage file must successfully receive mode 0600 before the atomic rename.

A permission failure is treated as a failed store rather than publishing a file with weaker permissions.


🧪 Validation

The complete local test suite passed after the final hardening:

12042/12042 PASSED

Dedicated tests cover:

  • real Python persistent karma;
  • real Tcl cron execution;
  • restart-safe daily announcements;
  • year-aware calendar identity;
  • strict 200-entry karma pruning;
  • storage byte gauges;
  • successful write counters;
  • bounded rejection reasons;
  • invalid-read counters;
  • initial metrics from existing storage;
  • gauge reset after deletion;
  • oversized and symbolic-link storage cases;
  • malformed runner diagnostics;
  • restrictive storage file permissions.

✨ What this means for plugin authors

The persistence contract is no longer an abstract API.

There are now complete examples showing how to:

read state
modify it
store it
observe it
schedule work around time
combine it with operator configuration

while Mediabot continues to own the dangerous parts:

  • file paths;
  • permissions;
  • atomic writes;
  • size and depth limits;
  • metrics;
  • validation;
  • lifecycle.

The script asks to remember something.

The bot decides whether that memory is safe to keep.


🏰 Final parchment

The Pensieve now has real memories inside it.

One portrait keeps score without filling the shelves forever. Another listens to the tower clock and remembers whether it already spoke today. And the caretaker can see exactly how much each memory weighs, how often it changes, and why the castle refused to store it.

The persistence layer is no longer just implemented.

It is demonstrated, measurable and guarded.

You must be logged in to reply.