Forum teuk.org

πŸ”₯ Mediabot v3 β€” Teaching the Update Spell How to Rise Again

in Mediabot Β· started by TeuK Β· 1w ago

TeuK Β· 1w ago

The update spell was already doing most of its magic correctly.

It could fetch the latest Mediabot release, validate it, preserve the configuration and Hailo brain, archive the previous installation and activate the new tree.

There was just one rather inconvenient detail:

after the spell finished, the bot could remain dead.

Not exactly the triumphant return one expects from advanced wizardry.

πŸͺ„ The mystery

A normal update followed this sequence:

m update
    ↓
clone latest release
    ↓
syntax + integrity checks
    ↓
SIGTERM current Mediabot
    ↓
archive old release
    ↓
activate new release

The updater expected systemd to bring Mediabot back.

But the recommended service used:

Restart=on-failure

Mediabot handles SIGTERM as a clean shutdown.

So systemd saw:

exit 0

and quite reasonably concluded:

Nothing failed. No restart required.

The deployment succeeded.

The bot stayed asleep.

πŸ”Ž Following the right clues

The first temptation was simply to replace the policy with:

Restart=always

That would indeed restart Mediabot after an update.

Unfortunately it would also resurrect the bot after an intentional:

m die

A remarkably stubborn sort of immortality.

There was another problem too: restarting after a fixed delay could race against the updater while it was still replacing the live release.

So MB645 establishes an explicit contract between Mediabot, the updater and systemd instead of relying on timing or assumptions.

πŸ¦β€πŸ”₯ The Phoenix contract

The recommended systemd template now uses the following principles:

ExitType=cgroup
Restart=always
SuccessExitStatus=75
RestartPreventExitStatus=75

and marks compatible instances with:

MEDIABOT_SYSTEMD_UPDATE_SAFE=1

During an update

The deployment worker remains inside the service cgroup while the main Mediabot process exits.

With:

ExitType=cgroup

systemd waits until the complete update worker has finished before considering the service gone.

Only then can the new Mediabot release start.

old Mediabot
      ↓
update worker
      ↓
SIGTERM bot
      ↓
bot exits
      β”‚
      └── updater still alive
              ↓
         archive release
              ↓
         activate new release
              ↓
         updater exits
              ↓
         cgroup becomes empty
              ↓
         systemd restart
              ↓
         NEW Mediabot

No arbitrary race against RestartSec.

☠️ And m die still means die

An intentional shutdown now uses exit status:

75

The service declares:

SuccessExitStatus=75
RestartPreventExitStatus=75

This gives systemd two pieces of information:

  • exit 75 is an intentional successful shutdown;
  • exit 75 must not trigger Restart=always.

So:

m die
    ↓
exit 75
    ↓
clean shutdown
    ↓
NO restart

Exactly as requested.

A normal administrative:

systemctl stop mediabot@instance

also remains a real stop.

πŸ›‘οΈ The updater now checks the contract

deploy_update.sh no longer blindly assumes that systemd will restart Mediabot.

Before stopping the running bot, it verifies that the detected unit has the required restart policy.

If the environment is unsafe, the update aborts before the destructive part begins.

That is considerably better than discovering the problem after Mediabot has already disappeared.

🏰 Multi-instance support stays intact

The long-standing template architecture remains unchanged:

mediabot@dev
mediabot@undernet
mediabot@anything

with instance-specific configuration provided through:

/etc/default/mediabot-<instance>

and:

BOT_DIR
BOT_BIN
BOT_CONF

The existing bash -lc, stdbuf and multi-instance layout were deliberately preserved after checking the systemd configuration already running reliably in production.

The temporary hard-coded service used during diagnosis was just that: a diagnostic workaround, not the public design.

πŸ§ͺ Ministry inspection

The targeted MB645 tests passed:

PASSED : 107/107

The entire Mediabot suite then passed:

RC=0

PASSED : 12918/12918

Runtime checks also covered the important behaviours:

update-style SIGTERM
    β†’ bot restarts

m die
    β†’ clean exit
    β†’ bot stays stopped

systemctl stop
    β†’ bot stays stopped

The distinction between a crash, an update and an intentional shutdown is now explicit.

⚑ Why this matters

MB645 does not merely add a restart.

It defines who is responsible for the complete lifecycle of an update.

Mediabot handles the application.

The updater handles the release transition.

systemd supervises the complete operation and starts the new release only when the previous update process has truly finished.

The result is a much safer spell:

m update

can now replace Mediabot and bring Mediabot back.

Which, considering the amount of magic involved, feels like a fairly reasonable requirement.

You must be logged in to reply.