The update spell was already doing most of its magic correctly.
It could fetch the latest Mediabot release, validate it, preserve the configuration and Hailo brain, archive the previous installation and activate the new tree.
There was just one rather inconvenient detail:
after the spell finished, the bot could remain dead.
Not exactly the triumphant return one expects from advanced wizardry.
A normal update followed this sequence:
m update
β
clone latest release
β
syntax + integrity checks
β
SIGTERM current Mediabot
β
archive old release
β
activate new release
The updater expected systemd to bring Mediabot back.
But the recommended service used:
Restart=on-failure
Mediabot handles SIGTERM as a clean shutdown.
So systemd saw:
exit 0
and quite reasonably concluded:
Nothing failed. No restart required.
The deployment succeeded.
The bot stayed asleep.
The first temptation was simply to replace the policy with:
Restart=always
That would indeed restart Mediabot after an update.
Unfortunately it would also resurrect the bot after an intentional:
m die
A remarkably stubborn sort of immortality.
There was another problem too: restarting after a fixed delay could race against the updater while it was still replacing the live release.
So MB645 establishes an explicit contract between Mediabot, the updater and systemd instead of relying on timing or assumptions.
The recommended systemd template now uses the following principles:
ExitType=cgroup
Restart=always
SuccessExitStatus=75
RestartPreventExitStatus=75
and marks compatible instances with:
MEDIABOT_SYSTEMD_UPDATE_SAFE=1
The deployment worker remains inside the service cgroup while the main Mediabot process exits.
With:
ExitType=cgroup
systemd waits until the complete update worker has finished before considering the service gone.
Only then can the new Mediabot release start.
old Mediabot
β
update worker
β
SIGTERM bot
β
bot exits
β
βββ updater still alive
β
archive release
β
activate new release
β
updater exits
β
cgroup becomes empty
β
systemd restart
β
NEW Mediabot
No arbitrary race against RestartSec.
m die still means dieAn intentional shutdown now uses exit status:
75
The service declares:
SuccessExitStatus=75
RestartPreventExitStatus=75
This gives systemd two pieces of information:
Restart=always.So:
m die
β
exit 75
β
clean shutdown
β
NO restart
Exactly as requested.
A normal administrative:
systemctl stop mediabot@instance
also remains a real stop.
deploy_update.sh no longer blindly assumes that systemd will restart Mediabot.
Before stopping the running bot, it verifies that the detected unit has the required restart policy.
If the environment is unsafe, the update aborts before the destructive part begins.
That is considerably better than discovering the problem after Mediabot has already disappeared.
The long-standing template architecture remains unchanged:
mediabot@dev
mediabot@undernet
mediabot@anything
with instance-specific configuration provided through:
/etc/default/mediabot-<instance>
and:
BOT_DIR
BOT_BIN
BOT_CONF
The existing bash -lc, stdbuf and multi-instance layout were deliberately preserved after checking the systemd configuration already running reliably in production.
The temporary hard-coded service used during diagnosis was just that: a diagnostic workaround, not the public design.
The targeted MB645 tests passed:
PASSED : 107/107
The entire Mediabot suite then passed:
RC=0
PASSED : 12918/12918
Runtime checks also covered the important behaviours:
update-style SIGTERM
β bot restarts
m die
β clean exit
β bot stays stopped
systemctl stop
β bot stays stopped
The distinction between a crash, an update and an intentional shutdown is now explicit.
MB645 does not merely add a restart.
It defines who is responsible for the complete lifecycle of an update.
Mediabot handles the application.
The updater handles the release transition.
systemd supervises the complete operation and starts the new release only when the previous update process has truly finished.
The result is a much safer spell:
m update
can now replace Mediabot and bring Mediabot back.
Which, considering the amount of magic involved, feels like a fairly reasonable requirement.
You must be logged in to reply.