Forum teuk.org

🛠️ MB687 — Teaching Debian 13 the Systemd Installation Charm

in Mediabot · started by TeuK · yesterday

TeuK · yesterday

Mediabot’s Debian 13 readiness work has now crossed the systemd boundary.

MB684 proved the fresh configuration/bootstrap path.

MB685 proved fresh MariaDB installation.

MB686 proved the stable 3.3 → current database migration path.

MB687 makes the published systemd service contract installable, testable, fail-closed — and it has now been validated on the real development service.

🧰 A supported systemd installer

MB687 adds:

install/systemd_install.sh

The helper installs Mediabot’s published multi-instance systemd template and optionally creates an instance environment file.

It is intentionally conservative:

identical existing file
    → accepted

different existing file
    → refused

different file + explicit replacement
    → backup + controlled replacement

symlink target
    → refused

Supported replacement controls include:

--replace-template
--replace-instance
--template-only

The installer does not implicitly start, restart, stop, or enable Mediabot.

When live systemd files change, its automatic systemd action is limited to:

systemctl daemon-reload

Operational activation remains an administrator decision.

🧙 The MB645 lifecycle contract becomes the real shared template

The published template carries the lifecycle previously proven through the MB645 development-instance override:

ExitType=cgroup
Environment=MEDIABOT_SYSTEMD_UPDATE_SAFE=1
Restart=always
SuccessExitStatus=75
RestartPreventExitStatus=75
RestartSec=10s

This allows updater work to remain inside the service cgroup while still preserving an explicit final shutdown through exit status 75.

Because ExitType=cgroup requires systemd 250 or newer, the installer refuses unsupported live systemd versions instead of pretending the lifecycle is portable.

🧪 Debian 13 CI gate

MB687 adds:

t/cases/889_mb687_debian13_systemd_install_gate.t

The Debian 13 gate exercises the real installer against an isolated filesystem tree and validates:

  • initial installation;
  • exact template installation;
  • instance environment creation;
  • idempotency;
  • fail-closed divergent-file handling;
  • explicit controlled replacement;
  • symlink refusal;
  • lifecycle contract presence;
  • systemd-analyze verify.

Validation completed successfully:

Targeted : 218 / 218
Fast     : 5992 / 5992
Full     : 15605 / 15605

Commit:

596ab5e 🛠️ Teach Debian 13 the Systemd Installation Charm

Version:

3.4dev-20260822_101942

🏰 Real host audit exposed the migration boundary

Before changing the live system, the production host showed two different effective contracts.

The shared template still used the old lifecycle:

Restart=on-failure
ExitType=main

The development instance had already received the MB645 lifecycle through:

/etc/systemd/system/mediabot@dev.service.d/mb645-test.conf

while the Undernet instance had no override and therefore still used the legacy shared template.

That meant replacing the shared template could not honestly be called a “dev-only” change.

🛡️ Controlled DEV-first live rollout

The live rollout therefore preserved Undernet’s old lifecycle explicitly before replacing the shared template.

A temporary compatibility drop-in was installed for Undernet:

/etc/systemd/system/mediabot@undernet.service.d/mb687-legacy-preserve.conf

It preserves:

ExitType=main
Restart=on-failure
SuccessExitStatus=
RestartPreventExitStatus=
UnsetEnvironment=MEDIABOT_SYSTEMD_UPDATE_SAFE

Only after that protection was active and verified did the rollout install the new shared template.

The obsolete development-only MB645 test drop-in was then removed.

✅ Live result

Before migration:

DEV PID      : 3626224
Undernet PID : 2899306

After installing the shared template and removing the obsolete development drop-in, neither process had restarted.

Then DEV alone was explicitly restarted:

DEV PID before : 3626224
DEV PID after  : 3664795

Undernet remained:

Undernet PID : 2899306

unchanged throughout the operation.

Final DEV contract:

ExitType=cgroup
Restart=always
SuccessExitStatus=75
RestartPreventExitStatus=75
Environment=MEDIABOT_SYSTEMD_UPDATE_SAFE=1
DropInPaths=
ActiveState=active
SubState=running

Final Undernet contract:

ExitType=main
Restart=on-failure
SuccessExitStatus=
RestartPreventExitStatus=
Environment=MEDIABOT_SYSTEMD_UPDATE_SAFE=1
UnsetEnvironment=MEDIABOT_SYSTEMD_UPDATE_SAFE
DropInPaths=/etc/systemd/system/mediabot@undernet.service.d/mb687-legacy-preserve.conf
ActiveState=active
SubState=running

The visible Environment= value reflects the shared template configuration; the compatibility drop-in explicitly removes the marker from the spawned Undernet process through UnsetEnvironment=.

🔁 Fail-closed rollout proved itself too

The first rollout attempt intentionally stopped when one validation rule was too strict.

The automatic rollback restored:

  • the original shared template;
  • the DEV MB645 drop-in;
  • the previous effective contracts;
  • both original process IDs.

No bot was inadvertently restarted.

The corrected rollout then completed successfully.

That is useful evidence in itself: the migration tooling did not merely succeed on the happy path — its rollback path was exercised on the real host before the successful deployment.

📜 Clean shutdown during DEV restart

The journal recorded the expected controlled shutdown:

Received SIGTERM. Initiating clean shutdown.
Cleaning and exiting...
Achievements: final save() before exit
mediabot@dev.service: Deactivated successfully.
Started mediabot@dev.service

The systemd migration therefore reached the live boundary without forcing an unclean Mediabot termination.

⚠️ Undernet intentionally remains a separate migration

Undernet has not been migrated to the new lifecycle yet.

Its compatibility drop-in must remain present until a separate validated Undernet migration is performed.

This separation is intentional:

shared template modernized
        ↓
DEV validated first
        ↓
Undernet preserved
        ↓
Undernet migration later, explicitly

MB687 therefore closes the systemd installation and DEV live-rollout boundary without pretending that every production instance has already been migrated.

A reliable deployment spell is not the one that changes everything at once. It is the one that proves exactly what changed — and proves just as carefully what did not. 🪄

You must be logged in to reply.