Forum teuk.org
Login Register

📻 Mediabot v3 Radio Integration: Liquidsoap Queue Control, MP3 Cache Playback, and Safer YouTube Downloads

in Mediabot · started by TeuK · 3d ago

TeuK · 3d ago

Over the last development cycle, Mediabot v3 received a serious radio upgrade — not a quick spell taped to the side of the bot, but a proper bit of wizardry forged carefully in the dungeons.

The goal was simple enough to say, but not simple to do:

make Mediabot able to control the radio stack cleanly, play from a local MP3 cache first, and use yt-dlp safely without freezing the IRC loop.

In other words: less cursed blocking magic, more disciplined Hogwarts engineering.

This work now gives Mediabot a safer foundation for radio playback around Liquidsoap, Icecast, a local MP3 cache, and controlled yt-dlp downloads.

It is not the final spellbook for the radio system. YouTube authentication and cookies remain a moving target. But this is now a solid base to build on.

🧙 Why this work was needed

The old radio-related behavior had several limitations:

  • radio controls were not centralized cleanly;
  • YouTube downloads could become operationally risky;
  • local MP3 cache handling needed to be stronger;
  • IRC commands needed clearer feedback;
  • production instances needed protection from accidental remote downloads;
  • yt-dlp failures could be noisy and unpleasant on IRC;
  • some status commands were listed in help but not properly wired in public dispatch.

The new approach is deliberately conservative:

Local cache first.
Remote download second.
Production safety always.

That is the core charm behind this integration.

🧪 Development radio stack

The current focus was the development radio stack on teuk.org.

The dev setup uses a separate Icecast/Liquidsoap path so the radio work can be tested without disturbing older production radio services.

Expected dev runtime settings:

LIQUIDSOAP_TELNET_HOST=127.0.0.1
LIQUIDSOAP_TELNET_PORT=1235
LIQUIDSOAP_QUEUE_ID=mediabot_queue

RADIO_ICECAST_STATUS_BASE_URL=http://127.0.0.1:8000
RADIO_ICECAST_PUBLIC_BASE_URL=http://teuk.org:8000
RADIO_ICECAST_PRIMARY_MOUNT=/radio.mp3

The Liquidsoap queue commands verified manually are:

mediabot_queue.push <uri>
mediabot_queue.queue
mediabot_queue.skip
mediabot_queue.flush_and_skip

There is no mediabot_queue.status command in this Liquidsoap setup, so Mediabot does not rely on one.

That avoided summoning a non-existent spell and then wondering why the cauldron exploded.

🪄 New Liquidsoap client module

A new module was added:

Mediabot/Liquidsoap.pm

Its job is intentionally small:

  • connect to the Liquidsoap telnet socket;
  • send one command;
  • send quit;
  • collect the response;
  • return clean success/error information.

The exposed methods are:

push($uri)
queue()
skip()
flush_and_skip()
command($raw_command)

This keeps Liquidsoap socket handling out of random command code and gives the bot a cleaner magical conduit to the radio queue.

📻 New and improved radio commands

The radio command set now includes queue control, cache handling, diagnostics, playback, and download administration.

Queue control

m radioqueue
m radiopush /absolute/path/file.mp3
m radioskip
m radioflush

Playback

m play <query>

MP3 cache administration

m radioimport /absolute/path/file.mp3 [artist - title]
m radioimportdir [directory]
m radiocache
m radiocacheprune
m radiocacheprune confirm

Download administration

m radiodlstatus
m radiodlcancel

Status commands

m radiocheck
m radiostatus
m radiomounts
m song
m listeners

Most of the new radio administration commands are Master-only during rollout.

m play is also still Master-only for now. That is deliberate: this spell is powerful enough that it should not be handed to first-year students in the corridor.

🧺 MP3 cache-first playback

The main playback logic now lives in:

Mediabot/Radio/Request.pm

When someone runs:

m play <query>

Mediabot now follows this order:

  1. search the local MP3 table/cache;
  2. verify that the referenced MP3 file is readable;
  3. push the cached file to Liquidsoap if found;
  4. only try yt-dlp if no usable cache match exists;
  5. respect RADIO_DOWNLOAD_ENABLED before starting any remote download.

This is the most important design decision in the whole integration.

Cache hit = play immediately.
Cache miss = download only if explicitly allowed.

That gives production instances a safe cache-only mode, while the dev instance can still populate the cache.

🧱 MP3 cache hardening

The cache logic was improved to avoid duplicate or stale rows.

The code now:

  • extracts YouTube IDs from URLs or raw IDs when possible;
  • first searches by id_youtube;
  • falls back to fuzzy artist/title search;
  • checks that the file on disk is readable before using a DB row;
  • updates existing rows by id_youtube;
  • updates existing rows by folder + filename;
  • inserts only when no matching row already exists.

This avoids repeated imports and repeated downloads creating a messy MP3 table.

No database schema change was required.

🗂 Local MP3 import commands

Manual SQL insertion is no longer needed for normal radio cache operations.

A single MP3 can be imported from IRC:

m radioimport /home/mp3/TeuK/ab.mp3 TeuK - ab

A directory can be scanned recursively:

m radioimportdir /home/mp3/TeuK

The import logic:

  • accepts readable .mp3 files;
  • can read nearby .info.json metadata;
  • can use a provided artist - title;
  • can fall back to the filename;
  • reuses the duplicate-protection logic.

This makes it much easier to rebuild or repair the cache without hand-editing SQL.

🧹 Cache diagnostics and pruning

A new cache diagnostic command exists:

m radiocache

It reports useful cache information such as:

  • total rows in MP3;
  • readable rows;
  • broken/missing rows;
  • .mp3 files found in the incoming directory;
  • examples of files missing from the database.

A dry-run prune is available:

m radiocacheprune

To actually delete broken DB rows:

m radiocacheprune confirm

Important: this only deletes rows from the MP3 table. It does not delete MP3 files from disk.

No Vanishing Charm is cast on the music.

🛡 Production safety: download opt-in

Remote downloads are controlled by:

RADIO_DOWNLOAD_ENABLED=0

Behavior:

RADIO_DOWNLOAD_ENABLED=0
- m play uses local cache only
- cache misses return a clean message
- no yt-dlp process is started

RADIO_DOWNLOAD_ENABLED=1
- m play uses local cache first
- cache misses may start yt-dlp asynchronously

Recommended policy:

# Development instance
RADIO_DOWNLOAD_ENABLED=1

# Production / public / Undernet instances
RADIO_DOWNLOAD_ENABLED=0

This prevents production bots from accidentally becoming public download endpoints.

It is the difference between a controlled spell and a room full of Cornish Pixies.

⚡ Non-blocking yt-dlp downloads

yt-dlp is now launched as a child process instead of blocking the bot.

The bot monitors the job through the event loop.

Safeguards include:

  • only one active download at a time;
  • timeout enforcement;
  • captured stdout/stderr;
  • temporary job files;
  • job cleanup;
  • DB insert/update after success;
  • Liquidsoap push after success;
  • status and cancellation commands.

Useful commands:

m radiodlstatus
m radiodlcancel

Timeout is configured with:

YTDLP_TIMEOUT=180

Temporary job files live under:

YOUTUBEDL_INCOMING/.mediabot_jobs

The IRC loop stays alive. The bot does not freeze while yt-dlp fights the YouTube basilisk.

🍪 YouTube cookies and EJS support

YouTube now requires extra care.

Mediabot supports a cookies file:

YTDLP_COOKIES_FILE=/home/mediabot/.config/yt-dlp/youtube.cookies.txt

That file must stay outside Git and outside public snapshots.

Recommended permissions:

chown mediabot:mediabot /home/mediabot/.config/yt-dlp/youtube.cookies.txt
chmod 600 /home/mediabot/.config/yt-dlp/youtube.cookies.txt

Mediabot also supports the newer yt-dlp EJS remote components workflow:

YTDLP_REMOTE_COMPONENTS=ejs:github

A direct manual test looks like this:

/usr/local/bin/yt-dlp \
  --cookies /home/mediabot/.config/yt-dlp/youtube.cookies.txt \
  --remote-components ejs:github \
  --skip-download \
  --no-playlist \
  --print "%(title)s" \
  "https://www.youtube.com/watch?v=dQw4w9WgXcQ"

This worked while the cookies were valid.

But cookies are fragile. Google/YouTube can rotate or reject them at any time. Mediabot can detect and explain the problem, but it cannot magically make expired YouTube sessions valid again.

Even Dumbledore would need a fresh session.

🧾 Cleaner IRC messages for yt-dlp failures

Raw yt-dlp output can be huge, technical, and ugly on IRC.

The bot now classifies common failure cases and returns clearer messages for:

  • expired or rotated cookies;
  • missing or rejected cookies;
  • YouTube bot checks;
  • HTTP 403 / player-token / PO token challenges;
  • EJS / JavaScript challenge failures;
  • missing audio formats;
  • rate limiting;
  • unavailable/private videos;
  • local permission errors;
  • disk full errors.

Example of the new style:

Radio: download failed: YouTube blocked this download with a 403/player-token challenge. cookies.txt may be stale, or YouTube changed its checks. Refresh cookies.txt; if it persists, revisit the Chromium/browser-profile plan. See Mediabot v3 wiki: Radio / YouTube cookies.

That future wiki page is not required for the commit, but the message prepares the documentation path.

The important part is that IRC users no longer get a cursed scroll of raw extractor noise.

🩺 radiocheck: the radio preflight spell

A new command gives a quick runtime overview:

m radiocheck

It reports:

  • incoming MP3 directory;
  • whether downloads are enabled;
  • yt-dlp path/version;
  • cookies file readability;
  • remote components setting;
  • timeout;
  • Liquidsoap host/port/queue;
  • Icecast status URL;
  • public stream URL.

Example style:

Radio check:
incoming: OK writable (/home/mediabot/mediabot_v3/mp3)
downloads: enabled
yt-dlp: OK 2026.03.17 (/usr/local/bin/yt-dlp)
cookies: OK readable (/home/mediabot/.config/yt-dlp/youtube.cookies.txt)
yt-dlp remote components: ejs:github
yt-dlp timeout: 180 seconds
Liquidsoap: OK 127.0.0.1:1235 queue=mediabot_queue
Icecast status base: http://127.0.0.1:8000
Icecast public stream: http://teuk.org:8000/radio.mp3

Important detail: radiocheck can verify that the cookies file is readable. It cannot guarantee that YouTube will accept the session.

🎨 Icecast display polish

The old fallback mount was cleaned up from:

/radio160.mp3

to:

/radio.mp3

The m song and m listeners output was also polished with IRC formatting:

  • no unsafe background colors;
  • usable on both dark and white IRC clients;
  • m song keeps the stream URL underlined;
  • m song does not print a [ LIVE ! ] badge;
  • m listeners uses English wording;
  • the visual style stays close to the old capsule spirit without becoming unreadable.

Small detail, but it matters. A spell can work and still look like it came from Filch’s broom cupboard.

📚 Help system improvements

The radio work also exposed a flood risk in:

m help commands
m commands

Instead of dumping too much at once, help output is now categorized and paced.

Examples:

m commands
m help commands
m help commands radio
m help commands admin
m help commands dynamic

This makes command discovery safer on real IRC networks.

No one wants to get kicked for excess flood while reading the spellbook.

🐛 Small bug fixed outside the radio scope

While testing the radio work, a small IRC parsing bug was found.

Sending a public or private message containing only spaces could trigger warnings like:

Use of uninitialized value $sCommand in substr
Use of uninitialized value $sCommand in string eq

The fix is simple: whitespace-only messages are now ignored before command parsing.

That keeps daemon logs cleaner and prevents harmless blank messages from making noise.

⚙️ Configuration example

Development instance:

[radio]
YOUTUBEDL_INCOMING=/home/mediabot/mediabot_v3/mp3

RADIO_DOWNLOAD_ENABLED=1

YTDLP_PATH=/usr/local/bin/yt-dlp
YTDLP_TIMEOUT=180
YTDLP_COOKIES_FILE=/home/mediabot/.config/yt-dlp/youtube.cookies.txt
YTDLP_REMOTE_COMPONENTS=ejs:github

LIQUIDSOAP_TELNET_HOST=127.0.0.1
LIQUIDSOAP_TELNET_PORT=1235
LIQUIDSOAP_QUEUE_ID=mediabot_queue

RADIO_ICECAST_STATUS_BASE_URL=http://127.0.0.1:8000
RADIO_ICECAST_PUBLIC_BASE_URL=http://teuk.org:8000
RADIO_ICECAST_TIMEOUT=5
RADIO_ICECAST_PRIMARY_MOUNT=/radio.mp3

Production/cache-only instance:

RADIO_DOWNLOAD_ENABLED=0

🧪 Suggested test checklist

On the development instance:

m check
m radiocheck
m radiocache
m radiodlstatus
m radiostatus
m radiomounts
m song
m listeners
m radioqueue
m radioimport /home/mp3/TeuK/ab.mp3 TeuK - ab
m play ab
m radioskip
m radioflush
m play Rick Astley Never Gonna Give You Up
m radiodlstatus
m help commands radio

Also test a blank or whitespace-only IRC message. The bot should ignore it cleanly.

🦉 What is deliberately not finished

The automatic Chromium/browser-profile workflow is not implemented yet.

A possible future direction is:

yt-dlp --cookies-from-browser chromium:/home/mediabot/.config/chromium-mediabot-youtube

and a future config key such as:

YTDLP_COOKIES_FROM_BROWSER=chromium:/home/mediabot/.config/chromium-mediabot-youtube

But this needs more work and care.

A fully automatic headless Google login is not a clean target. Google can ask for 2FA, CAPTCHA, browser verification, or rotate sessions at any time.

The current decision is sane:

  • commit the radio foundation now;
  • keep production cache-only;
  • document cookies as fragile;
  • revisit Chromium/browser-profile support later.

🏁 Final status

This is not the final form of Mediabot radio playback.

It is the safe foundation.

Mediabot can now:

  • control Liquidsoap cleanly;
  • query and manage the queue from IRC;
  • play from a local MP3 cache first;
  • import and diagnose MP3 cache entries;
  • run yt-dlp without blocking the IRC loop;
  • protect production instances with RADIO_DOWNLOAD_ENABLED=0;
  • explain YouTube/cookie failures clearly;
  • keep Icecast status commands usable;
  • present radio output more cleanly on IRC;
  • avoid warning noise on blank messages.

That is a pretty good spell.

Or, in proper Hogwarts commit language:

📻 Accio Airwaves: add safe radio cache playback and Liquidsoap controls

You must be logged in to reply.