Forum teuk.org

πŸ”— Mediabot v3 β€” MB492 β†’ MB495: URL previews that finally feel alive

in Mediabot Β· started by TeuK Β· 1mo ago

TeuK Β· 1mo ago

Mediabot v3 keeps moving on the 3.2dev-* line while the future 3.3 is being polished.
This pass focuses on a very visible daily feature: what happens when someone simply pastes a URL into IRC.

The goal was clear: no abstract tooling, no side quest β€” when a user drops a link, the bot should answer with a clean badge and useful details.


🎡 Apple Music gets proper rich details

Apple Music was already detected, but the output was too thin: mostly a raw title.

MB492 makes Apple Music much closer to the Spotify experience by extracting useful metadata from the page:

[AppleMusic] Abbey Road (2019 Mix) - by The Beatles - album - 1969 - 17 tracks

The enrichment uses data already present in the fetched HTML, including JSON-LD and Open Graph metadata.

The existing fallback cascade is preserved: if no rich metadata is found, the bot still falls back to the previous title behavior.


🐦 X / Twitter now shows the actual post text

Before this work, an X URL could produce something too vague, such as:

[X] jack on X

Now the bot can include the post text:

[X] jack on X: "just setting up my twttr"

MB494 then makes this much faster by using the public fxtwitter JSON path first for status URLs.
Chromium remains available as a fallback, but it is no longer the normal path for common X posts.

The richer output can also include compact engagement stats when available:

[X] Name (@screen) on X: "post text" (12.3k likes, 4.2k RTs)

πŸ“˜ Facebook becomes readable

Facebook was the painful one.

The bot could fall back to poor labels such as:

[Facebook] Facebook reel

or show ugly encoded entities like:

377 тыс. ... | Would you be so kind...🥹

This pass fixes the root causes:

hex HTML entities are decoded
Facebook reel counter prefixes are stripped
crawler-style HTTP fetch is used before Chromium
Accept-Language is stabilized for Facebook and Instagram

The expected result is now closer to:

[Facebook] Would you be so kind as to feed her?πŸ₯Ή #viralreels - Some reel description

πŸ“Έ Instagram gets a fast path too

Instagram had the same problem pattern as Facebook: the normal browser-like fetch could miss useful Open Graph metadata, then Chromium could be slow or blocked.

MB494 switches the first HTTP fetch to a crawler-style User-Agent for Instagram too.

Result: many Instagram links can now be answered quickly, without starting a browser.


⚑ Chromium is now the fallback, not the main road

A big production issue was found: Chromium could die immediately with SIGTRAP under systemd.

MB493 hardens the Chromium wrapper:

unique throwaway user-data-dir under /tmp
old profile cleanup
--disable-crash-reporter
--disable-breakpad
--no-first-run
--headless instead of --headless=new
stderr logged when Chromium dies by signal

This is important, but the bigger performance win is architectural: the fast HTTP/API paths are now tried first.

Chromium is still there, but only as a rescue path.


🧼 Entity decoding fixed at the root

MB495 fixes a real decoding bug in _decode_html.

Named entities and decimal entities were already decoded, but hexadecimal entities were not triggering decoding.

Now these forms are handled too:

 
т
🥹

That benefits Facebook immediately, but it also improves all URL handlers that rely on the shared decoder.


πŸ›‘οΈ What stayed intact

This pass deliberately preserves the existing safety boundaries:

no schema migration
no database change
existing URL chansets preserved
private/internal URL skip still present
Chromium URL remains a final argv argument after validation
badge reset invariant preserved
fallbacks preserved
cache behavior preserved

This was not a rewrite. It was targeted surgery on working code.


βœ… Validation target

Before commit, run the focused tests:

perl t/test_commands.pl --filter '702_mb492|703_mb493|704_mb494|705_mb495'

Then the usual guards:

perl tools/security_audit.pl
perl tools/startup_integrity_check.pl
git diff --check

And finally the full suite:

perl t/test_commands.pl

πŸ§ͺ Live checks after deploy

Paste real links in a channel:

Apple Music album / song
X status URL
Facebook reel URL
Instagram reel URL
Spotify link
generic web page

Useful examples:

https://music.apple.com/us/album/abbey-road-2019-mix/1474815798
https://x.com/jack/status/20
https://www.facebook.com/reel/1672309937302951

Expected behavior:

Apple Music shows artist/type/year/tracks when available
X shows post text and possibly stats
Facebook does not show raw &#x... entities
Facebook reel counter prefixes disappear
Instagram answers quickly instead of staying silent
Chromium stderr is visible in logs if fallback crashes

πŸ¦‰ Summary

This pass makes URL previews feel much more modern:

richer Apple Music
real X post text
faster X via fxtwitter
faster Facebook and Instagram via crawler-style fetch
more robust Chromium fallback
hex entity decoding fixed globally
Facebook reels cleaned up

This is exactly the kind of polish that makes Mediabot feel alive in a real IRC channel.

You must be logged in to reply.