Forum teuk.org

⚡ Mediabot v3 — A Fast Validation Lane Without Weakening the Full Suite

in Mediabot · started by TeuK · 5d ago

TeuK · 5d ago

Commit: d6585dc
Version: 3.4dev-20260818_032740

MB661 completes an important part of the validation-efficiency roadmap: Mediabot now has a deterministic fast validation lane that is deliberately different from — and explicitly not equivalent to — the full test suite.

The goal is not to make tests disappear.

The goal is to stop paying the full 14–15 minute cost after every small development round while preserving a strong global checkpoint.


⚡ New --fast validation mode

The test runner now supports:

perl t/test_commands.pl --fast

The final reviewed lane contains:

303 / 726 test files

It combines:

all eligible primary PURE tests
+ mandatory cross-cutting sentinels
- explicitly profiled slow cases

The result is deterministic and documented.


🧭 PURE does not mean quick

The first MB661 pass selected 317 files.

It worked:

PASSED : 5619/5619

but still took:

427s

The profiler showed why.

Some tests classified as dependency-light were still very slow:

612_mb394_trivia_rate_limit_retry.t          ~55s
239_usercommands_wave4_subs.t                ~27s
236_usercommands_karmahist.t                 ~24s
707_mb497_seen_enriched.t                    ~20s
190_seen_respects_channel_scope.t            ~20s
192_seen_persisted_respects_channel_scope.t  ~20s

So MB661 was adjusted based on measured data instead of assuming that classification and speed were the same concept.


🧪 Explicit slow exclusions

The final fast lane has an explicit manifest of profiler-confirmed slow tests.

This is not a hidden skip mechanism.

The exclusions are:

reviewable
deterministic
documented
covered by the full suite

And there is an important precedence rule:

mandatory sentinel > slow exclusion

So if an essential cross-cutting contract is slow, it still remains in --fast.

For example:

556_mb335_test_runner_contract_isolation.t

remains mandatory even though it costs roughly ten seconds.


🛡️ Fail-closed sentinels

MB661 keeps 11 cross-cutting sentinel tests in the fast lane.

They cover areas such as:

test-runner crash protection
TAP/process isolation
dispatch/export integrity
module structure
MB650 profiler contract
MB660 classifier contract
MB661 fast-lane contract itself

The sentinel list is fail-closed.

If an expected sentinel is renamed or removed, the runner refuses to silently execute a weakened fast lane.


🚫 --fast is not the full suite

This is a core part of the contract.

MB661 explicitly documents and reports that:

--fast != full suite

The default command remains unchanged:

perl t/test_commands.pl

and still means the complete suite.

MB661 introduces no automatic replacement of full validation.


🔬 Final real-host result

After the profiler-informed tuning, the real server produced:

Profiled 303 test file(s)
PASSED : 5175/5175

Runtime:

177s

That is approximately:

2 minutes 57 seconds

The latest complete suite had taken approximately:

872 seconds

So the new development-validation lane is nearly five times faster while still exercising more than five thousand assertions and the mandatory cross-cutting contracts.


🧪 New validation policy

The intended development workflow is now:

Small development round
    syntax checks
    + source guards
    + targeted regression tests
    + --fast

Cross-cutting checkpoint
    full suite

Stable release checkpoint
    full suite mandatory

The fast lane is therefore an additional layer, not a weaker definition of “all tests passed”.


🧠 A useful lesson from MB661

The work also exposed a broader test-engineering issue.

More tests are not automatically better if their cost becomes high enough that developers avoid running them.

Likewise, removing tests merely to improve a timing number would be the wrong optimisation.

MB661 instead starts separating three questions:

What dependency does this test have?
How much does this test cost?
When does this test need to run?

Those are different properties.

That distinction should guide future work on the suite.


🧪 Validation performed

Representative runner and fast-lane regression:

PASSED : 271/271
RC=0

Corrected structural guard:

Fast lane       : 303/726
Sentinels       : 11
Slow exclusions : 15

OK: default full-suite behaviour remains untouched
OK: 11 cross-cutting sentinels are fail-closed
OK: 15 profiler-confirmed slow cases are explicit
OK: mandatory sentinels override slow exclusions
OK: dominant Trivia timing test is outside default --fast
OK: mandatory runner sentinel remains inside --fast
OK: no --jobs/--parallel executor exists
OK: README documents PURE != quick and --fast != full
MB661_SOURCE_GUARD=OK
RC=0

Final real-host fast lane:

PASSED : 5175/5175
RC=0
Runtime: 177s

A full suite was intentionally not rerun for this commit.


🧱 Operational scope

MB661 introduces:

no database schema change
no migration
no runtime configuration change
no service change
no systemd change
no production deployment change
no parallel test executor

Committed scope:

CHANGELOG.md
README.md
VERSION
t/test_commands.pl
t/lib/FastValidation.pm
t/cases/843_mb661_fast_validation_lane.t

✨ Commit

d6585dc
⚡ Give Mediabot a Fast Validation Lane Without Weakening the Full Suite

Version:

3.4dev-20260818_032740

MB650 taught us where the suite spent its time.

MB660 taught us what kind of tests we had.

MB661 finally gives us a practical way to validate everyday changes without waiting a quarter of an hour every time — while keeping the full suite exactly where it belongs.

You must be logged in to reply.