Forum teuk.org

⏳ MB679 — Give the Test Suite a Progress Charm Without Making Noise

in Mediabot · started by TeuK · 2d ago

TeuK · 2d ago

MB679 improves the developer experience of Mediabot’s test runner without changing test semantics.

The full and fast suites were already deliberately redirected to log files during long validation rounds because printing thousands of TAP lines is noisy and difficult to follow. The new --progress option provides a compact live view directly inside the official t/test_commands.pl runner.

The goal was intentionally modest:

  • one dynamic terminal line;
  • no TUI;
  • no spinner framework;
  • no threads;
  • no additional dependency;
  • no hard-coded assertion total;
  • failure details preserved;
  • existing runner behaviour unchanged unless --progress is requested.

⏳ New option

The runner now accepts:

--progress

Examples:

perl t/test_commands.pl --progress

and:

perl t/test_commands.pl --fast --progress

A typical live line looks like:

[=========>          ] 43% [324/754 files | 6318 tests] | running: 575_mb356_scheduler_lifecycle_generation.t

The same terminal line is continuously rewritten.

Passing test output stays hidden while progress mode is active.


📊 What the counters mean

The percentage is based on the exact number of selected test files:

324 / 754 files

The assertion counter is the real number of assertions completed so far:

6318 tests

The runner does not hard-code a total such as 14739.

That total can change whenever tests are added or modified and is only known naturally as the suite executes.

At completion, the final line is clean:

[====================] 100% [754/754 files | 14739 tests]

No running: suffix remains after the last file has completed.


🧹 Terminal cleanup

During validation, one subtle terminal behaviour was discovered.

A carriage return:

\r

moves the cursor back to the beginning of the current line but does not erase characters remaining from a previous, longer line.

This initially produced a visual artefact such as:

[====================] 100% [2/2 files | 23 tests] | running: 831_...

even though the underlying final progress string no longer contained running:.

The final implementation explicitly clears any leftover characters before rendering the shorter completion line.

This solution stays deliberately simple:

  • no ANSI dependency;
  • no cursor-control library;
  • no asynchronous timer;
  • no background thread.

🧪 Failure behaviour

Passing cases remain quiet in progress mode.

Failure output is retained and reported after the progress display rather than flooding the terminal while the suite is running.

The normal final summary remains unchanged.

This preserves the useful distinction between:

live progress

and:

actionable failure diagnostics

🔒 Compatibility

--progress is strictly opt-in.

Running:

perl t/test_commands.pl

without the new option keeps the historical runner behaviour.

The fast lane is also supported:

perl t/test_commands.pl --fast --progress

The incompatible combination:

--progress --verbose

is explicitly rejected because verbose TAP output and single-line progress output serve opposite purposes.


🧪 Runner contracts

A dedicated contract was added:

t/cases/871_mb679_test_runner_progress.t

It protects:

  • option parsing;
  • progress rendering;
  • current-test display;
  • completion rendering;
  • final-line cleanup;
  • output capture;
  • fast compatibility;
  • failure deferral;
  • absence of hard-coded test totals;
  • preservation of historical runner semantics.

The relevant runner contract group reached:

PASSED : 144/144

⚡ Fast suite with progress

Real server validation:

perl t/test_commands.pl --fast --progress

The runner selected:

318 of 754 discovered test files

Result:

[====================] 100% [318/318 files | 5787 tests]

PASSED : 5787/5787  (64s)

🏰 Full suite with progress

The complete suite was then executed using the new mode:

perl t/test_commands.pl --progress

Result:

[====================] 100% [754/754 files | 14739 tests]

PASSED : 14739/14739  (209s)

The new display therefore survived the complete real test workload, not only synthetic or focused runner tests.


🧠 Why this matters

Mediabot’s suite has grown beyond fourteen thousand assertions.

At that size, developers need visibility during a full run, but printing every TAP line is not useful feedback.

MB679 adds enough observability to answer the important questions immediately:

Is the suite still moving?
How far through the selected files are we?
How many assertions have completed?
Which test file is currently running?

without turning the runner into a dashboard framework.

The implementation remains small, dependency-free and optional.


📦 Commit

Version : 3.4dev-20260821_084504
Commit  : 169e979
Branch  : master

⏳ Give the Test Suite a Progress Charm Without Making Noise

Commit summary:

3 files changed
349 insertions
2 deletions

New test:

t/cases/871_mb679_test_runner_progress.t

The commit was successfully pushed to GitHub.


The spell does not make the tests run faster. It simply lets you see the staircase moving.

You must be logged in to reply.