.nornir/bench-vs-test-for-idiots.md · nordisk/nornir

Bench vs test — for idiots

Short version. The long version is bencher.md; the per-repo work-list is bench-test-classification-audit.md.

They are two different things

Test = does the code give the right answer? Runs automatically on every nornir release, every viz "Run full matrix", every gate. Output: pass/fail.
Bench = how fast is it? Runs only when you ask (nornir bench run). Output: a number (MB/s, rows/s, ms). Historized; gates on regression.

A plain nornir test runs no benchmarks. (The bench-smoke column just does cargo bench --no-run = "does the bench code compile", nothing more.)

The trap

Some tests secretly do bench work — download 26 JARs, build a graph over 50 crates, scan a 15 GB table. That's what made nornir test take a minute and spin the fans. Fix it per test:

Does it produce a meaningful speed number (MB/s, rows/s, ms-for-N)?
- Yes, and a bench already measures it → just shrink the test (fetch 1 JAR, not 26). The bench keeps the full workload.
- Yes, but no bench exists → make a bench from it (full scale) and keep a shrunk test for correctness.
- No (the slow part is a download or a server boot) → leave it a heavy test.

Heavy = opt-in (the new rule)

Heavy / #[ignore]'d tests (big corpora, network, model downloads, scale matrices) now run only if you pass --heavy. Default OFF for release, nornir test, and viz. --skip-heavy does nothing now (off is already default).

nornir test            # fast: no heavy arms
nornir test --heavy    # includes the heavy/network/scale tests
nornir release         # fast: synthetic tier gates; heavy OFF
nornir release --heavy # includes heavy arms

Benches are NOT affected by --heavy. You ran nornir bench run on purpose — you want the heavy measurement. A bench that needs network/a giant fixture skips itself via nornir.toml's network_required + a fixture check, not --heavy.

Don't chain tests by order (the idiot trap)

Tempting: "test #1 unpacks the OSM file, test #2 reads it." Don't. Tests run in parallel and in any order, and if #1 is heavy it's skipped by default — then #2 blows up for no good reason.

A test must make its own inputs (a OnceCell / ensure_xxx() helper that downloads-or-reuses a cache). Then order doesn't matter and nothing breaks when a heavy arm is off. (This is how znippy already does it — copy that.)
If a step is genuinely expensive and shared, it's a bench stage with an explicit order (register_bench_ordered!) that checks its prerequisite and skips gracefully — not a sibling #[test].

Coming soon: nornir will draw the chain graph (in the Test tab + warehouse) and refuse a light test that depends on a heavy one. See test-bench-chains.md.