nordisk/nornir

Test & bench chains — ordering, dependencies, and the heavy-skip guard

Design note (durable; a chapter of nornir docs book). Companion to bencher.md and the heavy-opt-in rule there. Beginner version: bench-vs-test-for-idiots.md.

The problem

A test or bench may produce an artifact a later one consumes — znippy's canonical example: unpack OSM XML/PBF first, then downstream steps read it. Two hazards follow once heavy tests are skipped by default (see bencher.md):

1. Order isn't guaranteed for tests. The matrix runs cargo nextest run (parallel) / cargo test; there is no dependency ordering between #[test]s.
2. Heavy-skip can orphan a dependent. If a heavy producer (#[ignore]'d, now skipped by default) feeds a non-heavy consumer, the consumer runs against a missing/stale fixture and fails for the wrong reason — an "idiot problem".

Current state (audited 2026-06-17)

• Benches DO have ordering. register_bench_ordered!(n, …) pins a stable sequence; run_main_json sorts by (order, id). Staged pipelines check their prerequisite explicitly and degrade to a recorded-red skip, not a crash:

// znippy xtask/examples/nornir-bench.rs — stage 1 guards on stage 0's output
if !input.exists() {
    return Err(anyhow!("planet pbf missing at {} — stage 0 must produce it first", …));
}

NORNIR_BENCH_STOP_ON_ERROR=1 makes a stage failure abort the rest (pipelines where stage N+1 can't run without N). This is the pattern to generalize.

• Tests do NOT have ordering — and rely on the LAZY-FIXTURE convention instead. znippy is clean: every test is self-sufficient via on-demand fixtures (ensure_jar(), assets::ensure(), in-memory fixtures()), no #[ignore], no #[serial], no producer-test. So heavy-skip is safe there today. But nothing enforces the convention — the footgun is one careless test away.

LAW — tests use lazy on-demand fixtures, not inter-test ordering

The first and cheapest defence (no new machinery):

A test MUST create/ensure its own inputs on demand (a OnceCell / ensure_*() / cache-dir-checked-then-populated helper). A test MUST NOT depend on another test having run first. If a producer step is genuinely expensive and shared, it is a fixture helper (any caller triggers it) or a bench stage (ordered) — never a sibling #[test] that must run first.

This is exactly znippy's pattern; it makes execution order irrelevant and makes heavy-skip safe (the consumer lazily builds what it needs).

The chain graph (warehouse + viz) — the detector

A convention isn't a guarantee. We make chains first-class and detectable:

Model

A chain is a DAG of steps (a test case or a bench stage) with produces / consumes artifact keys. Edges = producer→consumer. Each step carries its heavy flag (tests) / order (benches).

Warehouse

A new historized table test_chains (mirrors test_inventory / dep_graph): rows (repo, chain_id, step, kind=test|bench, produces, consumes, heavy, order, ts_micros, git_sha). Populated by the syn-scan inventory (#[nornir::chain(produces="osm.pbf")] / consumes=… attribute, or a per-repo chains.toml) so it is discoverable without running anything — same philosophy as the build-free test inventory.

Detection / gate (the "idiot guard")

Over that DAG, nornir enforces:

1. No light-depends-on-heavy. A non-heavy step MUST NOT consume what only a heavy step produces. Flagged RED in the matrix + a release gate (no_light_on_heavy): "step B (light) consumes osm.pbf produced only by step A (heavy) — promote B to heavy, or make the producer a lazy fixture."
2. Skip-propagation. When a heavy producer is skipped (default run), its transitive dependents are auto-skipped and recorded as skipped:heavy-prereq (NOT failed) — the generalization of the bench PlanetPbfToGeo graceful skip. The matrix shows them greyed, not red.
3. Order from the DAG. Execution order is the topological order of the chain (benches already do this via order; tests inside a chain get --test-threads=1 + topo order for that chain only).

Viz

A chain graph in the 🧪 Test tab, rendered with the shared facett node/edge Scene (the same renderer as the dep graph / arch graph): nodes = steps coloured by state (ok/fail/skip/heavy-skipped/not-run), edges = produces→consumes. A skipped heavy root visibly greys its downstream subtree, so "heavy #1 skipped ⇒ light #2 can't run" is obvious at a glance. CLI parity: nornir test chains [--repo R] prints the same DAG + the gate verdict (per the CLI-parity LAW).

Sequencing

Per the owner: do the agreed clone→bench + downscale work first (see bench-test-classification-audit.md), then build this chain graph. The LAW above (lazy fixtures) is in force immediately and costs nothing; the warehouse table + gate + viz graph are the "big job" that follows.

Status

• Ordering audit (benches ordered; tests lazy-fixture; znippy clean — no regression)
• LAW documented: lazy fixtures, no inter-test ordering
• test_chains warehouse table + syn-scan / chains.toml populate
• no_light_on_heavy detection + skip-propagation in the matrix runner
• chain-graph viz in the Test tab + nornir test chains CLI