nordisk/nornir

The test function — reasoning

Class: reasoning. This file holds the why, the trade-offs, and every design we rejected. The descriptive shape we actually run is in test-design.md, which by rule mentions none of this; the CLI is in test-guide.md. If you want "why not X?" — X is here.

f0 — why one gate at all

The faces (f1…f6) have separate runners, tables, and viz tabs on purpose (see the per-function rejections below), yet they converge on one row type flowing through one sink into one warehouse, read back as one green/red board. The reason is a single failure mode we kept hitting: drift — a test that "passed" locally, a README number that went stale, a surface no test ever reached. If every verdict is a persisted, queryable row, then "is this green?" is a warehouse question with one answer, and a release can assert the whole is green rather than trusting a pile of independently-run tools. The creed behind it — green ⟺ the shipped surface, on real data, proven ran-and-correct from emitted data, and the same oracle goes RED when it breaks — is the eight-law doctrine in agent-ui-test-fixing.md; the persistence is what makes those laws enforceable instead of aspirational.

Why persist to a warehouse at all, not just exit codes? An exit code is a fact that exists for one second. A row time-travels: nornir test history, the trend sparkline, the "was this green in the last green run?" regression check, and the completeness gate all need yesterday's verdict, not just today's. The warehouse is the only place that has it.

f1 — why one multi-aspect matrix

Why fold build/clippy/fmt/audit into the test matrix instead of leaving them as separate CI steps? Because to an operator they answer the same question — "is this repo healthy?" — and a scattered set of tools each with its own output format is exactly the drift surface we are trying to kill. One Aspect enum, one row schema, one board means clippy red and a unit-test red are the same kind of fact, comparable and historized together. A missing tool becomes a neutral skip row rather than a hard fail so a repo without cargo-audit still renders a matrix instead of a crash.

Rejected: run tests on workspace load. The tempting design is "scan a workspace, show its live test results." But running a test needs a cargo build — slow, fan-spinning, and wrong to do on every populate/poll. So we split discovery from execution: the syn knowledge scan discovers every #[test] for free (it already parses every .rs), giving each an immediate X (not-run) cell; execution stays gated to nornir release / nornir test run. X is its own state — never conflated with pass or skip — precisely so "exists but never run" is visible, which is the anti-drift win. (Detail: test-matrix-always-visible.md.)

Rejected: cargo test --format terse. Terse collapses each test to a single character and loses the names. A matrix needs names to be a matrix, so we parse the default human grammar (or, better, nextest's structured libtest-json, which additionally gives per-test durations + captured failure messages). Installing cargo-nextest upgrades any repo to full fidelity with zero config — a carrot, not a mandate.

Rejected: a hand-maintained test registry. Any registry drifts from the code the moment someone adds a #[test] and forgets to list it. Discovery is data-driven (syn scan + runner --list) so the inventory is always exactly what the source contains.

Rejected: heavy/network tests run by default. ~90% of #[ignore]'d tests are infra (containers, VM boots, model downloads, live RPCs), not scale. A flaky network arm turning a release red is the opposite of a gate — it makes red meaningless. So heavy is opt-in (--heavy), default OFF for test, release, and the viz "Run full matrix"; the always-run synthetic tier is the gate. --skip-heavy is kept as a no-op only so old scripts don't break.

Rejected: inter-test ordering / a producer #[test]. "Test #1 unpacks the OSM file, test #2 reads it" is a footgun: tests run in parallel and in any order, and if #1 is heavy it is skipped by default, so #2 blows up for the wrong reason. The LAW is lazy on-demand fixtures — every test builds its own inputs (a OnceCell/ensure_*() helper) so order is irrelevant and heavy-skip is safe. A genuinely shared expensive step is a fixture helper or an ordered bench stage (register_bench_ordered!), never a sibling test. Making chains first-class + a no_light_on_heavy gate is the follow-up designed in test-bench-chains.md.

Rejected: "one value passed ⇒ covered." A green test that exercises one input proves almost nothing. utfallsrum (sample space, Ω) records the shape a test swept — boundary-value + equivalence partitions — and scores coverage as exercised/declared classes, so a single-value smoke scores low even when green. The completeness gate consumes it so a surface flips to covered only when it both ran end-to-end AND swept ≥K outcome classes.

Rejected: a coverage ratchet. A "coverage may only go up" ratchet tolerates permanent gaps as long as they don't grow. The surface-completeness gate is instead a HARD zero: Gap = Surface − Covered − Allowlist must be ∅, and a stale allowlist entry (its surface now covered, or gone) re-reds the gate. An excuse must not outlive its surface. Why an allowlist at all rather than just failing? To let a real gap be named with a reason and burned down by deletion as tests land, instead of blocking all work at once. (Detail: completeness-gate.md, aut5-surface-coverage-gate.md.)

f2 — why bench is a gate, and why it's separate from test

Why gate on speed? A performance regression is a defect that no correctness test catches. Because every BenchRun is historized (tagged machine + git SHA + dep-graph snapshot), we can compare a new run to the newest same-machine baseline and fail the release if any metric drops past max_regression_pct — "did it get slower" becomes a release-blocking fact, not a thing someone notices three versions later. The regression gate runs before the history append so a candidate can never compare against itself, and a red run is never persisted (a red run must not become a baseline).

Why keep bench a separate subsystem from test? They answer different questions (correct? vs fast?) and have opposite economics. Tests run implicitly on every gate, so they need a default-off switch for heavy work. Benches run only when you explicitly nornir bench run — you already chose the work, and a bench's whole point is to measure the heavy workload. So heavy is a test-only concept; benches are never --heavy-gated. Gating benches behind --heavy would also silently gut regression gating now that release defaults to no-heavy — the exact thing the gate exists to prevent. A bench that needs network/a giant fixture gates the existing way (network_required + a fixture-present check that degrades to skip). The only overlap kept is the BenchSmoke aspect (cargo bench --no-run, compile-only) — "does the bench code still compile" is a cheap correctness check, so it lives in the test matrix.

Why stdout-JSON as the only coupling? So nornir never links a child repo's bench code and stays invisible to the child's production build (dev-dependency only). The child prints one BenchRun line; nornir subprocess-parses it. This is the same contract shape as cargo metadata, and it is what lets the free-form metrics map carry paired "VS" metrics (ours + legacy) in one result without any schema change — which is what makes "how much faster than legacy" renderable and gateable. Scaling this across many machines (RPC ingest, no JSON hop) has its own design: scalable-bench-design.md. A worked head-to-head illustrating the pattern: scip-bench.md.

f3 — why maven-style source coverage, with a boundary view

Surface-completeness (f1) answers "is every testable surface reached?"; this answers "what fraction of the source lines did the run actually execute?" — the complementary question. We classify each function by an ui / gRPC / emitter / core boundary because those seams are the hard-to-cover ones and the whole doctrine is that they must be proven tested; an un-exercised state_json emitter reds the boundary view by name, which a raw overall-percentage would hide. The parser is kept pure (a JSON blob → asserted numbers) so it is testable without running cargo llvm-cov at all. (Detail: coverage-maven.md.)

f4 — why a headless robot over pixel-diffing

This is the crux, and it is inherited from facett. We test a GUI without a GUI. A facett component owns all its observable state in one serializable Model and exposes it as state_json, so a test can drive it with a Msg/action script, read the resulting state back, and assert on data — not pixels. The reasons pixel-diffing is not the primary oracle:

• A pixel diff is brittle (a font hint, a 1px theme tweak, a GPU driver reds it) and, worse, uninformative — it tells you something changed, not what is wrong. state_json says exactly which field diverged.
• Data assertions run with no window, no GPU (facett_core::harness::snapshot_json / headless_render), so the matrix runs anywhere, including CI and airgapped hosts.

This is why the facett Elm contract exists at all — the reasoning is spelled out in facett elm-reasoning ("testing a GUI without a GUI", why view is pure, why caches stay out of Model). nornir is the downstream consumer: probe_facett_components drives that harness at scale, and nornir-robotui generalises it to any egui app via the same Observable::state_json pattern. The wgpu snapshot is kept — but as the visual proof alongside the data assertion (EMIT-DOCTRINE: validate as data, keep the picture as evidence), never as the sole oracle. Rejected: pixel snapshots as the primary verdict — for the brittleness + uninformativeness above; they ride along as a golden PNG, asserted-on-data-first.

Why two robot engines (in-process egui + WebDriver browser)? LAW #1 of the doctrine: shipped ≠ tested — the deployed artifact (a wasm bundle behind a browser) is a different subject from the native egui build. The in-process robot proves the component; the browser robot proves the deployed surface; requiring native↔wasm parity closes the gap between them.

f5 — why introspect feeds test rather than standing alone

The dependency graph, symbol table, and call graph are not tests, but the test function consumes them: the surface-completeness closure (f1) walks call_edges to decide what is test-reachable, and the coverage boundary (f3) classifies functions the introspect layer names. Keeping introspection a first-class face (not buried in docs) is what lets both gates be data-driven off the same graphs the depgraph/symbols/callgraph renderers draw. The self-contained SVG output (never Mermaid) is a deliberate no-JS-no-build-step choice shared with the docs pipeline.

f6 — why record→replay as data

A recorded session (SagaEvents) is compiled to a ReplayPlan of pure ReplaySteps, and nornir-saga's contribution stops there — it emits data, never drives anything. The reason for the clean cut: the same plan can be replayed by either robot engine (plan_to_scenario for in-process, replay_plan_to_web_scenario for the browser), and each step carries the recorded expect_state so a divergence names the exact source_seq that broke. A real user session thus becomes a regression test for free, and it closes the loop back to f1 as an ordinary red row — no bespoke replay-verdict machinery, because the matrix already knows how to be red.