nordisk/edda

nornir-testmatrix

The portable core of nornir's multi-aspect test matrix — wrap a repo's native Rust test framework and test many aspects of its health (build, doctest, clippy, fmt, audit, coverage, feature-powerset, msrv, examples), parse the results into rows, and ship them to any TestSink.

Pure std + serde/serde_json + anyhow. No iceberg, arrow, eframe, tantivy, or skade — so any leaf repo can pull it cheaply.

use nornir_testmatrix::{run_full_matrix, Aspect, JsonFileSink, TestSink};
use std::path::Path;

let aspects = Aspect::DEFAULT; // build, unit, doctest, clippy, fmt, audit
let rows = run_full_matrix(Path::new("."), aspects);
JsonFileSink::new("target/nornir-testmatrix.json").append(&rows).unwrap();

A missing tool (clippy/audit/llvm-cov/hack/fmt) is recorded as a neutral skip row — never a hard failure. See .nornir/testmatrix-crate.md.

Functional-status mode (testmatrix feature)

A component can report whether it actually works — even when the result can't be eyeballed (a map that renders nothing). Behind the testmatrix cargo feature (off for release → zero cost), call from your self-test path:

nornir_testmatrix::functional_status("facett-map", "basemap_rendered", ok, "0 ways, blank framebuffer");

Aspect::Functional drains these into test_results rows (ok=false → a RED matrix row), so a dead render shows up in nornir test / the viz Test pane without anyone looking at a GUI. With the feature off, functional_status is an inlined no-op. See .nornir/testmatrix-functional-status.md.

Silenced-test guard (gatedtests module, Aspect::GatedTests)

Silence is not success. A tests/*.rs carrying a crate-level #![cfg(feature = "X")] where X is not in that crate's default set compiles to an empty test binary: cargo test prints running 0 tests ... ok and the file is green by vacuum. A 2026-07-21 sweep found 78 such files across 11 repos, hiding ~216 test functions — most in the camouflage shape all(default-on, default-on, default-OFF), which reads exactly like its running siblings.

let rep = nornir_testmatrix::audit_repo(std::path::Path::new("."), &nornir_testmatrix::new_run_id())?;
assert!(rep.is_green(), "{}", rep.summary());
// 93 test targets scanned · 21 feature-gated · 0 rescued by declared arms ·
// 15 SILENCED (66 hidden test fns) — RED

The guard walks every test target (cargo metadata), parses the leading #![cfg] (all/any/not, nested; non-feature predicates evaluate to unknown, never false), resolves each crate's transitive default closure, and emits a RED TestResultRow on the gated-tests aspect per dark file. A file IS covered when the repo declares the arm that re-invokes it, in .nornir/testmatrix-arms.json (holger's xtask/tests/holger_matrix.rs pattern):

{ "arms": [ { "package": "holger-ui", "features": ["gui"], "test": "robot_ui_app" },
            { "features": ["testmatrix"] } ] }

It is in Aspect::DEFAULT, so every repo running the matrix inherits it. See .nornir/testmatrix-gated-tests.md.

Test targets that do not COMPILE (targetcheck, Aspect::TargetCompile)

cargo check is not a proxy for cargo test, and a test target that does not compile is reported by nothing. cargo test never lists it, the matrix has no row for it, and the silenced-test guard above is looking at the other failure — a file that compiles to an empty binary. A file that does not compile at all never reaches that question.

Four independent instances, all measured on 2026-08-22:

let crates  = nornir_testmatrix::workspace_crates(root)?;
let configs = nornir_testmatrix::configs_from_metadata(&crates);
let report  = nornir_testmatrix::check_targets(root, &crates, &configs);
assert!(report.is_green(), "{}", report.summary());
// nornir: 26 target×config FAILURE(S)
//   nornir::viz_buttons  [default / default]
//     E0599: no method named `build_state` found for struct `HarnessBuilder<State>`

Three things make it a guard rather than a build:

• cargo build --tests --keep-going, not cargo test --no-run — cargo 1.97.1 rejects --keep-going on test, and without it cargo stops at the first broken target and the other twenty-five stay invisible. That is the failure being fixed, so it cannot be reproduced by the fix.
• The verdict comes from the JSON stream, never an exit code. Built, failed, skipped-because-required-features, and not accounted for are four states. The fourth is not read as a pass.
• An empty scan FAILS. Zero declared targets, zero configurations, or zero targets that compiled anywhere is RED and publishes a RED row, because a run that examined nothing is shaped exactly like a clean one.

Which configurations — enumerated from data, never a list somebody writes

Not covered, plainly: the feature powerset (Aspect::FeaturePowerset owns that axis and it is exponential), cross-compile targets, non-dev profiles, and any combination a human runs that no file in the repo declares. The claim is "every combination this repo declares", not "every combination that exists".

configs_from_variants takes plain VariantSpecs: nornir-build-thing is the one writer of that TOML, this crate's dependency budget is serde + anyhow, and the edge between them is a cargo metadata cycle. examples/target_compile.rs carries the six-line bridge, and doubles as the multi-repo driver:

cargo run --example target_compile -- ../korp ../nornir [--only-default] [--json rows.json]

Cost, and where this belongs

Compiling is the whole cost, and the default depth is cargo check --all-targets, not a build. Measured on oden (32 cores, CARGO_BUILD_JOBS=8, CARGO_INCREMENTAL=0), nornir at origin/main, cold target dir:

Same twenty-six, 4.2× the wall-clock and 60× the disk — 170 GB per repo is how /home hit 100 % five times in one night. Depth::Build is still not redundant: it is the only one that reaches codegen and the linker, so it catches a monomorphisation error in a generic nothing instantiates while checking, a const evaluated at codegen, and a missing native library. Every instance measured so far is a type or name error, which check sees.

Per-repo, one default configuration at check depth, cold shared target dir, 2026-08-22: funnel-core 5 s · draupnir 6 s · Skidbladnir 13 s · korp-collectors 13 s · znippy-zoomies 18 s · tunnr 21 s · skade 39 s · gunnar 58 s · korp 78 s · facett 83 s · holger 84 s · knut 85 s · nornir 94 s · dwarves 185 s · znippy 223 s · edda 65 s. Eighteen repos in eighteen minutes. Each extra configuration is another pass of the same size — korp's own twenty declared configurations (default + 11 arms + 7 build variants + required-features) take 354 s together.

What the sweep found (2026-08-22, worktrees off each repo's origin/HEAD)

55 of 1 115 declared test targets across 18 repos cannot be compiled today — 4.9 %, in 7 of 18 repos. Not all the same cause: 29 are per-target compile errors (nornir 26, dwarves 1, and edda 2 — found by pointing the guard at the repo it was written in), 22 are two repos whose dependencies do not resolve at all (modgunn, ordning), 4 sit behind a broken lib (tunnr 3, knut 1).

That is the DEFAULT configuration only, and it is a floor rather than the number. korp is the one repo swept across everything it declares, and it goes from 158/158 green on a default build to 38 target×config failures — including two targets that are red under every one of its seven declared build variants, i.e. every binary that ships.

Aspect::TargetCompile is deliberately not in Aspect::DEFAULT — a guard that makes the default matrix unaffordable is a guard that gets switched off. It belongs in an xtask verb or an explicit --aspect target-compile run, not on every cargo test. Note that a repo with broken targets pays the cost every run: a target that fails never caches a success.

UI-plane reachability — LAW 9 (uiplane module)

Isolation tests green "the component renders" while the running app can't navigate to it. uiplane models the UI as planes (UiPlane: a tab / dialog / view) + a navigation plan (UiPlan: planes as nodes, transitions as edges) and RobotPlan::walk(&plan, &mut driver) walks the REAL app via a PlaneDriver seam — reaching every plane and asserting each declared surface is present + RAN (read from emitted data, not pixels). A surface that is unreachable (orphan plane / missing transition) or reachable-but-didn't-run is a RED row. Generic + UI-agnostic (a headless library has an empty plan → trivially green); the native robot-UI and the deployed-wasm headless browser are two backends behind the same PlaneDriver. See .nornir/testmatrix-uiplane.md.