crates/nornir-testmatrix/.nornir/testmatrix-crate.md

nornir-testmatrix — design + guide (EPIC L)

The reusable, multi-aspect test-matrix engine, extracted from nornir so any leaf repo (znippy, skade, lbzip2-rs …) runs the same matrix with a tiny dep set: std + serde/serde_json + anyhow. No iceberg, arrow, eframe, tantivy, skade.

1. Why a crate

The C6 runner + test_results model lived inside nornir, which pulls iceberg, arrow, tantivy, eframe and friends. A leaf repo that just wants "run my matrix and record the rows" should not pay for that. EPIC L lifts the pure core into crates/nornir-testmatrix; nornir depends on it and re-exports every item, so the old paths (nornir::test_matrix::*, nornir::warehouse::test_results::{TestResultRow, …}) keep working unchanged.

2. The four modules

nornir-testmatrix (std + serde + anyhow)
├─ runner — wrap native cargo test/nextest (detect_runner, run_matrix, stall watchdog)
├─ aspect — the multi-aspect engine (Aspect, run_aspect, run_full_matrix)
├─ model  — pure rows + renderers (TestResultRow, RunSummary, status, render_matrix)
├─ gatedtests — the SILENCED-TEST GUARD (Aspect::GatedTests): every tests/*.rs
│              whose crate-level #![cfg(feature=…)] can never be satisfied by
│              `default` and that no DECLARED matrix arm re-invokes is a RED row.
│              See `.nornir/testmatrix-gated-tests.md`.
└─ sink   — TestSink trait + JsonFileSink + NullSink
   edges: aspect → model · runner → model · sink → model · aspect → runner

consumers ──▶ nornir-testmatrix
├─ nornir: IcebergTestSink + Iceberg schema (re-exports the crate)
└─ leaf repo: JsonFileSink / NullSink

runner — detect_runner() (nextest if on PATH, else cargo test), run_matrix(repo_root, runner) -> MatrixRun, the stall watchdog (NORNIR_TEST_STALL_SECS, default 120). Parsers for nextest libtest-json and the plain cargo test default grammar. Unchanged from C6 except the status import now points at model::status.
aspect — the NEW engine (L2). See §3.
model — the pure types + renderers. Std-only: a tiny SplitMix64 UUIDv4 (new_run_id) and an RFC3339 formatter (rfc3339_from_micros, Howard Hinnant civil-date math) replace the old uuid/chrono deps so the crate stays lean.
sink — the TestSink seam + built-ins (§4).

3. The multi-aspect engine (L2)

run_aspect(repo_root, aspect) -> AspectOutcome shells the right cargo subcommand per [Aspect] and parses the result:

Aspect	command	metric	skip when
`Build`	`cargo build --all-features`	—	(never)
`Unit`	`run_matrix` (nextest/cargo test)	failed count	—
`Doctest`	`cargo test --doc`	—	—
`Clippy`	`cargo clippy --all-targets -- -D warnings`	warning count	cargo-clippy absent
`Fmt`	`cargo fmt --check`	files needing fmt	rustfmt absent
`Audit`	`cargo audit`	advisory count	cargo-audit absent
`BenchSmoke`	`cargo bench --no-run`	—	—
`Coverage`	`cargo llvm-cov --summary-only`	coverage %	cargo-llvm-cov absent
`FeaturePowerset`	`cargo hack check --feature-powerset`	—	cargo-hack absent
`Msrv`	read `rust-version` + `cargo +<msrv> check`	—	no rust-version / toolchain absent
`Examples`	`cargo build --examples`	—	—

Graceful skip (LAW-shaped): an absent tool records ONE neutral skip row (status::SKIP) with a reason — never a hard fail. skip is neither red (is_red false) nor green (is_green false); is_neutral true.

run_full_matrix(repo_root, aspects) -> Vec<TestResultRow> runs a set of aspects under ONE run id. The unit aspect expands to one row per test case; every other aspect collapses to one synthetic row tagged with its aspect + metric. Aspect::DEFAULT = build,unit,doctest,clippy,fmt,audit (fast); Aspect::ALL = all eleven.

The aspect parsers are pure (parse_clippy, parse_audit, parse_coverage, parse_rust_version) — fed canned cargo output by the tests and asserted on (LAW #1), so the matrix logic is verified without a toolchain.

4. Sinks (L1)

pub trait TestSink { fn append(&self, rows: &[TestResultRow]) -> anyhow::Result<()>; }

JsonFileSink — append-only JSONL (one compact JSON array per run); read_all() flattens it back. A leaf repo's portable matrix history.
NullSink — drops everything (dry-runs / tests).
nornir ships IcebergTestSink (in warehouse::test_results) over its Iceberg test_results table.

5. The new `TestResultRow` fields

aspect: String (default "unit") and metric: f64 (default 0.0) were added with #[serde(default …)] so existing call sites, JSON, and pre-L Iceberg rows stay valid. TestResultRow::unit(...) is the back-compat constructor for plain test cases.

6. How a leaf repo (e.g. znippy) uses it

// Cargo.toml: nornir-testmatrix = "0.1"
use nornir_testmatrix::{run_full_matrix, Aspect, JsonFileSink, TestSink};
use std::path::Path;

let rows = run_full_matrix(Path::new("."), Aspect::DEFAULT);
JsonFileSink::new("target/nornir-testmatrix.jsonl").append(&rows)?;
// or a custom sink that POSTs to nornir-server, etc.

7. Tests (LAW #1 — inject + assert)

model: status classification incl. skip; summarize counting every status; newest-first ordering; JSON carrying aspect+metric; render tagging the red case with its aspect; RFC3339 for known epochs; UUIDv4 shape + uniqueness; legacy-row deserialize defaulting aspect/metric.
runner: cargo + nextest grammar parsers; matrix counts/green; watchdog red row; suite-from-path; detect_runner returns a real variant.
aspect: every label round-trips through parse_aspect; clippy/audit/coverage parsers fed canned output assert status+metric; parse_rust_version; outcome_to_rows (synthetic + unit-expansion); real run_aspect(Fmt) on this worktree; real run_full_matrix sharing one run id.
sink: NullSink swallows; JsonFileSink two-run round-trip with exact value assertions.