Your browser sucks, JavaScript disabled! Gunnar's console needs JavaScript and WebAssembly. Git over SSH on port 2222 does not.

nordisk/nornir

.nornir/test-matrix.md

C6 — Standardized nornir test-matrix (nornir test)

Design + guide for the C6 epic: ONE standardized way to run every repo's tests, land the results in the warehouse, and watch them light up green/red in a viz Test pane. Mirrors the nornir bench run→persist→view shape and the release_events warehouse-DAG pattern.

0. LAW — how an agent fixes a matrix test (no human in the loop)

This is the canonical, mandatory way a Claude agent fixes any red / missing / "looks wrong" UI / gRPC / render / CLI test in this matrix — and validates the fix with NO human ever looking at a screen. The single source is agent-ui-test-fixing.md (a chapter of the docs book, docs/book.pdf); read it before touching a red row. Its creed: green ⟺ the shipped surface, on real data, through the server, across its whole outcome space, was proven RAN-and-CORRECT from emitted data — and the same oracle goes RED when it breaks. It is structured as eight named LAWS, each born from a real false-green this constellation shipped:

  1. SHIPPED ≠ TESTED — the subject is the DEPLOYED artifact (wasm bundle / installed binary), not the native build; require native↔wasm parity + a headless-wasm robot that proves each surface present AND ran.
  2. RAGNARÖK — empty / zero data on a data-bearing surface SHALL NOT PASS; name the populate gap; prove the RED with the empty-input inverse.
  3. ONE VALUE ≠ COVERED — sweep the SAMPLE SPACE per-function AND whole-chain (the facett-map3d chain_sweep/partstep_sweep exemplar; ≥K classes).
  4. END-TO-END = THROUGH THE SERVER — drive thin (viz --server + CLI --server) → gRPC → server → warehouse → back; assert the round-trip, not the local fat path.
  5. WORKSPACE BLIND — the surface is workspace × {tab@{fat,thin}, cli, mcp} × chain for EVERY served workspace, clean-slate→populate→verify (not seeded).
  6. VERIFIED BY EYEBALL is not verified — every emptiable render emits its populated-state as DATA; correctness read from the emit, never "looks right?".
  7. PESSIMISM — green only when proven ran-and-passed-and-swept; an unverified surface is presumed BROKEN; distrust every "done."
  8. WIRE ≠ HOME — the terminus is the Iceberg WAREHOUSE via the lock-owning server; a /tmp JSONL row is NOT done — reconcile + read it back.

Plus the supporting technique: validate as DATA not pixels (EMIT-DOCTRINE), the oracle-must-go-RED contract bound, scan-the-pixels, multivariate/metamorphic stress, binary-search debug, follow-the-data — and a false-green checklist an agent runs before claiming done.

The 🧪 Test pane surfaces this LAW as a "How agents fix tests" link (§5, state_json["test"]["doctrine"]) so it's discoverable from the running viz, and it ships as a chapter of the rendered docs book. This applies to every UI / gRPC / render surface in the constellation (nornir CLI fat+thin, nornir viz, facett-*, korp-poc, and any future one).

1. What it is

nornir test <repo> runs a repo's native Rust test framework (cargo nextest run if cargo-nextest is on PATH, else cargo test), parses each test's pass/fail + duration, and appends one row per case to the warehouse test_results table under a fresh run_id. Any plain #[test] — including the inject-value + snapshot tests THE_BIG_PLAN mandates — is ingested automatically; there is no bespoke runner to register tests with.

nornir test run <repo>        # run one repo's matrix, persist to test_results
nornir test all               # every repo in nornir.toml, one run id per repo
nornir test history <repo>    # read the matrix back (green/red, newest first)
nornir test history <repo> --json --limit N

The viz 🧪 Test tab reads the same table as a per-repo green/red board: run list on the left (newest on top), the selected run's cases coloured by status on the right. CLI parity with the GUI is total — the pane shows nothing the history CLI can't print.

stallednornir test history

2. The runner (src/test_matrix/)

nornir test run repo
└─ detect_runner
   ├─ [cargo-nextest on PATH] cargo nextest run --message-format libtest-json
   │     └─ Parser::feed_json (one JSON event/line) ─┐
   └─ [else] cargo test --no-fail-fast (default      │
             human grammar)                          │
             └─ Parser::feed_cargo                    │
                (test NAME ... ok|FAILED|ignored) ────┤
                                                      ▼
                                          Vec<TestCase>
                                          └─ TestResultRow per case (shared run_id + ts)
                                             └─ append_test_results → Iceberg test_results

stall watchdog (NORNIR_TEST_STALL_SECS) ⋯⋯▶ kills silent child, mints red `stalled` case → Vec<TestCase>

Runner capability note

3. Stall watchdog (C6f)

The runner arms a watchdog off the output channel: if no subprocess output arrives for NORNIR_TEST_STALL_SECS seconds (default 120, 0 disables), the child is killed, a single synthetic red stalled TestCase is appended (message = "no test output for Ns — watchdog killed the run"), and the run is recorded red. A hung/crap test is therefore visible in the matrix, never silently eating the wall clock.

4. The test_results table (C6b)

src/warehouse/iceberg_schema.rs::test_results(), partitioned by repo (per-repo reads prune, like bench_runs). One row per (run_id, suite, test_name):

col type n notes
run_id string req groups a run's rows; one per nornir test invocation
repo string req partition key
suite string req crate / test target the case lives in
test_name string req module::path::test_fn
status string req pass | fail | ignored | stalled
duration_ms double req wall time (0.0 when the runner gave none)
ts_micros timestamptz req run time (shared across a run's rows)
message string req failure / stall detail ("" = none, never null)

Write/read live in src/warehouse/test_results.rs:

Relation to test_outcomes

test_outcomes is bench-shaped — keyed to a bench_runs.run_id, co-written when a bench runs. test_results is the standalone green/red board: no bench, no release coupling. The two coexist; this one is what the C6 matrix + Test pane use.

5. The viz Test pane (C6c — src/viz/test_tab.rs)

A new Tab::Test (🧪 Test), added to the tab enum + toolbar in src/viz/app.rs. Local mode opens the warehouse read-only (lock-tolerant open_read_only, so it coexists with a live server holding the catalog lock — the same fallback the Release tab + release events CLI use) and calls query_test_results. Remote mode shows a note (no Test.Results RPC yet — a documented follow-up, same split as the Release tab).

6. Tests (LAW #1 inject-and-assert, LAW #4 snapshot)

7. EPIC L — extracted crate + multi-aspect engine + strong test all

The C6 runner + pure test_results model were extracted into the reusable nornir-testmatrix crate (a workspace member; nornir is now the root crate of a 2-member workspace). nornir depends on it and re-exports every item, so nornir::test_matrix::* and nornir::warehouse::test_results::{TestResultRow, render_matrix, …} keep working. Full design: crates/nornir-testmatrix/.nornir/testmatrix-crate.md.

L1 — the crate + the TestSink seam

L2 — multi-aspect engine (test MANY aspects)

L3 — strong nornir test

nornir test run <repo> [--aspects build,unit,clippy,…]   # default: build,unit,doctest,clippy,fmt,audit
nornir test all        [--aspects …]                     # full matrix across every repo + per-repo×aspect grid
nornir test history <repo> [--json] [--limit N]          # unchanged

test all runs the aspect set across every nornir.toml repo, persists every row via the IcebergTestSink, prints a per-repo×aspect terminal grid (✓ pass · ✗ red · ○ skip · · not-run), and exits red if any red.

7b. n-001 — test inventory via runner --list (the list-first phase)

Current discovery is two-tier. The PRIMARY, build-free inventory comes from the syn knowledge scan on initial workspace load (no cargo build) — see test-matrix-always-visible.md: every #[test] shows ok | fail | X immediately, the test_inventory table LEFT JOIN test_results. The runner --list phase below still runs as part of nornir test run (the on-request / release execution path, where a build happens anyway) to seed listed rows for tests about to run. Initial load uses the syn inventory only; --list never runs on load.

The results matrix only recorded tests that happened to run, so an existing #[test] that wasn't reached stayed invisible (the autonom anti-drift problem). n-001 makes nornir test run discover every test first, so the 🧪 matrix shows EVERY test that EXISTS with status NotRun / pass / fail — data-driven, the runner lists them, never a hand registry.

Storage — REUSE test_results, new listed status (no new table)

status::LISTED ("listed") joins the existing status set in crates/nornir-testmatrix/src/model.rs. A listed row is discovered-but-not- run:

List-first — PHASE 1 of nornir test run

runner::list_tests(repo_root, runner) discovers tests without running them:

nextest:    cargo nextest list --message-format json   → parse_nextest_list(json)
fallback:   cargo test -- --list --format terse         → parse_cargo_test_list(text)

parse_nextest_list walks the rust-suites → testcases map and emits <binary-id>::<case>; parse_cargo_test_list keeps the libtest name: test lines (dropping : benchmark). Both are PURE (fed canned output by unit tests), sorted + deduped. model::listed_rows(names, run_id, repo, ts) turns the names into listed-status TestResultRows (suite = everything before the last ::).

The run_test_one driver (src/bin/nornir.rs) seeds those listed rows first (per repo, under the run's shared run_id/ts), prints 📋 discovered N test(s), then runs the aspect matrix as before. Discovery never hard-fails the run (a launch error just skips the inventory). The crate stays lean (no iceberg); the nornir driver persists via the existing sink. Aspect columns are unchanged (from the Aspect enum).

Grid — listed renders as NotRun (src/viz/test_matrix_grid.rs)

build_cell drops listed rows before rolling up a cell: a cell of only listed seeds falls through to Cell::not_run() (ran() = false, counted in total_blank, NOT skip/neutral), while a real pass/fail/ignored row in the same run overrides the seed. So:

Two inject-assert grid tests pin this (listed_only_cell_is_not_run, listed_then_pass_in_same_run_is_green), plus model tests asserting the exact discovered names from canned nextest list JSON, the listed_rows conversion, and the new status predicates; a runner test runs the real list phase against the crate's own dir and asserts it names this very test.

7c. utfallsrum — outcome-space coverage in the matrix (the n-005 gate's quality bar)

The matrix records pass/fail per check; utfallsrum (English math term: sample space, Ω) records the SHAPE of the check — how much of a function's OUTCOME SPACE a test swept. It is classic boundary-value + equivalence-partition analysis, made first-class and persistable. This is the machinery behind agent-ui-test-fixing.md LAW 3 (ONE VALUE ≠ COVERED): sample-space coverage of a chain f1 ∘ … ∘ fn is BOTH per-function (each fi swept across its full outcome space — the facett-map3d/tests/chain_sweep.rs shape) AND the composed chain across the same input space (partstep_sweep.rs). Keep the two layers separate: the primitive is GENERIC + UI-agnostic (works for a headless library with no viz/server/workspaces); nornir's workspace × {tab@{fat,thin}, cli, mcp} surface gate (LAW 5) is nornir applying it.

The n-005 gate consumes it (DELIVERABLE — the workspace × utfallsrum extension)

The autonom completeness gate (nornir test coverage, autonom.md / aut5-surface-coverage-gate.md) gained two dimensions:

Worked slice: tests/viz_utfallsrum_slice.rs (gate viz,server,testmatrix) drives two workspaces × {Test,Bench}@{fat,thin} clean-slate→populate→verify, records the outcome class each workspace exercised (the pair sweeps one + many), scores the outcome space, and persists + reads back the utfallsrum-scored coverage rows.

8. Deferred (noted, not built here)