Your browser sucks, JavaScript disabled! Gunnar's console needs JavaScript and WebAssembly. Git over SSH on port 2222 does not.

nordisk/nornir

.nornir/test-design.md

The test function — design

Class: design · Function axis: f0 (one gate) + f1 matrix engine · f2 bench · f3 coverage · f4 robot · f5 introspect · f6 saga. Strict: this file states only the shape we run, every claim source-backed. Every why, every rejected alternative, lives in test-reasoning.md. Siblings: test-guide.md (drive it) · test-for-idiots.md (every keystroke) · test-reasoning.md (why).

f0 — one gate (the whole)

The test function is one gate with many faces. Every face runs a check, folds its verdict into a row, appends the rows through a sink into the warehouse, and a release/CLI/viz reads them back as a green/red board. The faces do not share a runner or a table, but they share the row→sink→warehouse→read shape, and a red row from any of them fails the gate.

face f produces lands in (table) red ⇒ gate fails
multi-aspect matrix f1 TestResultRow per aspect×case test_results yes (release::gate::tests_gate)
bench f2 BenchRun (results + TestOutcomes) bench_runs / bench_results / bench_telemetry yes (no_regression, nexus_floor)
source coverage f3 CoverageReport rows coverage / coverage_fn yes ([gates].coverage)
robot f4 TestResultRow per (action→assert) test_results (suite robot-ui) yes — a red step is a red row
introspect f5 Graph / symbol tables dep_graph / DWARF-derived feeds docs + coverage boundary
saga f6 SagaEvents → ReplayPlan saga_events (record); replay → test_results a diverged replay step is a red row

Bench GATES like a test. f2 is a separate subsystem (own runner, tables, viz tab), but its verdict enters the same release gate set: a bench that panics degrades to a red TestOutcome (nornir_bench::TestOutcome, crates/nornir-bench/src/lib.rs:53), and a metric that regresses past max_regression_pct fails release::gate::no_regression (src/release/gate.rs:150). So "did it get slower" is a release-blocking check next to "did it give the right answer."

Reuse map (what the test function uses): f1/f4/f6 all emit nornir_testmatrix::TestResultRow through a TestSink (crates/nornir-testmatrix/src/sink.rs:18); f4 (robot) drives facett Elm components through the facett headless harness — the cross-repo seam described in facett elm-design f3 and in f4 below. f3 classifies coverage by the boundaries f5 names. nornir is the root crate of a workspace whose members (nornir-testmatrix, nornir-bench, nornir-coverage, nornir-robotui, nornir-introspect, nornir-saga, nornir-selftest) are re-exported so nornir::test_matrix::*, nornir::bench::*, nornir::coverage::* resolve unchanged.

The scaling companion to f2 (many machines → one warehouse over RPC) is its own design doc: scalable-bench-design.md.

f1 — the matrix engine (nornir-testmatrix)

nornir test <repo> runs a repo's native Rust test framework across a set of aspects, parses each into a row, and appends the rows under one run_id through a TestSink.

The pure core (crates/nornir-testmatrix/)

Pure std + serde + anyhow — no iceberg/arrow/eframe.

item pointer is
Aspect (14 variants) src/aspect.rs:21 Build · Unit · Doctest · Clippy · Fmt · Audit · BenchSmoke · Coverage · FeaturePowerset · Msrv · Examples · FunnelDecompose · FunnelDemo · Functional
run_aspect src/aspect.rs:186 shell the right cargo subcommand for one aspect + parse
run_full_matrix src/aspect.rs:213 run a set of aspects under one run id → Vec<TestResultRow>
AspectOutcome src/aspect.rs:155 one aspect's parsed verdict
detect_runner src/runner.rs:125 prefer cargo nextest run --message-format libtest-json, else cargo test
run_matrix src/runner.rs:185 spawn the runner with repo_root as CWD, fold lines → MatrixRun
list_tests src/runner.rs:345 discover tests without running (nextest list / cargo test -- --list) → listed rows
heavy_enabled src/runner.rs:162 single source of truth for the --heavy opt-in (adds --include-ignored)
TestCase src/runner.rs:71 { suite, name, status, duration_ms, message }
TestResultRow src/model.rs:64 the durable row: adds run_id, repo, ts_micros, aspect, metric
status constants src/model.rs:18 pass · fail · ignored · stalled · skip · listed; is_red :33, is_green :46
RunSummary / runs_to_json src/model.rs:287 per-run rollup (P/F/I/S/skip/listed + green/total), newest-first
TestSink src/sink.rs:18 fn append(&self, rows: &[TestResultRow]) -> Result<()> — the durable write seam
NullSink / JsonFileSink src/sink.rs built-in sinks (dry-run; portable JSONL history)

A stall watchdog inside run_matrix kills a child that produces no output for NORNIR_TEST_STALL_SECS (default 120) and mints one red stalled case, so a hung test is visible, never a silent wall-clock eater. Status buckets are disjoint: listed (discovered-not-run, renders X) and skip (absent tool) are neither red nor green.

The nornir-side sink + warehouse

nornir supplies the durable sink and the tables:

The viz 🧪 Test tab (src/viz/test_tab.rs) reads the same tables as a green/red board, exposing every rendered datum under state_json()["test"] for headless assertion. CLI parity is total.

f2 — bench (nornir-bench)

A project-agnostic harness a child repo implements so a release can drive its benchmarks, historize them, gate on regression, and render them into the README. The only coupling is stdout JSON.

item pointer is
Bencher trait crates/nornir-bench/src/api.rs:88 id() + run() -> BenchResult; one small struct per bench
register_bench! / _ordered! src/api.rs:122 / :148 inventory-collected at link time; _ordered! pins stage order
run_main_json src/api.rs:195 collect, sort by (order, id), run sequentially, stamp envelope, print one JSON line
run_pass src/api.rs wraps a unit, folds the telemetry Sampler summary into metrics
BenchResult src/lib.rs:39 { name, metrics: Map<String,Value> } — free-form metric keys
BenchRun src/lib.rs:77 { date, timestamp, version, machine, cores, results, tests }
TestOutcome src/lib.rs:53 a test/bench that ran beside the benches; a red one fails the release
Sampler src/telemetry.rs 1 Hz /proc-based CPU/mem sampler → telem_* keys → bench_telemetry
progress src/progress.rs live bench: stderr lines + $NORNIR_VIZ_ACTIONLOG sink

The contract. A child opts in with three things and nothing else: nornir under [dev-dependencies] only; one examples/nornir-bench.rs (or xtask/- / bench/-crate variant) whose body is nornir::bench::api::run_main_json(); and one impl Bencher + register_bench! per bench. nornir bench run subprocess-spawns the example, parses the one JSON line, and appends to the Iceberg bench_runs / bench_results (+ test_outcomes, + bench_telemetry) tables, tagged with release_id + per-repo git SHA. Full reference: bencher.md.

Gates (release-blocking). no_regression (src/release/gate.rs:150, wired at src/release/pipeline.rs:339) compares a new run to the newest same-machine baseline and fails on any metric drop past max_regression_pct; nexus_floor (gate.rs:128) enforces a paired floor; BenchSpec::validate (src/config.rs) fails the release if a required_results name is missing.

Comparative / "VS" benches carry paired metrics in one BenchResult (e.g. ljar_mbs + unzip_mbs); the renderers bold the winner. Renderers (src/docs/sections.rs): render_bench (flat table), render_benches (pivot, bold-winner, mashup), render_bench_chart (native-typst bar chart) — spec in docs-generation.md Built-in section renderers. A worked head-to-head (in-process index vs rust-analyzer scip) lives in scip-bench.md.

Bench and test are separate subsystems — a plain nornir test runs no benches; the only overlap is the BenchSmoke aspect (cargo bench --no-run, compile-only, lands in test_results). Heavy is a test-only concept; benches are never --heavy-gated. Ordering + the light-depends-on-heavy guard: test-bench-chains.md.

f3 — coverage, maven-style (nornir-coverage)

LLVM source-based line/region coverage → warehouse → docs + viz.

item pointer is
runner crates/nornir-coverage/src/runner.rs shells cargo llvm-cov --json in the repo dir
parse_llvm_cov_json crates/nornir-coverage/src/lib.rs:359 pure parser: per-file line/region totals + per-fn regions; demangles v0/legacy
classify src/lib.rs:87 tag a fn's Boundary from name + path
Boundary src/lib.rs:48 ui · grpc · emitter · core
CoverageReport src/lib.rs:212 overall + per-crate + per-file + per-fn rows

Persist (src/warehouse/coverage.rs): one snapshot per run into coverage (one row per overall | crate:<name> | file:<path>) and coverage_fn (one row per function, carrying its boundary), partitioned by repo, time-travellable. Surface: the nornir:gen:coverage [worst=N] marker renders a maven-style table into the book/PDF; the viz Test pane's 📐 panel (src/viz/coverage_live.rs) reads the same rows and exposes them under state_json["test"]["src_coverage"]. The boundary view proves the hard-to-cover seams (ui/grpc/emitter) are exercised. Full detail: coverage-maven.md.

f4 — robot (the facett harness at scale)

The robot drives a real UI headless, reads back the rendered state as data (never pixels), and turns each (action → expected outcome) into a matrix row. Three engines, one row type.

The facett harness seam — src/autonom/facett_probe.rs. This is the direct call site into facett's headless harness. probe_facett_components (src/autonom/facett_probe.rs:107) drives each facett component (facett-jobview, facett-docview, facett-graphview, facett-graph3d, facett-helix, …) through facett_core::harness::snapshot_json (:55, apply every Msg through update() and capture the resulting Model as json) and harness::headless_render (:60, render the post-drive state off-screen and assert RenderReport::drew) — a Msg script in, a Model/render out, deterministically, no window, no GPU. record (:75) runs each probe under a panic guard and emits a functional row (skip-never-fail on panic). This is exactly the seam facett's elm-design f3 names: every facett component is Elm → Facet with a serializable Model/state_json via impl_facet_via_elm!, and the test-matrix + robot consume that harness at scale.

In-process egui robot — nornir-robotui. RobotSession (crates/nornir-robotui/src/lib.rs:246) wraps an egui_kittest::Harness and drives real AccessKit interactions (click_label/click_button/ click_by_id, set_field, type_into, key/scroll/drag, inject_events, step/run frames). It observes through the Observable::state_json blob (trait Observable, :232) — the same blob a facett Facet exposes — and asserts against it. run_scenario (:593) emits Vec<nornir_testmatrix::TestResultRow> (suite robot-ui), so one call drives every button and literally fills the test matrix. Error detection walks the live AccessKit tree for ERROR_MARKERS (error_atoms, :101), reddening a surface that rendered a failure. The scenario DSL (Target/FieldValue/Action/Assert/Step/Scenario, crates/nornir-robotui/src/scenario.rs) is plain data; a pixel-snapshot oracle lives in src/snapshot.rs. Richer widget-level verbs + live-channel parity: robotui-richer-livedrive.md.

Browser / deployed web UI — src/robot.rs. A WebDriver (fantoccini, feature robot) robot that drives a real browser through declarative TOML scenarios (ScenarioFile/Scenario/Step). Locators anchor on ARIA role + accessible name + name= + URL, never CSS classes; the scenario set is a plan-tests-DAG (needs = [...], Kahn topo-order, a failure skips transitive dependents). Results append to robot_history.jsonl (the bench_history.jsonl pattern).

f5 — introspect (nornir-introspect)

Extracts the graphs the other faces consume.

item pointer is
Graph + to_svg / to_text crates/nornir-introspect/src/lib.rs:54 / :62 the node/edge graph + self-contained SVG (never Mermaid)
depgraph::extract src/depgraph.rs:11 workspace crate-dependency graph from cargo metadata
depgraph::extract_with_path_deps src/depgraph.rs:51 resolves sibling path-deps before metadata (headless-server fix)
artifact::extract_symbols src/artifact.rs:49 top public symbols by size from DWARF
artifact::lookup / defined_in src/artifact.rs:234 / :246 query the symbol set
callgraph_dwarf::extract_callgraph src/callgraph_dwarf.rs:55 most-called callees by inline-edge count
CallGraph::{callers_of, callees_of, path_between} src/callgraph_dwarf.rs:326… navigate the call graph
callgraph_llvm::extract_from_dir src/callgraph_llvm.rs:44 LLVM-IR call edges
pathdeps::materialize_path_dep_siblings src/pathdeps.rs:24 clone+symlink siblings for a flat checkout

These feed the depgraph / symbols / callgraph doc-section renderers (docs-generation.md), the surface-completeness closure (f1), and the coverage boundary classification (f3).

f6 — saga (replay, nornir-saga)

Records a UI session as events, then compiles them into a replayable test plan.

item pointer is
EventKind / SagaEvent crates/nornir-saga/src/event.rs:18 / :65 Nav · Click · Log · Error · State; a captured moment
SessionCtx src/event.rs:192 session id + user + app + version + git SHA stamped on each event
record / drain_saga_events src/record.rs:65 / :80 thread-local capture buffer
SagaSink src/sink.rs:25 durable write seam (NullSink, JsonFileSink); Iceberg impl IcebergSagaSink → table saga_events
ReplayStep / StepKind src/replay.rs:27 / :51 one actionable gesture: Navigate or Activate, + expect_state
ReplayPlan src/replay.rs:92 a session's ordered steps; for_session :105, all_sessions :122, is_replayable :144

ReplayPlan::for_session filters a mixed event stream to one session_id, sorts by capture seq, and keeps only actionable events (Nav→Navigate, Click→Activate), each carrying the recorded expect_state. nornir-saga's contribution stops at plain data; the bridge to f4 turns a plan into a runnable scenario: nornir_robotui::replay::plan_to_scenario (crates/nornir-robotui/src/replay.rs:44) for the in-process robot, robot::replay_plan_to_web_scenario (src/robot.rs:205) for the browser robot. A step whose replayed state_json diverges from expect_state becomes a red matrix row naming the exact recorded source_seq that broke — closing the loop back to f1.