.nornir/test-guide.md

The test function — guide

Class: guide · audience: a techie running or driving nornir's tests, benches, coverage, and robot. The what is in test-design.md; the why in test-reasoning.md; every keystroke in test-for-idiots.md. This file is the CLI + worked examples.

f0 — one green board (the whole workflow)

The end-to-end loop that spans every face: discover → run → persist → read back.

nornir test list <repo>          # the ok|fail|X matrix, discovered build-free (no cargo build)
nornir test run  <repo>          # run the aspect matrix, persist to test_results
nornir bench run <repo>          # measure, persist to bench_runs, gate on regression
nornir coverage  <repo>          # llvm-cov, persist to coverage, print the maven table
nornir test history <repo>       # read the runs back, newest-first, green/red

Every verb persists to the warehouse and every verb has viz + (mostly) MCP parity — the 🧪 Test tab shows nothing a CLI can't print. A nornir release runs the matrix, bench-regression, and coverage gates together; a red row from any face fails it.

f1 — the matrix (`nornir test`)

nornir test run <repo> [--aspects build,unit,clippy,…]   # default: build,unit,doctest,clippy,fmt,audit
nornir test all        [--aspects …]                     # every nornir.toml repo, one run id per repo
nornir test list [<repo>] [--json] [--heavy-only] [--not-run] [--server <url>]
nornir test history <repo> [--json] [--limit N]

test all prints a per-repo×aspect terminal grid (✓ pass · ✗ red · ○ skip · · not-run) and exits red if any red row.
Runner fidelity. With cargo-nextest on PATH you get per-test durations and failure messages; plain cargo test gives pass/fail/names only (duration = 0.0, message = ""). Install nextest to upgrade any repo to full fidelity, zero config.
Aspects ∈ build · unit · doctest · clippy · fmt · audit · bench-smoke · coverage · feature-powerset · msrv · examples · funnel-decompose · funnel-demo · functional. Pick a subset with --aspects. A missing tool (clippy/audit/llvm-cov/…) records a neutral skip, never a hard fail.
Heavy is opt-in. #[ignore]'d / large-corpus / network / container tests run only with the global --heavy flag (or NORNIR_HEAVY_TESTS=1); default OFF for test, release, and the viz "Run full matrix". --skip-heavy is now a no-op. See bench-vs-test-for-idiots.md.

Worked example — run just the fast lint aspects across the workspace and read them back:

nornir test run nornir --aspects fmt,clippy
nornir test history nornir --json --limit 3

MCP parity: test_history returns the same newest-first per-run summaries (test-history.md); test_coverage / release_gate_coverage drive the surface-completeness gate (aut5-surface-coverage-gate.md).

f2 — bench (`nornir bench`)

nornir bench run <repo>              # measure + persist + regression-gate
nornir bench history-show <repo> [--json] [--limit N]

Authoring a bench in a child repo — three things, nothing else:

# Cargo.toml
[dev-dependencies]
nornir = { path = "../nornir" }

// examples/nornir-bench.rs
fn main() -> anyhow::Result<()> { nornir::bench::api::run_main_json() }

struct JarDecompress;
impl nornir::bench::api::Bencher for JarDecompress {
    fn id(&self) -> &'static str { "ljar.jar_decompress" }        // <repo>.<scenario>
    fn run(&self) -> anyhow::Result<nornir::bench::BenchResult> { /* measure, fill metrics */ }
}
nornir::register_bench!(JarDecompress);

Then nornir bench run <repo> subprocess-runs the example, parses its one JSON line, and appends to bench_runs. Paired "VS" metrics (ljar_mbs + unzip_mbs) in one BenchResult render a "how much faster" table via the benches marker; suffix scalars _mbs/_ms/_pct/_ops_sec so the renderers and the no_regression gate can reason about direction. Pin pipeline order with register_bench_ordered!(N, …). Watch a run live in the viz:

NORNIR_VIZ_ACTIONLOG=/tmp/nornir_viz_actions.log nornir viz            # terminal A, toggle 🐞 panel
NORNIR_VIZ_ACTIONLOG=/tmp/nornir_viz_actions.log nornir bench run znippy   # terminal B

Full contract + renderers: bencher.md; scaling across machines: scalable-bench-design.md.

f3 — coverage (`nornir coverage`)

nornir coverage [REPO] [--show] [--boundary] [--json] [-- <extra cargo-llvm-cov args>]

nornir coverage nornir — measure + persist + print the maven table (overall %, per-crate, worst files).
--boundary — add the ui / gRPC / emitter view: a per-boundary tally + every boundary fn with ✓/✗.
--show — skip the run, READ the latest persisted run (the viz/docs source of truth).
-- --all-features — anything after -- is passed verbatim to cargo llvm-cov.

If cargo-llvm-cov / llvm-tools-preview is missing the verb prints the exact install hint. Detail: coverage-maven.md.

f4 — robot (drive a UI headless)

In-process egui / the facett seam — drive any Observable component, assert on its state_json, and each step becomes a matrix row:

use nornir_robotui::{RobotSession, Scenario, Step, Action, Assert, Target, FieldValue};

let mut robot = RobotSession::new(harness, "my-app");       // harness = egui_kittest::Harness over your app
let scenario = Scenario::new("smoke").step(
    Step::new("bump increments")
        .action(Action::ClickButton("Bump".into()))
        .assert(Assert::State { pointer: "/clicks".into(), expected: serde_json::json!(1) }),
);
let rows = robot.run_scenario(&scenario);                    // Vec<TestResultRow>, suite = "robot-ui"

// widget-level verbs:
robot.set_field(&Target::label("paste:"), &FieldValue::Text("hello".into()))?;
let id = robot.node_id("💡 Submit idea").unwrap();
robot.click_by_id(id)?;

Drive a live viz over MCP with the same verbs: viz.click set_field="funnel.intake=hello" click_id="funnel.submit_intake". Browser / deployed-web robot (WebDriver + TOML scenarios) lives in src/robot.rs (feature robot); point --webdriver at geckodriver/chromedriver. Richer verbs + live-channel parity: robotui-richer-livedrive.md. The facett component side of the seam: facett elm-guide.

f5 — introspect

The graphs render into docs through markers, not a standalone verb — put them in a .nornir/*.md:

<!-- nornir:gen:start:depgraph -->
<!-- nornir:gen:end:depgraph -->
<!-- nornir:gen:start:symbols binary=target/debug/nornir limit=20 -->
<!-- nornir:gen:end:symbols -->

nornir docs render <repo> fills them (depgraph from cargo metadata; symbols/callgraph need a built binary first). Spec: docs-generation.md.

f6 — saga (record → replay)

A recorded UI session (SagaEvents under a SessionCtx) compiles to a ReplayPlan:

use nornir_saga::{replay::ReplayPlan, sink::JsonFileSink};

let events = JsonFileSink::new("saga.jsonl").read_all()?;
for plan in ReplayPlan::all_sessions(&events) {
    if !plan.is_replayable() { continue; }          // a log-only session has no actionable steps
    for step in &plan.steps {
        // a driver (f4 robot) navigates/activates, then asserts state_json == step.expect_state;
        // a divergence → a red matrix row naming step.source_seq
    }
}

ReplayPlan::for_session(id, &events) builds one session's plan; all_sessions builds one per session_id, oldest first.