.nornir/testmatrix-functional-emitters.md

Functional-status emitters — feeding the `nornir test` matrix

Design + guide for holger's functional_status wiring (branch feat/testmatrix-emitters). Companion to .nornir/test-matrix.md (the grid) and xtask/tests/holger_matrix.rs (the per-component cells).

What this is

Every meaningful holger test now emits one functional-status row per real check into the nordisk nornir-testmatrix engine, so a nornir test run sees not just "the test binary exited 0" but the individual checks each test made and the value each measured. The emitter is nornir_testmatrix::functional_status (added in nornir-testmatrix 0.2.0 alongside the testmatrix cargo feature).

nornir_testmatrix::functional_status(
    component, // the unit under test  -> row.suite
    check,     // the specific assertion -> row.test_name
    ok,        // the REAL assertion result -> pass | fail
    detail,    // the MEASURED value -> row.message
);

Each row carries aspect = "functional", repo from $NORNIR_TESTMATRIX_REPO (holger in CI), and metric = 1.0 (pass) / 0.0 (fail).

Compiled OUT for release (the whole point)

The emitter is dead weight in a shipped binary, so it is gated twice:

nornir-testmatrix is an optional dependency of each consumer crate, enabled only by that crate's passthrough feature:

[features]
testmatrix = ["dep:nornir-testmatrix", "nornir-testmatrix/testmatrix"]

Every call site is wrapped in #[cfg(feature = "testmatrix")].

With the feature off (the default, i.e. every release build) the dependency is never compiled and the call sites are stripped — cargo build --workspace pulls nothing extra and the code is identical to before this change. With --features testmatrix the emitter is live and rows are appended.

functional_status itself is also a #[cfg(not(feature = "testmatrix"))] no-op in the engine, so even an un-gated call compiles to nothing in release.

Where rows go

The engine appends one compact JSON array per row to the JSONL sink at $NORNIR_TESTMATRIX_OUT (default target/nornir-functional.json), using an O_APPEND single-write so parallel test threads/processes never clobber each other. The file is JsonFileSink::read_all-compatible, so nornir test reads it straight back into the matrix. xtask/tests/holger_matrix.rs proves this round trip in testmatrix_feature_emits_functional_rows_into_the_sink.

Coverage — which tests emit

97 functional rows across 25 components when the full feature-on suite runs:

Component (suite)	Where	Rows
`config`, `wire_holger`, `rust_api`	`server/lib/tests/unit_tests.rs`	6
`proxy`, `tls`	`server/lib/src/proxy.rs`, `exposed/tls.rs`	9
`file-rust`	`server/repository/file-rust/src/lib.rs`	13
`file-oci`	`server/repository/file-oci/src/lib.rs`	5
`znippy-rust`	`…/znippy-rust/src/lib.rs` + integration smoke	10
`znippy-helm`	`…/znippy-helm/src/{lib,index}.rs`	8
`znippy-{maven,python}`	resp. crate `src/lib.rs`	3+4
`znippy-{composer,conda,deb,docker,gem,go,npm,nuget,package,rpm}`	each crate `src/lib.rs`	2 each
`traits`	`traits/src/lib.rs`	2
`world-connector`	`agent/connector/world/src/pip.rs`	5
`holger-server-ui`	`server/ui/tests/{local_core,remote_grpc,robot_ui}.rs` + `src/app.rs` look_tests	9
`agent-tar-zstd`	`agent/cli/tests/integration_tar_zstd.rs`	3

The holger-server-ui rows include the headless-render self-test paths the matrix calls "robot-UI" cells: each view's observable state_json() (Browse / Archive listings, the look-&-feel preset + resolved palette) is asserted and emitted.

LAW conformance

inject real input + assert real output — every ok is the same boolean the test's assert! checks; every detail is the measured count/hash/route, not a constant. No "didn't panic" rows.
implementation gets a test in the matrix — the emitter wiring itself is covered by testmatrix_feature_emits_functional_rows_into_the_sink, which runs a real crate with the feature, reads the sink back, and asserts real rows.
components expose readable data — the UI rows are exactly "what the operator sees, as data" (state_json), not pixels.

How to run the emitters

# Release path — feature off, nothing compiled, nothing emitted:
cargo build --workspace
cargo test --workspace          # green, zero extra deps

# Emit rows for the matrix (per crate that has the feature):
NORNIR_TESTMATRIX_OUT=target/holger-functional.json \
NORNIR_TESTMATRIX_REPO=holger \
  cargo test -p repository-znippy-rust --features testmatrix --lib

# The matrix-side proof:
cargo test -p xtask --test holger_matrix \
  testmatrix_feature_emits_functional_rows_into_the_sink

NORNIR_TESTMATRIX_OUT / NORNIR_TESTMATRIX_REPO / NORNIR_TESTMATRIX_RUN override the sink path / repo partition / run-id (all optional).

DEV-only patch

Cargo.toml carries a [patch.crates-io] mapping nornir-testmatrix to the local nornir checkout (it resolves 0.2.0 before that lands on crates.io). Like the znippy patches, remove it before publishing; it never affects a release build because testmatrix is off there.

Functional-status emitters — feeding the nornir test matrix