nordisk/edda

nornir-testmatrix — the silenced-test guard (gatedtests)

Silence is not success. A tests/*.rs file whose crate-level #![cfg(feature = "X")] names a feature outside its crate's default set compiles to an empty test binary. cargo test prints running 0 tests ... ok and the file is green by vacuum.

And one degree quieter: a [[test]] target with required-features = ["X"] is not compiled empty, it is skipped outright — no warning, no error, not even a running 0 tests line to notice.

1. Why

The 2026-07-21 constellation sweep (workspace_nordisk/silenced-tests-audit-2026-07-21.md) found 78 such files across 11 of 15 repos, hiding ~216 test functions — knut 15/66, modgunn 9/38, korp 13/28, facett 19/27, korp-collectors 5/23, nornir 10/20, and two in edda itself. Nothing was broken; nothing ran.

Three camouflage forms defeat review:

1. all(default-on, default-on, default-OFF) — the file reads exactly like its many running siblings. Only the trailing conjunct is dark. nornir's all(viz, server, testmatrix) group (14 tests) is the worst offender.
2. Near-miss arm names — skade/tests/lineage_http.rs needs lineage-http; the CI matrix passes lineage, a different feature that does not imply it.
3. Sibling asymmetry — korp-collectors runs collectors_http.rs (parsers, default-on) while eight CSV tests beside it are dark.

holger already had the mitigation: xtask/tests/holger_matrix.rs explicitly re-invokes cargo test -p holger-ui --features gui --test <name> per gated file (the holger_ui_robot_app_cell / demo_walk / login_pane cells, plus a workspace --features testmatrix arm). It is a convention, so it drifts — warehouse_scan.rs is the one gui file that matrix forgot. This module turns the convention into a gate, in the shared crate, so every repo inherits it.

2. The model

cargo metadata --no-deps
  └─ per workspace member: name · [features] table · every `test`-kind target
       │                                            · the target's required-features
       │                                            · the member's dependency EDGES
       ├─ testsurface::resolve(default_config)  — cargo's workspace-wide feature
       │                                          unification, run to a fixed point
       └─ read the target's src file
            ├─ extract_crate_cfg  — the leading crate-level #![cfg(...)]  (never #![cfg_attr])
            ├─ parse_cfg_expr     — CfgExpr::{Feature, All, Any, Not, Other}
            ├─ default_features   — transitive closure of `default` in THIS crate's table
            └─ classify           — Default | Rescued | ViaEdge | Silenced
                                       ↑ MatrixArms (.nornir/testmatrix-arms.json)
GatedTestReport → is_green() / summary() / rows() → Vec<TestResultRow> (aspect = "gated-tests")

The FOUR sources of "which feature is on"

A test runs only if cargo selects the target and the file compiles to something. Four things can make that true, and a verdict that weighs fewer of them is a guess:

Sources 3 and 4 both landed on 2026-08-26. Source 3 had been parsed into TestTarget::required_features since the module existed and never consulted in the verdict — a field parsed and dropped, which is worse than one never read: the code looks like it handles the case, so nobody checks.

required-features is the quietest gate of the four

An unsatisfied #![cfg] compiles the file to an empty binary that at least prints running 0 tests ... ok. An unsatisfied required-features makes cargo skip the target outright — no warning, no error, no line of output. And such a target usually carries no crate-level #![cfg] at all, so classify bailed on its first line and emitted no row of any kind.

Measured 2026-08-26 with cargo metadata --no-deps in every workspace under ~/git, counting test-kind targets with a non-empty required-features: 9 targets in 4 repos (dwarves 1, facett 4, gunnar 3, holger 1), and not one of them carries a crate-level #![cfg] — so all nine were invisible. (The 2026-08-25 skid-forge note said 12 in 7; re-measured here it is 9 in 4, or 10 in 5 counting this crate's own broken_target fixture. The likely difference is .claude/worktrees copies counted as separate repos.)

ViaEdge — the verdict that refuses to cry wolf

Five of those nine are not dark. facett/facett-demo/Cargo.toml:24 names facett = { …, features = ["model3d", "docview", "video", "plotters"] } and gunnar/crates/gunnar-cli/Cargo.toml:366 names gunnar-lfs = { …, features = ["fs"] }, both on unconditional edges — so cargo's workspace-wide feature unification really does build and run all five under a plain cargo test --workspace. A default-closure-only reading would have called them SILENCED: five false reds out of nine findings, and a detector that reports a live test as dark is a detector people switch off.

GateVerdict::ViaEdge passes and names the sibling holding the file up. It is not folded into Default, because nothing DECLARES the route: run cargo test -p facett on its own, or drop the feature from that one edge, and the file goes dark in silence. It is the only verdict that says "green today, undeclared, one edit from dark".

The fourth source also removed an existing FALSE red: gunnar/crates/gunnar-lfs/tests/swallowed_errors.rs (#![cfg(feature = "fs")], fs ∉ default) was reported SILENCED and is in fact lit by gunnar-cli's edge.

Constellation-wide effect of the two new sources, measured old-rule-beside-new over every repo under ~/git: 2 new REDs — holger/server/cli/tests/appliance_gate.rs (6 test fns, required-features = ["appliance"]) and dwarves/crates/dwarves/tests/min_req_floor_live.rs (4 fns, required-features = ["release"]) — and 1 lost red, the gunnar false positive above. Nothing else in any repo changes verdict.

Three-valued cfg evaluation

CfgExpr::eval returns Option<bool>; a non-feature predicate (unix, target_os = "linux", test, docsrs) is None — unknown, never false. All is Some(false) if any conjunct is; otherwise None if any is unknown. Any mirrors it. A file is dark iff eval(default_closure) == Some(false).

Consequences, all pinned by tests:

• not(target_os = "windows") is never flagged — no feature is mentioned.
• all(unix, feature = "heavy") with heavy off is flagged — the feature conjunct is decisive regardless of the platform one.
• not(feature = "x") with x off is satisfied, not silenced.
• A gate mentioning no feature at all is not even listed as a finding.

Feature closure

feature_closure BFSes the crate's own [features] table. dep:serde and cross-crate other/feat (and weak other?/feat) entries enable features on dependencies, never on self, so they are skipped. default = ["bundle"], bundle = ["light"] therefore makes a #![cfg(feature = "light")] file Default, not a false alarm — the transitive chain is the common shape.

Cargo's implicit optional-dependency features come free: they are already in cargo metadata's features map.

3. How a repo declares its rescue arms

<repo>/.nornir/testmatrix-arms.json — a declaration of the cargo test re-invocations the repo actually runs. Modelled directly on holger's cells:

{
  "arms": [
    { "package": "holger-ui", "features": ["gui"], "test": "robot_ui_app",
      "note": "xtask/tests/holger_matrix.rs :: holger_ui_robot_app_cell" },
    { "package": "holger-ui", "features": ["gui"], "test": "demo_walk" },
    { "package": "holger-ui", "features": ["gui"] },
    { "features": ["testmatrix"], "note": "workspace functional lane" }
  ]
}

An arm rescues a file iff it targets that (package, test) and the file's cfg compiles under arm_set = closure(features ∪ default unless suppressed). So a near-miss arm (lineage for a lineage-http file) rescues nothing — which is precisely the bug the audit found in skade. MatrixArm::features([..]) .package(..).test(..) builds the same thing in Rust for repos that prefer to declare arms in code and pass a MatrixArms directly.

Missing declaration file ⇒ empty arm set (every gated file must then be default-reachable). MatrixArms::read returns Ok(None) for "no file" vs a parse error, MatrixArms::load collapses it.

4. API

GatedTestFile::cfg is empty when the target's only gate is its required-features; gate_phrase() renders whichever halves are present, so a report line never attributes a manifest skip to a #![cfg] the file does not carry.

rows() emits one TestResultRow per gated file on the ASPECT_GATED_TESTS ("gated-tests") aspect: suite = crate, test_name = gated-tests::<target>, status = fail for Silenced / pass otherwise, metric = the hidden-test count so the matrix can rank by damage. Rescued files emit a green row, not nothing — the excuse stays visible, the same way Gap::allowlisted records its excuses.

is_green() is the HARD-zero gate: green ⟺ zero silenced files.

5. Proof

tests/gated_tests_guard.rs drives three standalone fixture repos under tests/fixtures/ through the real audit_repo path (real cargo metadata, real Cargo.toml, real declaration file) — the leaf_emitter convention:

Red-when-broken: stub classify's dark branch to GateVerdict::Default and gated_dark goes green → assertion 1 fails. Remove the arm-matching branch and gated_rescued goes red → assertion 3 fails. Drop the closure BFS to a literal default array lookup and gated_default_on false-alarms → assertion 2 fails. The three fixtures pin each other.

tests/required_features_gate.rs does the same for sources 3 and 4:

Red-when-broken, both directions: revert classify's required_features handling and broken_target reports files: [] → the first test fails on silenced().len() == 1 becoming 0, and assert_no_silenced_tests goes GREEN on a workspace holding a test target cargo never compiles. Delete the ViaEdge branch and required_features_edge's gpu_probe becomes SILENCED → the second fails the other way, on the false red. The two fixtures pin each other exactly as the first three do.

The guard's own test file is deliberately ungated — a silenced-test guard that is itself silenced would be the joke that writes itself.

6. Using it as a gate

use nornir_testmatrix::{audit_repo, JsonFileSink, TestSink};

let rep = audit_repo(std::path::Path::new("."), &nornir_testmatrix::new_run_id())?;
JsonFileSink::new("target/nornir-testmatrix.json").append(&rep.rows())?;
assert!(rep.is_green(), "{}", rep.summary());

Pointed at a repo with known findings it prints e.g.

93 test targets scanned · 21 feature-gated (3 of them by `required-features`) ·
0 rescued by declared arms · 2 run only through a dependency edge ·
15 SILENCED (66 hidden test fns) — RED

The fix for a finding is one of two things, both explicit: move the feature into default, or declare the arm that runs it. Never a third thing, and never silence.

A ViaEdge count in a GREEN summary is not a finding but it is not nothing either: it says n targets run only because some sibling's dependency edge happens to name their feature, and nothing in the repo declares that. Give each one an arm and it becomes Rescued — a statement instead of an accident.