xtask/src/gate.rs
//! `cargo xtask gate` — the gate holger did not have.
//!
//! ## The measurement that made this necessary
//!
//! At `542e296` the repository had no automatic build and no automatic test
//! run. `.forgejo/workflows/security.yml` was the only workflow file, and its
//! own comment at lines 12-13 states that it runs no `cargo build`, no
//! `cargo test` and no `cargo deny`. `.github/` contains one Markdown file.
//!
//! Two consequences are on the record:
//!
//! * `cargo build -p xtask --example nornir-bench` failed with ten `E0063`
//! errors for weeks. `nornir-bench` 0.1.8 added a struct field and holger
//! constructs that struct by literal at ten sites. Nothing built the example,
//! because an `--example` of a `publish = false` crate is built by nothing in
//! the normal flow. The benchmark numbers the repository publishes were
//! therefore not reproducible.
//! * Three feature leaks (`front`, `auth-passkey`, `auth-oidc`/`auth-oauth2`
//! with its OpenSSL closure) each reached a shipped binary and each was found
//! by a person inspecting a deployed artifact after the fact.
//!
//! ## What this verb is
//!
//! One entry point, five phases, all Rust, no YAML and no shell script:
//!
//! | phase | what it checks | where it runs |
//! |---|---|---|
//! | 1 rules | RULE 1 and RULE 2 from `holger_recipe::honesty` | in process, no cargo |
//! | 2 rows | every declared cargo row builds | in the declared toolchain container |
//! | 3 tests | the workspace test suites, which is where RULE 1 and RULE 2 also run as tests | in the declared toolchain container |
//! | 4 bench | RULE 3: every published bench driver compiles | in the declared toolchain container |
//! | 5 arms | the SILENCED guards: nothing is dark, and every declared rescue arm is RUN | on the host (named exception, below) |
//!
//! ## Why phase 5 exists — the hole phase 3 cannot see
//!
//! Phase 3 runs `cargo test --workspace` with DEFAULT features. A `tests/*.rs`
//! whose crate-level `#![cfg(feature = "X")]` is not in its crate's `default`
//! closure does not skip under that command: it compiles to an EMPTY test binary
//! and libtest prints `running 0 tests ... ok`. Phase 3 therefore reports green
//! for a file that executed nothing, and holger has no CI to notice.
//!
//! MEASURED on `main` at `b8562f3`, by `nornir_testmatrix::audit_repo`:
//!
//! ```text
//! 88 test targets scanned · 7 feature-gated · 0 rescued by declared arms
//! · 6 SILENCED (47 hidden test fns) — RED
//! ```
//!
//! All six are security guards, `open_appliance_loads_no_paid_module` — the OPEN
//! appliance's module census — among them. The repository already carried both
//! halves of the fix and used neither: `xtask/tests/silenced_tests_guard.rs`
//! names the dark files, and `.nornir/testmatrix-arms.json` is where the
//! re-invocation that rescues each one is declared. What was missing is the
//! thing that RUNS the declared arms; the arms file's own closing note said so
//! ("What they do NOT prove: that anything runs them on a schedule"). This phase
//! is that thing.
//!
//! It runs the audit first (a new dark file is a gate failure, not just a test
//! failure) and then executes every declared arm through
//! `nornir_testmatrix::runner::run_arms`, which parses each arm's per-test
//! verdicts. `ArmRun::green()` refuses an arm that compiled but ran ZERO tests —
//! the same vacuum one level up, because a rescue arm that rescues nothing is
//! exactly what this phase exists to catch.
//!
//! ### The named host exception
//!
//! Phases 2-4 run in the declared toolchain container. Phase 5 runs on the host,
//! deliberately and as a NAMED exception: `run_arm` is the one writer of an
//! arm's argv and of the run parsing, and it drives `cargo` directly with its own
//! target dir (the caller is itself under `cargo test`, which holds the build
//! lock on the shared one). Re-rendering that argv here to push it through
//! `run_cargo_in_container` would be a second spelling of the arm grammar, which
//! is the defect this whole lane is about. Moving phase 5 into the container is
//! a change to `nornir-testmatrix` — `run_arm` taking a runner — not a change
//! here.
//!
//! Four of the six arms load a REAL wasm guest. The guards build it on demand
//! (`traits::test_support::wasm_guest`, host cargo, no container), so the
//! precondition is the test's own, not an operator's memory.
//!
//! Phase 1 duplicates no mechanism: RULE 1 and RULE 2 are plain
//! `#[test]` functions in `holger-recipe/tests/feature_honesty.rs`, which phase
//! 3 runs. Phase 1 runs the same two functions directly so that an operator
//! gets the finding list in under a second without compiling the workspace.
//!
//! ## No build knowledge lives here
//!
//! Phase 2 calls `crate::run_build_with(&[])`, the existing `cargo xtask
//! build` body, which resolves the rows through `holger-recipe`, enters the
//! declared toolchain container once, and builds each cargo row with
//! `nornir_build_thing::cargo_container::build_variant_in_container`. Rows whose
//! `kind` cargo cannot build (`container`, `iso`, `installer`) are listed and
//! refused by name with the door that does build them, which is that function's
//! existing behaviour.
//!
//! Phases 3 and 4 use `run_cargo_in_container` with the same toolchain. The only
//! list in this file is read from `holger_recipe::honesty::BENCH_DRIVERS`; this
//! module names no package, no feature and no target triple of its own.
//!
//! ## History
//!
//! Phase 2's first real runs (loki and oden, 2026-09-20) found two defects the declared rows
//! had hidden: `bench` in the default pulled `openssl-sys` into the online rows, and the
//! `full` tier's federation code (`server/lib/src/upstream/mod.rs`) no longer compiled
//! because no row built it. Both were fixed at the source. The ORIGINAL and OPEN deliveries
//! those runs measured were retired by HOL-PRODUCTS (2026-10-30): one delivery, MVP, the
//! four products and the estate's builds (`git log -- xtask/src/gate.rs` has the tables).
//!
//! The finding worth more than the fix: **a build row nobody builds is a
//! compile check that never runs**, and `cargo check --workspace --all-targets`
//! cannot stand in for it — `--all-targets` widens the TARGET axis and says
//! nothing about the FEATURE axis. [`rows_check`] (`cargo xtask rows-check`) is
//! the cheap standing answer: the same declared rows, `cargo check` instead of
//! a release build, so the feature axis is compiled by the machine before a
//! commit rather than by a person months later.
//!
//! The MVP delivery — the one the appliance ships — is clean.
//!
//! ## What phase 4 found on its first real run
//!
//! MEASURED on loki, 2026-09-20, `cargo xtask gate --no-rows --no-tests` in the
//! declared toolchain container `nordisk-cargo-toolchain@sha256:3d5115ba…`:
//!
//! * `cargo build -p xtask --example nornir-bench` FAILS at
//! `openssl-sys`: *"Could not find openssl via pkg-config"*. The container
//! carries no OpenSSL development files, correctly — holger's airgap tiers are
//! judged on exactly that. So the bench driver's dependency closure contains
//! `openssl-sys`, which is the crate-level half of feature leak 3, and the
//! driver cannot be built in the environment the declaration pins. The
//! published performance numbers are therefore reproducible only on a box that
//! happens to have OpenSSL headers, which is the defect the container rule
//! exists to prevent.
//! * `cargo build -p holger-bench-scenarios --example secure_update_bench
//! --features secure-update` compiles.
//! * The `holger_ds` example named by `.nornir/stinky/09-*` §4.3 DOES NOT EXIST.
//! `bench-scenarios/examples/` is `art_probe`, `pip_probe`,
//! `retention_probe`, `secure_update_bench`. The three probes do not compile
//! either (`E0433` on `testcontainers` and `tokio`, and they declare no
//! `required-features`); they are recorded in
//! `honesty::EXAMPLES_THAT_DO_NOT_BUILD` rather than gated on, because they
//! are container probes and not drivers whose numbers the house publishes.
//!
//! ## Relationship to dwarves
//!
//! dwarves is the constellation's build system and can drive the same declared
//! rows. This verb does not depend on a dwarves pipeline, because those
//! pipelines are, as of 2026-09-20, little exercised. `cargo xtask gate` is a
//! local and offline door onto the same build thing.
use std::path::Path;
use anyhow::{bail, Context, Result};
use holger_recipe::honesty::{self, BENCH_DRIVERS};
use nornir_build_thing::cargo_container as cc;
/// Which phases the operator asked for.
#[derive(Debug, Clone)]
pub struct Options {
pub rules: bool,
pub rows: bool,
pub tests: bool,
pub bench: bool,
pub arms: bool,
}
impl Default for Options {
fn default() -> Self {
Self {
rules: true,
rows: true,
tests: true,
bench: true,
arms: true,
}
}
}
impl Options {
pub fn from_args<I: Iterator<Item = String>>(args: I) -> Options {
let args: Vec<String> = args.collect();
let has = |f: &str| args.iter().any(|a| a == f);
let mut o = Options::default();
// `--rules` is the offline sub-gate: it needs no container engine, so a
// box with no podman socket can still be told about a feature rider.
if has("--rules") {
o.rows = false;
o.tests = false;
o.bench = false;
o.arms = false;
}
// `--arms` is the silenced-guard sub-gate on its own: it needs cargo but
// no container engine, and it is the one phase an operator wants to
// repeat while turning a dark test file back on.
if has("--arms") {
o.rules = false;
o.rows = false;
o.tests = false;
o.bench = false;
}
if has("--no-rows") {
o.rows = false;
}
if has("--no-tests") {
o.tests = false;
}
if has("--no-bench") {
o.bench = false;
}
if has("--no-arms") {
o.arms = false;
}
o
}
}
pub fn run() -> Result<()> {
let opts = Options::from_args(std::env::args().skip(2));
let root = honesty::find_workspace_root(Path::new(env!("CARGO_MANIFEST_DIR")))?;
let mut failures: Vec<String> = Vec::new();
if opts.rules {
println!("=== gate 1/5 — RULE 1 and RULE 2 over the manifests (no cargo) ===");
let report = honesty::check(&root)?;
if report.is_green() {
println!(" no feature riders, no bench or test crate in a shipped graph");
} else {
for line in report.lines() {
println!(" FAIL {line}");
}
failures.push(format!(
"rules: {} rider(s), {} bench/test edge(s)",
report.riders.len(),
report.bench_edges.len()
));
}
}
// The container road is entered ONCE, and only when a phase needs it.
let road = if opts.rows || opts.tests || opts.bench {
Some(crate::cargo_road(&root)?)
} else {
None
};
if opts.rows {
println!("\n=== gate 2/5 — every declared cargo row builds in the container ===");
// ONE delivery at a time, and EVERY delivery, read off the declaration (one, MVP,
// since HOL-PRODUCTS; a second would be pressed without a change here).
for suite in declared_suites() {
println!("\n--- delivery {suite} ---");
if let Err(e) = crate::run_build_with(&["--suite".to_string(), suite.clone()]) {
failures.push(format!("rows in delivery '{suite}': {e:#}"));
}
}
}
if let Some(road) = &road {
if opts.tests {
println!(
"\n=== gate 3/5 — the workspace test suites (RULE 1 and RULE 2 run here too) ==="
);
let args: Vec<String> = ["test", "--workspace", "--no-fail-fast"]
.iter()
.map(|s| s.to_string())
.collect();
if !cargo_in_container(road, &root, "gate-tests", &args)? {
failures.push("tests: the workspace suites are red".into());
}
// The `estate/` workspace (H1): xtask, the build declaration, holger-fleet,
// holger-live, the monetize guest and the estate tests against the real ONE
// box. The root `--workspace` does not reach it, so it runs here, every time.
let args: Vec<String> = [
"test",
"--manifest-path",
"estate/Cargo.toml",
"--workspace",
"--no-fail-fast",
]
.iter()
.map(|s| s.to_string())
.collect();
if !cargo_in_container(road, &root, "gate-estate-tests", &args)? {
failures.push("tests: the estate/ workspace suites are red".into());
}
// holger-estate-front is its own workspace too (it links monetize and
// holger-ui), so `--workspace` never ran its suite (A18 §4 #1). Both ways:
// the open front and the metered one.
for (label, extra) in [
("gate-estate-front", None),
("gate-estate-front-metered", Some("metered")),
] {
let mut args: Vec<String> = [
"test",
"--manifest-path",
"holger-estate-front/Cargo.toml",
"--no-fail-fast",
]
.iter()
.map(|s| s.to_string())
.collect();
if let Some(f) = extra {
args.extend(["--features".to_string(), f.to_string()]);
}
if !cargo_in_container(road, &root, label, &args)? {
failures.push(format!(
"tests: holger-estate-front's suite is red ({label})"
));
}
}
}
if opts.bench {
println!("\n=== gate 4/5 — RULE 3: the published bench drivers compile ===");
for (pkg, example, feats) in BENCH_DRIVERS {
let mut args: Vec<String> = vec![
"build".into(),
"-p".into(),
(*pkg).into(),
"--example".into(),
(*example).into(),
];
if !feats.is_empty() {
args.push("--features".into());
args.push(feats.join(","));
}
let label = format!("gate-bench-{pkg}-{example}");
println!(" cargo {}", args.join(" "));
if !cargo_in_container(road, &root, &label, &args)? {
failures.push(format!(
"RULE 3: bench driver {pkg} --example {example} does not compile, so the \
numbers it produces cannot be reproduced"
));
}
}
}
}
if opts.arms {
println!("\n=== gate 5/5 — the SILENCED guards: nothing dark, every declared arm RUN ===");
if let Err(e) = run_arms_phase(&root, &mut failures) {
failures.push(format!("arms: the phase could not run: {e:#}"));
}
}
if failures.is_empty() {
println!("\ngate GREEN");
return Ok(());
}
println!("\ngate RED:");
for f in &failures {
println!(" - {f}");
}
bail!("{} gate failure(s)", failures.len());
}
/// **Phase 5.** The audit, then every declared rescue arm RUN.
///
/// Two halves, and neither replaces the other:
///
/// 1. `nornir_testmatrix::audit_repo` compares the tree against
/// `.nornir/testmatrix-arms.json`. A `tests/*.rs` behind a crate-level
/// `#![cfg(feature = …)]` that neither the crate's `default` closure nor a
/// declared arm turns on is SILENCED — it compiles to an empty binary and
/// prints `ok`. Finding one is a gate failure with the file and its hidden
/// test count named.
/// 2. Every declared arm is then RUN. A declaration nothing executes is how the
/// arms file spent its first seven weeks, and `ArmRun::green()` additionally
/// refuses an arm that compiled and ran ZERO tests — a rescue arm that
/// rescues nothing is the same vacuum one level up.
///
/// Four of the arms load a real wasm guest; each guard builds it on demand.
fn run_arms_phase(root: &Path, failures: &mut Vec<String>) -> Result<()> {
use nornir_testmatrix::gatedtests::{self, MatrixArms};
let report = gatedtests::audit_repo(root, "xtask-gate")
.with_context(|| format!("auditing gated test targets under {}", root.display()))?;
println!(" {}", report.summary());
// A walker that scanned nothing would make "no silenced targets" trivially
// true, which is the disease this phase was written for.
if report.scanned == 0 {
failures.push(format!(
"arms: the gated-test audit scanned ZERO test targets under {} — it proved nothing",
root.display()
));
return Ok(());
}
if !report.is_green() {
for f in report.silenced() {
println!(" FAIL {}", f.message());
}
failures.push(format!(
"arms: {} test target(s) are SILENCED — they compile to an empty binary and print \
`ok`. Declare a rescue arm in {} for each, or move the feature into `default`.",
report.silenced().len(),
gatedtests::ARMS_FILE,
));
}
let arms =
MatrixArms::load(root).with_context(|| format!("reading {}", gatedtests::ARMS_FILE))?;
if arms.arms.is_empty() {
println!(" no rescue arms declared — nothing to run");
return Ok(());
}
// ONE target dir for the guests AND for the arms, and it is not the ordinary
// one. `run_arm` builds each arm under `armcheck::arm_target_dir` — a child
// cargo pointed at a dir whose build lock the parent holds would block
// forever rather than fail — and the guards resolve their module from
// `CARGO_TARGET_DIR`, which `run_arm` sets to that same dir. Building the
// guests into `target/` therefore puts them where no arm looks, and the
// census fails on a missing file instead of on anything it is about.
// MEASURED before this line existed: 5 of 6 census tests and 6 of 8
// linker-census tests failed on `…/armcheck/wasm32-unknown-unknown/release/
// holger_plugin_allocation_wasm.wasm has not been built`.
let arm_dir = nornir_testmatrix::armcheck::arm_target_dir(root);
std::env::set_var("NORNIR_ARM_TARGET_DIR", &arm_dir);
std::env::set_var("CARGO_TARGET_DIR", &arm_dir);
// The guests the arms load are built on demand by the guards themselves
// (`traits::test_support::wasm_guest`, host cargo, `<CARGO_TARGET_DIR>/wasm-guests`),
// so this phase needs no container engine.
let runs = nornir_testmatrix::runner::run_arms(root, &arms)
.context("running the declared rescue arms")?;
for r in &runs {
println!(
" {} · {} ran, {} passed, {} failed",
r.command,
r.ran(),
r.passed(),
r.failed()
);
if r.green() {
continue;
}
// Say WHICH of the three ways it failed, because they need different
// fixes — and in THIS order. A red test also exits non-zero, so asking
// `!compiled_and_ran` first reports every ordinary test failure as "it
// did not compile", which is the least useful of the three answers.
// MEASURED: a deliberately broken grant check printed exactly that.
let why = if r.ran() == 0 && r.compiled_and_ran {
"it compiled and ran ZERO tests — the arm rescues nothing, which is the vacuum \
this phase exists to catch"
.to_string()
} else if r.failed() > 0 {
format!(
"{} of {} test(s) FAILED — re-run that line for the assertions",
r.failed(),
r.ran(),
)
} else {
"it did not compile, or the run was killed before it reported".to_string()
};
failures.push(format!("arms: `{}` — {}", r.command, why));
}
Ok(())
}
/// Every delivery the declaration tags, in declaration order.
///
/// Read off the rows' own `suites` list, never restated here: xtask owns no
/// build knowledge, and a hard-coded `["MVP"]` would be a second spelling of the
/// declaration. A row with no suite tag is counted under `ORIGINAL`, which is the
/// shared resolver's name for an untagged row (holger tags every row).
fn declared_suites() -> Vec<String> {
let mut out: Vec<String> = Vec::new();
for v in &holger_recipe::declared().variants {
if v.suites.is_empty() {
if !out.iter().any(|s| s == "ORIGINAL") {
out.push("ORIGINAL".to_string());
}
continue;
}
for s in &v.suites {
if !out.contains(s) {
out.push(s.clone());
}
}
}
out
}
/// Run one cargo command inside the declared toolchain container.
///
/// The same toolchain, target dir and registry bind `cargo xtask build` uses, so
/// a test run and a row build see one environment. The host contributes a
/// container engine socket and a workdir and nothing else.
fn cargo_in_container(
road: &crate::CargoRoad,
root: &Path,
label: &str,
args: &[String],
) -> Result<bool> {
let inv = cc::CargoInvocation {
label: label.to_string(),
repo_root: root.to_path_buf(),
target_dir: road.target_dir.clone(),
registry: road.registry.clone(),
cargo_args: args.to_vec(),
cfg: None,
env: cc::RunEnv::new(),
extra_binds: Vec::new(),
};
let out = cc::run_cargo_in_container(&road.tc, &inv, &road.runner).with_context(|| {
format!(
"running cargo {} in the toolchain container",
args.join(" ")
)
})?;
Ok(out.success())
}
/// `cargo xtask rows-check` — **the cheap half of phase 2**: `cargo check` every
/// declared cargo row, in the declared toolchain container, in one pass.
///
/// ## The defect that asked for it
///
/// MEASURED 2026-09-20, lane `lane/full-builds`.
/// `.nornir/vec-to-bytes-migration.md` moved `RepositoryBackendTrait::fetch` from
/// `Vec<u8>` to `bytes::Bytes` and named its blast radius "1 upstream". There were
/// two: `server/lib/src/upstream/mod.rs` sits behind the `federation` feature,
/// which only the `full` row selects, so it was never recompiled and carried two
/// `E0308`s until someone built that row.
///
/// `cargo check --workspace --all-targets` does NOT catch this and cannot:
/// `--all-targets` widens the TARGET axis (bins, tests, examples, benches) and
/// says nothing about the FEATURE axis. Code behind a feature no default set
/// selects is not compiled by any number of targets. The feature axis is the
/// declarations axis, and the declaration is the rows.
///
/// ## Why this is not phase 2 again
///
/// Phase 2 BUILDS every row: release, linked. MEASURED on oden 2026-09-20 a server
/// row costs minutes and produces a 35-42 MB artifact. That is the
/// right gate before a release and the wrong one before a commit, and a gate too
/// expensive to run before a commit is the gate that does not run. This verb
/// answers the same question — does every declared configuration still compile? —
/// with `cargo check`: no codegen, no link, no artifact.
///
/// ## The cost, measured
///
/// MEASURED on oden 2026-09-20 in the declared toolchain container, warm target
/// dir, all eleven cargo rows in one pass:
///
/// ```text
/// embedded 8.1 s · mini 8.3 s · full 11.5 s · mvp-server 6.1 s ·
/// mvp-server-metered 8.1 s · mvp-server-musl 8.2 s · server-local 8.4 s ·
/// agent 2.1 s · agent-online 4.1 s · mvp-bundle 2.1 s · mvp-verify 2.1 s
/// === rows-check: 11 of 11 cargo row(s) compile ===
/// ```
///
/// **69.1 s for the whole feature axis**, against the several minutes per row a
/// release build of the same eleven costs. That is the difference between a gate
/// somebody runs before a commit and a gate that waits for a release.
///
/// ## It owns no build knowledge either
///
/// The argv is the ROWs own `cargo_args()` with the verb swapped, so a rows
/// package, bin, features, `--no-default-features`, profile and target are
/// whatever the declaration says and this file names none of them. Non-cargo
/// kinds (`container`, `iso`, `installer`) are skipped by `ArtifactKind::is_cargo`,
/// the same predicate the build door uses.
pub fn rows_check() -> Result<()> {
let root = honesty::find_workspace_root(Path::new(env!("CARGO_MANIFEST_DIR")))?;
let set = holger_build_space::build_variant_set()?;
let road = crate::cargo_road(&root)?;
let mut failures: Vec<String> = Vec::new();
let mut checked = 0usize;
let mut skipped: Vec<String> = Vec::new();
for v in &set.variants {
if !v.kind.is_cargo() {
skipped.push(format!("{} ({})", v.name, v.kind.as_str()));
continue;
}
// The rows own argv, with `build` swapped for `check`. Everything else —
// package, bin, features, no-default-features, profile, target — is the
// declarations, untouched.
let mut args = v.cargo_args();
args[0] = "check".to_string();
println!("\n--- {} : cargo {}", v.name, args.join(" "));
let t0 = std::time::Instant::now();
let ok = cargo_in_container(&road, &root, &format!("rows-check-{}", v.name), &args)?;
checked += 1;
if ok {
println!(" ok in {:.1} s", t0.elapsed().as_secs_f64());
} else {
failures.push(v.name.clone());
}
}
println!(
"\n=== rows-check: {} of {} cargo row(s) compile ===",
checked - failures.len(),
checked
);
if !skipped.is_empty() {
println!(" not cargo rows, not checked here: {}", skipped.join(", "));
}
if failures.is_empty() {
return Ok(());
}
bail!("rows-check RED: {}", failures.join(", "));
}