Your browser sucks, JavaScript disabled! Gunnar's console needs JavaScript and WebAssembly. Git over SSH on port 2222 does not.

nordisk/edda

crates/nornir-testmatrix/src/chain.rs

//! **The CHAIN MATRIX** — the whole road from a declared build thing to a
//! server that answers, expressed as a grid you can look at.
//!
//! # The idea, in one paragraph
//!
//! A test is not "the build succeeded". A test is **the whole chain until a
//! working server**. The matrix is therefore a PRODUCER, not a scoreboard: its
//! rows are every build thing the suite DECLARES, its columns are the seven
//! stages of that road, and a cell is only reachable if the cell to its left
//! passed. Running the matrix builds the artifacts, boots them, mints their
//! identities and pushes git through them; the grid is what falls out.
//!
//! ```text
//!   1 BUILT        the Build Thing produced the artifact
//!   2 INSTANTIATED it is RUNNING in its own kind's shape
//!                    exe       → executed
//!                    service   → systemd unit up (Skidbladnir)
//!                    container → podman run, volume mounted
//!                    iso       → KVM booted off the medium
//!                    installer → KVM install, ISO DETACHED, POWER-CYCLED,
//!                                booted off the DISK (a SECOND VM lifetime)
//!   3 IDENTITY     hub key in; appliance key minted-or-loaded; its OWN public
//!                  key printed AND on disk       — CEREMONY stage
//!   4 TUNNEL       handshake AND traffic that CROSSED
//!                                                — CEREMONY stage
//!   5 SERVES       IT ANSWERED. What a correct answer IS comes from the row's
//!                  declared [`Answer`]: a git forge's is push + clone +
//!                  fsck --full --strict, a map server's is a correct RPC reply,
//!                  a CLI's is correct stdout.
//!   6 DURABLE      reboot / replace / upgrade → same public key, repo still there
//!   7 READY        TERMINAL: the installed server came back up and answered
//! ```
//!
//! # The ceremony is not the point — ANSWERING is
//!
//! MEASURED 2026-08-15: `korp-server` answered a real client
//! (`{"cases":0,…,"error":null,"seq":0}` — a Hugin `StreamState` + Munin
//! `ListCases` round trip) and **that correct answer had no cell to land in**,
//! because stages 3–4 are a WireGuard appliance ceremony korp deliberately does
//! not run (the 2026-08-03 `suite=ORIGINAL` ruling: `korp-appliance` is not a
//! variant). The chain hard-ordered IDENTITY → TUNNEL → SERVES, so korp's named
//! skip at stage 3 made stages 4–7 permanently `unreachable`, and the ONE thing
//! the matrix exists to prove could not be recorded.
//!
//! The fix is not to skip-and-pass. It is that "runs the appliance identity
//! ceremony" and "the thing works" are TWO axes, and the row now declares both:
//!
//! * [`Ceremony`] — does this row run the wg-appliance-core ceremony at all?
//!   [`Ceremony::None`] makes IDENTITY and TUNNEL [`ProofVerdict::Na`] (this row
//!   never had them), **never** `Absent` (no evidence) and **never** an
//!   auto-green (a fabricated pass). An `Na` stage is stepped over, so SERVES
//!   becomes reachable straight out of INSTANTIATED.
//! * [`Answer`] — what a CORRECT ANSWER is for this row, supplied by the caller.
//!   The driver never guesses it from a name, and the SERVES arm asks for
//!   exactly what the row declared.
//!
//! Both default to `Undeclared`, which changes nothing: an undeclared ceremony
//! is still a HOLE (`Absent`), because **`Na` must be EARNED by a declaration**.
//!
//! # Four verdicts, and conflating them is the whole failure mode
//!
//! [`ProofVerdict`] — shared with [`crate::appliance`] on purpose, so the suite
//! has ONE verdict vocabulary rather than two that drift:
//!
//! * [`Green`](ProofVerdict::Green)  — it RAN and it PASSED.
//! * [`Red`](ProofVerdict::Red)      — it RAN and it FAILED.
//! * [`Absent`](ProofVerdict::Absent)— it NEVER RAN. **No evidence.** Never
//!   silent green. A stage whose prerequisite failed is `Absent`, *not* `Red`:
//!   it was never given the chance to fail.
//! * [`Na`](ProofVerdict::Na)        — this KIND has no such stage. A library
//!   was never going to boot off a disk, and a green that would be impossible
//!   must not read as a hole someone should go fill.
//!
//! # Two honesty rules this module encodes rather than documents
//!
//! 1. **Install and boot-installed are NEVER the same VM lifetime.** The
//!    installer arm is required to power-cycle between them; a single-lifetime
//!    "install and it came up" claims stage 2 and stage 7 with one observation.
//!    [`ChainCtx::vm_lifetimes`] is the counter an installer arm must bump, and
//!    [`Stage::Ready`] is refused unless it reads ≥ 2.
//! 2. **`iso-kvm` and `iso-metal` are DISTINCT rows.** A faked machine's green
//!    must never sit where a real-hardware green would. See [`Form`].
//!
//! # The emit door is the one the suite already has
//!
//! Every cell becomes an ordinary [`TestResultRow`] on the `functional` aspect,
//! under [`crate::appliance`]'s naming convention extended with the build
//! thing's kind and name:
//!
//! ```text
//!   suite     = "<product>-appliance/<form>#<kind>:<variant>"
//!   test_name = the stage label ("built", "instantiated", …)
//!   status    = pass → Green · fail → Red · skip → Absent-with-a-REASON
//! ```
//!
//! [`crate::appliance_product`] still reads the product out of that, so the
//! same rows fold BOTH ways: coarsely into the (product × proof) appliance
//! rollup, and finely into this (build thing × stage) chain grid. One writer,
//! two views — never two writers.
//!
//! # What is NOT here, deliberately
//!
//! The doing. This module owns the ORDER, the SHORT-CIRCUIT and the VERDICTS;
//! [`ChainArms`] is the seam where a caller plugs in the machinery that
//! actually builds (nornir-build-thing, in-process) and instantiates
//! (draupnir's `BootSpec`, one spec for KVM / OCI / Redfish). Every arm defaults
//! to a **loud skip that names itself**, so an unwired stage reports
//! `absent: no INSTANTIATED arm wired for gunnar-appliance-iso` — never green,
//! and never anonymous.

use std::cmp::Ordering;
use std::collections::BTreeMap;
use std::hash::{Hash, Hasher};
use std::path::{Path, PathBuf};

use crate::appliance::{ProofVerdict, APPLIANCE_PRODUCTS};
use crate::model::{status, TestResultRow};

// ─────────────────────────── the seven stages ───────────────────────────────

/// The chain, in order. The ordinal IS the prerequisite: stage `n` is only
/// reachable when every applicable stage `< n` went [`ProofVerdict::Green`].
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub enum Stage {
    /// 1 — the Build Thing produced the artifact.
    Built,
    /// 2 — it is RUNNING in its own kind's shape.
    Instantiated,
    /// 3 — **CEREMONY stage.** Hub key in, appliance key minted-or-loaded, own
    /// public key printed AND on disk. [`Ceremony::None`] rows never had this
    /// stage: it is [`ProofVerdict::Na`] for them, and it does not block
    /// [`Stage::Serves`].
    Identity,
    /// 4 — **CEREMONY stage.** Handshake AND traffic that CROSSED. A handshake
    /// alone is a LIE: it was measured on this box that a sabotaged
    /// `AllowedIPs` router completes the handshake while losing 100% of packets.
    Tunnel,
    /// 5 — **IT ANSWERED.** What "a correct answer" means is the row's declared
    /// [`Answer`], not a hardcoded git ceremony: [`Answer::GitForge`] is push +
    /// clone + `fsck --full --strict`, [`Answer::Rpc`] is a correct reply,
    /// [`Answer::Stdout`] is correct output. This is the stage that says the
    /// thing WORKS, and it is reachable with or without stages 3–4.
    Serves,
    /// 6 — reboot / container replace / image upgrade → same public key, repo
    /// still there.
    Durable,
    /// 7 — TERMINAL: the installed server came back up and answered.
    Ready,
}

impl Stage {
    /// All seven, in chain order. Index in this array is the prerequisite depth.
    pub const ALL: [Stage; 7] = [
        Stage::Built,
        Stage::Instantiated,
        Stage::Identity,
        Stage::Tunnel,
        Stage::Serves,
        Stage::Durable,
        Stage::Ready,
    ];

    /// The stable string a row's `test_name` carries. Never change these: they
    /// are the join key between an emitted row and its cell.
    pub fn label(self) -> &'static str {
        match self {
            Stage::Built => "built",
            Stage::Instantiated => "instantiated",
            Stage::Identity => "identity",
            Stage::Tunnel => "tunnel",
            Stage::Serves => "serves",
            Stage::Durable => "durable",
            Stage::Ready => "ready",
        }
    }

    /// Parse a stage label back (the row → cell direction).
    pub fn parse(label: &str) -> Option<Stage> {
        Stage::ALL.into_iter().find(|s| s.label() == label)
    }

    /// 1-based column number, as printed in the grid header.
    pub fn column(self) -> usize {
        Stage::ALL.iter().position(|s| *s == self).unwrap_or(0) + 1
    }
}

// ─────────────────────────── the kinds ──────────────────────────────────────

/// What a declared build thing IS — the `kind` a repo's build-variant
/// declaration carries. This is the only thing that decides which stages are
/// [`ProofVerdict::Na`], so it is parsed from the declaration and never guessed
/// from a name.
///
/// ⚠ MEASURED LIMITATION, 2026-08-15: `kind` alone under-determines stages 3–6.
/// `gunnar-server` is declared `kind = "exe"` and yet it is THE server; a plain
/// client exe is also `kind = "exe"`. The table below is therefore conservative
/// for `Exe` — it marks the serving stages `Na` rather than inventing an
/// `Absent` hole nobody can fill — and the real fix is a capability flag on the
/// declaration itself (see this crate's README / the chain report). When
/// nornir-build-thing grows that flag, [`Kind::has_stage`] is the ONE function
/// that changes.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub enum Kind {
    /// A plain executable — runs, exits.
    Exe,
    /// A long-lived server under an init system (Skidbladnir writes the unit).
    Service,
    /// An OCI image — `podman run` with the volume mounted.
    Container,
    /// A disk image booted as a virtual machine.
    Vm,
    /// A bootable medium — KVM boots OFF it.
    Iso,
    /// An installer medium — installs onto a disk, then the DISK is booted in a
    /// second VM lifetime.
    Installer,
    /// A library — built, never instantiated.
    Lib,
    /// A rendered document.
    Text,
    /// A WebAssembly module — built, loaded in a runtime, no identity.
    Wasm,
    /// A kind this crate does not know. Only [`Stage::Built`] is knowable; every
    /// other stage is `Na` because we cannot honestly say what it would mean.
    Other(String),
}

impl Kind {
    /// Parse the declared `kind` string. Unknown strings are preserved in
    /// [`Kind::Other`] rather than collapsed to a default — an unrecognised kind
    /// must stay visible in the grid, spelled the way the repo spelled it.
    pub fn parse(s: &str) -> Kind {
        match s.trim().to_ascii_lowercase().as_str() {
            "exe" | "bin" | "binary" => Kind::Exe,
            "service" | "daemon" => Kind::Service,
            "container" | "image" | "oci" => Kind::Container,
            "vm" => Kind::Vm,
            "iso" => Kind::Iso,
            "installer" => Kind::Installer,
            "lib" | "library" | "rlib" | "cdylib" => Kind::Lib,
            "text" | "doc" | "docs" | "pdf" => Kind::Text,
            "wasm" => Kind::Wasm,
            other => Kind::Other(other.to_string()),
        }
    }

    /// The label a row carries and the grid prints.
    pub fn label(&self) -> &str {
        match self {
            Kind::Exe => "exe",
            Kind::Service => "service",
            Kind::Container => "container",
            Kind::Vm => "vm",
            Kind::Iso => "iso",
            Kind::Installer => "installer",
            Kind::Lib => "lib",
            Kind::Text => "text",
            Kind::Wasm => "wasm",
            Kind::Other(s) => s,
        }
    }

    /// **The n/a table.** `false` ⇒ the cell is [`ProofVerdict::Na`]: this kind
    /// has no such stage, so no green was ever possible there and no hole is
    /// being hidden.
    ///
    /// [`Stage::Ready`] is `service`-only by decree: READY means *the installed
    /// server came back up and answered*, and a thing that is not a server has
    /// nothing to come back up.
    pub fn has_stage(&self, stage: Stage) -> bool {
        use Stage::*;
        match self {
            // Built + run. The serving ladder is Na, not Absent — see the
            // MEASURED LIMITATION note on `Kind`.
            Kind::Exe => matches!(stage, Built | Instantiated),
            // The only kind that reaches the terminal stage.
            Kind::Service => true,
            // Instantiated in their own shape and can hold an identity and
            // serve — but READY (came back up after being INSTALLED) is not
            // their shape.
            Kind::Container | Kind::Vm | Kind::Iso => !matches!(stage, Ready),
            // An installer's stage 2 already spans install → detach →
            // power-cycle → boot-off-disk; READY is what that second lifetime
            // answers with, so it is the one non-service kind that keeps it.
            Kind::Installer => true,
            Kind::Lib | Kind::Text => matches!(stage, Built),
            Kind::Wasm => matches!(stage, Built | Instantiated),
            Kind::Other(_) => matches!(stage, Built),
        }
    }

    /// How many of the seven stages this kind can ever reach — the denominator
    /// the grid divides by, so a repo of libraries is not reported as 6/7 short.
    pub fn applicable_stages(&self) -> usize {
        Stage::ALL.iter().filter(|s| self.has_stage(**s)).count()
    }
}

/// The FORM a thing was proven in — the second honesty rule, made a key.
///
/// `iso-kvm` and `iso-metal` are DISTINCT rows: a green earned inside a faked
/// machine must never sit in the cell a real-hardware green would occupy.
/// `iso-metal` stays `Absent` until a real Redfish burn happens, and it stays
/// visibly, on its own row.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub struct Form(String);

impl Form {
    /// The ordinary form: whatever the kind's default shape is.
    pub fn native() -> Form {
        Form("native".into())
    }
    /// Instantiated inside KVM — a faked machine.
    pub fn kvm() -> Form {
        Form("kvm".into())
    }
    /// Instantiated on real hardware via Redfish.
    pub fn metal() -> Form {
        Form("metal".into())
    }
    /// Instantiated as an OCI container.
    pub fn container() -> Form {
        Form("container".into())
    }
    pub fn new(s: impl Into<String>) -> Form {
        Form(s.into())
    }
    pub fn as_str(&self) -> &str {
        &self.0
    }
}

// ───────────────── the two DECLARED axes: ceremony and answer ───────────────

/// **Does this row run the WireGuard appliance identity ceremony?**
///
/// The axis that stages 3–4 hang on, and it is DECLARED by the caller — never
/// inferred from a kind or a name, because `gunnar-server` and `korp-server` are
/// the same [`Kind::Service`] and only one of them has a ceremony.
///
/// The three states are three DIFFERENT facts and collapsing them is the whole
/// failure mode:
///
/// * [`Undeclared`](Ceremony::Undeclared) — nobody said. Stages 3–4 stay
///   applicable and read `Absent`: a hole someone can go fill. **This is the
///   default**, so `Na` can only ever be EARNED.
/// * [`WireGuard`](Ceremony::WireGuard) — this row DOES run it (gunnar, holger).
///   Stages 3–4 are real work and stay `Absent` until proven.
/// * [`None`](Ceremony::None) — this row DELIBERATELY has no ceremony
///   (korp: the 2026-08-03 `suite=ORIGINAL` ruling makes `korp-appliance` not a
///   variant). Stages 3–4 are [`ProofVerdict::Na`] — not a hole, and not a pass.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub enum Ceremony {
    /// Nobody declared either way → the ceremony stages stay `Absent` holes.
    #[default]
    Undeclared,
    /// Declared PRESENT: the wg-appliance-core mint/handshake ceremony applies.
    WireGuard,
    /// Declared ABSENT: this row has no identity ceremony, so stages 3–4 are
    /// `Na` and SERVES is reachable straight out of INSTANTIATED.
    None,
}

impl Ceremony {
    /// Whether stages 3–4 exist for this row at all.
    pub fn has_ceremony_stages(self) -> bool {
        !matches!(self, Ceremony::None)
    }
    /// The word a report prints.
    pub fn label(self) -> &'static str {
        match self {
            Ceremony::Undeclared => "undeclared",
            Ceremony::WireGuard => "wireguard",
            Ceremony::None => "none",
        }
    }
}

/// **What "a correct answer" IS for this row** — [`Stage::Serves`]'s parameter,
/// supplied by the caller.
///
/// SERVES used to mean one thing, git push + clone, which is right for a git
/// forge and meaningless for a map server. It is now a declaration the row
/// carries and the SERVES arm reads: the driver still owns the ORDER and the
/// VERDICT, and the caller owns what question gets asked.
///
/// The payloads are plain strings on purpose — this crate stays pure `std` +
/// serde, so it never names a probe type from `skidbladnir`, `draupnir` or a
/// gRPC stack. The arm that can honestly ask the question does the mapping, and
/// NAMES the gap when it cannot.
#[derive(Debug, Clone, Default, PartialEq, Eq, PartialOrd, Ord, Hash)]
pub enum Answer {
    /// Nobody said what a correct answer is. SERVES stays applicable wherever
    /// the KIND has it, and reads `Absent` — never `Na`, never green.
    #[default]
    Undeclared,
    /// A git forge: push + clone, object graphs compared, `fsck --full --strict`.
    GitForge,
    /// A protocol banner: connect to `addr`, the first bytes must contain
    /// `expect` (`SSH-2.0-gunnar`).
    Banner {
        /// `host:port` to connect to.
        addr: String,
        /// The substring the banner must contain.
        expect: String,
    },
    /// A correct RPC reply: `call` against `endpoint`, and `expect` is the shape
    /// the reply must have (`{"cases":0,…,"error":null,"seq":0}`).
    Rpc {
        /// `host:port` of the service.
        endpoint: String,
        /// The call that was made, named.
        call: String,
        /// The shape a correct reply has.
        expect: String,
    },
    /// Correct stdout from a real invocation — a CLI's whole contract.
    Stdout {
        /// The arguments the invocation used.
        args: String,
        /// The substring correct output must contain.
        expect: String,
    },
    /// A correct HTTP response from `url` containing `expect`.
    Http {
        /// The URL asked.
        url: String,
        /// The substring a correct body must contain.
        expect: String,
    },
    /// Something else this enum does not model, spelled out by the caller.
    Other(String),
    /// DECLARED: this thing answers NOTHING — there is no question to ask, so
    /// [`Stage::Serves`] is [`ProofVerdict::Na`].
    None,
}

impl Answer {
    /// Whether the caller said anything at all. An `Undeclared` answer is not a
    /// declaration, and neither is [`Answer::None`] a *question*.
    pub fn is_declared(&self) -> bool {
        !matches!(self, Answer::Undeclared | Answer::None)
    }

    /// The one line a SERVES cell (or a skip) carries: what would have to be
    /// true for this row to have answered.
    pub fn describe(&self) -> String {
        match self {
            Answer::Undeclared => {
                "no Answer declared — nobody said what a correct answer IS for this row"
                    .to_string()
            }
            Answer::GitForge => {
                "git push + clone, object graphs compared, fsck --full --strict".to_string()
            }
            Answer::Banner { addr, expect } => {
                format!("a banner on {addr} containing {expect:?}")
            }
            Answer::Rpc { endpoint, call, expect } => {
                format!("{call} on {endpoint} replying {expect}")
            }
            Answer::Stdout { args, expect } => {
                format!("stdout of `{args}` containing {expect:?}")
            }
            Answer::Http { url, expect } => format!("HTTP {url} returning {expect:?}"),
            Answer::Other(s) => s.clone(),
            Answer::None => "declared to answer NOTHING".to_string(),
        }
    }

    /// A banner answer from an address and the expected greeting.
    pub fn banner(addr: impl Into<String>, expect: impl Into<String>) -> Answer {
        Answer::Banner { addr: addr.into(), expect: expect.into() }
    }
    /// An RPC answer: the call, where it goes, and the shape of a correct reply.
    pub fn rpc(
        endpoint: impl Into<String>,
        call: impl Into<String>,
        expect: impl Into<String>,
    ) -> Answer {
        Answer::Rpc { endpoint: endpoint.into(), call: call.into(), expect: expect.into() }
    }
}

// ─────────────────────────── the row key ────────────────────────────────────

/// One ROW of the chain matrix: a declared build thing, in one form.
///
/// # The KEY is `(product, variant, kind, form)` — and ONLY that
///
/// `form` is part of the key and not a detail string precisely so
/// `gunnar-appliance-iso` in `kvm` and the same ISO on `metal` are two rows that
/// can disagree.
///
/// [`ceremony`](Self::ceremony), [`answer`](Self::answer) and
/// [`marker`](Self::marker) are **declarations ABOUT the row, not a second
/// row**, so `Eq`/`Ord`/`Hash` are hand-written over [`BuildThing::key`] and
/// exclude them. This is load-bearing: [`BuildThing::component`] cannot encode
/// them (it is the on-the-wire join key an emitted [`TestResultRow`] carries),
/// so a row parsed back by [`parse_chain_component`] has DEFAULT declarations —
/// and it must still land in the same grid cell as the roster entry that
/// declares them. Two entries differing only in declaration are one row with a
/// declaration conflict, never two pieces of evidence.
#[derive(Debug, Clone)]
pub struct BuildThing {
    /// The repo/product that declares it (`gunnar`, `holger`, `tunnr`, …).
    pub product: String,
    /// The declared variant name (`gunnar-server`, `holger-appliance-iso`, …).
    pub variant: String,
    /// The declared kind — the KIND half of the n/a table.
    pub kind: Kind,
    /// Which shape it was proven in. [`Form::native`] unless the arm says
    /// otherwise.
    pub form: Form,
    /// **Not part of the key.** Whether stages 3–4 exist for this row.
    pub ceremony: Ceremony,
    /// **Not part of the key.** What a correct answer IS at stage 5.
    pub answer: Answer,
    /// **Not part of the key.** The NAMED string this thing prints when it is
    /// up — stage 2's readiness evidence, and the one thing that separates "the
    /// process forked" from "the process is ready". `None` means the caller
    /// declared no marker, which an arm must report as a gap, never as ready.
    pub marker: Option<String>,
    /// **Not part of the key.** How the artifact is INVOKED at stage 2
    /// (`["serve", "--ssh-bind", "127.0.0.1:22222"]`). A serving binary almost
    /// never serves when run bare, and a stage-2 arm that guesses the argv is
    /// guessing what it is testing. Empty means "no arguments declared".
    ///
    /// MEASURED GAP 2026-08-15: `nornir-build-thing`'s declaration does not
    /// carry an invocation, so today this comes from the caller's own table.
    pub args: Vec<String>,
}

impl BuildThing {
    /// A build thing in its native form, with NOTHING declared: the ceremony and
    /// the answer are both [`Undeclared`](Ceremony::Undeclared), so every stage
    /// its KIND has stays a visible `Absent` hole until a caller says otherwise.
    pub fn new(product: impl Into<String>, variant: impl Into<String>, kind: Kind) -> Self {
        Self {
            product: product.into(),
            variant: variant.into(),
            kind,
            form: Form::native(),
            ceremony: Ceremony::default(),
            answer: Answer::default(),
            marker: None,
            args: Vec::new(),
        }
    }

    /// The same thing, keyed to a specific form (`kvm` vs `metal` vs …).
    pub fn in_form(mut self, form: Form) -> Self {
        self.form = form;
        self
    }

    /// **The identity of the row.** Everything not in here is a declaration
    /// about the row rather than a different row — see the type docs.
    pub fn key(&self) -> (&str, &str, &Kind, &Form) {
        (&self.product, &self.variant, &self.kind, &self.form)
    }

    /// Declare whether this row runs the WireGuard appliance ceremony.
    pub fn with_ceremony(mut self, ceremony: Ceremony) -> Self {
        self.ceremony = ceremony;
        self
    }

    /// Declare what a CORRECT ANSWER is for this row — [`Stage::Serves`]'s
    /// parameter.
    pub fn answering(mut self, answer: Answer) -> Self {
        self.answer = answer;
        self
    }

    /// Declare the readiness marker this thing prints when it is up.
    pub fn with_marker(mut self, marker: impl Into<String>) -> Self {
        self.marker = Some(marker.into());
        self
    }

    /// Declare the argv stage 2 invokes the artifact with.
    pub fn invoked_with<S: Into<String>, I: IntoIterator<Item = S>>(mut self, args: I) -> Self {
        self.args = args.into_iter().map(Into::into).collect();
        self
    }

    /// **The n/a rule, COMPLETE.** [`Kind::has_stage`] is the kind's half; this
    /// adds the two DECLARED halves, and it is the only function the driver and
    /// the grid ask.
    ///
    /// * stages 3–4 need [`Ceremony::has_ceremony_stages`]. A `Ceremony::None`
    ///   row has no ceremony, so those cells are `Na`, are stepped over, and
    ///   never block stage 5.
    /// * stage 5 is refused outright by [`Answer::None`] (nothing to ask), and
    ///   is PROMOTED to applicable by any declared [`Answer`] even for a kind
    ///   whose conservative table says no — a serving `exe` becomes a visible
    ///   `Absent` hole instead of a `Na` nobody can fill.
    ///
    /// Note what is NOT here: no declaration can turn a cell GREEN, and no arm
    /// can reach this function. Only the KIND and the CALLER'S DECLARATION
    /// decide `Na`.
    pub fn has_stage(&self, stage: Stage) -> bool {
        match stage {
            Stage::Identity | Stage::Tunnel => {
                self.ceremony.has_ceremony_stages() && self.kind.has_stage(stage)
            }
            Stage::Serves => {
                self.answer != Answer::None
                    && (self.kind.has_stage(stage) || self.answer.is_declared())
            }
            _ => self.kind.has_stage(stage),
        }
    }

    /// How many of the seven stages this ROW can ever reach — kind and
    /// declarations together. The honest denominator for this row.
    pub fn applicable_stages(&self) -> usize {
        Stage::ALL.iter().filter(|s| self.has_stage(**s)).count()
    }

    /// The `suite` string a row carries — [`crate::appliance`]'s convention,
    /// extended with kind + variant so the same row folds both coarsely
    /// (product × proof) and finely (build thing × stage).
    ///
    /// `"<product>-appliance/<form>#<kind>:<variant>"`
    pub fn component(&self) -> String {
        format!(
            "{}-appliance/{}#{}:{}",
            self.product,
            self.form.as_str(),
            self.kind.label(),
            self.variant
        )
    }

    /// Short human id for reports: `"gunnar/gunnar-server"` (native) or
    /// `"gunnar/gunnar-appliance-iso@metal"`.
    pub fn id(&self) -> String {
        if self.form == Form::native() {
            format!("{}/{}", self.product, self.variant)
        } else {
            format!("{}/{}@{}", self.product, self.variant, self.form.as_str())
        }
    }
}

// The four trait impls that make `key()` the identity. Hand-written, never
// derived, because a derive would silently fold `ceremony`/`answer`/`marker`
// into the key the moment someone adds a field — and then a roster row and the
// SAME row parsed back out of its own emitted evidence would be two different
// grid cells, which is a rendering lie (see the `BuildThing` docs).
impl PartialEq for BuildThing {
    fn eq(&self, other: &Self) -> bool {
        self.key() == other.key()
    }
}
impl Eq for BuildThing {}
impl Ord for BuildThing {
    fn cmp(&self, other: &Self) -> Ordering {
        self.key().cmp(&other.key())
    }
}
impl PartialOrd for BuildThing {
    fn partial_cmp(&self, other: &Self) -> Option<Ordering> {
        Some(self.cmp(other))
    }
}
impl Hash for BuildThing {
    fn hash<H: Hasher>(&self, state: &mut H) {
        self.key().hash(state);
    }
}

/// Parse a [`BuildThing::component`] string back into its key. Returns `None`
/// for any component that does not carry the full chain encoding — a plain
/// `holger-appliance/container` row from the older appliance emitters is NOT a
/// chain row and must not be invented into one.
pub fn parse_chain_component(component: &str) -> Option<BuildThing> {
    let (product_part, rest) = component.split_once('/')?;
    let product = product_part.strip_suffix("-appliance")?;
    if product.is_empty() {
        return None;
    }
    let (form, rest) = rest.split_once('#')?;
    let (kind, variant) = rest.split_once(':')?;
    if form.is_empty() || kind.is_empty() || variant.is_empty() {
        return None;
    }
    // DEFAULT declarations on purpose: the component string is the KEY, and the
    // key does not carry them. The grid re-derives `Na` from the ROSTER's row
    // (which does carry them) — see `ChainGrid::verdict`.
    Some(
        BuildThing::new(product, variant, Kind::parse(kind))
            .in_form(Form::new(form)),
    )
}

// ─────────────────────────── outcomes + cells ───────────────────────────────

/// What an arm reports back. Note there is no "it did not apply" variant: only
/// the KIND decides `Na`, never an arm — an arm that could decide its own stage
/// away is an arm that can hide.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum StageOutcome {
    /// It ran and it passed. The string is the APPLIED OUTPUT that proves it —
    /// the serial line, the emitted row, the object-graph hash. Not "ok".
    Pass(String),
    /// It ran and it failed. The string names what failed.
    Fail(String),
    /// It did NOT run, and here is exactly why — the loud skip. Folds to
    /// [`ProofVerdict::Absent`], never to green.
    Skipped(String),
}

impl StageOutcome {
    /// Build an outcome from a bool plus detail — for arms whose underlying
    /// check is already a predicate.
    pub fn from_ok(ok: bool, detail: impl Into<String>) -> StageOutcome {
        if ok {
            StageOutcome::Pass(detail.into())
        } else {
            StageOutcome::Fail(detail.into())
        }
    }

    fn split(self) -> (ProofVerdict, String) {
        match self {
            StageOutcome::Pass(d) => (ProofVerdict::Green, d),
            StageOutcome::Fail(d) => (ProofVerdict::Red, d),
            StageOutcome::Skipped(d) => (ProofVerdict::Absent, d),
        }
    }
}

/// One CELL: a verdict plus the detail that earned it.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Cell {
    pub verdict: ProofVerdict,
    /// Applied output on green, the failure on red, the NAMED reason on absent,
    /// the kind on n/a. Never empty for absent — that is the point of the loud
    /// skip.
    pub detail: String,
}

impl Cell {
    /// The `Na` cell for a stage this ROW does not have, with the rule that
    /// decided it NAMED. An `Na` whose reason is anonymous is indistinguishable
    /// from a shrug, and a shrug is how a skipped stage becomes a pass.
    fn na(thing: &BuildThing, stage: Stage) -> Cell {
        let detail = match stage {
            Stage::Identity | Stage::Tunnel if !thing.ceremony.has_ceremony_stages() => {
                format!(
                    "ceremony=none: this row DECLARES no WireGuard identity ceremony, so {} \
                     never existed for it — not a hole, and not a pass. SERVES is reached \
                     straight out of INSTANTIATED.",
                    stage.label().to_uppercase()
                )
            }
            Stage::Serves if thing.answer == Answer::None => {
                "answer=none: this row DECLARES that it answers nothing, so there is no \
                 correct answer to ask for"
                    .to_string()
            }
            _ => format!("kind {} has no such stage", thing.kind.label()),
        };
        Cell { verdict: ProofVerdict::Na, detail }
    }
}

// ─────────────────────────── the walk context ───────────────────────────────

/// State carried DOWN the chain: what stage 1 produced is what stage 2 boots,
/// what stage 3 minted is what stage 6 must find unchanged.
#[derive(Debug, Clone)]
pub struct ChainCtx {
    /// The row being walked.
    pub thing: BuildThing,
    /// The repo the declaration lives in.
    pub repo_root: PathBuf,
    /// Artifacts stage 1 produced, as the Build Thing reported them.
    pub artifacts: Vec<PathBuf>,
    /// An opaque handle id stage 2 produced (a draupnir instance, a unit name, a
    /// container id) — whatever later stages need to talk to it.
    pub instance: Option<String>,
    /// Where later stages reach it (`127.0.0.1:2222`, a tunnel IP, …).
    pub endpoint: Option<String>,
    /// The appliance public key stage 3 observed — the ONE canonical identity
    /// line wg-appliance-core emits, never a driver-private regex. Stage 6
    /// compares against exactly this.
    pub pubkey: Option<String>,
    /// **How many DISTINCT VM lifetimes this row has used.** The installer
    /// honesty rule: install happens in lifetime 1, the ISO is detached, the
    /// machine is power-cycled, and the disk boots in lifetime 2. [`Stage::Ready`]
    /// is refused while this reads < 2 for an [`Kind::Installer`] row, so one
    /// observation can never claim both ends.
    pub vm_lifetimes: u32,
    /// Free-form notes an arm wants later stages (or the report) to see.
    pub notes: BTreeMap<String, String>,
}

impl ChainCtx {
    pub fn new(thing: BuildThing, repo_root: impl Into<PathBuf>) -> Self {
        Self {
            thing,
            repo_root: repo_root.into(),
            artifacts: Vec::new(),
            instance: None,
            endpoint: None,
            pubkey: None,
            vm_lifetimes: 0,
            notes: BTreeMap::new(),
        }
    }
}

// ─────────────────────────── the arms seam ──────────────────────────────────

/// The machinery that actually DOES each stage. The driver owns the order and
/// the short-circuit; an implementor owns the doing, and only for the stages it
/// can honestly do.
///
/// **Every method defaults to a loud skip that names itself and the row.** An
/// unwired stage is therefore `absent: no TUNNEL arm wired for
/// gunnar/gunnar-appliance-iso` — visible, attributable, and never green. There
/// is no default that returns `Pass`, and there never may be.
pub trait ChainArms {
    /// 1 — build THROUGH the declared Build Thing, in-process. Push the produced
    /// paths into [`ChainCtx::artifacts`].
    fn built(&mut self, ctx: &mut ChainCtx) -> StageOutcome {
        skip(Stage::Built, ctx)
    }
    /// 2 — instantiate in the kind's own shape, THROUGH draupnir's `BootSpec`
    /// (KVM / OCI / Redfish are three backends behind one spec). Never a
    /// hand-rolled qemu command line or podman argv.
    ///
    /// # ⚠ A SERVING APPLIANCE NEVER EXITS — that is the NORMAL case
    ///
    /// MEASURED 2026-08-15: `wg-appliance-core`'s container rung used
    /// `Command::output()`, which WAITS FOR EXIT. gunnar's declared cell is
    /// `cmd = ["serve"]` — it mints its identity and then serves forever — so
    /// the rung hung for 927 s with `MINTED new keypair` already printed and
    /// both listeners bound. holger's identical rung "passed" only because ITS
    /// command *fails and exits* on a fresh volume: a green that depended on the
    /// appliance being BROKEN.
    ///
    /// So no stage may be designed around a process terminating. An arm
    /// instantiates, watches the stream for the APPLIED OUTPUT it needs (for
    /// stage 3 that is the one canonical identity line
    /// `wg_appliance_core::parse_identity_line` reads), and leaves the thing
    /// RUNNING for the stages above it — recording the handle in
    /// [`ChainCtx::instance`]. A backstop timeout must return WHAT IT SAW, never
    /// a verdict: the stage's own checks then render red against the transcript
    /// rather than the helper inventing an answer.
    fn instantiated(&mut self, ctx: &mut ChainCtx) -> StageOutcome {
        skip(Stage::Instantiated, ctx)
    }
    /// 3 — CEREMONY. Watch for the ONE canonical identity line
    /// `wg-appliance-core` emits (`wg_appliance_core::IDENTITY_LINE_REGEX`), and
    /// confirm the key is on disk too. Store it in [`ChainCtx::pubkey`].
    ///
    /// Never called for a [`Ceremony::None`] row — that cell is `Na` and the
    /// driver steps over it.
    fn identity(&mut self, ctx: &mut ChainCtx) -> StageOutcome {
        skip(Stage::Identity, ctx)
    }
    /// 4 — CEREMONY. Handshake AND traffic that CROSSED. A handshake alone is a
    /// LIE. Never called for a [`Ceremony::None`] row.
    fn tunnel(&mut self, ctx: &mut ChainCtx) -> StageOutcome {
        skip(Stage::Tunnel, ctx)
    }
    /// 5 — **IT ANSWERED.** Ask the question `ctx.thing.answer` declares and
    /// grade the reply against it. This arm must NOT hardcode a git ceremony:
    /// [`Answer::GitForge`] is one arm of a `match` on the row's declaration,
    /// alongside [`Answer::Rpc`], [`Answer::Banner`], [`Answer::Stdout`] and
    /// [`Answer::Http`].
    ///
    /// An [`Answer::Undeclared`] row reaches this arm too, and the honest thing
    /// to return there is a loud [`StageOutcome::Skipped`] naming the missing
    /// declaration — the driver deliberately does not fabricate one, because a
    /// question nobody wrote down cannot have been answered.
    fn serves(&mut self, ctx: &mut ChainCtx) -> StageOutcome {
        skip(Stage::Serves, ctx)
    }
    /// 6 — reboot / replace / upgrade and ask again: for a ceremony row the SAME
    /// [`ChainCtx::pubkey`] with the repo intact; for a no-ceremony row the same
    /// declared [`Answer`], still correct, after the restart.
    fn durable(&mut self, ctx: &mut ChainCtx) -> StageOutcome {
        skip(Stage::Durable, ctx)
    }
    /// 7 — TERMINAL. The installed server came back up and answered.
    fn ready(&mut self, ctx: &mut ChainCtx) -> StageOutcome {
        skip(Stage::Ready, ctx)
    }

    /// Dispatch. Overriding this instead of the per-stage methods is allowed but
    /// unusual; the driver only ever calls THIS.
    fn stage(&mut self, stage: Stage, ctx: &mut ChainCtx) -> StageOutcome {
        match stage {
            Stage::Built => self.built(ctx),
            Stage::Instantiated => self.instantiated(ctx),
            Stage::Identity => self.identity(ctx),
            Stage::Tunnel => self.tunnel(ctx),
            Stage::Serves => self.serves(ctx),
            Stage::Durable => self.durable(ctx),
            Stage::Ready => self.ready(ctx),
        }
    }
}

/// The loud skip every unwired arm reports: names the STAGE and the ROW.
fn skip(stage: Stage, ctx: &ChainCtx) -> StageOutcome {
    StageOutcome::Skipped(format!(
        "no {} arm wired for {}",
        stage.label().to_uppercase(),
        ctx.thing.id()
    ))
}

/// Arms that wire NOTHING — every stage a loud, named skip. The honest baseline
/// a fresh consumer starts from, and the fixture the absent-not-green guard
/// uses.
#[derive(Debug, Default, Clone, Copy)]
pub struct UnwiredArms;

impl ChainArms for UnwiredArms {}

// ─────────────────────────── the driver ─────────────────────────────────────

/// One row's walk: the seven cells, plus the context the arms left behind.
#[derive(Debug, Clone)]
pub struct ChainRun {
    pub thing: BuildThing,
    pub cells: Vec<Cell>,
    pub ctx: ChainCtx,
}

impl ChainRun {
    pub fn cell(&self, stage: Stage) -> &Cell {
        &self.cells[stage.column() - 1]
    }

    /// How far up the ladder this row actually got — the last stage that is
    /// Green, counting Na as "passed through".
    pub fn reached(&self) -> Option<Stage> {
        Stage::ALL
            .into_iter()
            .filter(|s| self.cell(*s).verdict == ProofVerdict::Green)
            .next_back()
    }

    /// Every cell as a matrix row, ready for [`crate::functional::record_rows`]
    /// or any [`crate::sink::TestSink`]. `Na` cells emit nothing: a stage that
    /// cannot exist is not evidence of anything, and a warehouse full of `n/a`
    /// rows is noise. The grid re-derives `Na` from the roster, so nothing is
    /// lost.
    pub fn rows(&self) -> Vec<TestResultRow> {
        Stage::ALL
            .into_iter()
            .filter(|s| self.cell(*s).verdict != ProofVerdict::Na)
            .map(|s| chain_row(&self.thing, s, self.cell(s)))
            .collect()
    }
}

/// Build the one canonical row for a chain cell. Shaped exactly like
/// [`crate::functional::functional_row`] but able to carry the third status —
/// `skip` — because [`ProofVerdict::Absent`] is a verdict `bool` cannot express.
pub fn chain_row(thing: &BuildThing, stage: Stage, cell: &Cell) -> TestResultRow {
    let st = match cell.verdict {
        ProofVerdict::Green => status::PASS,
        ProofVerdict::Red => status::FAIL,
        // Absent is NOT red (it never ran) and NOT green (no evidence). `skip`
        // is the existing neutral tag, and the reason travels in `message`.
        ProofVerdict::Absent | ProofVerdict::Na => status::SKIP,
    };
    TestResultRow {
        run_id: String::new(),
        repo: thing.product.clone(),
        suite: thing.component(),
        test_name: stage.label().to_string(),
        status: st.to_string(),
        duration_ms: 0.0,
        ts_micros: crate::harness::now_micros(),
        message: cell.detail.clone(),
        aspect: crate::functional::ASPECT_FUNCTIONAL.to_string(),
        metric: stage.column() as f64,
    }
}

/// **Walk one build thing up the chain.**
///
/// The three rules, all of them here and nowhere else:
///
/// 1. A stage the KIND does not have is [`ProofVerdict::Na`] and is stepped
///    over — it neither runs nor blocks.
/// 2. A stage whose prerequisite did not go green is [`ProofVerdict::Absent`]
///    with the BLOCKER NAMED, and **the arm is never called**. Unreachable is
///    absent, not red.
/// 3. A stage that ran is Green or Red on its own merit.
pub fn run_chain<A: ChainArms + ?Sized>(
    arms: &mut A,
    thing: &BuildThing,
    repo_root: &Path,
) -> ChainRun {
    let mut ctx = ChainCtx::new(thing.clone(), repo_root);
    let mut cells: Vec<Cell> = Vec::with_capacity(Stage::ALL.len());
    // Set once a stage fails or is skipped; every later stage is unreachable.
    let mut blocked: Option<(Stage, ProofVerdict)> = None;

    for stage in Stage::ALL {
        // Rule 1 asks the ROW (kind + declared ceremony + declared answer), not
        // the kind alone — that conflation is what made a no-ceremony server's
        // stages 4-7 permanently unreachable.
        if !thing.has_stage(stage) {
            cells.push(Cell::na(thing, stage));
            continue;
        }
        if let Some((blocker, verdict)) = &blocked {
            cells.push(Cell {
                verdict: ProofVerdict::Absent,
                detail: format!(
                    "unreachable: {} was {} — this stage never ran",
                    blocker.label().to_uppercase(),
                    match verdict {
                        ProofVerdict::Red => "RED",
                        _ => "ABSENT",
                    }
                ),
            });
            continue;
        }
        let (verdict, detail) = arms.stage(stage, &mut ctx).split();
        // Honesty rule 1, enforced rather than documented: an installer's
        // terminal green requires TWO distinct VM lifetimes. Install and
        // boot-installed measured in one lifetime is one observation claiming
        // two cells, so it is downgraded to RED with the count named.
        let (verdict, detail) = if stage == Stage::Ready
            && verdict == ProofVerdict::Green
            && thing.kind == Kind::Installer
            && ctx.vm_lifetimes < 2
        {
            (
                ProofVerdict::Red,
                format!(
                    "REFUSED: install and boot-installed must be DISTINCT VM lifetimes, \
                     ctx.vm_lifetimes={} (need ≥2). Claimed: {detail}",
                    ctx.vm_lifetimes
                ),
            )
        } else {
            (verdict, detail)
        };
        if verdict != ProofVerdict::Green {
            blocked = Some((stage, verdict));
        }
        cells.push(Cell { verdict, detail });
    }
    ChainRun { thing: thing.clone(), cells, ctx }
}

/// Walk a whole roster. `repo_root_of` maps a product to its checkout — the one
/// place the driver learns where anything lives.
pub fn run_chain_all<A: ChainArms + ?Sized>(
    arms: &mut A,
    roster: &[BuildThing],
    repo_root_of: &dyn Fn(&str) -> PathBuf,
) -> Vec<ChainRun> {
    roster.iter().map(|t| run_chain(arms, t, &repo_root_of(&t.product))).collect()
}

// ─────────────────────────── the grid ───────────────────────────────────────

/// The rendered chain matrix: **the roster floors it**, so a build thing that
/// emitted nothing is a row of `Absent`, never a missing row. Rows found in the
/// evidence but not in the roster are appended (floor, not cap).
#[derive(Debug, Clone)]
pub struct ChainGrid {
    /// Every row, roster order then extras.
    pub rows: Vec<BuildThing>,
    cells: BTreeMap<(BuildThing, Stage), Cell>,
}

impl ChainGrid {
    /// **The one constructor.** Roster × stages is the SHAPE; the rows are the
    /// EVIDENCE. Anything the evidence does not cover is `Na` (the kind has no
    /// such stage) or `Absent` (no evidence) — and never green.
    pub fn build<'a, I: IntoIterator<Item = &'a TestResultRow>>(
        roster: &[BuildThing],
        evidence: I,
    ) -> Self {
        let mut cells: BTreeMap<(BuildThing, Stage), Cell> = BTreeMap::new();
        let mut extras: Vec<BuildThing> = Vec::new();
        for row in evidence {
            let Some(thing) = parse_chain_component(&row.suite) else { continue };
            let Some(stage) = Stage::parse(&row.test_name) else { continue };
            let verdict = if status::is_green(&row.status) {
                ProofVerdict::Green
            } else if status::is_red(&row.status) {
                ProofVerdict::Red
            } else {
                ProofVerdict::Absent
            };
            if !roster.contains(&thing) && !extras.contains(&thing) {
                extras.push(thing.clone());
            }
            let key = (thing, stage);
            match cells.get_mut(&key) {
                // RED WINS. Two forms of the same evidence disagreeing is not
                // proven — the same law the appliance rollup already keeps.
                Some(existing) if existing.verdict == ProofVerdict::Red => {}
                Some(existing) => {
                    if verdict == ProofVerdict::Red || existing.verdict == ProofVerdict::Absent {
                        *existing = Cell { verdict, detail: row.message.clone() };
                    }
                }
                None => {
                    cells.insert(key, Cell { verdict, detail: row.message.clone() });
                }
            }
        }
        extras.sort();
        let mut rows: Vec<BuildThing> = roster.to_vec();
        rows.extend(extras);
        Self { rows, cells }
    }

    /// The grid straight from a walk — the roster IS what was walked.
    pub fn from_runs(runs: &[ChainRun]) -> Self {
        let roster: Vec<BuildThing> = runs.iter().map(|r| r.thing.clone()).collect();
        let evidence: Vec<TestResultRow> = runs.iter().flat_map(ChainRun::rows).collect();
        Self::build(&roster, &evidence)
    }

    /// One cell. Absence of evidence for an applicable stage is `Absent`; a
    /// stage the ROW does not have — by its kind, or by its declared
    /// [`Ceremony`]/[`Answer`] — is `Na`. These two are the ones that must never
    /// be confused, so they are decided HERE, from the row, and not from whether
    /// a row of evidence happened to exist.
    ///
    /// ⚠ `thing` must be the ROSTER's row: the declarations do not survive
    /// [`BuildThing::component`], so a row reconstructed from evidence alone
    /// carries defaults. That direction is safe (a cell reverts from `Na` to
    /// `Absent`, never to green), but it is why the grid re-derives from the
    /// roster.
    pub fn verdict(&self, thing: &BuildThing, stage: Stage) -> ProofVerdict {
        if !thing.has_stage(stage) {
            return ProofVerdict::Na;
        }
        self.cells.get(&(thing.clone(), stage)).map(|c| c.verdict).unwrap_or(ProofVerdict::Absent)
    }

    /// The detail behind a cell (`""` when nothing was ever recorded).
    pub fn detail(&self, thing: &BuildThing, stage: Stage) -> &str {
        self.cells.get(&(thing.clone(), stage)).map(|c| c.detail.as_str()).unwrap_or("")
    }

    /// Cells by verdict across the WHOLE grid: `(green, red, absent, na)`.
    /// `green + red + absent + na == rows.len() * 7`, always.
    pub fn tally(&self) -> Tally {
        let mut t = Tally::default();
        for thing in &self.rows {
            for stage in Stage::ALL {
                match self.verdict(thing, stage) {
                    ProofVerdict::Green => t.green += 1,
                    ProofVerdict::Red => t.red += 1,
                    ProofVerdict::Absent => t.absent += 1,
                    ProofVerdict::Na => t.na += 1,
                }
            }
        }
        t
    }

    /// Per-product tallies, in row order — the rollup that keeps a 700-cell grid
    /// readable.
    pub fn by_product(&self) -> Vec<(String, Tally)> {
        let mut order: Vec<String> = Vec::new();
        let mut acc: BTreeMap<String, Tally> = BTreeMap::new();
        for thing in &self.rows {
            if !order.contains(&thing.product) {
                order.push(thing.product.clone());
            }
            let t = acc.entry(thing.product.clone()).or_default();
            for stage in Stage::ALL {
                match self.verdict(thing, stage) {
                    ProofVerdict::Green => t.green += 1,
                    ProofVerdict::Red => t.red += 1,
                    ProofVerdict::Absent => t.absent += 1,
                    ProofVerdict::Na => t.na += 1,
                }
            }
            t.rows += 1;
        }
        order.into_iter().map(|p| (p.clone(), acc.remove(&p).unwrap_or_default())).collect()
    }

    /// How many rows can ever reach [`Stage::Ready`] — the size of the terminal
    /// column, which is the number of SERVERS in the constellation.
    pub fn ready_capable(&self) -> usize {
        self.rows.iter().filter(|t| t.kind.has_stage(Stage::Ready)).count()
    }

    /// The per-product summary table — lead with this, not with 700 cells.
    pub fn render_rollup(&self) -> String {
        let total = self.tally();
        let mut out = String::new();
        out.push_str(&format!(
            "CHAIN MATRIX rollup — {} build things × {} stages = {} cells\n",
            self.rows.len(),
            Stage::ALL.len(),
            self.rows.len() * Stage::ALL.len()
        ));
        out.push_str(&format!(
            "  GREEN {}  RED {}  ABSENT {}  N/A {}   (ready-capable rows: {})\n",
            total.green,
            total.red,
            total.absent,
            total.na,
            self.ready_capable()
        ));
        let width = self.rows.iter().map(|t| t.product.len()).max().unwrap_or(7).max(7);
        out.push_str(&format!("{:width$} {:>5} {:>6} {:>5} {:>7} {:>5}\n", "product", "rows", "green", "red", "absent", "n/a"));
        for (product, t) in self.by_product() {
            out.push_str(&format!(
                "{product:width$} {:>5} {:>6} {:>5} {:>7} {:>5}\n",
                t.rows, t.green, t.red, t.absent, t.na
            ));
        }
        out
    }

    /// The full grid, grouped by product. `✔` green · `✘` RED · `—` absent (no
    /// evidence) · `·` n/a (this kind has no such stage). The last two are
    /// DIFFERENT glyphs on purpose: one is a hole someone must go fill, the
    /// other never was one.
    pub fn render(&self) -> String {
        let mut out = self.render_rollup();
        out.push('\n');
        out.push_str(
            "✔ green(ran+passed) · ✘ RED(ran+failed) · — absent(NO EVIDENCE) · \
             · n/a(this ROW has no such stage: its kind, or its DECLARED ceremony/answer)\n",
        );
        let width = self.rows.iter().map(|t| t.id().len()).max().unwrap_or(20).max(20);
        out.push_str(&format!("{:width$} ", "build thing"));
        for s in Stage::ALL {
            out.push_str(&format!("{:>3}", s.column()));
        }
        out.push_str("  kind\n");
        let mut current = String::new();
        for thing in &self.rows {
            if thing.product != current {
                current = thing.product.clone();
                out.push_str(&format!("── {current} {}\n", "─".repeat(width.saturating_sub(current.len() + 4))));
            }
            out.push_str(&format!("{:width$} ", thing.id()));
            for s in Stage::ALL {
                out.push_str(&format!("{:>3}", self.verdict(thing, s).glyph()));
            }
            out.push_str(&format!("  {}\n", thing.kind.label()));
        }
        for s in Stage::ALL {
            out.push_str(&format!("  {} = {}\n", s.column(), s.label()));
        }
        out
    }
}

/// Cell counts. Kept as a struct rather than a tuple so a caller cannot swap
/// `red` and `absent` by accident — which is exactly the mistake this whole
/// module is built to prevent.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub struct Tally {
    pub rows: usize,
    pub green: usize,
    pub red: usize,
    pub absent: usize,
    pub na: usize,
}

// ─────────────────── the roster: discovered, never hardcoded ────────────────

/// The nordisk SUITE PROGRAMS the chain matrix covers, in the owner's order.
///
/// A **FLOOR, not a cap**: [`discover_products`] returns these four whether or
/// not they declare anything, so a product that emits nothing is a block of
/// visible holes rather than a missing block. Anything else a caller adds is
/// appended.
///
/// The libraries — tunnr, knut, znippy, skade, Skidbladnir, draupnir, modgunn,
/// nornir — are NOT rows here. They are the machinery the matrix USES: draupnir
/// instantiates, nornir-build-thing enumerates and builds, wg-appliance-core
/// owns the ceremony. That they declare build things of their own is their
/// concern, not this grid's.
pub const CHAIN_PRODUCTS: [&str; 5] = APPLIANCE_PRODUCTS;

/// Locate the product checkouts under a flat parent directory (`~/git`), in
/// roster order. A product with no checkout is returned anyway, with its
/// EXPECTED path — the caller then reports `absent: no checkout at …` rather
/// than silently shrinking the grid.
///
/// Pure std. Reuses nothing from `nornir-build-thing` on purpose: this answers
/// *where*, and [`RosterSource`] answers *what*.
pub fn discover_products(git_root: &Path) -> Vec<(String, PathBuf)> {
    CHAIN_PRODUCTS.iter().map(|p| (p.to_string(), git_root.join(p))).collect()
}

/// **The ONE seam onto `nornir-build-thing`.**
///
/// `nornir-testmatrix` CANNOT depend on `nornir-build-thing` as a normal
/// dependency — MEASURED 2026-08-15, `cargo metadata` refuses it with
/// `cyclic package dependency`:
///
/// ```text
///   nornir-build-thing → nornir-git → nornir-hash → nornir-testmatrix
/// ```
///
/// So the declarations arrive through this trait instead, and the ~10-line
/// adapter that calls `BuildVariantSet::discover(repo_root)?.declarations()`
/// lives in whichever caller already links the build thing (a product's xtask,
/// this crate's `tests/`, the dwarves pane). Isolating it here means the whole
/// dependency is ONE `impl` — and when the build thing's unified lookup lands,
/// exactly one function body changes.
///
/// # The API this crate needs from `nornir-build-thing`
/// 1. `BuildVariantSet::discover(repo_root) -> Result<Option<BuildVariantSet>>`
/// 2. one call returning EVERY declared buildable with its kind — today
///    `declarations() -> Vec<Declared>` (`Declared { name, kind, .. }`) or
///    `artifacts() -> Vec<(ArtifactKind, &str)>`
/// 3. `ArtifactKind::as_str()` — mapped by [`Kind::parse`]
pub trait RosterSource {
    /// Every build thing `product` declares at `repo_root`, in declaration
    /// order. `Err` is a DISCOVERY failure (unreadable/ill-formed declaration)
    /// and must be reported, never swallowed into an empty roster — an empty
    /// roster and a broken one look identical in a grid, and only one of them is
    /// honest.
    fn declared(&self, product: &str, repo_root: &Path) -> Result<Vec<BuildThing>, String>;
}

/// Build the whole roster across the suite products. Discovery failures become
/// a NAMED entry in `problems` rather than a silently shorter grid.
pub fn build_roster(
    source: &dyn RosterSource,
    git_root: &Path,
) -> (Vec<BuildThing>, Vec<String>) {
    let mut roster = Vec::new();
    let mut problems = Vec::new();
    for (product, repo_root) in discover_products(git_root) {
        if !repo_root.exists() {
            problems.push(format!("{product}: no checkout at {}", repo_root.display()));
            continue;
        }
        match source.declared(&product, &repo_root) {
            Ok(things) if things.is_empty() => {
                problems.push(format!("{product}: declares NOTHING at {}", repo_root.display()))
            }
            Ok(things) => {
                for t in things {
                    // A DUPLICATE KEY is not a row, it is a rendering lie: the
                    // same build thing printed twice reads as two independent
                    // pieces of evidence. MEASURED 2026-08-15 —
                    // `gunnar-appliance-iso` is declared both as a
                    // `[[build.variant]]` with `kind = "iso"` and as a
                    // `[[build.iso]]`, so `declarations()` returns it twice.
                    // Keep the first, and NAME the collision.
                    if roster.contains(&t) {
                        problems.push(format!(
                            "{product}: DUPLICATE declaration '{}' (kind {}) — declared twice, \
                             counted once",
                            t.variant,
                            t.kind.label()
                        ));
                        continue;
                    }
                    roster.push(t);
                }
            }
            Err(e) => problems.push(format!("{product}: discovery FAILED: {e}")),
        }
    }
    (roster, problems)
}

/// The product floor: the canonical appliance roster is always represented, so
/// a product that declares nothing still has a visible presence. Extras come
/// from discovery and are appended, never capped.
pub fn floor_products() -> Vec<String> {
    APPLIANCE_PRODUCTS.iter().map(|p| p.to_string()).collect()
}

#[cfg(test)]
mod tests {
    use super::*;

    fn thing(kind: Kind) -> BuildThing {
        BuildThing::new("gunnar", "gunnar-server", kind)
    }

    // ── the load-bearing distinction: absent ≠ red ≠ n/a ────────────────────

    #[test]
    fn an_unreachable_stage_is_absent_not_red_and_the_arm_never_ran() {
        /// Fails BUILT, and records every stage it was asked to run so we can
        /// prove the later arms were never called.
        struct FailBuild {
            called: Vec<Stage>,
        }
        impl ChainArms for FailBuild {
            fn stage(&mut self, stage: Stage, _ctx: &mut ChainCtx) -> StageOutcome {
                self.called.push(stage);
                match stage {
                    Stage::Built => StageOutcome::Fail("linker died".into()),
                    // Every other arm would LIE green if it were ever called.
                    _ => StageOutcome::Pass("THIS MUST NEVER BE REACHED".into()),
                }
            }
        }
        let mut arms = FailBuild { called: Vec::new() };
        let t = thing(Kind::Service);
        let run = run_chain(&mut arms, &t, Path::new("."));

        assert_eq!(run.cell(Stage::Built).verdict, ProofVerdict::Red, "it RAN and FAILED");
        assert!(run.cell(Stage::Built).detail.contains("linker died"));
        for later in [Stage::Instantiated, Stage::Identity, Stage::Tunnel, Stage::Serves, Stage::Durable, Stage::Ready] {
            let c = run.cell(later);
            assert_eq!(c.verdict, ProofVerdict::Absent, "{} never ran → ABSENT, not red, not green", later.label());
            assert!(c.detail.contains("BUILT"), "the blocker is NAMED: {}", c.detail);
            assert!(!c.detail.is_empty(), "an absent cell always says why");
        }
        assert_eq!(arms.called, vec![Stage::Built], "the short-circuit means the later arms were NEVER CALLED");
        assert_eq!(run.reached(), None, "nothing was reached");
    }

    #[test]
    fn a_kind_without_a_stage_is_na_not_absent_and_they_render_differently() {
        let lib = BuildThing::new("znippy", "znippy-lib", Kind::Lib);
        let mut arms = UnwiredArms;
        let run = run_chain(&mut arms, &lib, Path::new("."));
        assert_eq!(run.cell(Stage::Built).verdict, ProofVerdict::Absent, "BUILT applies and was never wired → ABSENT");
        for later in [Stage::Instantiated, Stage::Ready] {
            assert_eq!(run.cell(later).verdict, ProofVerdict::Na, "a lib has no {}", later.label());
        }
        assert_ne!(ProofVerdict::Absent.glyph(), ProofVerdict::Na.glyph(), "the two must never look the same");
        assert_eq!(ProofVerdict::Na.glyph(), "·");
        assert_eq!(ProofVerdict::Absent.glyph(), "—");
    }

    // ── the CEREMONY axis: a product can answer without a WireGuard ceremony ─

    /// Records every stage it was asked to run, and LIES GREEN for the two
    /// ceremony stages — so if the driver ever calls them the assertions below
    /// catch a fabricated pass rather than a crash.
    struct Recorder {
        called: Vec<Stage>,
    }
    impl ChainArms for Recorder {
        fn stage(&mut self, stage: Stage, ctx: &mut ChainCtx) -> StageOutcome {
            self.called.push(stage);
            match stage {
                Stage::Identity | Stage::Tunnel => {
                    StageOutcome::Pass("A CEREMONY THAT NEVER HAPPENED".into())
                }
                Stage::Serves => StageOutcome::Pass(format!(
                    "asked for: {}",
                    ctx.thing.answer.describe()
                )),
                _ => StageOutcome::Pass(format!("{} applied output", stage.label())),
            }
        }
    }

    /// **DEFECT 1, the whole point.** korp-server's real shape, MEASURED
    /// 2026-08-15: a `service` that answers gRPC and deliberately runs NO
    /// WireGuard ceremony. Before this change its stage 3 was a named skip, which
    /// made stages 4–7 permanently `unreachable` and left a REAL correct answer
    /// with no cell to land in.
    #[test]
    fn a_no_ceremony_server_reaches_serves_without_a_single_stage_being_fabricated() {
        let korp = BuildThing::new("korp", "korp-server", Kind::Service)
            .with_ceremony(Ceremony::None)
            .answering(Answer::rpc(
                "100.115.167.95:50051",
                "Hugin StreamState + Munin ListCases",
                r#"{"cases":0,…,"error":null,"seq":0}"#,
            ));
        let mut arms = Recorder { called: Vec::new() };
        let run = run_chain(&mut arms, &korp, Path::new("."));

        // The ceremony stages are Na — the row never had them.
        for s in [Stage::Identity, Stage::Tunnel] {
            let c = run.cell(s);
            assert_eq!(c.verdict, ProofVerdict::Na, "{} is inapplicable, not a hole", s.label());
            assert!(c.detail.contains("ceremony=none"), "the RULE is named: {}", c.detail);
        }
        // …and the arms for them were NEVER CALLED, so no green was fabricated.
        assert!(
            !arms.called.contains(&Stage::Identity) && !arms.called.contains(&Stage::Tunnel),
            "an Na stage must be stepped OVER, never run: {:?}",
            arms.called
        );

        // THE CELL THAT DID NOT EXIST BEFORE: it answered.
        let serves = run.cell(Stage::Serves);
        assert_eq!(serves.verdict, ProofVerdict::Green, "the correct answer has a cell: {}", serves.detail);
        assert!(serves.detail.contains("Munin ListCases"), "and it carries the row's OWN question: {}", serves.detail);
        assert_eq!(run.reached(), Some(Stage::Ready), "the whole ladder is walkable without a ceremony");

        // Na is not green: the tally counts them apart, and no row is emitted.
        let grid = ChainGrid::build(&[korp.clone()], &run.rows());
        let t = grid.tally();
        assert_eq!(t.green, 5, "five REAL greens (1,2,5,6,7)");
        assert_eq!(t.na, 2, "and two cells that never existed");
        assert_eq!(t.absent + t.red, 0);
        assert!(
            !run.rows().iter().any(|r| r.test_name == "identity" || r.test_name == "tunnel"),
            "an Na cell emits no evidence at all"
        );
    }

    /// The same row with NOTHING wired: the ceremony stages must be `Na` and the
    /// applicable ones `Absent`. The distinction is the load-bearing one — an
    /// inapplicable stage must never look like an unfilled hole, and an unfilled
    /// hole must never look like a pass.
    #[test]
    fn a_no_ceremony_row_is_na_at_the_ceremony_stages_and_absent_everywhere_it_could_have_run() {
        let korp = BuildThing::new("korp", "korp-server", Kind::Service)
            .with_ceremony(Ceremony::None)
            .answering(Answer::rpc("127.0.0.1:50051", "ListCases", "{…}"));
        let run = run_chain(&mut UnwiredArms, &korp, Path::new("."));
        for s in [Stage::Identity, Stage::Tunnel] {
            assert_eq!(run.cell(s).verdict, ProofVerdict::Na);
        }
        for s in [Stage::Built, Stage::Instantiated, Stage::Serves, Stage::Durable, Stage::Ready] {
            assert_eq!(run.cell(s).verdict, ProofVerdict::Absent, "{} is a HOLE, not an n/a", s.label());
        }
        assert_eq!(ChainGrid::build(&[korp], &run.rows()).tally().green, 0, "nothing ran, nothing green");
    }

    /// THREE different absences in ONE grid, and no two of them may look alike:
    /// RED (ran, failed) · ABSENT (never ran, no evidence) · N/A (never existed).
    #[test]
    fn red_and_unreachable_absent_and_inapplicable_na_stay_three_distinct_cells() {
        struct FailInstantiate;
        impl ChainArms for FailInstantiate {
            fn stage(&mut self, stage: Stage, _c: &mut ChainCtx) -> StageOutcome {
                match stage {
                    Stage::Built => StageOutcome::Pass("built".into()),
                    Stage::Instantiated => StageOutcome::Fail("unit refused to start".into()),
                    _ => StageOutcome::Pass("MUST NEVER BE REACHED".into()),
                }
            }
        }
        let t = BuildThing::new("korp", "korp-server", Kind::Service)
            .with_ceremony(Ceremony::None)
            .answering(Answer::rpc("127.0.0.1:50051", "ListCases", "{…}"));
        let run = run_chain(&mut FailInstantiate, &t, Path::new("."));
        assert_eq!(run.cell(Stage::Instantiated).verdict, ProofVerdict::Red, "it RAN and FAILED");
        assert_eq!(run.cell(Stage::Identity).verdict, ProofVerdict::Na, "it never existed");
        assert_eq!(run.cell(Stage::Serves).verdict, ProofVerdict::Absent, "it COULD have run and did not");
        // …and an Na cell must not swallow the blocker: SERVES still names it.
        assert!(run.cell(Stage::Serves).detail.contains("INSTANTIATED"), "{}", run.cell(Stage::Serves).detail);
        let g = [ProofVerdict::Red.glyph(), ProofVerdict::Absent.glyph(), ProofVerdict::Na.glyph()];
        assert_eq!(g.len(), {
            let mut v = g.to_vec();
            v.sort();
            v.dedup();
            v.len()
        }, "three verdicts, three glyphs");
    }

    // ── the ANSWER axis: SERVES is parameterised, never hardcoded to git ─────

    /// A git forge and a map server are asked DIFFERENT questions by the SAME
    /// arm, because the question comes off the row.
    #[test]
    fn serves_asks_what_the_row_declared_and_is_never_hardcoded_to_git() {
        /// The shape a real SERVES arm has: one `match` on the declaration.
        struct Asker {
            asked: Vec<String>,
        }
        impl ChainArms for Asker {
            fn serves(&mut self, ctx: &mut ChainCtx) -> StageOutcome {
                let q = match &ctx.thing.answer {
                    Answer::GitForge => "git push + clone + fsck".to_string(),
                    Answer::Rpc { call, .. } => format!("rpc {call}"),
                    Answer::Banner { expect, .. } => format!("banner {expect}"),
                    // The honest arm for a row nobody wrote a question for.
                    a @ Answer::Undeclared => return StageOutcome::Skipped(a.describe()),
                    a => return StageOutcome::Skipped(format!("cannot ask for: {}", a.describe())),
                };
                self.asked.push(q.clone());
                StageOutcome::Pass(format!("ANSWERED: {q}"))
            }
            fn stage(&mut self, stage: Stage, ctx: &mut ChainCtx) -> StageOutcome {
                if stage == Stage::Serves {
                    return self.serves(ctx);
                }
                StageOutcome::Pass(format!("{} ok", stage.label()))
            }
        }
        let forge = BuildThing::new("gunnar", "gunnar-server", Kind::Service)
            .with_ceremony(Ceremony::WireGuard)
            .answering(Answer::GitForge);
        let map = BuildThing::new("korp", "korp-server", Kind::Service)
            .with_ceremony(Ceremony::None)
            .answering(Answer::rpc("127.0.0.1:50051", "ListCases", "{…}"));
        let mut arms = Asker { asked: Vec::new() };
        let a = run_chain(&mut arms, &forge, Path::new("."));
        let b = run_chain(&mut arms, &map, Path::new("."));
        assert_eq!(arms.asked, vec!["git push + clone + fsck", "rpc ListCases"], "two rows, two questions");
        assert_eq!(a.cell(Stage::Serves).verdict, ProofVerdict::Green);
        assert_eq!(b.cell(Stage::Serves).verdict, ProofVerdict::Green);
        // The ceremony row KEEPS its ceremony stages — this fix must not delete
        // them for the products that really do run one.
        assert_eq!(a.cell(Stage::Identity).verdict, ProofVerdict::Green);
        assert_eq!(b.cell(Stage::Identity).verdict, ProofVerdict::Na);

        // An UNDECLARED answer is a HOLE, never an n/a and never a pass.
        let mute = BuildThing::new("korp", "korp-solo", Kind::Service);
        let c = run_chain(&mut arms, &mute, Path::new("."));
        assert_eq!(c.cell(Stage::Serves).verdict, ProofVerdict::Absent, "nobody wrote the question down");
        assert!(c.cell(Stage::Serves).detail.contains("nobody said"), "{}", c.cell(Stage::Serves).detail);
    }

    /// `Answer::None` is the only thing that makes SERVES `Na`, and a DECLARED
    /// answer promotes a serving `exe` from `Na` to a visible hole.
    #[test]
    fn only_a_declared_none_makes_serves_na_and_a_declared_answer_promotes_an_exe() {
        let mute = BuildThing::new("korp", "korp-cli", Kind::Service).answering(Answer::None);
        assert!(!mute.has_stage(Stage::Serves));
        let run = run_chain(&mut AllPass, &mute, Path::new("."));
        assert_eq!(run.cell(Stage::Serves).verdict, ProofVerdict::Na);
        assert!(run.cell(Stage::Serves).detail.contains("answer=none"), "{}", run.cell(Stage::Serves).detail);

        // A plain exe: the conservative KIND table says no SERVES…
        let plain = BuildThing::new("gunnar", "gunnar-cli", Kind::Exe);
        assert!(!plain.has_stage(Stage::Serves), "an undeclared exe keeps the conservative Na");
        // …until the caller declares what a correct answer is for it.
        let cli = plain.clone().answering(Answer::Stdout {
            args: "gunnar-cli --version".into(),
            expect: "gunnar".into(),
        });
        assert!(cli.has_stage(Stage::Serves), "a DECLARED answer promotes it to applicable");
        assert_eq!(
            run_chain(&mut UnwiredArms, &cli, Path::new(".")).cell(Stage::Serves).verdict,
            ProofVerdict::Absent,
            "promoted to a HOLE, never to a pass"
        );
        assert_eq!(cli.applicable_stages(), 3, "built + instantiated + serves");
    }

    /// The declarations are ABOUT the row, not a second row: the key stays
    /// `(product, variant, kind, form)` so evidence emitted before a declaration
    /// existed still lands in the declared row's cell.
    #[test]
    fn the_declarations_are_not_part_of_the_row_key() {
        use std::collections::hash_map::DefaultHasher;
        let bare = BuildThing::new("korp", "korp-server", Kind::Service);
        let declared = bare
            .clone()
            .with_ceremony(Ceremony::None)
            .answering(Answer::rpc("127.0.0.1:50051", "ListCases", "{…}"))
            .with_marker("korp-server listening");
        assert_eq!(bare, declared, "same row, more known about it");
        let h = |t: &BuildThing| {
            let mut s = DefaultHasher::new();
            t.hash(&mut s);
            s.finish()
        };
        assert_eq!(h(&bare), h(&declared), "Eq and Hash must agree");
        assert_eq!(bare.cmp(&declared), Ordering::Equal);
        // The component string is the key and carries no declaration…
        assert_eq!(declared.component(), "korp-appliance/native#service:korp-server");
        assert_eq!(parse_chain_component(&declared.component()).as_ref(), Some(&declared));
        // …so evidence from the BARE row lands in the DECLARED row's cell, and
        // the grid re-derives Na from the ROSTER's declaration.
        let cell = Cell { verdict: ProofVerdict::Green, detail: "answered".into() };
        let grid = ChainGrid::build(&[declared.clone()], &[chain_row(&bare, Stage::Serves, &cell)]);
        assert_eq!(grid.verdict(&declared, Stage::Serves), ProofVerdict::Green);
        assert_eq!(grid.verdict(&declared, Stage::Identity), ProofVerdict::Na);
    }

    #[test]
    fn an_na_stage_does_not_block_the_stages_after_it() {
        // Contrived but load-bearing: Na must be stepped OVER, not treated as a
        // failure that blocks. (No real kind has a gap today; the driver must
        // still be right if one appears.)
        let mut arms = AllPass;
        let ex = thing(Kind::Exe);
        let run = run_chain(&mut arms, &ex, Path::new("."));
        assert_eq!(run.cell(Stage::Built).verdict, ProofVerdict::Green);
        assert_eq!(run.cell(Stage::Instantiated).verdict, ProofVerdict::Green);
        assert_eq!(run.cell(Stage::Ready).verdict, ProofVerdict::Na, "an exe never comes back up off a disk");
    }

    struct AllPass;
    impl ChainArms for AllPass {
        fn stage(&mut self, stage: Stage, _ctx: &mut ChainCtx) -> StageOutcome {
            StageOutcome::Pass(format!("{} applied output", stage.label()))
        }
    }

    #[test]
    fn unwired_arms_skip_loudly_by_name_never_silently() {
        let mut arms = UnwiredArms;
        let run = run_chain(&mut arms, &thing(Kind::Service), Path::new("."));
        let c = run.cell(Stage::Built);
        assert_eq!(c.verdict, ProofVerdict::Absent, "an unwired arm is ABSENT — never green");
        assert!(c.detail.contains("BUILT"), "names the stage: {}", c.detail);
        assert!(c.detail.contains("gunnar/gunnar-server"), "names the row: {}", c.detail);
    }

    // ── honesty rule 1: install and boot-installed are two VM lifetimes ─────

    #[test]
    fn an_installer_ready_green_is_refused_inside_one_vm_lifetime() {
        /// The classic lie: install it, watch it come up in the SAME machine,
        /// and claim both stage 2 and stage 7.
        struct OneLifetime;
        impl ChainArms for OneLifetime {
            fn stage(&mut self, stage: Stage, ctx: &mut ChainCtx) -> StageOutcome {
                if stage == Stage::Instantiated {
                    ctx.vm_lifetimes = 1; // installed and never power-cycled
                }
                StageOutcome::Pass(format!("{} ok", stage.label()))
            }
        }
        let inst = BuildThing::new("gunnar", "gunnar-installer", Kind::Installer);
        let run = run_chain(&mut OneLifetime, &inst, Path::new("."));
        let ready = run.cell(Stage::Ready);
        assert_eq!(ready.verdict, ProofVerdict::Red, "one lifetime cannot claim both ends");
        assert!(ready.detail.contains("vm_lifetimes=1"), "the count is NAMED: {}", ready.detail);
        assert!(ready.detail.contains("DISTINCT VM lifetimes"));

        // …and with a real power-cycle it is allowed to go green.
        struct TwoLifetimes;
        impl ChainArms for TwoLifetimes {
            fn stage(&mut self, stage: Stage, ctx: &mut ChainCtx) -> StageOutcome {
                if stage == Stage::Instantiated {
                    ctx.vm_lifetimes = 2; // install VM, ISO detached, disk VM
                }
                StageOutcome::Pass(format!("{} ok", stage.label()))
            }
        }
        let run = run_chain(&mut TwoLifetimes, &inst, Path::new("."));
        assert_eq!(run.cell(Stage::Ready).verdict, ProofVerdict::Green);
    }

    // ── iso-kvm and iso-metal are DISTINCT rows ────────────────────────────

    #[test]
    fn kvm_and_metal_are_distinct_rows_and_a_kvm_green_never_fills_the_metal_cell() {
        let iso = BuildThing::new("gunnar", "gunnar-appliance-iso", Kind::Iso);
        let kvm = iso.clone().in_form(Form::kvm());
        let metal = iso.clone().in_form(Form::metal());
        assert_ne!(kvm, metal, "different keys");
        assert_ne!(kvm.component(), metal.component(), "different rows in the evidence");

        let mut arms = AllPass;
        let kvm_run = run_chain(&mut arms, &kvm, Path::new("."));
        let grid = ChainGrid::build(&[kvm.clone(), metal.clone()], &kvm_run.rows());
        assert_eq!(grid.verdict(&kvm, Stage::Serves), ProofVerdict::Green, "the faked machine passed");
        assert_eq!(
            grid.verdict(&metal, Stage::Serves),
            ProofVerdict::Absent,
            "and real hardware is still UNPROVEN — a kvm green must not sit in the metal cell"
        );
    }

    // ── the row ⇄ cell round trip, and the appliance rollup still reads it ──

    #[test]
    fn the_component_encoding_round_trips_and_stays_readable_to_the_appliance_rollup() {
        let t = BuildThing::new("holger", "holger-appliance-iso", Kind::Iso).in_form(Form::metal());
        let c = t.component();
        assert_eq!(c, "holger-appliance/metal#iso:holger-appliance-iso");
        assert_eq!(parse_chain_component(&c).as_ref(), Some(&t));
        // The coarse fold still works: the SAME row feeds the (product × proof) grid.
        assert_eq!(crate::appliance::appliance_product(&c), Some("holger"));
    }

    #[test]
    fn non_chain_components_are_never_invented_into_chain_rows() {
        for s in [
            "holger-appliance",             // the old coarse emitter
            "holger-appliance/container",   // form only, no kind:variant
            "holger-appliance/kvm#iso",     // no variant
            "holger-appliance/kvm#:v",      // empty kind
            "-appliance/kvm#iso:v",         // empty product
            "holger-mvp",                   // not an appliance at all
        ] {
            assert!(parse_chain_component(s).is_none(), "{s} must NOT parse as a chain row");
        }
    }

    #[test]
    fn a_thing_that_emitted_nothing_is_absent_across_its_applicable_stages_never_missing() {
        let a = BuildThing::new("gunnar", "gunnar-server", Kind::Service);
        let b = BuildThing::new("korp", "korp-server", Kind::Service);
        let mut arms = AllPass;
        let only_a = run_chain(&mut arms, &a, Path::new("."));
        let grid = ChainGrid::build(&[a.clone(), b.clone()], &only_a.rows());
        assert_eq!(grid.rows.len(), 2, "the roster FLOORS the grid — korp keeps its row");
        for s in Stage::ALL {
            assert_eq!(grid.verdict(&b, s), ProofVerdict::Absent, "korp's {} is a visible hole", s.label());
        }
        let rendered = grid.render();
        assert!(rendered.contains("korp/korp-server"), "the empty row is ON the wall:\n{rendered}");
        let t = grid.tally();
        assert_eq!(t.green, 7, "only a's seven");
        assert_eq!(t.absent, 7, "and b's seven holes");
        assert_eq!(t.green + t.red + t.absent + t.na, 14, "every cell is accounted for");
    }

    #[test]
    fn red_wins_over_green_when_two_pieces_of_evidence_disagree() {
        let t = BuildThing::new("korp", "korp-server", Kind::Service);
        let cell_g = Cell { verdict: ProofVerdict::Green, detail: "ok here".into() };
        let cell_r = Cell { verdict: ProofVerdict::Red, detail: "broke there".into() };
        let rows = vec![
            chain_row(&t, Stage::Built, &cell_g),
            chain_row(&t, Stage::Built, &cell_r),
        ];
        let grid = ChainGrid::build(&[t.clone()], &rows);
        assert_eq!(grid.verdict(&t, Stage::Built), ProofVerdict::Red);
        // …and in the other order too.
        let rows = vec![
            chain_row(&t, Stage::Built, &cell_r),
            chain_row(&t, Stage::Built, &cell_g),
        ];
        let grid = ChainGrid::build(&[t.clone()], &rows);
        assert_eq!(grid.verdict(&t, Stage::Built), ProofVerdict::Red, "order must not matter");
    }

    #[test]
    fn a_skip_row_reads_back_as_absent_and_carries_its_reason_never_as_green() {
        let t = BuildThing::new("gunnar", "gunnar-server", Kind::Service);
        let run = run_chain(&mut UnwiredArms, &t, Path::new("."));
        let rows = run.rows();
        assert!(rows.iter().all(|r| r.status == status::SKIP), "absent emits `skip`, never `pass`");
        assert!(rows.iter().all(|r| !r.message.is_empty()), "every skip names its reason");
        let grid = ChainGrid::build(&[t.clone()], &rows);
        for s in Stage::ALL {
            assert_eq!(grid.verdict(&t, s), ProofVerdict::Absent, "a skip row is ABSENT after the round trip");
        }
        assert_eq!(grid.tally().green, 0, "nothing was proven and the grid says so");
    }

    #[test]
    fn na_cells_emit_no_rows_but_the_grid_still_shows_them() {
        let lib = BuildThing::new("znippy", "znippy", Kind::Lib);
        let run = run_chain(&mut AllPass, &lib, Path::new("."));
        let rows = run.rows();
        assert_eq!(rows.len(), 1, "only the one applicable stage emits: {rows:?}");
        assert_eq!(rows[0].test_name, "built");
        let grid = ChainGrid::build(&[lib.clone()], &rows);
        assert_eq!(grid.verdict(&lib, Stage::Built), ProofVerdict::Green);
        assert_eq!(grid.verdict(&lib, Stage::Ready), ProofVerdict::Na, "re-derived from the KIND, not from the rows");
        assert_eq!(grid.tally().na, 6);
    }

    #[test]
    fn ready_is_the_terminal_column_and_only_servers_are_in_it() {
        assert!(Kind::Service.has_stage(Stage::Ready));
        assert!(Kind::Installer.has_stage(Stage::Ready));
        for k in [Kind::Exe, Kind::Container, Kind::Vm, Kind::Iso, Kind::Lib, Kind::Text, Kind::Wasm, Kind::Other("mystery".into())] {
            assert!(!k.has_stage(Stage::Ready), "{} must not be in the READY column", k.label());
        }
        // …and BUILT is universal: everything declared can at least be built.
        for k in [Kind::Exe, Kind::Service, Kind::Container, Kind::Vm, Kind::Iso, Kind::Installer, Kind::Lib, Kind::Text, Kind::Wasm, Kind::Other("mystery".into())] {
            assert!(k.has_stage(Stage::Built), "{} must be buildable", k.label());
        }
    }

    #[test]
    fn unknown_kinds_stay_visible_spelled_the_way_the_repo_spelled_them() {
        let k = Kind::parse("firmware-blob");
        assert_eq!(k, Kind::Other("firmware-blob".into()));
        assert_eq!(k.label(), "firmware-blob");
        assert_eq!(k.applicable_stages(), 1, "only BUILT is knowable for a kind we do not model");
    }

    #[test]
    fn stage_labels_round_trip_and_the_column_numbers_are_the_chain_order() {
        for (i, s) in Stage::ALL.into_iter().enumerate() {
            assert_eq!(Stage::parse(s.label()), Some(s));
            assert_eq!(s.column(), i + 1);
        }
        assert_eq!(Stage::parse("READY"), None, "labels are exact, lowercase");
        assert_eq!(Stage::ALL.len(), 7);
    }

    #[test]
    fn the_rollup_leads_with_the_honest_number() {
        let a = BuildThing::new("gunnar", "gunnar-server", Kind::Service);
        let b = BuildThing::new("korp", "korp-server", Kind::Service);
        let run = run_chain(&mut AllPass, &a, Path::new("."));
        let grid = ChainGrid::build(&[a, b], &run.rows());
        let r = grid.render_rollup();
        assert!(r.contains("2 build things × 7 stages = 14 cells"), "{r}");
        assert!(r.contains("GREEN 7"), "{r}");
        assert!(r.contains("ABSENT 7"), "{r}");
        assert!(r.contains("ready-capable rows: 2"), "{r}");
        // Per-product rollup carries both products, one line each.
        assert!(r.contains("gunnar"), "{r}");
        assert!(r.contains("korp"), "{r}");
    }

    // A product with SEVERAL forms must count them all. The rows are the forms
    // (gunnar-server, gunnar-cli, gunnar-appliance-iso …), never the product
    // name — a per-product `rows` that always read 1 would hide exactly that.
    #[test]
    fn the_per_product_rollup_counts_every_form_not_one_row_per_product() {
        let roster = vec![
            BuildThing::new("gunnar", "gunnar-server", Kind::Service),
            BuildThing::new("gunnar", "gunnar-cli", Kind::Exe),
            BuildThing::new("gunnar", "gunnar-appliance-iso", Kind::Iso),
            BuildThing::new("korp", "korp-ui", Kind::Exe),
        ];
        let grid = ChainGrid::build(&roster, std::iter::empty());
        let by = grid.by_product();
        assert_eq!(by.len(), 2, "two products");
        assert_eq!(by[0].0, "gunnar");
        assert_eq!(by[0].1.rows, 3, "gunnar has THREE forms, not one");
        assert_eq!(by[1].1.rows, 1, "korp has one");
        // Cell arithmetic per product holds too.
        let g = by[0].1;
        assert_eq!(g.green + g.red + g.absent + g.na, 3 * 7);
        // Na comes only from the KIND: service 0, exe 5, iso 1 → 6.
        assert_eq!(g.na, 0 + 5 + 1, "exe and iso have stages that do not exist for them");
        assert_eq!(grid.tally().rows, 0, "the whole-grid tally does not use `rows`");
    }
}