installtest/tests/installer_drive.rs
//! **`gunnar-installer` — THE car, driven: compose the medium, INSTALL to a
//! blank disk, boot the INSTALLED disk with the medium ABSENT, and reboot it.**
//!
//! `.nornir/build-variants.toml` has declared a `[[build.installer]]` cell named
//! `gunnar-installer` since 2026-08-14 and — measured 2026-08-15, by grepping
//! every repo for `run_installer` — nothing in gunnar had ever composed it. It
//! was DECLARED and never once built, so every claim in its (excellent) comment
//! block was a claim about a file nobody had produced.
//!
//! # The three lifetimes
//!
//! ```text
//! 1 INSTALL boot off the ISO with a blank 64 GiB disk attached.
//! korp-installer is PID 1: GPT + ESP + payload + data
//! partition, verdict = korp_installer::INSTALL_OK_MARKER.
//! ── the medium is GONE from here on ──
//! 2 FIRST BOOT OVMF walks to \EFI\BOOT\BOOTX64.EFI on the disk lifetime 1
//! wrote. gunnar MINTS its identity onto P3 and SERVES.
//! 3 REBOOT the same disk again. It must NOT re-mint, must present the
//! SAME public key, and must answer a real TCP connection
//! with its own SSH banner through a host→guest forward.
//! ```
//!
//! Lifetime 3 is the point of the whole shape. The live ISO's `/data` is RAM, so
//! its key dies with the power and the operator would register a key the next
//! boot has already thrown away. Only an installed disk can hold an identity, so
//! "the same key came back after a power cycle" is the claim that has to be
//! MEASURED, not the mint.
//!
//! # Reuse, not a twin
//!
//! Almost nothing here is mechanism. It is all somebody else's, on purpose:
//!
//! * the medium is composed by `nornir_build_thing::suite_cli::run_installer`
//! with `korp_installer::isocompose::ForgeComposer` — the SAME one composer
//! `dwarves buildthing --installer` and holger's xtask reach;
//! * the payload is packed by `korp_installer::payload::pack_znippy`, under the
//! entry the cell's own `korp.entry=` token names;
//! * the disk size and its QEMU attachment argv come from the CELL
//! (`target_disk_bytes()`, `kvm_disk_args()`) — never a number spelled twice;
//! * the boots, the serial gates, the GPT/FAT readback and the rollup rows are
//! `korp_installer::isoproof`, whose own header says it exists so that korp's,
//! holger's and gunnar's installer proofs share ONE harness. `IsoInstallProof`
//! even takes `product` and `check` as inputs and names gunnar in its doc.
//!
//! # Two things this drive learned the hard way, both recorded
//!
//! Both were once reasons draupnir's KVM backend **could not** drive this
//! install. Both are now fixed upstream (2026-09-05), so they are recorded here
//! as measurements, not as limitations — the boots stay on `isoproof` because
//! `isoproof` owns the multi-disk attach order, the GPT/FAT readback, the ESP
//! `korp/install` record and the rollup rows this drive is assembled out of, and
//! moving them is a job of its own.
//!
//! 1. **The medium must be on virtio, not `-cdrom`.** MEASURED 2026-08-15 on
//! this box: driving the install through `draupnir::kvm::KvmBoot`
//! (`BootSpec::kvm_install_from_medium`, whose `medium_qemu_args` then
//! emitted `-cdrom <iso> -boot d` and nothing else) booted the firmware fine
//! and then died with `INSTALLER: volume: no block device carries the
//! ISO-9660 magic (CD001) … (probed: loop1, loop6, loop4, loop2, loop0,
//! loop7, loop5, vda, loop3)` — no `sr0` anywhere, because the appliance
//! kernel carries no ATA/AHCI/SCSI at all.
//! `isoproof::attach_iso_as_virtio` is the fix and its doc says so in as many
//! words; the harness applies it for us. draupnir `519afcf` now spells the
//! same attachment `MediumAttach::Both`, with that argv copied verbatim.
//! 2. **A NIC has to be asked for.** `isoproof::attach_slirp_nic` adds
//! virtio-net + a `hostfwd`, which is what makes lifetime 3's live SSH probe
//! possible at all. draupnir's KVM backend used to drop a port forward
//! outright (`BootSpec::validate` rejected `ports`/`port_maps` on any
//! non-container backend); `60ffb06` honours both on `Backend::Kvm` and folds
//! them onto tunnr's `NetConfig::User { hostfwd }`, which is what let
//! `src/kvm.rs` and `src/iso.rs` move onto draupnir.
//!
//! ```text
//! CARGO_TARGET_DIR=~/scratch/cargo/gunnar-install \
//! KORP_TEST_KERNEL=~/scratch/bzImage \
//! cargo test --manifest-path installtest/Cargo.toml --test installer_drive -- --ignored --nocapture
//! ```
//!
//! `GUNNAR_INSTALLER_ISO=<path>` reuses an already-composed medium and says so
//! LOUDLY — the compose rung then records a named skip instead of a pass,
//! because a run that did not compose has not proven composition (and the
//! fixture-hub rungs below are skipped by name too: a reused medium carries
//! nobody's fixture).
//!
//! # The declared wg-hub FIXTURE (2026-08-22)
//!
//! The cell declares `hub_conf` and `[build.installer.fixtures] hub_conf =
//! "wg-hub"`; this drive supplies `--param hub_conf=@fixture`, exactly as
//! `dwarves buildthing … --param hub_conf=@fixture` and the dwarves build pane
//! do, and the FORGE mints the hub: a userspace WireGuard hub reserved before
//! the appliance exists, private half 0600 under the drive's own state dir
//! (`NORDISK_FORGE_STATE_DIR`), public half rendered as the ceremony's
//! `hub.conf` with `Endpoint = 10.0.2.2:<port>` (slirp's host). What is then
//! MEASURED, not inferred:
//!
//! * the ISO volume carries `hub.conf` BYTE-IDENTICAL to the fixture's render
//! (read with `korp_installer::iso9660`, no mount);
//! * the installed P3 carries `wg/hub.conf` byte-identical (`expect_data_files`);
//! * the hub, re-opened FROM THE STAGED CONF ALONE (`HubState::for_hub_conf`)
//! and spawned OPEN on the reserved port before the first boot, sees ≥1
//! valid handshake INITIATION from the installed appliance, answers it, and
//! the key it LEARNED equals the identity line the appliance printed —
//! the appliance's userspace data plane (`wg::bring_up`, boringtun compiled
//! in) initiated toward the fixture's endpoint at first boot. `sealed_packets`
//! (boringtun's first-packet keepalive after the handshake) is reported as
//! its own row.
use std::path::PathBuf;
use std::time::Duration;
use korp_installer::isoproof::{
self, DataFileExpectation, FirstBootProof, IsoInstallProof, KvmPreflight, FORM_ISO_KVM,
};
use korp_installer::kvmproof;
use korp_installer::wgfixture::{self, HubState};
use nornir_build_thing::params::{FixtureKind, FIXTURE_VALUE};
use nornir_build_thing::variant::BuildVariantSet;
/// The proof name every skip is spoken under.
const PROOF: &str = "gunnar_installer_drive";
/// The declared installer cell.
const CELL: &str = "gunnar-installer";
/// The declared appliance variant it lays down.
const VARIANT: &str = "gunnar-server";
/// The product, as the rollup spells it.
const PRODUCT: &str = "gunnar";
/// gunnar's SERVING banner — `serve.rs:1413`, printed with the listener's own
/// address in the line above it. Deliberately NOT the identity line: a mint
/// with no server behind it is "boots but does not serve", which is exactly the
/// distinction `FirstBootProof::marker` documents.
const SERVING_MARKER: &str = "grpc disabled";
/// The canonical identity announcement, `wg_appliance_core::ceremony`'s
/// `IDENTITY_LINE_REGEX` in prefix form — product-qualified so a drive can
/// never mistake another product's appliance for gunnar's.
const IDENTITY_PREFIX: &str = "wg-appliance identity: product=gunnar pubkey=";
/// Scratch on DISK — a 64 GiB sparse image has no business in `/tmp` (RAM).
const SCRATCH: &str = "~/scratch/gunnar-installer-drive";
/// Where the declared variant's artifact lives.
fn build_target_dir() -> PathBuf {
std::env::var_os("GUNNAR_EXE_TARGET_DIR")
.map(PathBuf::from)
.unwrap_or_else(|| gunnar_installtest::expand_home("~/scratch/cargo/gunnar"))
}
/// One row in the rollup, through the harness's OWN emitter so the component
/// string (`<product>-appliance/<form>`) is composed in one place.
fn row(check: &str, pass: bool, detail: &str) {
isoproof::emit_kvm_row(PRODUCT, check, pass, detail);
}
#[test]
#[ignore = "heavy: composes the gunnar-installer ISO (builds korp-boot for UEFI + two musl PID-1s), then boots three KVM lifetimes"]
fn the_declared_installer_lays_gunnar_on_a_disk_that_boots_serves_and_keeps_its_identity() {
let repo = gunnar_installtest::workspace_root();
// `KORP_TEST_KERNEL` is the harness's own env; `KORP_KERNEL` is what the
// forge's ISO test uses on this box. Accept either rather than make an
// operator remember which proof wants which name.
if std::env::var_os(isoproof::TEST_KERNEL_ENV).is_none() {
if let Some(k) = std::env::var_os("KORP_KERNEL") {
std::env::set_var(isoproof::TEST_KERNEL_ENV, k);
}
}
let Some(pf) = KvmPreflight::detect(PROOF) else {
row(
"host_capable",
false,
"qemu / OVMF / KORP_TEST_KERNEL missing — see stderr",
);
return;
};
// ── 1. the DECLARATION, as the one writer ────────────────────────────
let set = BuildVariantSet::discover(&repo)
.expect("read .nornir/build-variants.toml")
.unwrap_or_else(|| panic!("gunnar declares a build space at {}", repo.display()));
let cell = set
.installers
.iter()
.find(|i| i.name == CELL)
.unwrap_or_else(|| {
panic!(
"gunnar declares a `[[build.installer]]` named {CELL} (declared: {})",
set.installers
.iter()
.map(|i| i.name.as_str())
.collect::<Vec<_>>()
.join(", ")
)
})
.clone();
assert_eq!(
cell.variant, VARIANT,
"the cell must lay down the SAME atom the live ISO boots, or the installed disk \
is not the regression baseline's binary"
);
// The gate is the DECLARATION's, never this drive's. An installer proof that
// typed GUNNAR_APPLIANCE=1 itself would prove nothing about the artifact.
let params = cell.effective_boot_params();
// `korp.env.` prefixed, and that prefix is load-bearing rather than
// cosmetic: `install::carry_site_tokens` forwards ONLY `korp.*` tokens onto
// the installed disk, so a bare `GUNNAR_APPLIANCE=1` here would gate the
// INSTALLER's PID 1 and die with that boot. Measured 2026-08-15; see the
// cell's own comment block and `expect_cmdline` below.
assert!(
params.iter().any(|p| p == "korp.env.GUNNAR_APPLIANCE=1"),
"the ceremony gate must ride the DECLARED boot params in the form that REACHES \
the installed disk (`korp.env.GUNNAR_APPLIANCE=1`): {params:?}"
);
assert!(
params.iter().any(|p| p.starts_with("korp.args=")),
"the app's argv tail must be declared as `korp.args=` — a kernel `--` tail is not \
carried to the installed cmdline, and PID 1 would exec a bare `gunnar`: {params:?}"
);
let entry = params
.iter()
.find_map(|p| p.strip_prefix("korp.entry="))
.expect("the cell names its product with korp.entry=")
.to_string();
assert_eq!(
entry, PRODUCT,
"the cell's korp.entry= names the rollup product"
);
let variant = set.get(VARIANT).expect("the chained variant is declared");
let mut bin = build_target_dir();
if let Some(t) = variant.target.as_deref() {
bin = bin.join(t);
}
let bin = bin
.join(variant.profile.label())
.join(variant.bin.as_deref().expect("the variant names a bin"));
// A missing payload is a FAILURE, never a skip (2026-09-22, T17): a skip
// row plus `ok` let a run that proved nothing exit green, and it did.
if let Some(refusal) = gunnar_installtest::missing_payload_refusal(
&bin,
&variant.command_line(),
&build_target_dir(),
) {
row("app_payload_packed", false, &refusal);
panic!("{refusal}");
}
let work = &gunnar_installtest::expand_home(SCRATCH);
std::fs::create_dir_all(work).expect("scratch on disk");
// ── 2. the app payload, through the ONE codec ────────────────────────
//
// `isocompose` reads `app` verbatim into `ComposeInputs::app_znippy` and
// streams it onto P2 — so what is handed over must ALREADY be a znippy
// archive whose single entry is the `korp.entry=` name korp-start resolves.
// gunnar has no `appliance-pack` verb (holger does,
// `xtask/src/appliance_pack.rs`), so the pack happens here with
// korp-installer's own function rather than a second spelling of the codec.
let raw = std::fs::read(&bin).expect("read the declared appliance binary");
let raw_len = raw.len();
let packed = korp_installer::payload::pack_znippy(&entry, raw, 1)
.expect("pack the app .znippy under the declared entry");
let app = work.join("gunnar-app.znippy");
std::fs::write(&app, &packed).expect("stage the app payload");
eprintln!(
"PACKED: {} → {} ({raw_len} → {} bytes, entry `{entry}`)",
bin.display(),
app.display(),
packed.len()
);
row(
"app_payload_packed",
true,
&format!(
"entry={entry}, {raw_len} → {} bytes via korp_installer::payload::pack_znippy",
packed.len()
),
);
// ── 3a. THE FIXTURE, DECLARED BY THE CELL, MINTED BY THE FORGE ───────
//
// Asserted on the DECLARATION first: the cell must declare `hub_conf`
// (text, staged as hub.conf) and a wg-hub fixture for it. The drive types
// nothing about hubs itself — it supplies the same `@fixture` word the
// CLI and the pane supply, and everything else is the forge's.
let hub_spec = cell
.params
.iter()
.find(|p| p.name == "hub_conf")
.expect("the cell declares a `hub_conf` parameter");
assert_eq!(
hub_spec.staged_file, "hub.conf",
"hub_conf lands as the ceremony's hub.conf"
);
assert_eq!(
cell.fixture_for("hub_conf"),
Some(FixtureKind::WgHub),
"the cell declares `[build.installer.fixtures] hub_conf = \"wg-hub\"` — the fixture \
is DECLARED data, never a hand-set hub"
);
// The forge's OWN state dir for this drive (the hub's 0600 private half
// lives here, never under /etc), and the endpoint a slirp guest reaches
// its host at. Both are the fixture's documented knobs.
let forge_state = work.join("forge-state");
let _ = std::fs::remove_dir_all(&forge_state);
std::fs::create_dir_all(&forge_state).expect("the drive's forge state dir");
std::env::set_var(wgfixture::STATE_DIR_ENV, &forge_state);
std::env::set_var(wgfixture::ENDPOINT_ENV, "10.0.2.2");
// ── AND THE BOOTSTRAP ADMIN, WITHOUT WHICH NOTHING COMPOSES AT ALL ───
//
// MEASURED 2026-08-26: this drive had been unable to reach the composer
// since `a8ba0665` landed the previous evening, dying at the line below
// with
//
// composing gunnar-installer: installer 'gunnar-installer': required
// parameter 'admin_key' (Bootstrap-admin (kundens OpenSSH-publika
// nyckel)) was not supplied — pass --param admin_key=<value>
//
// — the whole run red at step 3, every rung below it unexercised, for a
// day, unnoticed because this test is `#[ignore]`d and takes twenty
// minutes. The declaration is right and the refusal is right; the drive
// simply had not caught up with them.
//
// A drive mints its OWN pair rather than carrying a committed key, for the
// reason the cell's own comment gives: a committed key would ship an
// appliance that authorizes whoever holds the matching private half. The
// private half is kept beside the staging tree, and `keygen` is this
// crate's ONE key minter (its header records why `ssh-keygen` is
// admissible: gunnar's zero-shell rule is about what the SERVER does).
//
// What the resulting medium can then DO with that key — a real, live,
// authenticated SSH session against the installed disk — is
// `tests/installer_ssh_drive.rs`, the sibling this drive composes with by
// name. Here the key exists so that composition can happen at all.
let admin_key = work.join("admin_ed25519");
gunnar_installtest::keygen(&admin_key).expect("mint this drive's bootstrap admin key");
let fixture_params = vec![
format!("hub_conf={FIXTURE_VALUE}"),
format!("admin_key=@{}", admin_key.with_extension("pub").display()),
];
// ── 3. COMPOSE THE MEDIUM — the same door dwarves' CLI drives ────────
let iso = match std::env::var_os("GUNNAR_INSTALLER_ISO").map(PathBuf::from) {
Some(p) if p.is_file() => {
// Honest bookkeeping: this run did NOT exercise composition, so the
// compose cell must not read green. It is a named skip with the
// path, and the boot rungs below still prove the medium.
eprintln!(
"REUSING an already-composed medium: {} — composition is NOT exercised \
this run (unset GUNNAR_INSTALLER_ISO to compose)",
p.display()
);
row(
"installer_iso_composed",
false,
&format!("SKIPPED: reused {} via GUNNAR_INSTALLER_ISO", p.display()),
);
p
}
_ => {
// The root of trust is armed with the published DEVELOPMENT key, so
// the installed ESP carries a real `korp/app.sig` + `korp/root.pub`
// instead of a payload the start-app would have to wave through.
std::env::set_var(
korp_installer::rootkey::SIGNING_KEY_ENV,
korp_installer::rootkey::DEVELOPMENT_SELECTOR,
);
let composed = match nornir_build_thing::suite_cli::run_installer(
&repo,
PRODUCT,
Some(CELL),
&pf.kernel,
&app,
&fixture_params,
&korp_installer::isocompose::ForgeComposer,
) {
Ok(p) => p,
Err(e) => {
row("installer_iso_composed", false, &format!("{e:#}"));
panic!("composing {CELL}: {e:#}");
}
};
let bytes = std::fs::metadata(&composed)
.unwrap_or_else(|e| {
panic!(
"the composer claimed success but produced no {}: {e}",
composed.display()
)
})
.len();
assert!(
bytes > 8 * 1024 * 1024,
"an installer ISO carrying a kernel and a {} byte payload is not {bytes} bytes",
packed.len()
);
eprintln!("INSTALLER ISO: {} — {bytes} bytes", composed.display());
row(
"installer_iso_composed",
true,
&format!(
"{} ({bytes} bytes), volume_id={}",
composed.display(),
cell.volume_id
),
);
composed
}
};
// ── 3b. THE FIXTURE'S RENDER, read back OFF THE MEDIUM ───────────────
//
// The mint wrote the PUBLIC conf beside the 0600 key under the state dir;
// a fresh dir holds exactly one. The ISO's root-level `hub.conf` (the
// composition stages it on the volume, inspectable without booting) must
// be those bytes exactly — read with the crate's own ISO-9660 reader, no
// mount. A reused medium (GUNNAR_INSTALLER_ISO) carries nobody's fixture.
let fixture: Option<(String, HubState)> = if std::env::var_os("GUNNAR_INSTALLER_ISO").is_some()
{
row(
"fixture_conf_on_iso",
false,
"SKIPPED: reused medium via GUNNAR_INSTALLER_ISO",
);
None
} else {
let hub_dir = forge_state.join(wgfixture::STATE_SUBDIR);
let mut confs: Vec<PathBuf> = std::fs::read_dir(&hub_dir)
.expect("the forge minted into its state dir")
.flatten()
.map(|e| e.path())
.filter(|p| p.extension().is_some_and(|x| x == "conf"))
.collect();
assert_eq!(
confs.len(),
1,
"one mint, one conf under {}: {confs:?}",
hub_dir.display()
);
let conf_path = confs.pop().unwrap();
let render = std::fs::read_to_string(&conf_path).expect("read the fixture's render");
let key_path = conf_path.with_extension("key");
use std::os::unix::fs::PermissionsExt;
let mode = std::fs::metadata(&key_path)
.expect("the 0600 key beside it")
.permissions()
.mode()
& 0o777;
assert_eq!(
mode, 0o600,
"the hub's private half is 0600, under the forge's state dir"
);
assert!(
!render.contains("PrivateKey"),
"the render is PUBLIC text: {render}"
);
let mut vol = korp_installer::iso9660::Iso9660::open(
std::fs::File::open(&iso).expect("open the composed medium"),
)
.expect("the composed medium is ISO-9660");
let on_iso = vol
.read_file("hub.conf")
.expect("the ISO volume carries hub.conf at its root");
assert_eq!(
on_iso,
render.as_bytes(),
"the hub.conf ON THE ISO is the fixture's render, byte for byte"
);
eprintln!(
"FIXTURE hub.conf ({} B) on the ISO == {}:\n{render}",
on_iso.len(),
conf_path.display()
);
row(
"fixture_conf_on_iso",
true,
&format!(
"ISO root hub.conf == fixture render ({} B), key {} mode 0600",
on_iso.len(),
key_path.display()
),
);
// Re-open the hub FROM THE STAGED CONF ALONE — what a proof anywhere
// on this box can do with nothing but the medium and the state dir.
let st = HubState::for_hub_conf(&forge_state, &render)
.expect("the conf names a hub this box minted");
Some((render, st))
};
// ── 4. the two lifetimes are two VALUES — draupnir's refusal, asserted ──
//
// The boots below go through `isoproof` because it owns the multi-disk
// attach order, the GPT/FAT readback and this drive's rollup rows — not,
// any longer, because draupnir cannot attach a virtio medium or a forwarding
// NIC; it can (`MediumAttach::Both`, `with_port_map`). What is taken from
// draupnir here is the RULE that install and boot-installed can never be one
// VM lifetime: it is the one place in the constellation where "I detached
// the ISO" is refused rather than believed.
{
use draupnir::BootSpec;
let disk = work.join("target.img");
let refusal = BootSpec::kvm_boot_installed_disk("collapsed", disk.to_string_lossy())
.with_medium(iso.to_string_lossy())
.validate()
.expect_err("a spec carrying BOTH a medium and boot-installed must be REFUSED");
eprintln!("BY CONSTRUCTION: {refusal}");
assert!(
BootSpec::kvm_install_from_medium("l1", iso.to_string_lossy(), disk.to_string_lossy())
.validate()
.is_ok()
&& BootSpec::kvm_boot_installed_disk("l2", disk.to_string_lossy())
.validate()
.is_ok(),
"the two honest legs must both validate"
);
row(
"two_lifetimes_by_construction",
true,
&format!("draupnir refuses medium+boot-installed in one lifetime: {refusal}"),
);
}
let disk = work.join("target.img");
let disk_arg = disk.to_str().expect("a UTF-8 scratch path").to_string();
// ── 5. LIFETIME 1 — INSTALL onto a blank disk ────────────────────────
let detail = format!(
"{} ({} GiB target, {} payload), medium {}",
cell.appliance_label(),
cell.target_disk_gib,
packed.len(),
iso.display()
);
let installed = IsoInstallProof {
product: PRODUCT,
// The canonical ladder cell (`nornir_testmatrix::appliance::APPLIANCE_PROOFS`).
check: "install_ok",
iso: &iso,
disk: &disk,
// From the CELL, never a second spelling.
disk_bytes: cell.target_disk_bytes(),
disk_args: cell.kvm_disk_args(&disk_arg),
workdir: &work.join("install"),
marker: korp_installer::INSTALL_OK_MARKER,
timeout: Duration::from_secs(600),
expect_serial: &[
// The installer narrated its own steps, so INSTALL-OK is the end of
// a story rather than a line some other code could print.
"INSTALLER: partition",
"INSTALLER: esp",
],
// ── WHAT MUST BE TRUE OF THE INSTALLED DISK, read off its ESP ──
//
// SEEN RED, 2026-08-15, and it is the reason this drive exists. With
// the cell declaring bare `GUNNAR_APPLIANCE=1` the install reached
// INSTALL-OK and wrote this cmdline:
//
// console=ttyS0 korp.appliance=lean korp.entry=gunnar
// korp.znippy=part:korp-rootfs korp.require_signed=1 initrd=\initrd
//
// — the gate, the data root and the argv tail all silently absent,
// because `install::carry_site_tokens` forwards `korp.*` and nothing
// else. The cell now declares the `korp.env.`/`korp.args=` forms and
// THIS list is what holds it to them.
expect_cmdline: &[
"korp.entry=gunnar",
"korp.env.GUNNAR_APPLIANCE=1",
"korp.env.GUNNAR_DATA_ROOT=/data",
"korp.args=serve,--no-grpc",
],
// The fixture's hub.conf must reach P3 VERBATIM as wg/hub.conf — the
// file the first boot's ceremony reads.
expect_data_files: &fixture
.as_ref()
.map(|(render, _)| {
vec![DataFileExpectation {
path: "wg/hub.conf",
bytes: render.as_bytes(),
}]
})
.unwrap_or_default(),
detail: &detail,
}
.run(&pf);
eprintln!("INSTALLED cmdline: {}", installed.cmdline.trim());
if let Some((render, _)) = &fixture {
row(
"fixture_conf_on_p3",
true,
&format!(
"P3 wg/hub.conf == fixture render ({} B), asserted by the install proof",
render.len()
),
);
}
// ── 5b. THE HUB, UP AND OPEN, BEFORE THE APPLIANCE'S FIRST BOOT ─────
//
// The two-phase shape: the appliance's key does not exist yet, so the hub
// is spawned OPEN on the reserved port and LEARNS the key from the first
// initiation that decrypts under its private half. It stays up through
// the reboot below.
let hub = fixture.as_ref().map(|(_, st)| {
let hub = st
.spawn_open()
.expect("bind the reserved hub port (0.0.0.0)");
eprintln!(
"FIXTURE HUB up: 0.0.0.0:{} pub={}",
hub.port(),
hub.public_b64
);
hub
});
// The ESP the installer wrote carries the whole boot chain, by name.
let mut present = Vec::new();
for path in [
korp_installer::esp::FALLBACK_BOOT_PATH,
"vmlinuz",
korp_installer::esp::INITRD_PATH,
isoproof::ESP_CMDLINE_PATH,
korp_installer::esp::APP_PIN_PATH,
korp_installer::esp::APP_SIG_PATH,
korp_installer::esp::ROOT_PUBKEY_PATH,
] {
let bytes = installed
.esp_file(path)
.unwrap_or_else(|| panic!("the installed ESP carries {path}"));
assert!(!bytes.is_empty(), "the installed ESP's {path} is EMPTY");
present.push(format!("{path}({})", bytes.len()));
}
// P2 holds the app .znippy RAW — no filesystem — and the ESP's pin is the
// blake3 of exactly those bytes. A statement about the disk, not the log.
let pin = String::from_utf8_lossy(
&installed
.esp_file(korp_installer::esp::APP_PIN_PATH)
.expect("app.blake3 (asserted above)"),
)
.trim()
.to_string();
let ours = korp_installer::kernel::blake3_of(&app).expect("hash the staged payload");
assert_eq!(
pin, ours,
"the ESP's app pin must be the blake3 of the payload this drive packed"
);
let (p2_first, _p2_last) = kvmproof::gpt_entry(&disk, 1);
let head = kvmproof::read_at(&disk, p2_first * 512, packed.len().min(4096));
assert_eq!(
&head[..],
&packed[..packed.len().min(4096)],
"P2 must hold the app .znippy RAW — the first bytes on the partition are the \
archive's own, with no filesystem in front of them"
);
eprintln!("P1 ESP: {}", present.join(" "));
eprintln!("P2 RAW: app.znippy at LBA {p2_first}, pin {pin} agrees");
row(
"disk_layout",
true,
&format!(
"P1 ESP[{}] + P2 raw znippy pin {}",
present.join(","),
&pin[..16]
),
);
// ── 6. LIFETIME 2 — the medium is GONE; the disk boots and MINTS ─────
let first = FirstBootProof {
product: PRODUCT,
disk: &disk,
disk_args: cell.kvm_disk_args(&disk_arg),
workdir: &work.join("firstboot"),
marker: SERVING_MARKER,
hostfwd: None,
timeout: Duration::from_secs(300),
expect_serial: &["wg-appliance identity: product=gunnar pubkey="],
refuse_serial: &["Kernel panic", "FAILED exec"],
}
.run(&pf);
let key1 = first
.pubkey
.clone()
.expect("the first boot printed the canonical identity line (asserted above)");
// ONE identity per boot — counted as DISTINCT keys, not lines. MEASURED
// 2026-08-22 against gunnar-server v1.1.3-71: the canonical line now
// appears three times on one boot (the app's stdout copy, korp-start's
// echo, and the tracing INFO record), all carrying the SAME key. A second
// MINT would show as a second key, which is the thing this guards.
let announced: std::collections::BTreeSet<String> = first
.serial
.lines()
.filter_map(|l| wg_appliance_core::ceremony::parse_identity_line(l.trim_end()))
.filter(|(p, _)| *p == PRODUCT)
.map(|(_, k)| k.to_string())
.collect();
assert_eq!(
announced.len(),
1,
"exactly one IDENTITY announced per boot ({} identity lines, keys {announced:?})",
isoproof::identity_line_count(&first.serial, PRODUCT)
);
eprintln!(
"FIRST BOOT (no medium): pubkey={key1} ({} identity lines, one key)",
isoproof::identity_line_count(&first.serial, PRODUCT)
);
row(
"identity_minted",
true,
&format!("pubkey={key1} off the INSTALLED disk with the medium absent"),
);
// …and it is on P3, the durable home. The live ISO cannot do this: its
// /data is RAM.
// On an xfs data home (`korp.data.fs=xfs`, gunnar since 1.4.0) this lifetime
// was ended by `child.kill()` and the first boot's writes can still sit in
// the journal; the image readback (`korp_mkfs_xfs::read`) does not replay a
// log. MEASURED 2026-09-15: the first boot narrated `MINTED new keypair at
// /data/wg/identity.key` and the readback found no `wg/identity.pub`. That is
// not a lost key: the REBOOT below mounts P3, the kernel replays the journal,
// and it must present the SAME key with no re-mint — the durability proof
// this row then records. FAT has no journal, so there a miss stays the hard
// failure it always was.
let xfs_home = installed.cmdline.contains("korp.data.fs=xfs");
let durable_by_reboot = match installed
.data_file("wg/identity.pub")
.map(|b| String::from_utf8_lossy(&b).trim().to_string())
{
Some(on_p3) => {
assert_eq!(
on_p3, key1,
"the announced key and wg/identity.pub on P3 disagree"
);
row(
"identity_durable_on_p3",
true,
&format!("wg/identity.pub == {key1}"),
);
false
}
None if xfs_home => {
eprintln!(
"P3 (xfs): wg/identity.pub is not in the checkpointed metadata of a killed \
boot — the reboot's mount (journal replayed) decides durability"
);
true
}
None => {
row(
"identity_durable_on_p3",
false,
"no wg/identity.pub readable on P3",
);
panic!(
"the appliance announced {key1} but no wg/identity.pub is readable on P3 — \
a key that does not outlive the power is the exact defect this shape removes"
)
}
};
// ── 6a. THE TUNNEL ADDRESS, over the sealed box's OWN SSH door (T17) ──
//
// The medium carries the SITE only (edda wgfixture since 14d1472): no
// `[Interface] Address`, because one medium installs every box of a site.
// So the first boot above raised NO tunnel — it had a key and a hub and no
// address — and the hub must have heard nothing. The sealed box runs
// `serve --no-grpc` (asserted on its cmdline above, and kept), so the
// address arrives the one way it can: `gunnar-tunnel-address` over its SSH
// door, as the bootstrap admin this drive staged (`admin_key`). The guest is
// HELD past the serving marker (`kvmproof::boot_disk_until_verdict`) while
// the address is handed over and the hub waits for the handshake the box
// raises at once, with no restart.
let (want_addr, want_prefix) = wgfixture::APPLIANCE_ADDR;
let want = format!("{want_addr}/{want_prefix}");
let handover = hub.as_ref().map(|hub| {
let silent = hub.verdict();
row(
"fixture_hub_silent_without_address",
silent.initiations == 0,
&format!(
"initiations={} before the address was handed over (a site-only medium raises no tunnel)",
silent.initiations
),
);
let port = isoproof::free_port();
let mut args = cell.kvm_disk_args(&disk_arg);
assert!(
!args.iter().any(|a| a.ends_with(".iso") || a == "-cdrom"),
"the hand-over boot must carry NO medium: {args:?}"
);
isoproof::attach_slirp_nic(&mut args, Some((port, 2222)));
let wd = work.join("address");
std::fs::create_dir_all(&wd).expect("the hand-over workdir");
let known = wd.join("known_hosts");
let line = gunnar_installtest::installer_ssh::tunnel_address_line(want_addr, want_prefix);
let said = std::sync::Mutex::new(String::new());
let judged: std::sync::Mutex<Option<Result<gunnar_proto::AssignTunnelAddressResponse, String>>> =
std::sync::Mutex::new(None);
let serial = kvmproof::boot_disk_until_verdict(
&pf.qemu,
&pf.ovmf_code,
&pf.ovmf_vars,
&args,
&wd,
SERVING_MARKER,
Duration::from_secs(300),
|| {
let out = gunnar_installtest::installer_ssh::ssh_exec(&admin_key, &known, port, &line);
let verdict = gunnar_installtest::installer_ssh::judge_tunnel_answer(&out, &want);
*said.lock().expect("said") = out;
match verdict {
None => false,
Some(v) => {
let handed = v.is_ok();
*judged.lock().expect("judged") = Some(v);
if handed {
// The tunnel comes up at the hand-over: hold the
// guest while the hub sees it, not after the kill.
hub.wait_for_sealed(Duration::from_secs(20));
}
true
}
}
},
Duration::from_secs(120),
);
let said = said.into_inner().expect("said");
let judged = judged.into_inner().expect("judged");
let detail = match &judged {
Some(Ok(a)) => format!(
"ssh {line} as the staged admin -> address={} changed={} tunnel_up={} key={}",
a.address, a.changed, a.tunnel_up, a.wg_public_key
),
Some(Err(why)) => format!("REFUSED: {why}"),
None => format!("no answer from the SSH door: {}", said.lines().last().unwrap_or("")),
};
eprintln!("TUNNEL ADDRESS HAND-OVER: {detail}");
let ok = matches!(&judged, Some(Ok(a)) if a.wg_public_key == key1);
row("tunnel_address_over_ssh", ok, &detail);
assert!(
ok,
"the sealed box must take {want} over its SSH door from the staged admin, and name the key it \
minted ({key1}): {detail}\n{}",
tail(&serial, 40)
);
serial
});
// ── 6b. THE HUB'S VERDICT — cryptographic, not a log line ─────────────
//
// `wg::bring_up` (userspace boringtun, compiled into gunnar) read the
// fixture's hub.conf off P3 and, once 6a handed the address over,
// INITIATED toward 10.0.2.2:<port>. The
// fixture counts only initiations whose encrypted static decrypted under
// ITS key; the learned key must be the identity the appliance printed.
// `sealed_packets` is boringtun's first-packet keepalive right after the
// handshake — reported as its own row, because the boot is killed at the
// serving marker and the window between handshake and kill is the
// appliance's, not this drive's to widen.
if let Some(hub) = &hub {
let v = hub.wait_for_sealed(Duration::from_secs(10));
eprintln!("FIXTURE HUB verdict after the address hand-over: {v:?}");
let learned = v
.learned_peer
.map(|k| wg_appliance_core::keys::encode_key(&k));
let detail = format!(
"initiations={} responses={} sealed={} refused={} learned={}",
v.initiations,
v.responses,
v.sealed_packets,
v.refused,
learned.as_deref().unwrap_or("none")
);
row("fixture_hub_initiation", v.initiations >= 1, &detail);
assert!(
v.initiations >= 1 && v.responses >= 1,
"the installed appliance must INITIATE toward the fixture hub it was baked with once its \
address is handed over, and the hub must answer: {detail}\n{}",
tail(handover.as_deref().unwrap_or(&first.serial), 40)
);
row(
"fixture_hub_learned_identity",
learned.as_deref() == Some(key1.as_str()),
&format!(
"learned={} announced={key1}",
learned.as_deref().unwrap_or("none")
),
);
assert_eq!(
learned.as_deref(),
Some(key1.as_str()),
"the key the hub LEARNED must be the identity the appliance announced — an open \
hub still names whom it talked to"
);
row(
"fixture_hub_sealed",
v.sealed_packets >= 1,
&format!(
"sealed_packets={} (boringtun's post-handshake keepalive)",
v.sealed_packets
),
);
} else {
row(
"fixture_hub_initiation",
false,
"SKIPPED: no fixture on a reused medium",
);
}
// ── 7. LIFETIME 3 — POWER CYCLE: same key, and it ANSWERS ────────────
//
// `refuse_serial: ["MINTED"]` is the whole proof. A second mint would
// silently replace the identity the hub already registered, and it would
// look identical to a healthy boot on every other line.
// The gate is the IDENTITY line, not the serving banner, and the split is
// deliberate: on this leg the two have different answers and collapsing
// them into one marker would hide the interesting one. `refuse_serial`
// carries `MINTED` — a second mint would silently replace the identity the
// hub already registered and would look like a healthy boot on every other
// line.
let host_port = isoproof::free_port();
// The gate is the WHOLE identity line for the key the first boot announced,
// not its prefix. MEASURED 2026-09-15: gated on the prefix, the poll matched
// `…pubkey=` and killed the guest mid-line, and the capture ended at
// `pubkey=8RddVlV2` — an 8-character stub compared against the full key, red
// on a boot that had presented the right identity. A re-minted key never
// completes this marker, so it is still red (with the capture), not a skip.
let same_identity_line = format!("{IDENTITY_PREFIX}{key1}");
let again = FirstBootProof {
product: PRODUCT,
disk: &disk,
disk_args: cell.kvm_disk_args(&disk_arg),
// A DIFFERENT workdir: the doors delete and re-read `serial.log`, and a
// shared one would let the previous boot's bytes end this poll.
workdir: &work.join("reboot"),
marker: &same_identity_line,
hostfwd: Some((host_port, 2222)),
timeout: Duration::from_secs(300),
expect_serial: &[],
refuse_serial: &["MINTED"],
}
.run(&pf);
// `isoproof::minted_pubkey` delegates to
// `wg_appliance_core::ceremony::parse_identity_line`, whose regex is
// ANCHORED (`^wg-appliance identity: …$`). On installed metal korp-start
// prefixes the app's stdout with `INSTALLER: `, so the anchored parse
// misses it and only matches the SECOND, unprefixed copy gunnar emits once
// it is further along.
//
// MEASURED 2026-08-15, and it is why this fallback exists rather than an
// `.expect()`: on the first boot both copies appear (`firstboot/serial.log`
// lines 495 and 503) and `minted_pubkey` returns the key; on the reboot
// gunnar dies before the unprefixed copy, so the ONLY identity line on the
// console is `INSTALLER: wg-appliance identity: product=gunnar pubkey=…`
// and `minted_pubkey` returns `None` — the watcher missing the very line it
// exists to watch. Reported as a finding against the shared parser; here
// the line is taken by `contains`, which is what the marker gate above
// already used.
let key2 = again.pubkey.clone().unwrap_or_else(|| {
let line = again
.serial
.lines()
.find(|l| l.contains(IDENTITY_PREFIX))
.unwrap_or_else(|| {
panic!(
"the reboot printed no identity line at all:\n{}",
tail(&again.serial, 40)
)
});
row(
"shared_identity_parser_anchored",
false,
"isoproof::minted_pubkey (wg_appliance_core::parse_identity_line) is ^-anchored \
and MISSED korp-start's `INSTALLER: `-prefixed copy — the only identity line on \
this boot's console",
);
line.split_once(IDENTITY_PREFIX)
.map(|(_, k)| k.trim().to_string())
.expect("the line contains the prefix")
});
assert_eq!(
key2, key1,
"a power cycle must present the SAME identity — the hub has already registered it"
);
eprintln!("REBOOT: same pubkey={key2}, no re-mint");
if let Some(hub) = &hub {
let v = hub.wait_for_initiation(Duration::from_secs(5));
eprintln!("FIXTURE HUB verdict after reboot (cumulative): {v:?}");
}
row(
"identity_survives_reboot",
true,
&format!("same pubkey={key2} after a power cycle, no MINTED line"),
);
if durable_by_reboot {
row(
"identity_durable_on_p3",
true,
&format!(
"xfs: the reboot mounted P3, replayed its journal and presented pubkey={key2} \
with no re-mint"
),
);
}
// …and the DECLARED argv tail really reached the exec. This is the other
// half of the `korp.args=` fix and it is visible on the console verbatim.
//
// ── RENDERED FROM THE DECLARATION, NOT SPELLED ───────────────────────
//
// This line read `exec argv: ["gunnar", "serve", "--no-grpc"]` as a string
// literal, and `a8ba0665` — which appended `--admin-key @/data/admin.pub`
// to the cell's `korp.args=` — made it false the same evening. MEASURED
// 2026-08-26: the console says
//
// exec argv: ["gunnar", "serve", "--no-grpc", "--admin-key", "@/data/admin.pub"]
//
// and the assertion, being a second spelling of the declaration, went red
// on a change that was entirely correct. The lesson is this file's own,
// recorded twice already for `staging` and for the disk size: a test that
// re-types a declared value drifts the moment the declaration moves. So the
// expected argv is RENDERED from `korp.args=` — the same token korp-start
// decodes — and any future edit to the cell's tail carries this line with
// it for free.
let want_argv = {
let tail: Vec<String> = params
.iter()
.find_map(|p| p.strip_prefix("korp.args="))
.expect("the cell declares korp.args= (asserted at the top of this drive)")
.split(',')
.map(|a| format!("{a:?}"))
.collect();
format!("exec argv: [{:?}, {}]", entry, tail.join(", "))
};
// Read off either lifetime of THIS installed disk. The reboot above is gated
// on the identity line, which korp-start prints BEFORE `exec argv:` —
// MEASURED 2026-09-15, so that capture legitimately ends first. The first
// boot ran to SERVING on the same disk and cannot have served without the
// exec line, so it is the capture that must carry it.
assert!(
first.serial.contains(&want_argv) || again.serial.contains(&want_argv),
"korp-start must exec the DECLARED argv tail, not a bare `{entry}`. Expected to see \
`{want_argv}` on the first boot or the reboot:\n{}",
tail(&first.serial, 40)
);
row(
"declared_argv_reaches_exec",
true,
&format!("korp-start {want_argv}, rendered from the cell's own korp.args="),
);
// ── 8. THE WALL, MEASURED — a FAT data home does not survive a power cut ──
//
// The reboot keeps its identity (above) and then does NOT serve. Verbatim,
// 2026-08-15:
//
// [ 2.984613] FAT-fs (vda3): error, corrupted directory (invalid entries)
// gunnar: creating the repository root /data/repos: I/O error (os error 5)
// Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000100
//
// P3 is FAT — chosen so the pure-Rust installer can format and populate it
// with no kernel filesystem tooling — and FAT has no journal. The previous
// lifetime was ended by `kvmproof`'s boot doors, which `child.kill()` QEMU
// the instant the marker lands: no ACPI shutdown, no unmount, no sync. So
// the directory the first boot wrote is left half-committed, the kernel's
// vfat driver refuses it on the next mount, gunnar cannot create
// `/data/repos`, PID 1 exits and the kernel panics.
//
// Two separate things, and they must not be conflated:
// * TEST-SIDE: the harness has no clean-shutdown door. `boot_disk_until`
// and `boot_disk_until_verdict` both end in `child.kill()`. A
// `-device virtio-serial`/QMP `system_powerdown` + wait is what a
// durability leg needs, and it belongs in `korp_installer::kvmproof`
// (another repository), not in a hand-rolled boot here.
// * PRODUCT-SIDE: a real appliance loses power. On this shape that leaves
// an unbootable machine rather than a degraded one, because nothing
// fscks or repairs P3 and gunnar correctly refuses to serve on I/O
// errors. That is a durability decision for the data home, not a bug in
// this drive.
// On an xfs home lifetime 3 above is gated on the IDENTITY line and killed
// there, before gunnar can print its serving banner: MEASURED 2026-09-15 the
// reboot replayed the xfs journal cleanly, kept the key, reached gunnar's
// start-up lines and was killed — so asking that capture for the serving
// banner could never pass, whatever the appliance did. The question is
// asked of its own lifetime instead: power-cut again (lifetime 3 was
// killed too), boot until SERVING, no re-mint. A boot that never serves is
// a red with the capture, not a skip. FAT keeps the measured judgement.
let serves_after_cut = if xfs_home {
let served = FirstBootProof {
product: PRODUCT,
disk: &disk,
disk_args: cell.kvm_disk_args(&disk_arg),
workdir: &work.join("serve-after-cut"),
marker: SERVING_MARKER,
hostfwd: Some((isoproof::free_port(), 2222)),
timeout: Duration::from_secs(300),
expect_serial: &[],
refuse_serial: &["MINTED"],
}
.run(&pf);
served.serial.contains(SERVING_MARKER)
} else {
again.serial.contains(SERVING_MARKER)
};
let detail = if serves_after_cut {
"the reboot served".to_string()
} else {
format!(
"RED (measured): the reboot kept pubkey={key2} and did NOT serve — {}",
first_line_matching(&again.serial, &["FAT-fs", "Kernel panic"])
)
};
row("serves_after_power_cut", serves_after_cut, &detail);
eprintln!("REBOOT SERVES? {serves_after_cut} — {detail}");
// ── 8. THE WALL, NAMED ───────────────────────────────────────────────
//
// `boot_disk_until` KILLS the guest the moment the marker lands, so the
// hostfwd above is open only while the VM lives — there is no window in
// which this process can open a TCP connection to it. `kvmproof` has the
// door that would allow it (`boot_disk_until_verdict`, which HOLDS the
// guest while a caller's closure forms a verdict) but `FirstBootProof` does
// not expose it, and reaching around the harness to hand-roll the boot
// would be the twin this file exists not to be.
//
// ── BOTH HALVES OF THAT HAVE MOVED. CORRECTED 2026-08-26 ─────────────
//
// (a) `kvmproof::boot_disk_until_verdict` is PUBLIC and is exactly the
// held-open door this paragraph asked for. `tests/installer_ssh_drive.rs`
// calls it and reaches the appliance over a live `hostfwd`.
//
// (b) said the bootstrap channel "is a [[build.installer.params]] staged
// file that nornir-build-thing 0.1.5 — gunnar's own pin — would silently
// drop." Both clauses are FALSE and were checkable when written:
//
// * the pin is 0.1.7, not 0.1.5 — `crates/gunnar-server/Cargo.toml:334`
// and `crates/gunnar-build-thing/Cargo.toml:65` both say
// `version = "0.1.7"`, and `Cargo.lock:4350` resolves 0.1.7;
// * 0.1.7's `InstallerBuild` HAS `params` (and `env`, and `cmd`).
//
// Nothing was ever dropped. `a8ba0665` declared `admin_key` through that
// very channel and this drive supplies it thirty lines above; the
// installer narrates `payload admin.pub materialized off the ISO volume
// (100 bytes)` and it lands on P3 as `/data/admin.pub`.
//
// What remains true is the SECOND sentence and only it: a freshly installed
// P3 has no namespace, no repository and no principal, because the installer
// formats it fresh — and with `--no-grpc` there is no channel that could
// create one. `gunnar-ssh/src/command.rs` admits three verbs
// (`git-upload-pack`, `git-receive-pack`, `gunnar_lfs::TRANSFER_VERB`) and
// `shell_request`/`subsystem_request` both `refuse(...)` by construction, so
// the bootstrap admin the medium stages can AUTHENTICATE and then has no verb
// with which to create the first repository. A push is therefore correctly
// refused, and this rung stays unproven on this form — for that reason and
// no longer for a pin that moved.
row(
"serves_git_through_installed_disk",
false,
"SKIPPED BY NAME, one reason (the other two are FIXED): a freshly installed P3 is \
empty — no namespace, no repository, no principal — and `--no-grpc` leaves no verb \
that could create one, since gunnar's SSH admits git-upload-pack / git-receive-pack \
/ lfs-transfer and refuses shell and subsystem by construction. The held-open window \
now exists (kvmproof::boot_disk_until_verdict) and the staged-file bootstrap channel \
works (admin_key -> /data/admin.pub): tests/installer_ssh_drive.rs opens an \
AUTHENTICATED session on the installed disk with it. The exe and container forms DO \
prove push+clone against object graphs.",
);
eprintln!(
"SKIP {PROOF}/serves_git_through_installed_disk: the installed appliance AUTHENTICATES \
the staged admin key (see installer_ssh_drive) but a fresh P3 has no repository and \
--no-grpc leaves no verb to create one — git push/clone THROUGH the installed \
appliance is NOT proven on this form ({FORM_ISO_KVM})"
);
// The exit code must agree with the rows. Everything above is emitted
// first, so the matrix carries the whole truth even though this ends red.
assert!(
serves_after_cut,
"WALL (measured 2026-08-15): the installed appliance keeps its identity across a \
power cut but does NOT come back up — P3's FAT directory is corrupt after the \
previous lifetime was SIGKILLed, gunnar refuses on the I/O error and PID 1 exits. \
Fixing it needs a clean-shutdown door in korp_installer::kvmproof (ACPI \
system_powerdown instead of child.kill()) and a durability decision for the FAT \
data home. Every other rung of this drive is GREEN.\n{}",
tail(&again.serial, 25)
);
}
/// The last `n` lines of a capture.
fn tail(s: &str, n: usize) -> String {
let lines: Vec<&str> = s.lines().collect();
lines[lines.len().saturating_sub(n)..].join("\n")
}
/// The first line containing any of `needles` — the applied output a report
/// quotes, rather than a summary of it.
fn first_line_matching(s: &str, needles: &[&str]) -> String {
s.lines()
.find(|l| needles.iter().any(|n| l.contains(n)))
.map(|l| l.trim().to_string())
.unwrap_or_else(|| "no matching line".to_string())
}