installtest/tests/installer_ssh_drive.rs
//! **The installed appliance is REACHED: a live SSH session, authenticated by
//! the admin key the medium was composed with — one guest, then a fleet.**
//!
//! `tests/installer_drive.rs` proves the medium installs, that the disk boots
//! with the medium gone, and that the identity survives a power cycle. It ends
//! at a rung it names and does not clear:
//!
//! ```text
//! serves_git_through_installed_disk SKIPPED BY NAME:
//! (a) FirstBootProof kills the guest at the marker, so the hostfwd has no
//! open window — kvmproof::boot_disk_until_verdict is the door and the
//! harness does not expose it;
//! (b) a freshly installed P3 is empty …, and the bootstrap channel is a
//! [[build.installer.params]] staged file that nornir-build-thing 0.1.5
//! — gunnar's own pin — would silently drop.
//! ```
//!
//! Both halves have moved since that comment was written, and this file is what
//! moves with them.
//!
//! * **(a) is a door, and the door is public.**
//! [`korp_installer::kvmproof::boot_disk_until_verdict`] holds the guest
//! RUNNING after the marker until a caller's closure forms a verdict. That is
//! exactly one layer below `FirstBootProof` — the same qemu argv, the same
//! serial poll, the same OVMF staging — so reaching for it is reuse, not the
//! hand-rolled boot the sibling file exists not to be. The verdict this drive
//! forms inside the hold is a TCP connection and a real `ssh` userauth.
//!
//! * **(b) is STALE, and that is a doc claim this drive found false.**
//! `a8ba0665` added `[[build.installer.params]] admin_key`, `required = true`,
//! `staged_file = "admin.pub"`, and `installer_drive.rs` itself reads
//! `cell.params` for `hub_conf` twenty lines earlier. The staged-file channel
//! is not hypothetical and is not dropped: the composer writes the value onto
//! the ISO volume flat and `install.rs` stages it onto P3, where the declared
//! `korp.args=serve,--no-grpc,--admin-key,@/data/admin.pub` makes
//! `AuthTable::prepare_to_serve` install it as the bootstrap admin on the
//! first boot. What (b) says cannot be declared has been declared for eleven
//! days.
//!
//! # What "SSH in" can mean here, exactly
//!
//! gunnar has **no shell, ever** — `gunnar_ssh::server::Handler::shell_request`
//! and `subsystem_request` both answer `refuse(...)` by construction, and
//! `command.rs` admits three verbs and no others (`git-upload-pack`,
//! `git-receive-pack`, `gunnar_lfs::TRANSFER_VERB`). So an interactive prompt
//! on an installed gunnar is not a thing this drive failed to reach; it is a
//! thing the product refuses on purpose. The reachable, and load-bearing,
//! meaning of "SSH in" is therefore:
//!
//! ```text
//! 1 the listener on the guest is gunnar's own SSH-2.0-gunnar_<ver>
//! 2 the STAGED admin key passes userauth a session is opened
//! 3 the server answers IN ITS OWN VOICE "repository not found,
//! or you do not have
//! access: demo/hello"
//! 4 a stranger's key does NOT pass userauth Permission denied
//! (publickey)
//! ```
//!
//! 3 and 4 are the whole proof and they must be read as a PAIR. Rung 3 alone is
//! not enough — `DispatchError::client_message` deliberately fuses "no such
//! repository" and "you do not have access" into one string so the wire cannot
//! be used to enumerate private repositories, so that sentence says nothing
//! about *who* asked. Rung 4 is what makes rung 3 mean something: the same
//! sentence is unreachable for a key the appliance was not composed with,
//! because that key dies one protocol layer earlier, at userauth, before any
//! channel exists. Two different layers, two different failures, and only the
//! composed key crosses both.
//!
//! # The three traps, and where each is paid
//!
//! 1. **the medium must be virtio** — `IsoInstallProof::run` calls
//! `isoproof::attach_iso_as_virtio` itself; nothing here says `-cdrom`.
//! 2. **detach the medium after installing** — the boot legs take
//! `cell.kvm_disk_args(&disk)` and NOTHING else. The medium is not merely
//! later in the boot order, it is not attached, so the 35-reinstall loop is
//! unreachable rather than merely unlikely.
//! 3. **it is unattended and picks its own disk** — every disk this file names
//! is created by `IsoInstallProof` under [`SCRATCH`], per-guest.
//!
//! # Scale
//!
//! `the_fleet_…` composes ONE medium and installs it onto N blank disks, then
//! boots all N at once with N distinct `hostfwd` ports and SSHs into each. The
//! defects that only appear at N are what it is for: a host key reused across
//! guests (every appliance mints on P3, so N guests must present N DISTINCT
//! host keys — a shared one would mean the identity came off the medium, not
//! out of the ceremony), a DHCP collision, and the host's own memory.
//!
//! ```bash
//! CARGO_TARGET_DIR=~/scratch/cargo/iso2ssh-lane \
//! KORP_TEST_KERNEL=~/scratch/bzImage \
//! GUNNAR_EXE_TARGET_DIR=~/scratch/cargo/iso2ssh-lane \
//! cargo test --manifest-path installtest/Cargo.toml --test installer_ssh_drive \
//! -- --ignored --nocapture
//! ```
//!
//! `GUNNAR_SSH_FLEET=<n>` sets the fleet size (default 4).
//! `GUNNAR_INSTALLER_ISO=<path>` reuses a composed medium — and then the admin
//! key must be the one it was composed WITH, so the reuse path also takes
//! `GUNNAR_ADMIN_KEY=<private key path>` and says so rather than composing a
//! key the medium never saw and reporting the refusal as a product defect.
//! ## Where the mechanism lives
//!
//! In `gunnar_installtest::installer_ssh`, not here. This file is the PROOF —
//! it asserts and it emits the rollup rows — and `bench/examples/nornir-bench.rs`'s
//! `gunnar.installer_ssh` arm is the MEASUREMENT over the identical six
//! operations. Two copies of "compose, install, boot, authenticate" would drift
//! the first time one of them learned something, and this crate already keeps
//! `iso.rs` / `kvm.rs` / `container.rs` single for exactly that reason.
use std::time::Duration;
use gunnar_installtest::installer_ssh::{
row, Context, Guest, Verdict, SERVER_VOICE, USERAUTH_REFUSED,
};
// ── ONE GUEST ───────────────────────────────────────────────────────────────
#[test]
#[ignore = "heavy: composes the gunnar-installer medium, installs it to a disk under KVM, boots the \
installed disk with the medium DETACHED, and SSHs into it"]
fn an_installed_gunnar_answers_an_authenticated_ssh_session_for_the_staged_admin_key() {
let Some(ctx) = Context::open("one") else {
return;
};
let Some(medium) = ctx.compose() else { return };
let guest = ctx.install_one(&medium, 0);
let verdict = ctx.boot_and_reach(&guest, Duration::from_secs(300));
verdict.print_exchange();
verdict.assert_reached(&guest.name);
row(
"serves_ssh_through_installed_disk",
true,
&format!(
"one guest: banner {:?}, the STAGED admin key passed userauth and the server answered \
in its own voice ({SERVER_VOICE:?}); a stranger key was refused at userauth \
({USERAUTH_REFUSED:?}). boot→banner {} ms, boot→authenticated {} ms",
verdict.banner.as_deref().unwrap_or(""),
verdict.banner_ms.unwrap_or_default(),
verdict.auth_ms.unwrap_or_default(),
),
);
eprintln!(
"\nBENCH gunnar.installer_ssh (n=1) {}",
verdict.bench_line(&medium, &guest)
);
}
// ── THE FIRST-PUSH DOOR ─────────────────────────────────────────────────────
/// The sibling test above ends where the 2026-08-25 measurement ended: the
/// staged admin key opens a session and is told "repository not found" — the
/// enrolled brick. This is the unbricking, on the installed appliance itself:
/// the admin's first `git push` names a repository no control plane ever
/// created, and the push creates it (`gunnar_ssh::Control::create_for_push`).
///
/// The ladder, each rung asserted by `PushVerdict::assert_created`:
///
/// ```text
/// A admin READ of demo/hello still the fused refusal (reads never create)
/// B admin FIRST PUSH nordisk/first succeeds — the door
/// C ls-remote refs/heads/main is really there
/// D SECOND push the ordinary resolve now serves it
/// E stranger push Permission denied (publickey), one layer down
/// ```
///
/// A and E are what keep the door honest: it must change nothing for reads
/// and nothing for any other key, or it is not the smallest door.
#[test]
#[ignore = "heavy: composes the gunnar-installer medium, installs it to a disk under KVM, boots the \
installed disk with the medium DETACHED, and pushes the first repository into it"]
fn the_admins_first_push_creates_the_first_repository_on_an_installed_gunnar() {
let Some(ctx) = Context::open("door") else {
return;
};
let Some(medium) = ctx.compose() else { return };
let guest = ctx.install_one(&medium, 0);
let verdict = ctx.boot_and_first_push(&guest, Duration::from_secs(300));
verdict.print_exchange();
verdict.assert_created(&guest.name);
row(
"first_push_creates_repository",
true,
&format!(
"one guest: admin READ of an absent repo still refused ({SERVER_VOICE:?}); the \
admin's FIRST git push created {}; the ref listed back; a second push served; a \
stranger still refused at userauth ({USERAUTH_REFUSED:?})",
gunnar_installtest::installer_ssh::DOOR_REPO
),
);
}
// ── A FLEET ─────────────────────────────────────────────────────────────────
#[test]
#[ignore = "heavy: installs the SAME medium onto N blank disks and boots all N at once, SSHing into \
each — the scale leg"]
fn a_fleet_of_installed_gunnars_each_answer_ssh_with_a_distinct_host_identity() {
let n: usize = std::env::var("GUNNAR_SSH_FLEET")
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(4);
assert!(
n >= 2,
"a fleet leg with n={n} is the one-guest leg wearing a hat"
);
let Some(ctx) = Context::open("fleet") else {
return;
};
let Some(medium) = ctx.compose() else { return };
// The installs are SEQUENTIAL and the boots are CONCURRENT, deliberately.
// An install is one guest writing 64 GiB of GPT with nothing to collide
// with; N of them in parallel would measure this box's page cache. The
// interesting collisions — a reused host key, a DHCP answer taken by the
// wrong guest, N appliances minting at once — all live on the BOOT leg, so
// that is the leg that runs wide.
let guests: Vec<Guest> = (0..n).map(|i| ctx.install_one(&medium, i)).collect();
eprintln!(
"FLEET: {n} disks installed from ONE medium ({})",
medium.iso.display()
);
let verdicts: Vec<Verdict> = std::thread::scope(|s| {
let handles: Vec<_> = guests
.iter()
.map(|g| s.spawn(|| ctx.boot_and_reach(g, Duration::from_secs(600))))
.collect();
handles
.into_iter()
.map(|h| h.join().expect("a boot thread"))
.collect()
});
// Every guest, named, before any assert — a fleet that fails on guest 3
// must still report guests 0..2.
for (g, v) in guests.iter().zip(&verdicts) {
eprintln!(" {} → {}", g.name, v.summary());
}
// One guest's exchange in full. N copies would bury the fleet verdict, and
// the per-guest booleans above already say whether the others agreed.
if let Some(v) = verdicts.first() {
v.print_exchange();
}
let reached: Vec<&Verdict> = verdicts.iter().filter(|v| v.authenticated).collect();
// ── the defect a fleet is FOR: N guests, N DISTINCT host keys ──
//
// Every appliance mints its SSH host identity into its own /data on P3 at
// first boot. If two guests presented the same host key the identity would
// have come off the MEDIUM — one key mailed to every customer — and that
// would look perfectly healthy on a single-guest run. It is only visible at
// N, which is why it is here and not in the one-guest leg.
let keys: std::collections::BTreeSet<String> =
verdicts.iter().filter_map(|v| v.host_key.clone()).collect();
let detail = format!(
"{}/{n} guests authenticated; {} distinct SSH host keys across {} that presented one",
reached.len(),
keys.len(),
verdicts.iter().filter(|v| v.host_key.is_some()).count()
);
let distinct_ok = keys.len() == verdicts.iter().filter(|v| v.host_key.is_some()).count();
row(
"fleet_distinct_host_keys",
distinct_ok && reached.len() == n,
&detail,
);
eprintln!("FLEET VERDICT: {detail}");
for (g, v) in guests.iter().zip(&verdicts) {
v.assert_reached(&g.name);
}
assert!(
distinct_ok,
"N installed appliances must present N DISTINCT SSH host keys — a shared one means the \
identity rode the medium instead of being minted on P3: {keys:?}"
);
let slowest = verdicts
.iter()
.filter_map(|v| v.auth_ms)
.max()
.unwrap_or_default();
let fastest = verdicts
.iter()
.filter_map(|v| v.auth_ms)
.min()
.unwrap_or_default();
eprintln!(
"\nBENCH gunnar.installer_ssh (n={n}) cold boot → authenticated SSH: fastest {fastest} ms, \
slowest {slowest} ms, medium {} B",
medium.bytes
);
}