installtest/src/kvm.rs
//! The **KVM** arm: install gunnar into a real machine and prove it serves.
//!
//! This is the arm the container one cannot be. A container has no init and no
//! service manager, so "install" there means `ENTRYPOINT`. A KVM guest boots
//! systemd as PID 1 with a real `org.freedesktop.systemd1` on the system bus,
//! which is the only place Skidbladnir's installer, the one gunnar already
//! depends on (`crates/gunnar-cli/src/service.rs`), can actually do
//! its job. So this arm installs gunnar **as a systemd unit**,
//! through gunnar's own `service install` / `service start` / `service status`
//! verbs, and only then runs the push/clone/object-graph assertion from the host.
//!
//! # How this arm boots: draupnir, which drives tunnr
//!
//! The container arm goes through draupnir because draupnir owns the one OCI
//! engine in the constellation. This arm goes through draupnir for the same
//! reason — it owns the one KVM seam — and since 2026-09-05 it can, because the
//! two gaps that used to make that impossible are closed. Both were real, both
//! were measured here, and the measurements are the part worth keeping:
//!
//! **1. draupnir could not express a host→guest port forward.**
//! `KvmBoot::to_tunnr_spec` only ever mapped `NetMode::None` onto
//! `tunnr_vm::NetConfig::None`, and `BootSpec::validate` REJECTED
//! `ports`/`port_maps` outright on any non-container backend, on the grounds
//! that they were container vocabulary. tunnr itself had modelled the thing all
//! along (`NetConfig::User { hostfwd }`, rendered `hostfwd=tcp::<host>-:<guest>`),
//! and a forwarded port is the entire mechanism by which a host process reaches
//! a service inside the guest — so this arm, whose whole subject is "reach the
//! installed gunnar from the host", coded straight against `tunnr_vm::boot_test`
//! and said so here. draupnir `60ffb06` honours both forms on `Backend::Kvm`,
//! folds them onto tunnr's pair list exactly as the container backend folds
//! them, and refuses an airgap (`NetMode::None`) combined with a forward BY
//! NAME rather than resolving it in silence. [`crate::free_port`] picks the host
//! side; `.with_port_map(host, guest)` is the whole wiring now.
//!
//! **2. draupnir attached a medium only as `-cdrom`,** an IDE tray. That never
//! bit *this* arm — it installs into a stock cloud image whose kernel has ATA —
//! but it is why the sibling installer drive still boots through
//! `korp_installer::isoproof`: the appliance kernel is the `nuked` preset and
//! vetoes ATA, so the medium the firmware booted from was invisible to the PID 1
//! it started. MEASURED 2026-08-15: `INSTALLER: volume: no block device carries
//! the ISO-9660 magic (CD001) … (probed: loop1 … vda)`, no `sr0` anywhere.
//! draupnir `519afcf` adds `MediumAttach::{Cdrom, Virtio, Both}` with the virtio
//! argv copied verbatim from `isoproof::attach_iso_as_virtio`. This arm names no
//! medium at all, so it takes no attachment: the guest is a disk, not an ISO.
//!
//! **What is still tunnr's, directly.** `tunnr_vm::host_capabilities()`, the
//! /dev/kvm + QEMU preflight below. draupnir does not re-export it, and running
//! without it means QEMU silently falls back to TCG and every duration this arm
//! reports becomes a statement about an emulator.
//!
//! **One measured gap, stated plainly:** before this arm existed, tunnr's
//! `hostfwd` had never been exercised end to end. Its only coverage was an argv
//! string assertion (`tunnr/src/boot_primitive.rs:1304`). Nothing in the
//! constellation had ever booted a guest and connected to a forwarded port from a
//! host process. That is still what this arm is worth: it is now the live proof
//! for draupnir's `with_port_map` as well as for tunnr's `hostfwd`.
//!
//! # Shutting the guest down, and why it is `shutdown` and not `power_off`
//!
//! [`Lifecycle::power_off`] is an immediate SIGKILL to QEMU. QEMU's raw drive is
//! `cache=writeback`, so guest writes sit in QEMU's memory and a SIGKILL throws
//! them away — MEASURED 2026-08-16 on gunnar: INSTALL-OK, the payload streamed
//! onto p2 with one blake3, SIGKILL 4 s later, and the second boot hashed those
//! same bytes to a *different* blake3. So the teardown here is
//! `KvmBoot::shutdown(grace)`, which does the monitor/ACPI power-down first and
//! only then kills. A power cut is a thing an arm may want deliberately (the ISO
//! arm's [`crate::iso::restart`] wants exactly that), but it must be asked for,
//! never arrived at by default.
//!
//! # The libc floor, and why the guest image is Ubuntu 26.04
//!
//! The artefact installed here is the **same host-built binary** the container arm
//! installs — one artefact, two install targets, so a difference between the arms
//! cannot be a difference between two binaries. That binary is dynamically linked
//! against the host's glibc and needs `GLIBC_2.38`, so the guest must not be
//! older. Debian 12 (glibc 2.36) is on this box and would fail at exec with a
//! message that reads like a broken install; Ubuntu 26.04 is the host's own
//! release and is the default here for that reason. Override with
//! `GUNNAR_INSTALLTEST_VM_IMAGE` and keep the floor in mind.
use std::path::{Path, PathBuf};
use std::process::Command;
use std::time::{Duration, Instant};
use draupnir::kvm::KvmBoot;
use draupnir::{BootSpec, CloudInit, Lifecycle, Machine, PowerState};
use crate::subject::Subject;
use crate::{keygen, wait_for_ssh, ArmResult, Exchange, Fault, Probe};
/// The guest that gets installed into. See the libc note in the module header.
pub fn base_image() -> PathBuf {
match std::env::var("GUNNAR_INSTALLTEST_VM_IMAGE") {
Ok(v) => PathBuf::from(v),
Err(_) => {
crate::expand_home("~/scratch/skidbladnir/isos/ubuntu-26.04-server-cloudimg-amd64.img")
}
}
}
/// The administrative account cloud-init creates, and that this harness drives
/// the install over. Not `ubuntu`/`debian`: the account is this test's, so its
/// presence or absence is unambiguous.
const ADMIN: &str = "gunnaradm";
/// Where gunnar is installed inside the guest. `service install` defaults its
/// `ExecStart` to exactly this path.
const EXEC: &str = "/usr/local/bin/gunnar";
/// Where the served repositories live inside the guest.
const DATA_ROOT: &str = "/var/lib/gunnar";
/// gunnar's SSH listener inside the guest.
const IN_GUEST_GIT: u16 = 2222;
/// How long the guest gets to boot and finish cloud-init. Ubuntu cloudimg on
/// this box reaches sshd well inside this; the budget is generous because a
/// timeout here should mean "it never came up", never "the box was busy".
fn boot_budget() -> Duration {
Duration::from_secs(
std::env::var("GUNNAR_INSTALLTEST_VM_BOOT_SECS")
.ok()
.and_then(|s| s.parse().ok())
.unwrap_or(420),
)
}
// ── ssh into the guest ──────────────────────────────────────────────────────
/// A shell inside the guest, over the forwarded port.
///
/// Every call returns the **real** exit status and the real stdout/stderr of the
/// command that ran in the guest. Nothing here reports success on a command that
/// did not run: `ssh` exits 255 when it could not connect at all, which is a
/// different failure from the remote command's own non-zero exit, and both come
/// back as `Err` carrying the text.
pub struct GuestShell {
pub port: u16,
pub key: PathBuf,
}
impl GuestShell {
fn ssh_args(&self) -> Vec<String> {
vec![
"-i".into(),
self.key.display().to_string(),
"-o".into(),
"IdentitiesOnly=yes".into(),
"-o".into(),
"StrictHostKeyChecking=no".into(),
"-o".into(),
"UserKnownHostsFile=/dev/null".into(),
"-o".into(),
"LogLevel=ERROR".into(),
"-o".into(),
"BatchMode=yes".into(),
"-o".into(),
"ConnectTimeout=10".into(),
]
}
/// Run one command in the guest. `Ok(stdout)` only on exit 0.
pub fn run(&self, command: &str) -> Result<String, String> {
let out = Command::new("ssh")
.args(self.ssh_args())
.args(["-p", &self.port.to_string()])
.arg(format!("{ADMIN}@127.0.0.1"))
.arg(command)
.output()
.map_err(|e| format!("ssh: {e}"))?;
let stdout = String::from_utf8_lossy(&out.stdout).into_owned();
let stderr = String::from_utf8_lossy(&out.stderr).into_owned();
if !out.status.success() {
return Err(format!(
"in the guest, `{command}` exited {}\n stdout: {}\n stderr: {}",
out.status,
stdout.trim(),
stderr.trim()
));
}
Ok(stdout)
}
/// Run a command and hand back its status and output whatever they were.
/// For the places where a **non-zero is the expected answer** and swallowing
/// it into an `Err` would lose the evidence.
pub fn try_run(&self, command: &str) -> Result<(i32, String), String> {
let out = Command::new("ssh")
.args(self.ssh_args())
.args(["-p", &self.port.to_string()])
.arg(format!("{ADMIN}@127.0.0.1"))
.arg(command)
.output()
.map_err(|e| format!("ssh: {e}"))?;
let text = format!(
"{}{}",
String::from_utf8_lossy(&out.stdout),
String::from_utf8_lossy(&out.stderr)
);
Ok((out.status.code().unwrap_or(-1), text))
}
/// Copy a host file into the guest.
pub fn put(&self, local: &Path, remote: &str) -> Result<(), String> {
let out = Command::new("scp")
.args(self.ssh_args())
.args(["-P", &self.port.to_string()])
.arg(local)
.arg(format!("{ADMIN}@127.0.0.1:{remote}"))
.output()
.map_err(|e| format!("scp: {e}"))?;
if !out.status.success() {
return Err(format!(
"scp {} → {remote} failed: {}",
local.display(),
String::from_utf8_lossy(&out.stderr).trim()
));
}
Ok(())
}
/// Block until the guest answers a command. Port 22 accepting is not enough.
///
/// sshd accepting TCP and cloud-init having created the account are minutes
/// apart on a first boot, and a harness that stops at "the port is open"
/// would then fail its first real command and blame the install.
pub fn wait_usable(
&self,
budget: Duration,
alive: impl Fn() -> Result<(), String>,
console: impl Fn() -> String,
) -> Result<Duration, String> {
let started = Instant::now();
let deadline = started + budget;
let mut last;
loop {
alive()?;
match self.run("true") {
Ok(_) => return Ok(started.elapsed()),
Err(e) => last = e,
}
if Instant::now() >= deadline {
// The serial console comes with the failure, always. Without it
// "the guest never answered" is unactionable: a kernel panic, a
// cloud-init that never ran, and an sshd that never started all
// look identical from out here.
return Err(format!(
"the guest never accepted an ssh command within {budget:?}; last: {last}\n\
── serial console, last 30 lines ──\n{}",
tail_lines(&console(), 30)
));
}
std::thread::sleep(Duration::from_secs(2));
}
}
}
// ── the disk ────────────────────────────────────────────────────────────────
/// The four magic bytes at the head of a QCOW2 file: `QFI\xfb`.
const QCOW2_MAGIC: [u8; 4] = [0x51, 0x46, 0x49, 0xfb];
/// Read the base image's real format out of its first four bytes.
///
/// **This exists because of a bug found while writing this arm**, and it is worth
/// stating rather than hiding behind a working call. tunnr decides the QEMU
/// `format=` from the **file extension**:
///
/// ```text
/// let fmt = if disk.ends_with(".qcow2") { "qcow2" } else { "raw" };
/// // tunnr/src/boot_primitive.rs:741
/// ```
///
/// Every Ubuntu cloud image on this box is a QCOW2 file named `.img`. Handed to
/// tunnr directly it is opened as `raw`, OVMF finds nothing bootable, and the
/// guest produces **no console output at all**. The failure is a silent 420 s
/// timeout with an empty serial log, which reads exactly like "the network is
/// broken". Measured here on 2026-08-03.
///
/// So the format is taken from the file, and the overlay this arm builds is
/// named `.qcow2` so that tunnr's extension test also agrees with it. Two things
/// have to be wrong at once for the disk to be misread now.
fn disk_format(base: &Path) -> Result<&'static str, String> {
use std::io::Read as _;
let mut head = [0u8; 4];
let mut f = std::fs::File::open(base).map_err(|e| format!("{}: {e}", base.display()))?;
f.read_exact(&mut head)
.map_err(|e| format!("reading the head of {}: {e}", base.display()))?;
Ok(if head == QCOW2_MAGIC { "qcow2" } else { "raw" })
}
/// How much room the overlay adds on top of the base image's virtual size.
///
/// **This is not a round number picked for comfort, it is the floor gunnar
/// enforces.** `gunnar serve` refuses a push when the data root has less than
/// `--disk-floor 5%,2.0 GiB` free, and the Ubuntu 26.04 cloudimg is a 3.5 GiB
/// virtual disk with the OS already on it. MEASURED 2026-09-05 on this box, on
/// BOTH the pre-draupnir and post-draupnir boot paths, byte-identical:
///
/// ```text
/// fatal: remote error: /var/lib/gunnar has 296.9 MiB free, under the 2.0 GiB
/// floor (--disk-floor 5%,2.0 GiB): no new repository and no push until space
/// is reclaimed
/// ```
///
/// So the arm installed gunnar perfectly and then failed its own git exchange,
/// on a guest that could never have passed. Growing the *overlay* (never the
/// shared base) and letting cloud-init's `growpart` extend the root partition at
/// first boot is what makes the exchange a statement about gunnar rather than
/// about the size of a stock cloud image. 8 GiB leaves ~8.3 GiB free, which
/// clears both halves of the floor with room for the fixture to grow.
const OVERLAY_HEADROOM: &str = "+8G";
/// A throwaway copy-on-write overlay over the base image, grown to a size the
/// guest can actually serve from.
///
/// The guest gets a disk it may write to freely. The base is a shared, expensive
/// artefact several suites on this box depend on, and is never opened for writing.
/// An overlay is also instant, where copying 820 MiB per run is not — and the
/// resize is instant too, because a qcow2's virtual size is metadata.
fn overlay(base: &Path, work: &Path) -> Result<PathBuf, String> {
let fmt = disk_format(base)?;
let out = work.join("disk.qcow2");
let _ = std::fs::remove_file(&out);
let status = Command::new("qemu-img")
.args(["create", "-q", "-f", "qcow2", "-F", fmt, "-b"])
.arg(base)
.arg(&out)
.output()
.map_err(|e| format!("qemu-img: {e}"))?;
if !status.status.success() {
return Err(format!(
"qemu-img create over {} failed: {}",
base.display(),
String::from_utf8_lossy(&status.stderr).trim()
));
}
if !out.is_file() {
return Err(format!(
"qemu-img reported success and wrote no overlay at {}",
out.display()
));
}
// Grow it, and then READ BACK what it grew to. `qemu-img resize` exiting 0 is
// not the claim being made — the claim is that the guest gets a bigger disk,
// and a resize that silently did nothing looks exactly like one that worked
// if the exit code is all that is checked.
let before = virtual_size(&out)?;
let resize = Command::new("qemu-img")
.arg("resize")
.arg(&out)
.arg(OVERLAY_HEADROOM)
.output()
.map_err(|e| format!("qemu-img resize: {e}"))?;
if !resize.status.success() {
return Err(format!(
"qemu-img resize {} {OVERLAY_HEADROOM} failed: {}",
out.display(),
String::from_utf8_lossy(&resize.stderr).trim()
));
}
let after = virtual_size(&out)?;
if after <= before {
return Err(format!(
"qemu-img resize {OVERLAY_HEADROOM} reported success and the overlay's virtual \
size is unchanged at {before} bytes ({} MiB; it reads {after} after). The guest \
would boot with the base image's size and gunnar would refuse every push under \
its own --disk-floor; see OVERLAY_HEADROOM.",
before / (1024 * 1024)
));
}
Ok(out)
}
/// A qcow2's **virtual** size in bytes, read out of the image rather than
/// assumed.
///
/// # `--output=json` was tried first, and it lies to a naive reader
///
/// The obvious implementation is `qemu-img info --output=json` and the first
/// `"virtual-size"` in it. **That field is the wrong one**, and it is wrong in
/// the direction that looks plausible: qemu ≥ 6 nests a `children[0].info` block
/// describing the *file* protocol layer, and its `"virtual-size"` is the size of
/// the file on disk — `197120` for a fresh overlay over a 3.5 GiB base. Caught
/// 2026-09-05 by deliberately setting [`OVERLAY_HEADROOM`] to `+0` and reading
/// the refusal, which named `197120` where it should have named `3758096384`.
/// The comparison it fed still went green on a real resize (allocating the
/// larger L1 table nudges the file up a few clusters), so nothing would have
/// reported this: a guard that reads the wrong number and passes anyway.
///
/// The human-readable form has exactly one `virtual size:` line and it is the
/// image's, so that is what is parsed.
fn virtual_size(image: &Path) -> Result<u64, String> {
let out = Command::new("qemu-img")
.arg("info")
.arg(image)
.output()
.map_err(|e| format!("qemu-img info: {e}"))?;
if !out.status.success() {
return Err(format!(
"qemu-img info {} failed: {}",
image.display(),
String::from_utf8_lossy(&out.stderr).trim()
));
}
let text = String::from_utf8_lossy(&out.stdout);
// `virtual size: 3.5 GiB (3758096384 bytes)` — the parenthesised byte count,
// never the human unit, which rounds.
text.lines()
.find(|l| l.trim_start().starts_with("virtual size:"))
.and_then(|l| l.split('(').nth(1))
.and_then(|rest| rest.split_whitespace().next())
.and_then(|n| n.parse::<u64>().ok())
.ok_or_else(|| {
format!(
"qemu-img info {} carried no parseable `virtual size:` line: {}",
image.display(),
text.trim()
)
})
}
// ── cloud-init ──────────────────────────────────────────────────────────────
fn user_data(admin_pubkey: &str) -> String {
format!(
"#cloud-config\n\
hostname: gunnar-installtest\n\
users:\n\
\x20 - name: {ADMIN}\n\
\x20 sudo: 'ALL=(ALL) NOPASSWD:ALL'\n\
\x20 shell: /bin/bash\n\
\x20 lock_passwd: true\n\
\x20 ssh_authorized_keys:\n\
\x20 - {admin}\n\
ssh_pwauth: false\n\
package_update: false\n",
admin = admin_pubkey.trim()
)
}
// ── the arm ─────────────────────────────────────────────────────────────────
/// **The KVM arm.**
pub fn run(subject: &Subject, work: &Path, fault: Fault, keep: bool) -> ArmResult {
let mut ready = None;
let mut stages = crate::Stages::default();
let outcome = run_inner(subject, work, fault, keep, &mut ready, &mut stages);
ArmResult {
arm: "kvm (draupnir → tunnr · QEMU/OVMF/KVM)",
fault,
outcome,
ready,
stages: stages.0,
demands_red: false,
}
}
fn run_inner(
subject: &Subject,
work: &Path,
fault: Fault,
keep: bool,
ready_out: &mut Option<Duration>,
stages: &mut crate::Stages,
) -> Result<Exchange, String> {
let image = base_image();
if !image.is_file() {
return Err(format!(
"no guest image at {}; set GUNNAR_INSTALLTEST_VM_IMAGE to a cloud image \
whose glibc is at least the host's (this binary needs GLIBC_2.38)",
image.display()
));
}
// Refuse to run on a host that cannot actually virtualise: without /dev/kvm
// QEMU silently falls back to TCG and every timing below becomes a statement
// about an emulator. Silence is the failure mode measurement guards against.
let caps = tunnr_vm::host_capabilities();
let blockers = caps.blockers();
if !blockers.is_empty() {
return Err(format!("this host cannot boot a KVM guest: {blockers:?}"));
}
let _ = std::fs::remove_dir_all(work);
std::fs::create_dir_all(work).map_err(|e| format!("{}: {e}", work.display()))?;
// Two keys, and they are NOT the same one. The admin key is how this harness
// drives the install over the guest's own sshd; the git key is what gunnar
// authenticates. Sharing one would make an authorisation bug in gunnar
// invisible, because the key would already be trusted by the box.
let admin_key = work.join("id_admin");
let git_key = work.join("id_git");
keygen(&admin_key)?;
keygen(&git_key)?;
let admin_pub = std::fs::read_to_string(admin_key.with_extension("pub"))
.map_err(|e| format!("reading the admin public key: {e}"))?;
// ── seed + boot ─────────────────────────────────────────────────────────
let ci = CloudInit::user_data(user_data(&admin_pub));
let admin_port = crate::free_port()?;
let git_port = crate::free_port()?;
// `kvm_bootloader_disk` is the firmware boot: draupnir hands tunnr an EMPTY
// kernel path, so OVMF boots the cloud image directly, exactly as it would
// on metal, rather than taking QEMU's direct-kernel shortcut.
//
// The cloud-init seed rides `with_cloud_init`: draupnir authors the NoCloud
// vfat image (`draupnir::seed`, the same builder this arm used to call by
// hand) AND attaches it through tunnr's own `with_seed` seam. One owner for
// the seed and the drive it arrives on, rather than this crate authoring an
// image and wiring its own drive beside a boot it did not build.
//
// The two forwards are the whole point of the arm: 22 for the install shell,
// `IN_GUEST_GIT` for the git conversation the host drives afterwards.
//
// **The spec NAME must be unique per run, and that is not cosmetic.**
// `KvmBoot` authors the seed at `std::env::temp_dir()/draupnir-<name>-seed.img`,
// so every boot sharing a name shares one file on disk. A `red` sweep runs
// five faults back to back **in one process**, and with a constant name the
// second run rewrites the seed the first guest still has open as a virtio
// drive. Before this arm went through draupnir the seed was authored into
// `work/seed.img`, which is already per-fault, so the collision is new with
// the move and is closed by taking the name from the same place: the work
// directory (`kvm-none`, `kvm-missing-binary`, …).
let run = work
.file_name()
.map(|n| n.to_string_lossy().into_owned())
.unwrap_or_else(|| "run".to_string());
let disk = stages.time("overlay", || overlay(&image, work))?;
let mut spec =
BootSpec::kvm_bootloader_disk(format!("gunnar-installtest-{run}"), disk.to_string_lossy())
.with_cloud_init(ci)
.with_port_map(admin_port, 22)
.with_port_map(git_port, IN_GUEST_GIT)
.with_boot_timeout(boot_budget() + Duration::from_secs(120));
spec.mem_mb = 2048;
spec.cores = 2;
// COW protection OFF, deliberately. `kvm_bootloader_disk` defaults it ON
// (tunnr's `base_snapshot` → `snapshot=on`) because the usual caller boots a
// SHARED, content-addressed base disk that must never be mutated. This arm's
// disk is not that: `overlay` above already built a throwaway qcow2 layer
// over the shared base, this run owns it, and a second COW layer on top
// would protect nothing while costing a second copy-on-write path. That is
// exactly what `persist_disk` is documented for — "a caller that booted its
// OWN per-run disk copy" — and it can only ever turn COW off, never on.
spec.persist_disk = true;
// `draupnir::boot` validates the spec BEFORE any QEMU is launched, so a
// forward on an airgapped guest, an unsized VM or an empty disk path fails
// by name here rather than as a guest that never came up.
let kvm = KvmBoot::new();
let machine = draupnir::boot(&spec, &kvm).map_err(|e| format!("draupnir kvm boot: {e}"))?;
let guard = Vm {
kvm: kvm.clone(),
machine: machine.clone(),
};
let console = || kvm.serial_log(&machine).unwrap_or_default();
let alive = || match kvm.status(&machine) {
Ok(PowerState::On) => Ok(()),
other => Err(format!(
"the guest died before it could be used: {other:?}{}\n console tail: {}",
kvm.exit_reason(&machine)
.map(|r| format!(" ({r})"))
.unwrap_or_default(),
tail(&console(), 6)
)),
};
let shell = GuestShell {
port: admin_port,
key: admin_key,
};
let elapsed = shell.wait_usable(boot_budget(), alive, console)?;
*ready_out = Some(elapsed);
stages.record("boot_to_usable", elapsed);
// cloud-init is asynchronous; the account existing does not mean it has
// finished. Ask it, rather than sleeping and hoping.
stages.time("cloud_init_wait", || {
let _ = shell.run("cloud-init status --wait || true");
});
// ── the install ─────────────────────────────────────────────────────────
stages.time("install", || {
install(&shell, subject, &git_key, fault, git_port)
})?;
if fault == Fault::KilledService {
// Installed, started, healthy — and then stopped through its own verb.
// The exchange below must fail, which is what proves the assertion is on
// the git conversation and not on the status read a moment ago.
shell.run(&format!("sudo {EXEC} service stop"))?;
}
// ── the assertion, driven from the HOST over the forwarded port ─────────
let probe = Probe {
port: git_port,
key: git_key,
work: work.to_path_buf(),
user: "git".to_string(),
};
let repo = if fault == Fault::EmptyRepo {
"demo/empty"
} else {
"demo/hello"
};
let result = probe.push_then_clone(repo, fault);
if let Ok(x) = &result {
stages.0.push(("push", x.push_ms));
stages.0.push(("clone", x.clone_ms));
}
if keep {
eprintln!(
" --keep: guest still running, ssh -i {} -p {admin_port} {ADMIN}@127.0.0.1",
work.join("id_admin").display()
);
std::mem::forget(guard);
}
result
}
/// Everything between "a bare cloud image is up" and "gunnar is serving".
///
/// Each step is asserted on the guest's own report, not on the fact that the
/// command was sent.
fn install(
shell: &GuestShell,
subject: &Subject,
git_key: &Path,
fault: Fault,
guest_git_port: u16,
) -> Result<(), String> {
// 1 — the artefact, and then the artefact's OWN account of itself.
//
// The check used to be `version.to_lowercase().contains("gunnar")`, which
// every gunnar ever built passes — including one left in the guest image, or
// an `scp` that silently copied a stale file. `--version` carries the git
// description since `008d74b`, so the question "is the thing in the guest the
// thing this run built?" has an answer, and it is asked here rather than
// assumed.
shell.put(&subject.path, "/tmp/gunnar")?;
shell.run("sudo install -m 0755 -o root -g root /tmp/gunnar /usr/local/bin/gunnar")?;
let version = shell.run(&format!("{EXEC} --version"))?;
subject.assert_installed(&version, "in the KVM guest")?;
// 2 — the data root and the server's own state, through gunnar's verbs.
shell.run(&format!("sudo mkdir -p {DATA_ROOT}"))?;
let git_pub = std::fs::read_to_string(git_key.with_extension("pub"))
.map_err(|e| format!("reading the git public key: {e}"))?;
shell.run(&format!(
"printf '%s' {} | sudo tee {DATA_ROOT}/tester.pub >/dev/null",
shell_quote(git_pub.trim())
))?;
let keyfile = format!("@{DATA_ROOT}/tester.pub");
for verbs in crate::bootstrap_plan() {
let line = verbs
.into_iter()
.map(|v| {
if v == crate::KEYFILE {
keyfile.as_str()
} else {
v
}
})
.collect::<Vec<_>>()
.join(" ");
shell.run(&format!("sudo {EXEC} --data-root {DATA_ROOT} {line}"))?;
}
// 3 — the systemd install, through Skidbladnir. `--exec` is the injection
// point for Fault::MissingBinary: a unit whose ExecStart names a path
// that is not there installs perfectly and then never serves.
let exec = if fault == Fault::MissingBinary {
"/usr/local/bin/gunnar-not-installed"
} else {
EXEC
};
shell.run(
"sudo useradd --system --home-dir /var/lib/gunnar --shell /usr/sbin/nologin gunnar || true",
)?;
shell.run(&format!("sudo chown -R gunnar:gunnar {DATA_ROOT}"))?;
let unit = shell.run(&format!(
"sudo {EXEC} service unit --exec {exec} --ssh-bind 0.0.0.0:{IN_GUEST_GIT} \
--data-root {DATA_ROOT} --run-as gunnar"
))?;
if !unit.contains("ExecStart=") || !unit.contains("serve") {
return Err(format!(
"the unit Skidbladnir rendered does not start gunnar:\n{unit}"
));
}
shell.run(&format!(
"sudo {EXEC} service install --exec {exec} --ssh-bind 0.0.0.0:{IN_GUEST_GIT} \
--data-root {DATA_ROOT} --run-as gunnar"
))?;
// The unit really is on disk, where systemd looks. Asserted against the
// filesystem rather than against the installer's own cheerful stdout.
shell.run("test -f /etc/systemd/system/gunnar.service")?;
shell.run(&format!(
"sudo {EXEC} service start --exec {exec} --ssh-bind 0.0.0.0:{IN_GUEST_GIT} \
--data-root {DATA_ROOT} --run-as gunnar"
))?;
// 4 — systemd's own verdict, and the installer's. They must agree.
//
// Both are read, and cross-checked against each other, because either
// alone can be hollow: `gunnar service status` is Skidbladnir's D-Bus
// read and could report a stale or wrong unit, and `systemctl is-active`
// alone would say nothing about whether the API gunnar ships works.
// The `MissingBinary` fault drives them both to "inactive", so the
// agreement check is exercised on both sides.
let (_, systemd_says) = shell.try_run("systemctl is-active gunnar.service")?;
let (_, gunnar_says) = shell.try_run(&format!(
"sudo {EXEC} service status --ssh-bind 0.0.0.0:{IN_GUEST_GIT} --data-root {DATA_ROOT}"
))?;
let systemd_active = systemd_says.trim() == "active";
// The verb prints `<unit> active` or `<unit> inactive`; test for the word,
// not for a substring of it.
let gunnar_active = gunnar_says.split_whitespace().any(|w| w == "active");
if systemd_active != gunnar_active {
return Err(format!(
"systemd and gunnar disagree about the service: systemctl says {:?}, \
`gunnar service status` says {:?}",
systemd_says.trim(),
gunnar_says.trim()
));
}
if fault != Fault::MissingBinary && !systemd_active {
let (_, journal) = shell.try_run("journalctl -u gunnar.service --no-pager -n 20")?;
return Err(format!(
"the installed unit is not active: systemctl says {:?}\n{journal}",
systemd_says.trim()
));
}
// The service must also be REACHABLE, not just `active`. systemd reports a
// `Type=simple` unit active the moment it forks, before gunnar has bound
// anything. That is the measure-not-assert gap between "the unit started" and "the server
// is up", the same gap the container arm's readiness wait closes.
if fault != Fault::MissingBinary {
wait_for_service(guest_git_port, Duration::from_secs(60))?;
}
Ok(())
}
/// A single-quoted shell word. The only value that reaches it is an OpenSSH
/// public key line, but quoting it by construction is cheaper than reasoning
/// about what a key can contain.
fn shell_quote(s: &str) -> String {
format!("'{}'", s.replace('\'', "'\\''"))
}
fn tail(text: &str, n: usize) -> String {
let lines: Vec<&str> = text.lines().collect();
lines[lines.len().saturating_sub(n)..].join(" / ")
}
/// The last `n` lines, one per line. On a serial console the line breaks carry
/// the meaning.
fn tail_lines(text: &str, n: usize) -> String {
let lines: Vec<&str> = text.lines().collect();
lines[lines.len().saturating_sub(n)..].join("\n")
}
/// Stops the guest when the arm leaves, however it leaves.
struct Vm {
/// `KvmBoot` is a cheap `Arc` clone and the registry inside it is what maps
/// this machine back to the QEMU process group, so the guard can own one.
kvm: KvmBoot,
machine: Machine,
}
impl Drop for Vm {
fn drop(&mut self) {
// `KvmBoot::shutdown`, NOT `Lifecycle::power_off`: the graceful path
// does the monitor/ACPI power-down first and only SIGKILLs after the
// grace elapses, so the guest's filesystem is not torn out from under
// it. See the module header for the 2026-08-16 measurement that makes
// this load-bearing rather than tidy.
let _ = self.kvm.shutdown(&self.machine, Duration::from_secs(20));
}
}
/// Wait for gunnar's listener inside the guest to accept on the forwarded port.
///
/// Exposed because the KVM arm's readiness has two stages, the guest and then
/// the service, and the second one is worth naming.
pub fn wait_for_service(port: u16, budget: Duration) -> Result<Duration, String> {
wait_for_ssh(port, budget, || Ok(()))
}