installtest/src/main.rs
//! `gunnar-installtest`, the driver.
//!
//! ```text
//! gunnar-installtest bare gunnar as an ORDINARY PROCESS, must PASS
//! gunnar-installtest container the container arm, must PASS
//! gunnar-installtest kvm the KVM arm, must PASS
//! gunnar-installtest iso the OS-LESS APPLIANCE arm, must PASS
//! gunnar-installtest all all four
//! gunnar-installtest red every fault against every arm, each must go RED
//! gunnar-installtest red --arm bare the faults against one arm
//! gunnar-installtest pair --keep TWO appliances, both up, ports printed
//! gunnar-installtest pair --members main,twin,spare --keep …and a cold spare
//!
//! gunnar-installtest --help the usage above in full; runs NOTHING
//!
//! --fault NAME inject one fault into a single arm (wrong-port, missing-binary,
//! killed-service, empty-repo); that run must then FAIL
//! --life-fault N bare only: sabotage the LIFECYCLE instead (no-teardown,
//! port-still-bound, lock-holder-alive)
//! --binary PATH install THIS gunnar instead of building one. It is still made
//! to identify itself: a binary whose `--version` does not name
//! this checkout's HEAD is refused (`src/subject.rs`)
//! --keep leave the container / guest / process up afterwards
//! ```
//!
//! # What is installed, and how that stopped being a guess
//!
//! Every run resolves a [`Subject`]: a gunnar this binary **builds**
//! (`--no-default-features --features server,systemd`) and then makes identify
//! itself against `git rev-parse HEAD`. The `container` and `kvm` arms install
//! that exact binary and re-read `gunnar --version` afterwards, which must
//! report the same string back.
//!
//! The `iso` arm is the exception and says so: what it boots is the forge's own
//! `gunnar-server` variant packed inside the ISO, a different artefact. There
//! the subject contributes the **commit**, which the appliance's SSH banner has
//! to name — see `src/iso.rs`.
//!
//! Until 2026-08-05 there was no such check anywhere, the crate installed
//! whatever binary was lying around — measured once at **1998 minutes old**,
//! from another worktree's target directory — and `cargo xtask ci` refused to run it
//! for exactly that reason. It is now in `ci`.
//!
//! # The bare server a human drives
//!
//! The four verbs below are the point of the bare arm and are not a test: they
//! are for a person at a terminal who wants a gunnar to talk to.
//!
//! ```text
//! gunnar-installtest up start one, print the `git clone` line to paste
//! gunnar-installtest status is it still there, on what port, since when
//! gunnar-installtest down stop it, and PROVE it let go of port and lock
//! gunnar-installtest pressure ramp concurrent clones and pushes until it gives
//! ```
//!
//! They share one state file under `--root` (default `$GUNNAR_BARE_ROOT`, else
//! `~/scratch/gunnar-bare`), so `down` works from a different shell
//! from the one that ran `up`. Never `/tmp`: that is RAM on this box.
//!
//! `cargo xtask install-test` and `cargo xtask bare` are the intended callers and
//! pass these through.
//!
//! # The exit code is the verdict, and it is not the same as "no errors"
//!
//! A `red` run is **successful when every arm fails**. Reporting that with the
//! ordinary "any error means exit 1" rule would make a harness that is broken in
//! the direction of always failing look identical to one that works. So the
//! verdict is computed from [`ArmResult::as_demanded`], per run, and printed
//! next to what was demanded of it.
use std::path::{Path, PathBuf};
use std::process::ExitCode;
use gunnar_installtest::bare::{self, LifeFault};
use gunnar_installtest::subject::Subject;
use gunnar_installtest::{container, kvm, render, ArmResult, Fault};
fn main() -> ExitCode {
let args: Vec<String> = std::env::args().skip(1).collect();
let mode = args.first().cloned().unwrap_or_default();
// ── the strict parse, and it comes before everything ────────────────────
//
// Everything below this block reads arguments by NAME — `flag("--iso")`,
// `args.iter().any(|a| a == "--grpc-only")` — and every one of those reads
// is blind to the arguments it does not name. Until 2026-09-17 nothing
// looked at the rest, so an argument this binary did not know was simply
// dropped, in silence, and the run continued as if it had never been typed.
//
// MEASURED that day, twice, at the cost of two full re-image cycles: the
// ops xtask gated a TWIN medium — which serves no git, the guest says so
// itself (`this box is a TWIN: the git listener is bound to loopback only
// asked=0.0.0.0:2222`) — with a `gunnar-installtest` on PATH that predated
// `--grpc-only`. The flag fell through, the SSH-banner door ran instead,
// `B2 serve` waited 180 s for `:2222` and could never pass. A stale binary
// and a current one behaved IDENTICALLY on the flag that decides which test
// runs, which is the one thing a gate may never do.
//
// So: an argument no verb accepts stops the run and is named. The same rule
// `deploy/upcloud/src/main.rs` already applies to its own argv ("nothing is
// ignored here"), now applied by the gate as well.
if args.is_empty() {
eprintln!("install tests: no verb given");
eprint!("{}", usage());
return ExitCode::FAILURE;
}
// `--help` PRINTS, on stdout, and runs nothing. It used to be an unknown
// MODE, so it fell through to the arm dispatch and built a gunnar before
// reporting itself unknown on stderr — which is how the staleness above hid
// for a day: `gunnar-installtest --help | grep grpc-only` printed nothing
// and read as proof that the flag did not exist.
if mode == "help" || args.iter().any(|a| a == "--help" || a == "-h") {
print!("{}", usage());
return ExitCode::SUCCESS;
}
if let Err(refusal) = check_args(&mode, &args[1..]) {
eprintln!("{refusal}");
return ExitCode::FAILURE;
}
let flag = |name: &str| -> Option<String> {
args.iter()
.position(|a| a == name)
.and_then(|i| args.get(i + 1))
.cloned()
};
let keep = args.iter().any(|a| a == "--keep");
// The operator verbs run a gunnar the operator already has; the ARMS build
// and identify their own subject. Resolving the subject is therefore deferred
// until after the operator branch, so `gunnar-installtest down` does not
// compile a server in order to stop one.
let named = flag("--binary");
let named = named.as_deref().map(Path::new);
let fault = match flag("--fault") {
Some(f) => match Fault::parse(&f) {
Ok(f) => f,
Err(e) => {
eprintln!("install tests: {e}");
return ExitCode::FAILURE;
}
},
None => Fault::None,
};
let life = match flag("--life-fault") {
Some(f) => match LifeFault::parse(&f) {
Ok(f) => f,
Err(e) => {
eprintln!("install tests: {e}");
return ExitCode::FAILURE;
}
},
None => LifeFault::None,
};
let only_arm = flag("--arm");
let scratch = scratch_root();
// ── the operator verbs: not tests, and they take a different path ───────
//
// They print their own report and their own exit code, because a running
// server is not an `ArmResult` and forcing it into one would mean inventing
// an object-graph comparison that never happened.
match mode.as_str() {
// The firmware door: the INSTALLER medium under SeaBIOS or UEFI. Its
// subject is a medium, not a binary, so it runs before `Subject::prepare`
// for the same reason the operator verbs do — and it prints its own
// report, because three boot legs are not an `ArmResult`'s object-graph
// exchange and forcing them into one would invent a comparison that
// never happened.
"installer" => return installer_door(&args, &flag),
// TWO appliances, both alive at once — the rig the quick start's
// replication section is written against. Also not a test and also not
// an `ArmResult`: it installs a pair and hands the operator the ports,
// and what happens next is whatever they type. See `src/pair.rs`.
"pair" => return pair_rig(&flag, keep),
// `down` and `status` read the ROOT and never run the binary. Refusing them because no
// binary could be located made `down` exit 1 after an `up --binary <path>` whose
// binary is outside the located list (a prebuilt subject), leaving the server up.
"down" | "status" => return operator(&mode, &args, None, flag("--root")),
"up" | "pressure" => {
let binary = match container::locate_binary(named) {
Ok(b) => b,
Err(e) => {
eprintln!("install tests: {e}");
return ExitCode::FAILURE;
}
};
return operator(&mode, &args, Some(&binary), flag("--root"));
}
_ => {}
}
// **What is being installed.** Built here unless named, and made to identify
// itself against this checkout's HEAD before a single arm runs. See
// `installtest/src/subject.rs`: until 2026-08-05 this crate installed
// whatever binary was lying around, which is why `xtask ci` refused to run
// it at all.
let subject = match Subject::prepare(named) {
Ok(s) => s,
Err(e) => {
eprintln!("install tests: {e}");
return ExitCode::FAILURE;
}
};
let binary = subject.path.clone();
eprintln!("install tests");
eprintln!(" binary : {}", binary.display());
eprintln!(" scratch : {}", scratch.display());
eprintln!(" base image : {}", container::base_image());
eprintln!(" guest image: {}", kvm::base_image().display());
eprintln!();
let mut results: Vec<ArmResult> = Vec::new();
match mode.as_str() {
"bare" => results.push(one_bare(&binary, &scratch, fault, life, keep)),
"chain" => results.push(one_chain(&binary, &scratch, fault)),
"container" => results.push(one("container", &subject, &scratch, fault, keep)),
"kvm" => results.push(one("kvm", &subject, &scratch, fault, keep)),
"iso" => results.push(one("iso", &subject, &scratch, fault, keep)),
"all" => {
results.push(one_bare(&binary, &scratch, fault, life, keep));
results.push(one("container", &subject, &scratch, fault, keep));
results.push(one("kvm", &subject, &scratch, fault, keep));
results.push(one("iso", &subject, &scratch, fault, keep));
results.push(one_chain(&binary, &scratch, fault));
}
"red" => {
let arms: Vec<&str> = match only_arm.as_deref() {
Some(a) => vec![a],
None => vec!["bare", "container", "kvm", "iso"],
};
for arm in arms {
// The green first: a red-first proof against an arm that cannot
// pass proves nothing, because everything fails then.
if arm == "bare" {
results.push(one_bare(
&binary,
&scratch,
Fault::None,
LifeFault::None,
false,
));
for f in Fault::ALL_RED {
results.push(one_bare(&binary, &scratch, *f, LifeFault::None, false));
}
// The lifecycle matrix, which only this arm has.
for l in LifeFault::ALL_RED {
results.push(one_bare(&binary, &scratch, Fault::None, *l, false));
}
continue;
}
results.push(one(arm, &subject, &scratch, Fault::None, false));
// Per ARM, not one global list. The ISO arm cannot inject
// `empty-repo` — the appliance's baked state has no empty
// repository — and running it anyway would produce a red for
// the wrong reason, which passes the `must be Err` demand while
// proving nothing. Saying which faults an arm CAN inject is the
// honest version of skipping one. See `iso::RED`.
let reds: &[Fault] = if arm == "iso" {
gunnar_installtest::iso::RED
} else {
Fault::ALL_RED
};
for f in reds {
results.push(one(arm, &subject, &scratch, *f, false));
}
}
}
// Unreachable through `check_args`, which refuses an unknown mode
// before a subject is ever built — and kept anyway, because it catches
// the OTHER drift: a verb added to `accepted()` and to `usage()` but
// never wired to an arm would otherwise run nothing and report a green
// empty verdict.
other => {
eprintln!(
"install tests: unknown mode `{other}` — it is listed in `accepted()` but no arm \
runs it; the dispatch and the flag table have drifted"
);
eprint!("{}", usage());
return ExitCode::FAILURE;
}
}
print!("{}", render(&results));
if results.iter().all(ArmResult::as_demanded) {
ExitCode::SUCCESS
} else {
ExitCode::FAILURE
}
}
fn one(arm: &str, subject: &Subject, scratch: &Path, fault: Fault, keep: bool) -> ArmResult {
eprintln!("── {arm} · fault={} ──", fault.name());
let work = scratch.join(format!("{arm}-{}", fault.name()));
match arm {
"container" => container::run(subject, &work, fault, keep),
"kvm" => kvm::run(subject, &work, fault, keep),
"iso" => gunnar_installtest::iso::run(subject, &work, fault, keep),
other => ArmResult {
arm: "unknown",
fault,
outcome: Err(format!("unknown arm `{other}`")),
ready: None,
stages: Vec::new(),
demands_red: false,
},
}
}
/// The chain arm: push → build → registry → verified pull, against the
/// installed binary's own CLI.
///
/// No `keep`: it takes no container and no VM, so there is nothing to leave
/// running for an operator to poke at — the scratch root is the artefact, and
/// it is where it always was.
fn one_chain(binary: &Path, scratch: &Path, fault: Fault) -> ArmResult {
eprintln!("── chain · fault={} ──", fault.name());
let work = scratch.join(format!("chain-{}", fault.name()));
gunnar_installtest::chain::run(binary, &work, fault)
}
fn one_bare(binary: &Path, scratch: &Path, fault: Fault, life: LifeFault, keep: bool) -> ArmResult {
eprintln!("── bare · fault={} · life={} ──", fault.name(), life.name());
let work = scratch.join(format!("bare-{}-{}", fault.name(), life.name()));
bare::run(binary, &work, fault, life, keep)
}
// ── the firmware door ───────────────────────────────────────────────────────
/// `installer --firmware seabios|uefi [--serial none|uart] [--iso PATH] [--grpc-only]`.
///
/// `--grpc-only` proves B2/B3 by `Bootstrap.Status` (build stamp + declared role)
/// on the private interface instead of the SSH banner — a twin medium serves no
/// git (`src/control.rs`).
///
/// Default firmware stays `uefi` (every earlier proof's shape). `--serial none`
/// is the UpCloud machine: no UART, so the door gates on the disk and the
/// ports. The ISO defaults to `$GUNNAR_INSTALLER_ISO`, else the open medium the
/// forge writes under `.iso-out/`.
fn installer_door(args: &[String], flag: &dyn Fn(&str) -> Option<String>) -> ExitCode {
use gunnar_installtest::firmware::{self, Door, FirmwareChoice, Shape};
let firmware = match flag("--firmware").as_deref().map(FirmwareChoice::parse) {
Some(Ok(f)) => Some(f),
Some(Err(e)) => {
eprintln!("install tests: {e}");
return ExitCode::FAILURE;
}
None => None,
};
let serial = match flag("--serial").as_deref() {
None => None,
Some("uart") | Some("on") => Some(true),
Some("none") | Some("off") => Some(false),
Some(other) => {
eprintln!("install tests: unknown --serial `{other}`: uart | none");
return ExitCode::FAILURE;
}
};
// `--target <preset>` renders the machine from edda's roster; the two flags
// are overrides on top of it. Without a preset the flags ARE the shape
// (default uefi + uart, every earlier proof's machine).
let shape = match flag("--target") {
Some(name) => match Shape::from_preset(&name, firmware, serial) {
Ok(s) => s,
Err(e) => {
eprintln!("install tests: {e}");
return ExitCode::FAILURE;
}
},
None => Shape::flags(
firmware.unwrap_or(FirmwareChoice::Uefi),
serial.unwrap_or(true),
),
};
let iso = flag("--iso")
.map(PathBuf::from)
.or_else(|| std::env::var_os("GUNNAR_INSTALLER_ISO").map(PathBuf::from))
.unwrap_or_else(|| {
gunnar_installtest::workspace_root().join(".iso-out/gunnar-installer-open.iso")
});
let work = flag("--work").map(PathBuf::from).unwrap_or_else(|| {
scratch_root().join(match shape.preset {
Some(p) => format!("installer-target-{p}"),
None => format!(
"installer-{}-{}",
shape.firmware.label(),
if shape.serial_console {
"uart"
} else {
"noserial"
}
),
})
});
// `--twin target`: the medium's virtio face padded to the target's size and
// attached first — UpCloud's ISO-storage-beside-target shape, the one that
// wiped the medium on 2026-08-29. Default = the ISO file, read-only.
let twin = match flag("--twin").as_deref().map(firmware::Twin::parse) {
Some(Ok(t)) => t,
Some(Err(e)) => {
eprintln!("install tests: {e}");
return ExitCode::FAILURE;
}
None => firmware::Twin::Iso,
};
// `--upgrade-from OLD.iso`: the data-survives door — OLD installs, the
// door bootstraps + pushes, then THIS medium reinstalls on the same image
// and the appliance must come back closed with the repository (four legs).
// `--disks 75,75,75,75`: extra fixed disks (GiB) beside the target, the
// multi-disk /data door — see firmware::data_set_verdict.
let disks: Vec<u64> = match flag("--disks") {
Some(list) => match list
.split(',')
.map(|g| g.trim().parse::<u64>())
.collect::<Result<Vec<_>, _>>()
{
Ok(v) => v,
Err(_) => {
eprintln!("install tests: --disks wants a comma list of GiB, got {list:?}");
return ExitCode::FAILURE;
}
},
None => Vec::new(),
};
let upgrade_from = flag("--upgrade-from").map(PathBuf::from);
let cli = flag("--cli")
.map(PathBuf::from)
.unwrap_or_else(|| gunnar_installtest::workspace_root().join("target/release/gunnar-cli"));
// `--grpc-only`: prove the medium by its control plane — `Bootstrap.Status`
// names a build stamp and the role the medium declares — for a medium that
// serves no git (`korp.args=…,--role,twin`). Refused on a medium whose
// control plane is not bound off the loopback: there would be nothing to ask.
let grpc_only = if args.iter().any(|a| a == "--grpc-only") {
if !firmware::declared_grpc_open(&iso) {
eprintln!(
"install tests: --grpc-only needs a medium that binds its control plane off the \
loopback (`--grpc-bind eth1:50051`); {} declares none",
iso.display()
);
return ExitCode::FAILURE;
}
match firmware::declared_role(&iso) {
Ok(role) => Some(role),
Err(e) => {
eprintln!("install tests: --grpc-only: {e}");
return ExitCode::FAILURE;
}
}
} else {
None
};
let _ = args;
let door = Door {
expect_grpc: firmware::declared_grpc_open(&iso),
grpc_only,
iso,
shape,
work,
twin,
disks,
// `--drive-cmd PATH`: the appliance-rig hook — a host program run once
// in B2 the instant the ports answer, `PATH <ssh_port> <grpc_port>`.
drive_cmd: flag("--drive-cmd").map(PathBuf::from),
};
let (want, result) = match &upgrade_from {
Some(old) => (4usize, firmware::run_upgrade(&door, old, &cli)),
None => (3usize, firmware::run(&door)),
};
match result {
Ok(legs) => {
print!("{}", firmware::render(&door, &legs));
// passed / failed-or-refused / TIMED OUT, told apart in the exit
// code as well as the row — a caller reading only `$?` used to be
// unable to tell a 420-second hang from a green door.
ExitCode::from(firmware::exit_code(&legs, want))
}
Err(e) => {
eprintln!("install tests: installer door refused: {e}");
ExitCode::from(firmware::EXIT_NOT_PROVEN)
}
}
}
/// `gunnar-installtest pair` — install and boot TWO appliances and hand the
/// operator their ports. See [`gunnar_installtest::pair`].
fn pair_rig(flag: &dyn Fn(&str) -> Option<String>, keep: bool) -> ExitCode {
use gunnar_installtest::pair;
let repo = gunnar_installtest::workspace_root();
let mut ask = pair::Ask::from_env(&repo);
if let Some(p) = flag("--iso") {
ask.iso = PathBuf::from(p);
}
if let Some(p) = flag("--work") {
ask.kv = PathBuf::from(p);
}
if let Some(v) = flag("--mem-mb").and_then(|v| v.parse().ok()) {
ask.mem_mb = v;
}
if let Some(v) = flag("--vcpus").and_then(|v| v.parse().ok()) {
ask.vcpus = v;
}
if let Some(list) = flag("--members") {
ask.members = list
.split(',')
.map(str::trim)
.filter(|s| !s.is_empty())
.map(str::to_string)
.collect();
}
ask.drive_cmd = flag("--drive-cmd").map(PathBuf::from);
// `--keep` is what a person wants here far more often than not: the rig's
// whole purpose is that the two boxes are still up when it stops printing.
ask.keep = keep;
match pair::run(&repo, &ask) {
Ok(()) => ExitCode::SUCCESS,
Err(e) => {
eprintln!("install tests: pair: {e}");
ExitCode::FAILURE
}
}
}
// ── the operator verbs ──────────────────────────────────────────────────────
/// The default home for the server a human brings up. md1, never `/tmp`.
fn bare_root(explicit: Option<String>) -> PathBuf {
explicit
.map(PathBuf::from)
.or_else(|| std::env::var("GUNNAR_BARE_ROOT").ok().map(PathBuf::from))
.unwrap_or_else(|| gunnar_installtest::expand_home("~/scratch/gunnar-bare"))
}
fn operator(mode: &str, args: &[String], binary: Option<&Path>, root: Option<String>) -> ExitCode {
let needs =
|| binary.ok_or_else(|| format!("`{mode}` needs a gunnar binary and none was located"));
let root = bare_root(root);
let num = |name: &str, default: usize| -> usize {
args.iter()
.position(|a| a == name)
.and_then(|i| args.get(i + 1))
.and_then(|v| v.parse().ok())
.unwrap_or(default)
};
let result = match mode {
"up" => needs().and_then(|b| op_up(b, &root, args)),
"status" => op_status(&root),
"down" => op_down(&root, !args.iter().any(|a| a == "--keep-data")),
"pressure" => needs().and_then(|b| op_pressure(b, &root, args, &num)),
_ => Err(format!("unknown operator verb `{mode}`")),
};
match result {
Ok(()) => ExitCode::SUCCESS,
Err(e) => {
eprintln!("gunnar bare {mode}: {e}");
ExitCode::FAILURE
}
}
}
/// `up`: one command, and what comes back is a server plus the exact lines to
/// paste. The `export` lines matter as much as the server: without
/// `GIT_SSH_COMMAND` a stock `git` reaches for the operator's own agent and
/// their own `~/.ssh/config`, and then the failure is about their laptop.
fn op_up(binary: &Path, root: &Path, args: &[String]) -> Result<(), String> {
if let Ok(existing) = bare::Instance::load(root) {
if existing.alive() {
return Err(format!(
"a gunnar is already up at {} (pid {}, port {}). Stop it with \
`gunnar-installtest down` first",
root.display(),
existing.pid,
existing.ssh_port
));
}
}
let port = args
.iter()
.position(|a| a == "--port")
.and_then(|i| args.get(i + 1))
.and_then(|v| v.parse::<u16>().ok());
let opts = bare::UpOptions {
ssh_port: port,
http_port: None,
extra_repos: Vec::new(),
grpc: args.iter().any(|a| a == "--grpc"),
// `--reuse` keeps an existing data root, which is how an operator gets
// their repositories back after a `down`. A fresh root is the default
// because a surprise about which repositories exist is worse.
reuse_data: args.iter().any(|a| a == "--reuse") && root.join("data").is_dir(),
exec: None,
extra_principals: 0,
max_concurrent_transfers: None,
max_transfers_per_principal: None,
hard_per_principal_limit: false,
};
let mut server = bare::Instance::up(binary, root, &opts)?;
// The whole point of `up` is that the server is still there when this
// process exits. Without this the handle's Drop kills it on return.
server.detach();
let probe = server.probe();
println!("gunnar is up.");
println!(" pid : {}", server.pid);
println!(" ssh : 127.0.0.1:{}", server.ssh_port);
println!(" data root : {}", server.data.display());
println!(" log : {}", server.log.display());
println!(" ready in : {} ms", server.ready.as_millis());
println!();
println!(" export GIT_SSH_COMMAND='{}'", probe.ssh_command());
// The push comes FIRST, and it is not a style choice. `demo/hello` is
// created empty, and `git clone` of an empty repository exits zero and
// leaves an empty directory — the vacuous green this whole crate exists to
// refuse. Printing the clone line first invites an operator to run exactly
// that and conclude the server works.
println!(
" git push {} main:main # the repo starts EMPTY",
server.url("demo/hello")
);
println!(" git clone {}", server.url("demo/hello"));
println!(" git ls-remote {}", server.url("demo/hello"));
println!();
// Both spellings, because this binary cannot tell which one invoked it and
// an operator who reached it through xtask should not be told to go and
// find the other.
println!(" gunnar-installtest status (or: cargo xtask bare status)");
println!(" gunnar-installtest down (or: cargo xtask bare down) — stops it and");
println!(" proves it released the port and the store lock");
Ok(())
}
fn op_status(root: &Path) -> Result<(), String> {
let server = bare::Instance::load(root)?;
let alive = server.alive();
println!(" root : {}", root.display());
println!(
" pid : {} ({})",
server.pid,
if alive { "running" } else { "GONE" }
);
println!(" ssh : 127.0.0.1:{}", server.ssh_port);
println!(
" listening : {}",
if bare::listening_on(server.ssh_port) {
"yes"
} else {
"no"
}
);
// The lock is redb's on `control.redb` and is held by the *process*, so its
// holder is the pid above and there is no file to read a name out of. Saying
// that is better than the line this replaced, which read a path nothing has
// written since the redb move and printed "released" under a live server every
// time. See `bare::store_lock_path`.
println!(
" store lock: {} — {}",
bare::store_lock_path(&server.data).display(),
if alive {
"held by the pid above (redb, kernel-held: it is released when that process is)"
} else {
"not held: its owner is gone"
}
);
if !alive {
return Err("the recorded server is not running".to_string());
}
Ok(())
}
/// `down`: stop it, and then say what it actually released.
///
/// The exit code is the release check, not the signal. A server that exits and
/// leaves its lock behind has failed to shut down even though `kill` returned
/// zero, and that is the failure a stale store lock is.
fn op_down(root: &Path, remove_root: bool) -> Result<(), String> {
let server = bare::Instance::load(root)?;
let port = server.ssh_port;
let pid = server.pid;
let teardown = server.down("TERM", remove_root);
println!("gunnar {pid} on :{port} — {}", teardown.render());
teardown.check()
}
/// `pressure`: bring one up, load it until something gives, report where and
/// how, tear it down.
fn op_pressure(
binary: &Path,
root: &Path,
args: &[String],
num: &dyn Fn(&str, usize) -> usize,
) -> Result<(), String> {
let blobs = num("--blobs", 1200);
let blob_bytes = num("--blob-bytes", 4096);
let pushes = num("--pushes", 0);
// `--serial` is the red switch for the overlap measure, not a mode anyone
// should benchmark with: it runs the clients one at a time, so a correct
// harness must report ~1.0x and the floor below must FIRE.
let drive = if args.iter().any(|a| a == "--serial") {
bare::Drive::Serial
} else {
bare::Drive::Together
};
let ramp: Vec<usize> = args
.iter()
.position(|a| a == "--ramp")
.and_then(|i| args.get(i + 1))
.map(|v| v.split(',').filter_map(|n| n.trim().parse().ok()).collect())
.unwrap_or_else(|| vec![1, 2, 4, 8, 16, 32, 48, 64, 96, 128]);
// One extra principal per 8 clients at the top of the ramp, so the
// server-wide gate is reachable at all; see UpOptions::extra_principals.
let principals = num("--principals", 1);
// Shrinking the SERVER is how contention is reached without ~200 concurrent
// `git` processes: the gate's rule is a ratio (75 % of the server-wide
// limit), so a small server exercises the same arithmetic. `None` leaves
// the shipped 256/32.
let server_wide = args
.iter()
.position(|a| a == "--server-wide")
.and_then(|i| args.get(i + 1))
.and_then(|v| v.parse::<usize>().ok());
let per_principal = args
.iter()
.position(|a| a == "--per-principal")
.and_then(|i| args.get(i + 1))
.and_then(|v| v.parse::<usize>().ok());
let opts = bare::UpOptions {
extra_principals: principals.saturating_sub(1),
max_concurrent_transfers: server_wide,
max_transfers_per_principal: per_principal,
// The pre-max-min behaviour, on demand: `--hard-per-principal` turns
// the floor back into a wall so the admission regression can be re-measured
// against this very binary instead of taken on trust.
hard_per_principal_limit: args.iter().any(|a| a == "--hard-per-principal"),
..bare::UpOptions::default()
};
let server = bare::Instance::up(binary, root, &opts)?;
eprintln!(
"pressure: gunnar {} on 127.0.0.1:{}, ready in {} ms",
server.pid,
server.ssh_port,
server.ready.as_millis()
);
let objects = bare::build_pressure_fixture(&server, "demo/hello", blobs, blob_bytes)?;
eprintln!(
"pressure: fixture {objects} objects ({blobs} blobs × {blob_bytes} random bytes) pushed"
);
eprintln!("pressure: {principals} principal(s) driving the load");
eprintln!(
"pressure: {} cores, load average {}",
std::thread::available_parallelism()
.map(|n| n.get())
.unwrap_or(0),
std::fs::read_to_string("/proc/loadavg")
.unwrap_or_default()
.split_whitespace()
.take(3)
.collect::<Vec<_>>()
.join(" ")
);
println!();
println!("── pressure: concurrent stock-git clones against one bare gunnar ──");
let mut rounds = Vec::new();
let mut broke_at: Option<usize> = None;
for n in ramp {
let round = bare::pressure_round(&server, "demo/hello", n, pushes, drive)?;
println!("{}", round.render());
// A step that reports `n` clients and an overlap of ~1 measured a queue
// of one, whatever `n` said. Past the first step that is a broken
// fixture, not a result, and it must not be reported as one.
if round.concurrency >= 4 && round.overlap_x < 1.5 {
return Err(format!(
"at {} concurrent the measured overlap was only {:.2}x: the transfers did \
not run at the same time, so this ramp is not measuring concurrency. Use a \
bigger --blobs.",
round.concurrency, round.overlap_x
));
}
if broke_at.is_none() && (round.refused() > 0 || !round.faults.is_empty()) {
broke_at = Some(round.concurrency);
}
rounds.push(round);
}
println!();
match broke_at {
Some(n) => println!(" first refusal or fault at {n} concurrent transfers"),
None => println!(" nothing gave: every transfer at every step completed"),
}
for r in &rounds {
if !r.agrees() {
println!(
" DISAGREEMENT at {}: clients reported {} refusals, the server logged {}",
r.concurrency,
r.refused(),
r.log_refused_principal + r.log_refused_server
);
}
for f in &r.faults {
println!(" fault at {}: {f}", r.concurrency);
}
}
// A ramp in which NOTHING succeeded anywhere is not a pressure result, it
// is a broken harness reporting a very orderly failure.
let total_ok: usize = rounds.iter().map(|r| r.ok).sum();
if total_ok == 0 {
let teardown = bare::Instance::load(root).map(|s| s.down("TERM", false));
let _ = teardown;
return Err(
"not one transfer succeeded at any concurrency; the ramp measured nothing".to_string(),
);
}
let teardown = server.down("TERM", false);
println!();
println!(" teardown: {}", teardown.render());
teardown.check()
}
/// Scratch lives on md1, never in `/tmp` (that is RAM on this box) and never
/// inside the checkout, where a stray guest disk would end up in `git status`.
fn scratch_root() -> PathBuf {
std::env::var("GUNNAR_INSTALLTEST_SCRATCH")
.map(PathBuf::from)
.unwrap_or_else(|_| gunnar_installtest::expand_home("~/scratch/gunnar-installtest"))
}
// ── what each verb accepts ──────────────────────────────────────────────────
//
// **Why a table and not a CLI crate.** `installtest/` is a detached crate with a
// deliberate dependency budget — its manifest header records why, package by
// package — and the question here is not "parse a command line" but "is this
// argument one this binary READS?". The answer to that is a list, one row per
// verb, and it is small enough to hold in view: eleven flags on the largest
// verb. A parser framework would answer it too, and would also bring a
// dependency tree into the crate whose whole point is that it drags nothing into
// gunnar's own resolution.
//
// What keeps the list honest is not care, it is
// `strict_parse::every_flag_this_file_reads_is_one_some_verb_accepts` below:
// it reads THIS FILE and holds every `flag("--…")`, `num("--…")` and
// `a == "--…"` in it against the table. A flag that is read but not listed is a
// failing test, not a second silent difference between two runs.
/// An accepted argument: its spelling, and whether it consumes the next one.
type Flag = (&'static str, bool);
/// `--iso PATH` — takes the next argument as its value.
const VALUE: bool = true;
/// `--keep` — stands alone.
const SWITCH: bool = false;
/// Every verb, for the drift test and for the unknown-mode refusal. In the order
/// [`usage`] names them.
const VERBS: &str = "bare chain container kvm iso all red installer pair up status down pressure";
/// `bare` and `all`: the only arms that run the LIFECYCLE matrix, so the only
/// ones that take `--life-fault`.
const BARE: &[Flag] = &[
("--binary", VALUE),
("--fault", VALUE),
("--life-fault", VALUE),
("--keep", SWITCH),
];
/// container / kvm / iso.
const ARM: &[Flag] = &[("--binary", VALUE), ("--fault", VALUE), ("--keep", SWITCH)];
/// `chain` takes no `--keep`: it holds no container and no guest, so there is
/// nothing to leave up — its module docs say so, and this is that sentence made
/// enforceable.
const CHAIN: &[Flag] = &[("--binary", VALUE), ("--fault", VALUE)];
/// `red` drives its own fault list per arm and never keeps an instance, so
/// `--fault` and `--keep` are not quietly ignored here: they are refused, which
/// is the only way an operator finds out the run they asked for is not the run
/// they get.
const RED: &[Flag] = &[("--binary", VALUE), ("--arm", VALUE)];
const INSTALLER: &[Flag] = &[
("--firmware", VALUE),
("--serial", VALUE),
("--target", VALUE),
("--iso", VALUE),
("--work", VALUE),
("--twin", VALUE),
("--disks", VALUE),
("--upgrade-from", VALUE),
("--cli", VALUE),
("--drive-cmd", VALUE),
("--grpc-only", SWITCH),
];
const PAIR: &[Flag] = &[
("--iso", VALUE),
("--work", VALUE),
("--members", VALUE),
("--mem-mb", VALUE),
("--vcpus", VALUE),
("--drive-cmd", VALUE),
("--keep", SWITCH),
];
const UP: &[Flag] = &[
("--root", VALUE),
("--binary", VALUE),
("--port", VALUE),
("--grpc", SWITCH),
("--reuse", SWITCH),
];
const STATUS: &[Flag] = &[("--root", VALUE), ("--binary", VALUE)];
const DOWN: &[Flag] = &[
("--root", VALUE),
("--binary", VALUE),
("--keep-data", SWITCH),
];
const PRESSURE: &[Flag] = &[
("--root", VALUE),
("--binary", VALUE),
("--ramp", VALUE),
("--blobs", VALUE),
("--blob-bytes", VALUE),
("--pushes", VALUE),
("--principals", VALUE),
("--server-wide", VALUE),
("--per-principal", VALUE),
("--serial", SWITCH),
("--hard-per-principal", SWITCH),
];
/// The flags `mode` accepts, or `None` when this binary has no such verb.
fn accepted(mode: &str) -> Option<&'static [Flag]> {
Some(match mode {
"bare" | "all" => BARE,
"chain" => CHAIN,
"container" | "kvm" | "iso" => ARM,
"red" => RED,
"installer" => INSTALLER,
"pair" => PAIR,
"up" => UP,
"status" => STATUS,
"down" => DOWN,
"pressure" => PRESSURE,
_ => return None,
})
}
/// Every argument after the verb, held against [`accepted`].
///
/// Three refusals, and each of them used to be a silent difference between the
/// run that was asked for and the run that happened: an argument no verb
/// accepts, an argument this verb does not accept, and a flag that takes a value
/// and was given none (`--iso` at the end of a line used to fall back to the
/// DEFAULT medium and gate something nobody asked about).
///
/// A value is never inspected: `--firmware seabios` consumes `seabios` whatever
/// it looks like, exactly as `flag()` does two screens up. This function answers
/// "is this argument read at all", and the verbs keep answering "is this value a
/// good one" — `FirmwareChoice::parse` and its neighbours, which already refuse
/// by name.
fn check_args(mode: &str, rest: &[String]) -> Result<(), String> {
let Some(flags) = accepted(mode) else {
return Err(format!(
"install tests: unknown mode `{mode}` — one of: {VERBS}\n \
`gunnar-installtest --help` prints the whole usage."
));
};
let mut i = 0;
while i < rest.len() {
let arg = rest[i].as_str();
match flags.iter().find(|(name, _)| *name == arg) {
Some((name, VALUE)) => {
if i + 1 >= rest.len() {
return Err(format!(
"install tests: `{mode} {name}` wants a value and was given none.\n{}",
accepts(mode, flags)
));
}
i += 2;
}
Some(_) => i += 1,
None => {
return Err(format!(
"install tests: `{mode}` does not accept `{arg}`.\n{}",
accepts(mode, flags)
))
}
}
}
Ok(())
}
/// The second half of every refusal: what this verb DOES accept, and why the
/// argument was not simply ignored.
fn accepts(mode: &str, flags: &[Flag]) -> String {
let names: Vec<&str> = flags.iter().map(|(n, _)| *n).collect();
format!(
" `{mode}` accepts: {}\n \
Nothing here is dropped or defaulted in silence: otherwise a stale binary and\n \
a current one behave identically on the flag that decides WHICH test runs\n \
(measured 2026-09-17, on `--grpc-only`, twice).\n \
`gunnar-installtest --help` prints the whole usage.",
names.join(" ")
)
}
#[cfg(test)]
mod strict_parse {
use super::*;
fn args(list: &[&str]) -> Vec<String> {
list.iter().map(|a| (*a).to_string()).collect()
}
/// The list a refusal quotes and the table a refusal is computed from are
/// two objects, and only this holds them together.
#[test]
fn every_verb_named_in_the_refusal_has_a_flag_list() {
for verb in VERBS.split_whitespace() {
assert!(
accepted(verb).is_some(),
"`{verb}` is offered to the operator by name but no flag list answers for it"
);
}
assert!(accepted("definitely-not-a-mode").is_none());
}
/// **The drift guard, and the reason the table is allowed to be a table.**
///
/// It reads this file and collects every argument the code actually reaches
/// for — `flag("--…")`, `num("--…", …)` and `a == "--…"` — then demands that
/// some verb accept it. A flag added to the parse and forgotten here would
/// otherwise be refused at the door while the code below waited for it: the
/// same silent difference, pointing the other way.
#[test]
fn every_flag_this_file_reads_is_one_some_verb_accepts() {
let src = include_str!("main.rs");
let mut read: Vec<&str> = Vec::new();
for pattern in ["flag(\"", "num(\"", "a == \""] {
for piece in src.split(pattern).skip(1) {
if let Some(name) = piece.split('"').next() {
read.push(name);
}
}
}
let known: Vec<&str> = [
BARE, ARM, CHAIN, RED, INSTALLER, PAIR, UP, STATUS, DOWN, PRESSURE,
]
.iter()
.flat_map(|table| table.iter().map(|(n, _)| *n))
.collect();
for name in read {
// Only real spellings: ASCII lower-case, digits and dashes. The
// prose above quotes `flag("--…")` with an ellipsis, and a scanner
// that reads its own documentation as code is a scanner that fails
// for the wrong reason (measured, the first time this ran). A flag
// spelled with a capital would slip past this line, and there is
// none: every argument in the tables above is lower-case.
//
// `--help` is answered before any verb is chosen, so it belongs to
// no verb's table. Everything else does.
let spelled = name.strip_prefix("--").filter(|rest| {
!rest.is_empty()
&& rest
.chars()
.all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '-')
});
if spelled.is_none() || name == "--help" {
continue;
}
assert!(
known.contains(&name),
"`main.rs` reads `{name}` but no verb accepts it, so it is refused at the door \
while the code below waits for it. Add it to the table it belongs to."
);
}
}
/// The refusal, in the shape an operator reads it, without spawning
/// anything. Its end-to-end pair — exit code and stream — is
/// `tests/strict_flags.rs`.
#[test]
fn the_refusal_names_the_argument_and_lists_the_alternative() {
let e = check_args("installer", &args(&["--grpc_only"])).unwrap_err();
assert!(e.contains("`--grpc_only`"), "{e}");
assert!(
e.contains("--grpc-only"),
"the near miss must be visible: {e}"
);
assert!(e.contains("`installer` accepts:"), "{e}");
let e = check_args("bare", &args(&["--grpc-only"])).unwrap_err();
assert!(e.contains("`bare` does not accept `--grpc-only`"), "{e}");
let e = check_args("installer", &args(&["--iso"])).unwrap_err();
assert!(e.contains("wants a value"), "{e}");
let e = check_args("red", &args(&["bare"])).unwrap_err();
assert!(e.contains("does not accept `bare`"), "{e}");
let e = check_args("nope", &[]).unwrap_err();
assert!(e.contains("unknown mode `nope`"), "{e}");
}
/// The pair: what the gate actually runs must pass, or the refusals above
/// are met by a parse that refuses everything.
#[test]
fn the_calls_the_gate_makes_are_accepted() {
// `cargo xtask reimage --machine twin`, verbatim.
assert!(check_args(
"installer",
&args(&["--iso", "/x.iso", "--work", "/w", "--grpc-only"])
)
.is_ok());
// …and the primary's gate, which passes no door.
assert!(check_args("installer", &args(&["--iso", "/x.iso", "--work", "/w"])).is_ok());
// A value that looks like a flag is still a value.
assert!(check_args(
"installer",
&args(&["--firmware", "seabios", "--grpc-only"])
)
.is_ok());
assert!(check_args("red", &args(&["--arm", "bare"])).is_ok());
assert!(check_args("pair", &args(&["--members", "main,twin,spare", "--keep"])).is_ok());
assert!(check_args("down", &args(&["--root", "/r", "--keep-data"])).is_ok());
assert!(check_args("bare", &[]).is_ok());
}
}
fn usage() -> &'static str {
"usage: gunnar-installtest <bare|chain|container|kvm|iso|all|red|installer|pair> [options]
gunnar-installtest <up|status|down|pressure> [options]
gunnar-installtest --help
bare run gunnar as an ORDINARY PROCESS — no container, no VM, no ISO —
push to it and clone from it with stock git over ssh, compare the
object graphs, restart it on the same port, kill it and prove the
stale store lock is reclaimed, then stop it and prove it released
the port and the lock
chain the WHOLE chain against the installed binary's own CLI: push
source over ssh, build it in the sandbox, attest it, make an OCI
image of it, land it in the registry, pull it back over `/v2/`
and verify the chain from the image digest down to a line of
source. Three phases, because the control store is single-writer
and the chain's admin verbs are offline verbs — see `chain`'s
module docs, which record where that was measured
container install gunnar into an OCI container through draupnir, push to it
and clone from it over ssh, compare the object graphs
kvm install gunnar into a KVM guest through draupnir — as a systemd unit,
via Skidbladnir — then the same push/clone/object-graph assertion
driven from the host over a forwarded port
iso boot the OS-LESS appliance ISO the ⚒ Build Thing cell produced
(gunnar is PID 1, no systemd) under draupnir, run the same
push/clone/object-graph assertion plus the whole git matrix over a
forwarded port, check the build reference on its SSH banner, then
power-cut it and boot it again
all all four
red the green run plus every fault, and every fault must go red
installer the INSTALLER medium under the customer's firmware: install from
the ISO (IDE cdrom + virtio, target disk first, 2 NICs), boot the
installed disk and probe :2222/:50051, power-cut it, boot again
--target <preset> render the machine from edda's roster
(upcloud qemu-local qemu-local-bios
gce-virtio hetzner-cloud ovh-public-cloud;
a needs-real-cloud preset is refused by
name) and emit appliance-boot/<preset>
rows; --firmware/--serial override it
--firmware uefi|seabios (default uefi; seabios = UpCloud)
--serial uart|none (none = no UART at all, UpCloud)
--iso PATH (default $GUNNAR_INSTALLER_ISO, else
.iso-out/gunnar-installer-open.iso)
--disks G,G,… extra fixed disks (GiB) beside the target,
attached vdb, vdc, … on every leg: the
multi-disk /data door — the smallest
disk boots, the largest equal group is
one striped xfs, odd disks stay untouched
(rule + verdict in firmware.rs)
--upgrade-from OLD.iso the data-survives door: OLD installs
onto a blank target (U1), the door
bootstraps + pushes a repo (U2), THIS
medium reinstalls on the SAME image
(U3: korp/install = mode=upgrade,
korp-data blake3 identical), and the
appliance comes back closed with the
repo (U4). --cli PATH = the gunnar-cli
that drives it (default
target/release/gunnar-cli)
--grpc-only prove the medium by its CONTROL PLANE —
`Bootstrap.Status`, which names the
build stamp and the role the medium
declares — instead of the SSH banner.
THE DOOR FOR A TWIN MEDIUM, which
serves no git at all, so the banner leg
can never pass however good the medium
is. Refused on a medium that binds no
control plane off the loopback, so it
cannot wave a primary through
--work PATH where this run's disks and logs go
(default: scratch, named after the
shape or the preset)
--cli PATH the gunnar-cli that drives
--upgrade-from (default
target/release/gunnar-cli)
--drive-cmd PATH a host program run once in B2 the
instant the ports answer:
PATH <ssh_port> <grpc_port>
--twin iso|target|larger (target = the medium's virtio face
padded to the target's size and
attached FIRST — UpCloud's ISO storage
beside the target; larger = +1 GiB,
the shape capacity ranking always
wiped; the installer must pick the
blank disk, not the twin)
pair install and boot TWO appliances and hand the operator their ports.
Not a test: it is the rig the quick start's replication section is
written against, and what happens next is whatever a person types
--members main,twin,spare who to install (default main,twin;
the third is the cold spare the
primary-was-nuked drill needs)
--iso PATH the medium both are installed from
--work PATH where the guests live
--mem-mb N / --vcpus N the shape of each guest
--drive-cmd PATH run once when a guest's ports answer
--keep leave both up, which is the point
--fault NAME wrong-port | missing-binary | killed-service | empty-repo
--life-fault N bare only: no-teardown | port-still-bound | lock-holder-alive
--arm NAME restrict `red` to one arm
--binary PATH the gunnar to install
--keep leave the instance up
the bare server a human drives (state under --root, default $GUNNAR_BARE_ROOT
or ~/scratch/gunnar-bare):
up start one and print the git lines to paste
--port N bind this port instead of a free one
--grpc also bind the control plane
--reuse keep an existing data root and its repositories
status pid, port, whether anything is listening, who holds the store lock
down stop it and PROVE it released the port and the store lock
--keep-data leave the data root on disk
pressure ramp concurrent clones (and pushes) until something gives
--ramp 1,2,4,… the concurrency steps
--blobs N fixture size (default 1200)
--blob-bytes N bytes per blob (default 4096)
--pushes N concurrent pushes alongside each clone step
--principals N how many principals drive the load (default 1)
--server-wide N / --per-principal N shrink the SERVER's admission
limits, which is how contention is reached
without 200 concurrent gits
--serial RED switch: run the clients one at a time, so a
correct harness must report ~1.0x and fire the
overlap floor
--hard-per-principal the pre-max-min behaviour, on demand
"
}