installtest/src/bare.rs
//! **The bare arm**: gunnar as an ordinary process. No container, no VM, no ISO.
//!
//! Every other proof in this repository wraps the server in something —
//! [`crate::container`] puts it in an OCI container behind draupnir,
//! [`crate::kvm`] boots a machine, `cargo xtask iso` builds an El Torito image.
//! Each of those is a proof about the *packaging* as much as about gunnar, and
//! each brings a whole engine along that can fail on its own.
//!
//! This one brings nothing. `Command::new(gunnar).arg("serve")`, stock `git`
//! against the port it prints, `SIGTERM`, gone. It is the plainest statement the
//! repository can make and it is the one nobody had written down.
//!
//! # Why it is not redundant with the container arm
//!
//! Three things are only reachable here, and the third is the one that matters:
//!
//! 1. **A human can run it.** `gunnar-installtest up` leaves a server running and
//! prints the `git clone` line to paste. `down` stops it. That is a thing an
//! operator does at a terminal; a `cargo test` is not.
//! 2. **Teardown is observable.** A container's teardown is podman's teardown:
//! the namespace goes away and takes every socket and every file lock with it,
//! whatever gunnar did or failed to do. Here the process exits into a kernel
//! that is still holding gunnar's port and gunnar's lock file, so
//! "did it actually let go" is a question with an answer.
//! 3. **The stale-lock reclaim has never run against the SERVER.**
//! `gunnar-store`'s `LockFile` records `<pid> <start-time>` so a lock left by
//! a killed process can be reclaimed at the next start. That is
//! proved at the store level — `repack.rs::a_repack_killed_mid_flight_leaves_
//! the_old_state_serving` SIGKILLs a real child and reopens — but never for
//! `gunnar serve` itself, because the only two proofs that run the server as
//! a process put it in a container or a guest, and **every container run
//! starts from a fresh volume**. There was never a lock there to reclaim. So
//! the thing that reclaim actually exists for, an appliance that comes back after
//! a power cut, had no end-to-end proof.
//! [`LifeFault::LockHolderAlive`] is the red that keeps this honest: without
//! it, "kill it and start it again" passes just as happily against a gunnar
//! that reclaims locks a live process is holding.
//!
//! # The assertion is the same one
//!
//! [`Probe::push_then_clone`] and [`Graph::assert_matches`] are shared with the
//! other two arms, floor and all: two empty object graphs compare equal,
//! so equality alone is worthless and [`crate::MIN_OBJECTS`] is what makes a
//! match mean something.
//!
//! # What the liveness check deliberately does NOT reuse
//!
//! [`proc_start_time`] re-implements `gunnar-store`'s field-22 read instead of
//! calling it. That is a deviation from reuse-don't-twin, and it is on purpose: this
//! function exists to *check* the store's lock bookkeeping, and a check written
//! by calling the code under test agrees with it by construction. Fifteen lines
//! of duplication buys an independent witness. It is also the only duplication
//! here; everything else routes through the shared code.
use std::io::Read;
use std::path::{Path, PathBuf};
use std::process::{Child, Command, Stdio};
use std::time::{Duration, Instant};
use crate::{
bootstrap_plan, free_port, keygen, wait_for_ssh, ArmResult, Exchange, Fault, Graph, Probe,
Stages, KEYFILE,
};
/// Where `up` records what it started, so `down` in another shell can find it.
pub const STATE_FILE: &str = "instance";
/// How long a `SIGTERM`ed server gets to exit before teardown calls it a
/// failure.
///
/// `Server::shutdown` gives the gRPC listener `GRPC_SHUTDOWN_GRACE` (2 s) and
/// then aborts it, so anything past a handful of seconds is gunnar not
/// shutting down rather than gunnar shutting down slowly. Ten seconds is
/// generous enough not to flake on a loaded box and short enough that a hang is
/// reported as a hang.
pub const STOP_BUDGET: Duration = Duration::from_secs(10);
/// How long the server gets to bind and answer with an SSH banner.
pub const READY_BUDGET: Duration = Duration::from_secs(30);
// ── the lifecycle red switches ──────────────────────────────────────────────
/// A deliberate defect in the **lifecycle** half — up, down, and the stale-lock
/// reclaim — as opposed to [`Fault`], which sabotages the git exchange.
///
/// They are a separate enum rather than four more [`Fault`] variants because
/// [`Fault::ALL_RED`] is run against *every* arm, and none of these means
/// anything to a container: podman's teardown releases the port and the lock
/// whatever gunnar does, and a fresh volume has no lock to reclaim. Bolting
/// them onto the shared enum would force the other two arms to skip them, and a
/// suite that skips reports green having tested nothing.
///
/// Every variant must make the arm **red**. `gunnar-installtest bare-red` runs
/// each in turn and fails if any comes out green.
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub enum LifeFault {
/// No fault: the arm must PASS.
None,
/// Never send the stop signal. The teardown assertions — process gone, no
/// listener on the port, lock file released — must then all report the
/// server still there. Proves the teardown check is reading the world
/// rather than reporting what it just asked for.
NoTeardown,
/// Hold a second listener on the port after the server has stopped, so
/// [`listening_on`] must still see a `LISTEN` socket. Proves the port probe
/// can say no.
///
/// Bound with `SO_REUSEADDR` off is not enough — see [`listening_on`] for
/// why a `bind` probe is the wrong test and a `LISTEN`-state read is the
/// right one.
PortStillBound,
/// Run the reclaim step **without killing the lock holder first**: the
/// first server is left alive and a second is started on the same data
/// root. The second must be refused.
///
/// This is the one that matters. Without it, "start, kill, start again,
/// it served" is indistinguishable from "start, stop, start again", which
/// would pass against a gunnar that reclaimed *any* lock it found,
/// including one a live process is holding — the exact behaviour
/// `arrow_log.rs` says is far worse than refusing to boot.
LockHolderAlive,
// `LockOwnedByHarness` was here, and it is deleted rather than left to pass
// by accident. It rewrote `control.git/gunnar-store/gunnar-store.lock` with
// this process's own pid and start time so that the reclaim would find a
// live owner and refuse — a proof about the CONTENTS of a pid-file lock.
// The move to redb put the store lock in redb's kernel lock on `control.redb`
// (see [`store_lock_path`]), which has no contents to forge: a lock held by
// a live process is exactly what `LockHolderAlive` already arranges, and it
// arranges it with a real gunnar rather than with a file the server no
// longer reads. Kept as a variant it would have written a file nothing
// opens, the reclaim would have succeeded, and a fault that DEMANDS red
// would have come out green — which is worse than not having it.
}
impl LifeFault {
pub const ALL_RED: &'static [LifeFault] = &[
LifeFault::NoTeardown,
LifeFault::PortStillBound,
LifeFault::LockHolderAlive,
];
pub fn name(self) -> &'static str {
match self {
LifeFault::None => "none",
LifeFault::NoTeardown => "no-teardown",
LifeFault::PortStillBound => "port-still-bound",
LifeFault::LockHolderAlive => "lock-holder-alive",
}
}
pub fn parse(s: &str) -> Result<LifeFault, String> {
match s {
"none" => Ok(LifeFault::None),
"no-teardown" => Ok(LifeFault::NoTeardown),
"port-still-bound" => Ok(LifeFault::PortStillBound),
"lock-holder-alive" => Ok(LifeFault::LockHolderAlive),
other => Err(format!(
"unknown lifecycle fault `{other}`; known: none, no-teardown, \
port-still-bound, lock-holder-alive"
)),
}
}
}
// ── /proc, read directly ────────────────────────────────────────────────────
/// Field 22 of `/proc/<pid>/stat`, the process start time in clock ticks since
/// boot. `None` when there is no such process.
///
/// Field 2 is `comm`, parenthesised and free to contain spaces and close-parens,
/// so the split is anchored on the **last** `)`. Everything after it starts at
/// field 3, which puts field 22 at index 19.
///
/// Deliberately not `gunnar_store`'s copy; see this module's header.
pub fn proc_start_time(pid: u32) -> Option<u64> {
let stat = std::fs::read_to_string(format!("/proc/{pid}/stat")).ok()?;
let (_, fields) = stat.rsplit_once(')')?;
// Field 3 is the state. `Z` is a process that has already died and is only
// waiting for its parent to collect its exit status: it holds no
// descriptors and can write nothing, so for every question this module asks
// — is the server still serving, is it still holding the lock — it is gone.
//
// This is not decoration. It was measured: `SIGKILL` on a server this
// harness spawned left a zombie, `/proc/<pid>/stat` still read back field
// 22 unchanged, and the stale-lock proof reported "pid N survived SIGKILL"
// for a process that had been dead for seconds. `Instance::reap` is the
// other half of the fix; this half also covers a caller that has no handle
// to reap with.
if fields.split_whitespace().next() == Some("Z") {
return None;
}
fields.split_whitespace().nth(19)?.parse().ok()
}
/// Is this exactly the process that was started — not merely *a* process with
/// that pid?
///
/// Pids are recycled. `kill -0` and a bare `/proc/<pid>` test both answer "yes"
/// for a recycled pid, which would make every teardown assertion here able to
/// pass for the wrong reason.
pub fn same_process(pid: u32, started: Option<u64>) -> bool {
match (proc_start_time(pid), started) {
(Some(now), Some(then)) => now == then,
(Some(_), None) => true,
(None, _) => false,
}
}
/// Is anything **listening** on `127.0.0.1:port` right now?
///
/// Read out of `/proc/net/tcp` and `/proc/net/tcp6` rather than probed with a
/// `bind`, and the difference is not pedantry — it was measured on the desktop while
/// writing this file. A `bind` probe on a just-vacated port fails with
/// `EADDRINUSE` for around a minute, because the *connections* the server
/// accepted are sitting in `TIME-WAIT` on that local port long after the
/// listening socket is gone. A teardown check written that way reports
/// "the port was not released" every single time, and its red is therefore
/// blind: it can never distinguish a leaked listener from ordinary `TIME-WAIT`.
///
/// The state column is hex; `0A` is `TCP_LISTEN`. That says exactly the thing
/// worth asserting — nobody is accepting on this port any more — and says
/// nothing about sockets that are merely finishing.
pub fn listening_on(port: u16) -> bool {
for table in ["/proc/net/tcp", "/proc/net/tcp6"] {
let Ok(text) = std::fs::read_to_string(table) else {
continue;
};
for line in text.lines().skip(1) {
let mut f = line.split_whitespace();
let (Some(_sl), Some(local), Some(_rem), Some(state)) =
(f.next(), f.next(), f.next(), f.next())
else {
continue;
};
if state != "0A" {
continue;
}
let Some((_addr, p)) = local.rsplit_once(':') else {
continue;
};
if u16::from_str_radix(p, 16) == Ok(port) {
return true;
}
}
}
false
}
/// Send `sig` (`TERM`, `KILL`, `INT`, `HUP`, `QUIT`, `USR1`, `USR2`, or `0` to probe) to
/// `pid` through `kill(2)`. `Child::kill` can only send `SIGKILL`, and cannot reach a process
/// this process did not spawn, which is precisely the `down`-in-another-shell case. It was a
/// shell-out to `kill(1)`, which the CI image (ci4) does not carry: `down` then failed with
/// "No such file or directory" before any signal was sent.
pub fn signal(pid: u32, sig: &str) -> Result<(), String> {
let num = match sig.trim_start_matches("SIG") {
"TERM" => libc::SIGTERM,
"KILL" => libc::SIGKILL,
"INT" => libc::SIGINT,
"HUP" => libc::SIGHUP,
"QUIT" => libc::SIGQUIT,
"USR1" => libc::SIGUSR1,
"USR2" => libc::SIGUSR2,
"0" => 0,
other => {
return Err(format!(
"kill -{other} {pid}: not a signal this harness sends"
))
}
};
let target = libc::pid_t::try_from(pid).map_err(|_| format!("kill -{sig} {pid}: not a pid"))?;
// SAFETY: kill(2) takes two integers and touches no memory of ours.
if unsafe { libc::kill(target, num) } != 0 {
return Err(format!(
"kill -{sig} {pid} failed: {}",
std::io::Error::last_os_error()
));
}
Ok(())
}
// ── the instance ────────────────────────────────────────────────────────────
/// How [`Instance::up`] should bring a server up.
#[derive(Clone, Debug, Default)]
pub struct UpOptions {
/// The port to bind. `None` takes one the kernel says is free.
///
/// A concrete port rather than `:0` because the whole teardown proof is
/// *"start again on the same port"*, which needs a port to name. `:0` would
/// also mean parsing the bound address back out of a log that carries ANSI
/// escapes.
pub ssh_port: Option<u16>,
/// Also bind the gRPC control plane. Off by default: this arm is about the
/// git wire, and a second listener is a second thing to fail.
pub grpc: bool,
/// Repositories to create during provisioning, beyond the shared
/// [`crate::bootstrap_plan`]'s two.
///
/// Provisioned rather than created afterwards because **the control store
/// is single-writer**: while `gunnar serve` holds it, `gunnar repo create`
/// answers `Database already open. Cannot acquire lock.` That is the store's
/// discipline and the reason `serve --runner` runs the build runner
/// in-process; see [`crate::chain`], which measured it.
///
/// sha1, like the shared plan's, so stock `git` needs no object-format
/// negotiation.
pub extra_repos: Vec<String>,
/// Also bind smart HTTP, which is also where the OCI registry answers.
///
/// `None` leaves it unbound, and that is the default for the same reason
/// `grpc` is off: most arms are about the git wire over SSH. The **chain**
/// arm ([`crate::chain`]) sets it, because a registry with no listener is a
/// registry nothing can pull from.
pub http_port: Option<u16>,
/// Keep an existing data root instead of wiping and re-bootstrapping it.
/// This is what makes the restart and stale-lock steps a restart rather
/// than a fresh install.
pub reuse_data: bool,
/// Exec this instead of the located binary. Only [`Fault::MissingBinary`]
/// sets it, to a path that does not exist.
pub exec: Option<PathBuf>,
/// Provision this many EXTRA principals, each with its own key, on top of
/// the one `bootstrap_plan` creates.
///
/// Needed to reach the server-wide ceiling at all. `max_transfers_per_
/// principal` is 32 and `max_concurrent_transfers` is 256, so a load driven
/// from one key is refused at 32 and the server-wide limit is simply
/// unreachable however many clients are started — a fact worth knowing, and
/// one that makes "gunnar's concurrency limit is 256" wrong for every
/// single-key client in existence. Eight or more principals put the
/// server-wide gate back in reach.
pub extra_principals: usize,
/// Pass `--max-concurrent-transfers`. `None` leaves the appliance default
/// (256).
///
/// The pressure arm sets it. Reaching contention at the shipped 256 needs
/// 192 transfers genuinely in flight, i.e. ~200 `git` processes; shrinking
/// the *server* instead of growing the client count measures the same gate
/// at a size a bench can afford. That is a legitimate shrink because the
/// gate's rule is a ratio, not an absolute — unlike shrinking the fixture,
/// which changes whether the transfers overlap at all.
pub max_concurrent_transfers: Option<usize>,
/// Pass `--max-transfers-per-principal`. `None` leaves the default (32).
pub max_transfers_per_principal: Option<usize>,
/// Pass `--hard-per-principal-limit`, making the per-principal number an
/// unconditional wall instead of a max-min share.
///
/// This is exactly the **pre-2026-08-03 behaviour**, which is why the
/// pressure driver exposes it: it makes the connection-admission regression re-runnable
/// against the shipped binary rather than only describable. Same server,
/// same fixture, same clients, one flag — solo goes from every slot served
/// to one eighth of them.
pub hard_per_principal_limit: bool,
}
/// A gunnar running as a plain process, and everything needed to reach it and
/// to stop it.
#[derive(Debug)]
pub struct Instance {
pub root: PathBuf,
pub binary: PathBuf,
pub data: PathBuf,
pub key: PathBuf,
pub log: PathBuf,
pub work: PathBuf,
pub ssh_port: u16,
pub pid: u32,
/// `/proc` start time of `pid`, so a recycled pid cannot be mistaken for
/// the server.
pub started: Option<u64>,
/// The smart-HTTP / OCI port, if one was asked for.
pub http_port: Option<u16>,
/// How long from `spawn` to gunnar's own `SSH-2.0-…` banner.
pub ready: Duration,
/// `Some` only in the process that spawned it. `down` from another shell
/// has a pid and no handle, which is the case that forces [`signal`].
child: Option<Child>,
}
/// The file whose lock a running gunnar holds against its data root.
///
/// # This moved, and nothing here noticed for a whole release
///
/// It used to be `control.git/gunnar-store/gunnar-store.lock`: the
/// pid-and-start-time lock `gunnar_core::arrow_log::LockFile` writes beside an
/// Arrow log. The whole control plane then moved to `control.redb`, and
/// `gunnar_core::legacy_secret_log` now opens the Arrow log **only if one is
/// already there** — `gunnar-server/src/stores.rs` says so in as many words:
/// *"a fresh installation never creates"* it. So on any installation made after
/// that change there is no `gunnar-store.lock` anywhere under the data root, on
/// **either** engine.
///
/// MEASURED the desktop 2026-08-14, `gunnar-installtest up` on this checkout's binary,
/// `find <data root> -name '*lock*'`: **nothing**, znippy and gix alike. What is
/// there is `control.redb`, and redb takes an exclusive lock on it — a second
/// `gunnar serve` against a live data root dies with
/// `opening the control store …/control.redb: Database already open. Cannot
/// acquire lock.`
///
/// Everything the lifecycle arm asserted about the old path was therefore
/// either unreachable or vacuous:
///
/// * `Teardown::lock_released` was `!lock.exists()` over a file nothing
/// creates — **true on every run, including a run in which nothing was
/// released at all**. It is gone; the `restart` stage, which brings a second
/// server up on the same data root and makes it answer `ls-remote`, is the
/// applied output that says the store was let go.
/// * the reclaim step read that file and failed on `No such file or directory
/// (os error 2)`, which is `gunnar.bare_lifecycle`'s red on the **gix**
/// column verbatim.
///
/// A kernel-held lock has no stale-owner problem to solve — it dies with the
/// process — so the reclaim below no longer asks a file who owns it. It asks
/// the server, by trying to start a second one; see [`store_is_held`].
pub fn store_lock_path(data: &Path) -> PathBuf {
data.join("control.redb")
}
/// **What a gunnar says when the data root's store is already held**, and the
/// one place that string is written down.
///
/// MEASURED on the desktop 2026-08-14, both engines, a second `gunnar serve` and a
/// `gunnar gc` against a live data root:
///
/// ```text
/// gunnar: opening the control store …/control.redb: the control store failed:
/// opening the control store …/control.redb: Database already open.
/// Cannot acquire lock.
/// ```
///
/// It is redb's own words rather than the bare word "lock", which the server's
/// log says for unrelated reasons ("pack set lock poisoned") and which would
/// make a check pass on the wrong line. It is engine-independent: the control
/// store is redb whichever engine holds the repositories, so this needle does
/// not fork per store.
///
/// **It is `pub` because there were two readers of the old one and they drifted
/// apart.** `bench/examples/nornir-bench.rs`'s `gunnar.endurance` demanded
/// `"locked by another writer"` — the Arrow log's wording, from before the redb move
/// — and reported the refusal it asked for as *"`gunnar gc real/knut` failed
/// for a reason that is not the store lock"* while quoting the store lock
/// refusing it. `bench/` already depends on this crate, so there is now one
/// writer for the sentence and both readers take it from here.
pub const STORE_LOCK_HELD: &str = "Cannot acquire lock";
/// **Is this data root's store held right now?** Answered by trying to take it.
///
/// The only honest form of the question once the lock is the kernel's: start a
/// second `gunnar serve` against the same data root, on a port of its own, and
/// see whether it is refused. `Ok(Some(reason))` is a refusal and the message
/// that came with it; `Ok(None)` means the second server **came up**, which is
/// the answer that must never appear while a first one is live.
///
/// A port of its own is load-bearing (it is the point `LifeFault::
/// LockHolderAlive`'s doc already made about the old proof): sharing the port
/// would let a bind failure produce the refusal, and then the probe would say
/// "held" against a gunnar with no store lock at all.
///
/// # Seen red, the desktop 2026-08-14, gix engine
///
/// Each of the three callers' assertions was broken here in turn and the arm
/// went red with the message quoted, so none of them is a guard that cannot
/// fire:
///
/// * this function forced to `Ok(None)` →
/// *"a second `gunnar serve` came up against …/bare-none-none/data while pid
/// 3774638 was still serving it. Two servers over one data root is the
/// condition the store lock exists to prevent…"*
/// * forced to `Ok(Some("the port was already bound"))` →
/// *"the second server against a live data root was refused, but not by a
/// lock: \"the port was already bound\". Something other than the store
/// turned it away and this step is then not about the store"*
/// * forced to `Ok(None)` on its SECOND call only →
/// *"after the reclaim, a second `gunnar serve` came up against
/// …/bare-none-none/data alongside the running pid 3790417. The store was
/// reclaimed and then not held…"*
fn store_is_held(binary: &Path, root: &Path, reuse: &UpOptions) -> Result<Option<String>, String> {
let probe = UpOptions {
ssh_port: Some(free_port()?),
..reuse.clone()
};
match Instance::up(binary, root, &probe) {
Ok(second) => {
// It must not be left holding a port on this box.
let _ = second.down("KILL", false);
Ok(None)
}
Err(e) => Ok(Some(e)),
}
}
/// One line that says what is wrong with a scratch root, written once and used
/// by every step of [`Instance::up`] that touches it.
///
/// The failure this exists for: an earlier run under `sudo systemd-run --scope`
/// leaves `~/scratch/gunnar-bare` owned by `root`, and every
/// subsequent non-root run then dies on `Permission denied (os error 13)`
/// somewhere deep — the message names neither the root nor its owner, so it
/// reads like a gunnar bug rather than a stale directory. Naming the owner is
/// the whole point: `EACCES` on a root you own is a different problem from
/// `EACCES` on a root owned by uid 0, and the fix is different too.
pub fn unusable_root(path: &Path, e: &std::io::Error) -> String {
use std::os::unix::fs::MetadataExt;
// Walk up to the first component that exists: `create_dir_all` fails on the
// leaf, but the directory that actually denied us is an ancestor.
let existing = std::iter::successors(Some(path), |p| p.parent()).find(|p| p.exists());
let owner = existing.and_then(|p| p.metadata().ok()).map(|m| m.uid());
let mut s = format!("{}: {e}", path.display());
if let (Some(p), Some(uid)) = (existing, owner) {
if e.kind() == std::io::ErrorKind::PermissionDenied {
let who = if uid == 0 {
"root".to_string()
} else {
format!("uid {uid}")
};
s.push_str(&format!(
"\n {} is owned by {who}, and this process is not.",
p.display()
));
if uid == 0 {
s.push_str(
"\n An earlier run under `sudo systemd-run --scope` left it behind. \
These arms do not need that scope (peak RSS is ~283 MB), so run them \
as yourself.",
);
s.push_str(&format!("\n Clear it with: sudo rm -rf {}", p.display()));
}
}
}
s
}
impl Instance {
/// Bring a server up and return once it has answered with an SSH banner.
///
/// Readiness is the banner, never a successful `connect`: `wait_for_ssh`'s
/// header records a container that answered TCP at 0 ms while the gunnar
/// inside it had already died. Nothing forwards ports here, but the rule
/// costs nothing and the shared function is the one both other arms use.
pub fn up(binary: &Path, root: &Path, opts: &UpOptions) -> Result<Instance, String> {
let data = root.join("data");
let keys = root.join("keys");
let work = root.join("work");
let key = keys.join("id_ed25519");
if !opts.reuse_data {
// NOT `let _ =`. A root left behind by an earlier run under
// `sudo systemd-run --scope` is owned by root, and this call is the
// first thing that touches it. Discarding the error made the run
// carry on into `create_dir_all`, `keygen` and finally
// `gunnar namespace create demo`, so the report an operator saw was
// a bare `Permission denied (os error 13)` from four layers down
// with no mention of the root, its owner, or how to clear it.
if let Err(e) = std::fs::remove_dir_all(root) {
if e.kind() != std::io::ErrorKind::NotFound {
return Err(unusable_root(root, &e));
}
}
}
for d in [&data, &keys, &work] {
std::fs::create_dir_all(d).map_err(|e| unusable_root(d, &e))?;
}
if !key.exists() {
keygen(&key)?;
}
if !opts.reuse_data {
provision(binary, &data, &key)?;
for repo in &opts.extra_repos {
provision_repo(binary, &data, repo)?;
}
for i in 0..opts.extra_principals {
let k = keys.join(format!("id_ed25519_{i}"));
if !k.exists() {
keygen(&k)?;
}
provision_principal(binary, &data, &format!("tester{i}"), &k)?;
}
}
let port = match opts.ssh_port {
Some(p) => p,
None => free_port()?,
};
// Resolved before the process starts, so the caller learns the port
// without parsing it back out of a log that carries ANSI escapes.
let http_port = match opts.http_port {
Some(0) => Some(free_port()?),
other => other,
};
let log = root.join("server.log");
let file = std::fs::OpenOptions::new()
.create(true)
.append(true)
.open(&log)
.map_err(|e| format!("{}: {e}", log.display()))?;
let errfile = file
.try_clone()
.map_err(|e| format!("{}: {e}", log.display()))?;
let exec = opts.exec.clone().unwrap_or_else(|| binary.to_path_buf());
let mut cmd = Command::new(&exec);
cmd.arg("--data-root")
.arg(&data)
.arg("serve")
.arg("--ssh-bind")
.arg(format!("127.0.0.1:{port}"));
if opts.grpc {
cmd.arg("--grpc-bind").arg("127.0.0.1:0");
} else {
cmd.arg("--no-grpc");
}
if let Some(port) = http_port {
cmd.arg("--http-bind").arg(format!("127.0.0.1:{port}"));
}
if let Some(n) = opts.max_concurrent_transfers {
cmd.arg("--max-concurrent-transfers").arg(n.to_string());
}
if let Some(n) = opts.max_transfers_per_principal {
cmd.arg("--max-transfers-per-principal").arg(n.to_string());
}
if opts.hard_per_principal_limit {
cmd.arg("--hard-per-principal-limit");
}
let mut child = cmd
.stdin(Stdio::null())
.stdout(Stdio::from(file))
.stderr(Stdio::from(errfile))
.spawn()
.map_err(|e| format!("spawning `{} serve`: {e}", exec.display()))?;
let pid = child.id();
let log_for_wait = log.clone();
let ready = wait_for_ssh(port, READY_BUDGET, || match child.try_wait() {
Ok(Some(status)) => Err(format!(
"`gunnar serve` exited {status} before it served; log tail:\n{}",
tail(&log_for_wait, 6)
)),
Ok(None) => Ok(()),
Err(e) => Err(format!("waiting on `gunnar serve`: {e}")),
});
let ready = match ready {
Ok(d) => d,
Err(e) => {
let _ = child.kill();
let _ = child.wait();
// Give the dead child's stderr a moment to reach the log before
// quoting it. Without this the tail in `e` was taken at the
// instant `try_wait` reported the exit, which routinely beats
// the child's own error message to the file and produces a
// failure report that quotes the PREVIOUS server's log lines.
std::thread::sleep(Duration::from_millis(100));
return Err(format!("{e}\nfull log tail:\n{}", tail(&log, 3)));
}
};
// ── the banner is not proof that OUR child is the one serving ───────
//
// `wait_for_ssh` returns as soon as something on that port speaks SSH,
// and it never reaches its `alive` callback when the very first probe
// succeeds. Measured here on 2026-08-03: a second `gunnar serve` was
// started on a port an earlier one still held, exited immediately with
// a store-lock refusal, and `up` reported it READY in 0 ms — because
// the banner came from the server that was already there. That is
// exactly the hazard `wait_for_ssh`'s own header describes for podman's
// port forwarder, arriving by a different road, and it made a red-first
// proof pass for the wrong reason.
//
// So: the process this function spawned must still be running. Applied
// output, not a proxy.
if let Ok(Some(status)) = child.try_wait() {
std::thread::sleep(Duration::from_millis(100));
return Err(format!(
"something on 127.0.0.1:{port} answered with an SSH banner, but the \
`gunnar serve` this call started has already exited {status}; the banner \
came from a different process.\nlog tail:\n{}",
tail(&log, 3)
));
}
let instance = Instance {
root: root.to_path_buf(),
binary: binary.to_path_buf(),
data,
key,
log,
work,
ssh_port: port,
http_port,
pid,
started: proc_start_time(pid),
ready,
child: Some(child),
};
instance.write_state()?;
Ok(instance)
}
/// The `Probe` the shared object-graph exchange runs against.
pub fn probe(&self) -> Probe {
Probe {
port: self.ssh_port,
key: self.key.clone(),
work: self.work.clone(),
user: "git".to_string(),
}
}
/// Every client key on this instance: the bootstrap one first, then any
/// extras. A pressure round rotates through them so the load arrives as
/// several principals rather than one.
pub fn client_keys(&self) -> Vec<PathBuf> {
let mut keys = vec![self.key.clone()];
let dir = self.key.parent().unwrap_or(Path::new("."));
for i in 0.. {
let k = dir.join(format!("id_ed25519_{i}"));
if !k.exists() {
break;
}
keys.push(k);
}
keys
}
/// The `ssh://` URL a human should paste.
pub fn url(&self, repo: &str) -> String {
format!("ssh://git@127.0.0.1:{}/{repo}.git", self.ssh_port)
}
fn write_state(&self) -> Result<(), String> {
let path = self.root.join(STATE_FILE);
let body = format!(
"pid {}\nstarted {}\nport {}\nbinary {}\ndata {}\nkey {}\nlog {}\nwork {}\n",
self.pid,
self.started.map(|s| s.to_string()).unwrap_or_default(),
self.ssh_port,
self.binary.display(),
self.data.display(),
self.key.display(),
self.log.display(),
self.work.display(),
);
std::fs::write(&path, body).map_err(|e| format!("{}: {e}", path.display()))
}
/// Read back what a previous `up` left, so `down` works from another shell.
pub fn load(root: &Path) -> Result<Instance, String> {
let path = root.join(STATE_FILE);
let text = std::fs::read_to_string(&path).map_err(|e| {
format!(
"no running instance recorded at {}: {e}\n (bring one up with \
`gunnar-installtest up`)",
path.display()
)
})?;
let field = |name: &str| -> Option<String> {
text.lines()
.find_map(|l| l.strip_prefix(name)?.trim().to_string().into())
};
let need = |name: &str| -> Result<String, String> {
field(name).ok_or_else(|| format!("{} has no `{name}` line", path.display()))
};
let started = field("started ").and_then(|s| s.parse().ok());
Ok(Instance {
root: root.to_path_buf(),
binary: PathBuf::from(need("binary ")?),
data: PathBuf::from(need("data ")?),
key: PathBuf::from(need("key ")?),
log: PathBuf::from(need("log ")?),
work: PathBuf::from(need("work ")?),
ssh_port: need("port ")?
.parse()
.map_err(|e| format!("bad port in {}: {e}", path.display()))?,
pid: need("pid ")?
.parse()
.map_err(|e| format!("bad pid in {}: {e}", path.display()))?,
started,
// Not recorded in the state file: `down` from another shell needs a
// pid and a port to stop the process, and reading a field back that
// no verb uses would be a second thing to keep in agreement.
http_port: None,
ready: Duration::ZERO,
child: None,
})
}
/// Is the server this handle names still the running server?
pub fn alive(&self) -> bool {
same_process(self.pid, self.started)
}
/// Give up ownership so the server **outlives this handle**.
///
/// [`Instance`]'s `Drop` exists so a panicking proof can never leave a
/// gunnar holding a port, and that is right for every arm — and exactly
/// wrong for `up`, whose entire job is to leave one running. Measured on
/// the desktop 2026-08-03: `up` printed a `git clone` line and a port, and by the
/// time the operator pasted it the process was gone, because `up` returning
/// dropped the handle. The `status` verb in the next shell reported
/// `pid … (GONE)` beside a store lock still naming it.
///
/// `std::process::Child` does not kill on drop, so dropping the handle is
/// all that is needed; nothing leaks.
pub fn detach(&mut self) {
self.child = None;
}
/// Collect the exit status of a child this process spawned.
///
/// Without this a `SIGKILL`ed server stays in `/proc` as a zombie until the
/// harness itself exits, and every liveness question about it answers
/// "still there". Cheap, non-blocking, and safe to call when there is no
/// handle (the `down`-from-another-shell case), where the server's real
/// parent does the reaping.
pub fn reap(&mut self) {
if let Some(child) = self.child.as_mut() {
let _ = child.try_wait();
}
}
/// Stop it and **prove** it let go.
///
/// Three separate facts, because they can fail independently and the
/// interesting failures are the ones where only one of them does: a process
/// that exits while leaving its lock file behind is a bootable appliance
/// that will refuse to boot next time.
pub fn down(mut self, sig: &str, remove_root: bool) -> Teardown {
let started = Instant::now();
let signalled = if self.alive() {
signal(self.pid, sig)
} else {
Err("the server was already gone before the stop signal".to_string())
};
// A handle we own must be reaped, or the exit status is never collected
// and the pid stays live in `/proc` as a zombie — which would make
// `same_process` say the server is still there, forever.
let deadline = started + STOP_BUDGET;
loop {
if let Some(child) = self.child.as_mut() {
let _ = child.try_wait();
}
if !self.alive() {
break;
}
if Instant::now() >= deadline {
break;
}
std::thread::sleep(Duration::from_millis(50));
}
let exit_ms = started.elapsed().as_millis();
let teardown = Teardown {
signal: sig.to_string(),
signalled: signalled.err(),
exit_ms,
process_gone: !self.alive(),
listener_gone: !listening_on(self.ssh_port),
port: self.ssh_port,
root_removed: false,
};
let _ = std::fs::remove_file(self.root.join(STATE_FILE));
if remove_root {
let removed = std::fs::remove_dir_all(&self.root).is_ok();
return Teardown {
root_removed: removed,
..teardown
};
}
teardown
}
}
impl Drop for Instance {
/// A panicking or short-circuiting arm must never leave a gunnar holding a
/// port on this box. `SIGKILL` here rather than `SIGTERM` on purpose: this
/// is the path that runs when something has already gone wrong, and it must
/// not be able to hang.
fn drop(&mut self) {
if let Some(child) = self.child.as_mut() {
let _ = child.kill();
let _ = child.wait();
}
}
}
/// What one teardown actually released.
#[derive(Clone, Debug)]
pub struct Teardown {
pub signal: String,
/// `Some` when the signal itself could not be delivered.
pub signalled: Option<String>,
pub exit_ms: u128,
pub process_gone: bool,
/// No socket in `LISTEN` state on the port. **Not** "the port can be bound":
/// see [`listening_on`].
pub listener_gone: bool,
// There is deliberately no `lock_released` here. It used to be
// `!lock_path(data).exists()` over a file no installation has created since
// the redb move — always `true`, including on a teardown that released nothing.
// See [`store_lock_path`]. What replaced it is not another boolean but an
// act: the lifecycle arm restarts a server on the same data root and makes
// it answer `ls-remote`, which cannot succeed unless the store was let go.
pub port: u16,
pub root_removed: bool,
}
impl Teardown {
/// Every release assertion, as one verdict.
pub fn check(&self) -> Result<(), String> {
if let Some(e) = &self.signalled {
return Err(format!("the stop signal was not delivered: {e}"));
}
let mut bad = Vec::new();
if !self.process_gone {
bad.push(format!(
"the server process is still running {} ms after {}",
self.exit_ms, self.signal
));
}
if !self.listener_gone {
bad.push(format!(
"something is still LISTENing on 127.0.0.1:{}",
self.port
));
}
if bad.is_empty() {
Ok(())
} else {
Err(bad.join("; "))
}
}
pub fn render(&self) -> String {
format!(
"{} → exited in {} ms · process gone {} · no listener on :{} {}{}",
self.signal,
self.exit_ms,
yesno(self.process_gone),
self.port,
yesno(self.listener_gone),
if self.root_removed {
" · data root removed"
} else {
""
}
)
}
}
fn yesno(b: bool) -> &'static str {
if b {
"yes"
} else {
"NO"
}
}
/// Run the shared [`bootstrap_plan`] against a data root with the binary's own
/// verbs, one process at a time.
///
/// Sequentially, and it has to be: the Arrow log takes an `O_EXCL` lock at open,
/// so two verbs at once would collide. That is the same reason the container arm
/// runs each verb as its own container.
pub fn provision(binary: &Path, data: &Path, key: &Path) -> Result<(), String> {
let pubkey = format!("@{}.pub", key.display());
for verbs in bootstrap_plan() {
let verbs: Vec<String> = verbs
.into_iter()
.map(|v| {
if v == KEYFILE {
pubkey.clone()
} else {
v.to_string()
}
})
.collect();
let out = Command::new(binary)
.arg("--data-root")
.arg(data)
.args(&verbs)
.output()
.map_err(|e| format!("`gunnar {}`: {e}", verbs.join(" ")))?;
if !out.status.success() {
return Err(format!(
"`gunnar {}` exited {}: {}",
verbs.join(" "),
out.status,
String::from_utf8_lossy(&out.stderr).trim()
));
}
}
Ok(())
}
/// Create one more repository and let `tester` write to it.
///
/// Called from [`Instance::up`], between the bootstrap plan and the server
/// starting, because the control store is single-writer and a running server
/// holds it.
pub fn provision_repo(binary: &Path, data: &Path, repo: &str) -> Result<(), String> {
let plan: Vec<Vec<String>> = vec![
vec![
"repo".into(),
"create".into(),
repo.into(),
"--hash".into(),
"sha1".into(),
],
vec![
"grant".into(),
"tester".into(),
"write".into(),
"--repo".into(),
repo.into(),
],
];
for verbs in plan {
let out = Command::new(binary)
.arg("--data-root")
.arg(data)
.args(&verbs)
.output()
.map_err(|e| format!("`gunnar {}`: {e}", verbs.join(" ")))?;
if !out.status.success() {
return Err(format!(
"`gunnar {}` exited {}: {}",
verbs.join(" "),
out.status,
String::from_utf8_lossy(&out.stderr).trim()
));
}
}
Ok(())
}
/// Add one more principal, with its own key and the same grants.
///
/// Separate from [`provision`] because [`bootstrap_plan`] is shared with the
/// other two arms and describes the ONE identity they need; a loop bolted into
/// it would change what a container run bootstraps for no reason.
pub fn provision_principal(
binary: &Path,
data: &Path,
name: &str,
key: &Path,
) -> Result<(), String> {
let pubkey = format!("@{}.pub", key.display());
let plan: Vec<Vec<String>> = vec![
vec!["principal".into(), "add".into(), name.into()],
vec!["key".into(), "add".into(), name.into(), pubkey],
vec![
"grant".into(),
name.into(),
"write".into(),
"--repo".into(),
"demo/hello".into(),
],
];
for verbs in plan {
let out = Command::new(binary)
.arg("--data-root")
.arg(data)
.args(&verbs)
.output()
.map_err(|e| format!("`gunnar {}`: {e}", verbs.join(" ")))?;
if !out.status.success() {
return Err(format!(
"`gunnar {}` exited {}: {}",
verbs.join(" "),
out.status,
String::from_utf8_lossy(&out.stderr).trim()
));
}
}
Ok(())
}
/// The last `n` lines of a file, for an error message.
pub fn tail(path: &Path, n: usize) -> String {
let Ok(mut f) = std::fs::File::open(path) else {
return String::new();
};
let mut s = String::new();
let _ = f.read_to_string(&mut s);
let lines: Vec<&str> = s.lines().collect();
lines[lines.len().saturating_sub(n)..].join("\n")
}
/// How many times the server's own log recorded an admission refusal.
///
/// The server writes one `warn` per refusal naming the limit that was hit
/// (`gunnar-ssh`'s `admission.rs`). Counting them is the **server side** of the
/// pressure result; the client side is what `git` printed. Two independent
/// witnesses of the same event, and a pressure run in which they disagree is
/// reporting something other than what happened.
pub fn refusals_in_log(log: &Path) -> (usize, usize) {
let Ok(text) = std::fs::read_to_string(log) else {
return (0, 0);
};
let mut server_wide = 0;
let mut per_principal = 0;
for line in text.lines() {
if !line.contains("git.transfer.refused") {
continue;
}
// The log records the refusal by its `Display`, which is
// `Refusal::client_message` — the very sentence the client is handed.
// Matching on the variant names (`PrincipalAtCapacity`) was the first
// version of this function and it could never match anything: it would
// have reported zero server-side refusals forever, next to a client
// count that was right, and `PressureRound::agrees` would have blamed
// the server. Matching the message is also strictly better, because it
// is the SAME string both witnesses see.
if line.contains("per-principal limit") {
per_principal += 1;
} else if line.contains("concurrent git transfers") {
server_wide += 1;
}
}
(server_wide, per_principal)
}
/// The highest `peak_in_flight` the admission gate has reported in this log.
///
/// The gate's own count of how many transfers were genuinely running at once.
/// It is the number that says whether a concurrency step actually produced
/// concurrency: 64 clients that each finished before the next one started
/// report a peak of 1, and a ramp built on such a fixture measures nothing.
pub fn peak_in_flight(log: &Path) -> u64 {
let Ok(text) = std::fs::read_to_string(log) else {
return 0;
};
text.lines()
.filter_map(|l| crate::log_field(l, "peak_in_flight="))
.max()
.unwrap_or(0)
}
// ── the arm ─────────────────────────────────────────────────────────────────
/// **The bare arm.** Up, serve real git, restart on the same port, reclaim a
/// stale lock, down, and prove the release.
pub fn run(binary: &Path, work: &Path, fault: Fault, life: LifeFault, keep: bool) -> ArmResult {
let mut ready = None;
let mut stages = Stages::default();
let outcome = run_inner(binary, work, fault, life, keep, &mut ready, &mut stages);
ArmResult {
// The lifecycle fault has to reach the report somehow, and `arm` is the
// only `&'static str` on the record. Five possible strings, leaked once
// each per process; the set is closed by `LifeFault`, so it cannot grow.
arm: match life {
LifeFault::None => "bare (an ordinary process, no container, no VM)",
other => Box::leak(
format!(
"bare (an ordinary process) · lifecycle fault={}",
other.name()
)
.into_boxed_str(),
),
},
fault,
outcome,
ready,
stages: stages.0,
demands_red: life != LifeFault::None,
}
}
fn run_inner(
binary: &Path,
root: &Path,
fault: Fault,
life: LifeFault,
keep: bool,
ready_out: &mut Option<Duration>,
stages: &mut Stages,
) -> Result<Exchange, String> {
// A port chosen once and reused across every start in this arm, because
// "it came back on the same port" is one of the things being proved.
let port = free_port()?;
let opts = UpOptions {
ssh_port: Some(port),
http_port: None,
extra_repos: Vec::new(),
grpc: false,
reuse_data: false,
exec: if fault == Fault::MissingBinary {
Some(root.join("gunnar-not-installed"))
} else {
None
},
extra_principals: 0,
max_concurrent_transfers: None,
max_transfers_per_principal: None,
hard_per_principal_limit: false,
};
// ── up ──────────────────────────────────────────────────────────────────
let mut server = stages.time("up", || Instance::up(binary, root, &opts))?;
*ready_out = Some(server.ready);
stages.record("ready", server.ready);
// ── the git exchange, the shared assertion ──────────────────────────────
if fault == Fault::KilledService {
signal(server.pid, "KILL")?;
// Wait for it to actually be gone, or the push races the kill and the
// fault reports a flake instead of a red.
let deadline = Instant::now() + Duration::from_secs(5);
while Instant::now() < deadline {
server.reap();
if !server.alive() {
break;
}
std::thread::sleep(Duration::from_millis(20));
}
}
let repo = if fault == Fault::EmptyRepo {
"demo/empty"
} else {
"demo/hello"
};
let exchange = stages.time("exchange", || server.probe().push_then_clone(repo, fault))?;
// ── does it respond to git? asked eight more ways ───────────────────────
//
// Only on a green exchange: the matrix builds on the fixture the exchange
// pushed, so running it after a deliberate fault would fail for the fault's
// reason and say nothing about tags, deletes or negotiation.
if fault == Fault::None {
let ops = stages.time("git_matrix", || {
server
.probe()
.git_matrix(repo, &crate::ServerLog::file(&server.log))
})?;
for op in &ops {
eprintln!(" git {:<18} {} ({} ms)", op.name, op.detail, op.ms);
}
if ops.len() < 8 {
return Err(format!(
"the git matrix ran {} operations; it defines 8, so something short-circuited",
ops.len()
));
}
}
// Everything past here is the lifecycle half, and it only runs on a green
// exchange: a lifecycle proof layered on top of a broken server would be
// red for the wrong reason, which is worth nothing.
let ssh = server.probe().ssh_command();
let url = server.url(repo);
let head = exchange.head.clone();
let data = server.data.clone();
// ── down, and the release ───────────────────────────────────────────────
//
// `NoTeardown` keeps the server alive and reads the same three facts about
// it. The struct is built by hand there rather than by `down`, because the
// point of the fault is that nothing was stopped: routing it through `down`
// with the signal suppressed would leave `down` untested in the one branch
// that is supposed to be exercising it.
let down_started = Instant::now();
// Kept alive by the `NoTeardown` fault, and held here rather than forgotten
// so its `Drop` still kills the process when this function returns. A
// deliberately-failing proof must not leave a gunnar holding a port.
let mut held: Option<Instance> = None;
let teardown = if life == LifeFault::NoTeardown {
let t = Teardown {
signal: "none (no-teardown fault)".to_string(),
signalled: None,
exit_ms: 0,
process_gone: !same_process(server.pid, server.started),
listener_gone: !listening_on(port),
port,
root_removed: false,
};
held = Some(server);
t
} else {
server.down("TERM", false)
};
let _held = held;
stages.record("down", down_started.elapsed());
// Held only for the port fault, and only until the assertion has read it.
let squatter = if life == LifeFault::PortStillBound {
let l = std::net::TcpListener::bind(("127.0.0.1", port));
if l.is_err() {
return Err(format!(
"the port-still-bound fault could not bind 127.0.0.1:{port}, so it never \
created the condition it is meant to detect"
));
}
l.ok()
} else {
None
};
let teardown = if squatter.is_some() {
Teardown {
listener_gone: !listening_on(port),
..teardown
}
} else {
teardown
};
let verdict = teardown.check().map_err(|e| format!("teardown: {e}"));
drop(squatter);
verdict?;
// ── back up on the same port, on the same data, and still serving ───────
//
// The strongest statement teardown can make. "The port can be bound" is a
// proxy; "gunnar bound it again and answered a real `ls-remote` with the
// same oid" is the applied output.
let reuse = UpOptions {
reuse_data: true,
..opts.clone()
};
let mut again = stages.time("restart", || Instance::up(binary, root, &reuse))?;
let listed = stages.time("ls_remote", || {
crate::git(&again.work, &ssh, &["ls-remote", &url, "refs/heads/main"])
})?;
if !listed.contains(&head) {
return Err(format!(
"after a restart on the same port and the same data root, `ls-remote` did not \
report the pushed head {head}; it said {listed:?}"
));
}
// ── the store lock, asked of the SERVER rather than of a file ───────────
//
// The `owner_before` this used to read is gone with the file it read; see
// [`store_lock_path`] for the measurement that says nothing writes it any
// more. What is asserted instead is the same three-step statement in the
// form the current lock can actually make it:
//
// 1. while a server is live, a second one against the same data root is
// REFUSED, and refused by the store rather than by the port;
// 2. the live one is SIGKILLed, so nothing ran a `Drop` and nothing
// released anything on the way out;
// 3. a server started afterwards takes the root and serves the same head.
//
// Step 1 is what keeps step 3 from being vacuous, and it is a genuine
// applied output rather than a file read: a gunnar with no store lock at all
// passes 2 and 3 and fails 1.
let killed_pid = again.pid;
let killed_started = again.started;
let refusal = store_is_held(binary, root, &reuse)?.ok_or_else(|| {
format!(
"a second `gunnar serve` came up against {} while pid {killed_pid} was still \
serving it. Two servers over one data root is the condition the store lock \
exists to prevent, and with it absent the reclaim below proves nothing: \
\"it started again after a kill\" is what a store with NO lock does too",
data.display()
)
})?;
// Refused by the STORE, not by something incidental. The probe runs on a
// port of its own, so a bind failure cannot be the reason — but an
// unreadable data root or a missing binary would also produce an `Err`, and
// either would let this pass against a server that never locked anything.
//
if !refusal.contains(STORE_LOCK_HELD) {
return Err(format!(
"the second server against a live data root was refused, but not by the store \
lock — nothing in its output said {STORE_LOCK_HELD:?}. Something other than the \
store turned it away and this step is then not about the store. It said:\n{refusal}"
));
}
if life == LifeFault::LockHolderAlive {
// Do NOT kill it. The lock's owner is alive, so the next start must be
// refused. `again` stays in scope and its Drop cleans it up.
} else {
signal(killed_pid, "KILL")?;
let deadline = Instant::now() + Duration::from_secs(5);
while Instant::now() < deadline {
// Reap every round: this process is the killed server's parent, and
// an uncollected exit status leaves it in `/proc` looking alive.
again.reap();
if !same_process(killed_pid, killed_started) {
break;
}
std::thread::sleep(Duration::from_millis(20));
}
if same_process(killed_pid, killed_started) {
return Err(format!("pid {killed_pid} survived SIGKILL"));
}
}
// The reclaim attempt runs on a port of its OWN when the holder is still
// alive, and that is the difference between a proof and a coincidence.
//
// `LockHolderAlive` leaves the first server running, and a first server is
// holding two things: the store lock and the port. Re-using the port would
// let a bind failure produce the red, and the fault would then pass just as
// happily against a gunnar that reclaimed live locks — which is the exact
// behaviour it exists to forbid. On a free port the store lock is the only
// thing left that can refuse the second server.
let reclaim_opts = if life == LifeFault::LockHolderAlive {
UpOptions {
ssh_port: Some(free_port()?),
http_port: None,
extra_repos: Vec::new(),
..reuse.clone()
}
} else {
reuse.clone()
};
let reclaimed = stages.time("reclaim", || Instance::up(binary, root, &reclaim_opts))?;
// And it must actually serve, not merely have started. The reclaim is about an
// appliance that boots; a process that took the store and then cannot answer
// git has not recovered anything.
let listed = stages.time("reclaim_ls_remote", || {
crate::git(
&reclaimed.work,
&ssh,
&["ls-remote", &url, "refs/heads/main"],
)
})?;
if !listed.contains(&head) {
return Err(format!(
"after reclaiming the store from a killed owner the server did not report the \
pushed head {head}; it said {listed:?}"
));
}
// The other half of step 1, now on the far side of the kill: the reclaimed
// server must hold the store as exclusively as the killed one did. Without
// this, a gunnar that recovered by *dropping* the lock would pass every
// assertion above.
if store_is_held(binary, root, &reuse)?.is_none() {
return Err(format!(
"after the reclaim, a second `gunnar serve` came up against {} alongside the \
running pid {}. The store was reclaimed and then not held, which leaves the \
root open to exactly the two-writer case the lock exists for",
data.display(),
reclaimed.pid
));
}
// ── final teardown ──────────────────────────────────────────────────────
if !keep {
let last = stages.time("final_down", || reclaimed.down("TERM", false));
last.check()
.map_err(|e| format!("the final teardown did not release: {e}"))?;
} else {
// `--keep` means the operator wants it running when this returns.
let mut reclaimed = reclaimed;
reclaimed.detach();
}
Ok(exchange)
}
// ── pressure ────────────────────────────────────────────────────────────────
//
// Concurrent clones and pushes against one bare server, ramped until something
// gives, with the failure CLASSIFIED rather than counted.
//
// # Why classification is the whole point
//
// "37 of 64 clones failed" is not a result. A refusal that names a configured
// limit is a server degrading exactly as designed; a connection reset is a
// server falling over. `gunnar-ssh`'s admission control refuses over its ceiling
// instead of queueing, and says so in the client's own protocol
// (`Refusal::client_message`), so the two are distinguishable from the client
// side alone — and the server logs its own warn per refusal, which gives a
// second, independent witness. A pressure run in which the two disagree is
// describing something other than what happened, and [`PressureRound::agrees`]
// is what notices.
//
// # The ceiling that actually bites is the per-principal one
//
// `DEFAULT_MAX_CONCURRENT_TRANSFERS` is 256 and reads like the limit. It is not
// the one a single client meets: every clone here authenticates with the same
// key, so they are all one principal, and `DEFAULT_MAX_TRANSFERS_PER_PRINCIPAL`
// is 32. That is deliberate — it is what stops one identity taking the server
// away from everybody else — and it means the honest headline for a single-key
// load is 32, not 256.
/// How the fixture the pressure clones pull is built.
///
/// Random bytes, from a fixed LCG seed so the fixture is the same on any
/// machine. Random rather than compressible on purpose: a repository of
/// zero-filled blobs deflates to nothing, every clone finishes in microseconds,
/// and no two transfers are ever in flight at the same time — which would make a
/// concurrency ceiling unreachable and the whole run vacuous.
pub const PRESSURE_SEED: u64 = 0x9E3779B97F4A7C15;
/// One concurrency step's outcome.
#[derive(Clone, Debug)]
pub struct PressureRound {
pub concurrency: usize,
pub attempted: usize,
pub ok: usize,
/// Refused by the per-principal ceiling, as the client was told.
pub refused_principal: usize,
/// Refused by the server-wide ceiling, as the client was told.
pub refused_server: usize,
/// Everything else: resets, timeouts, protocol errors. These are the ones
/// that mean "fell over" rather than "degraded".
pub faults: Vec<String>,
pub wall_ms: u128,
pub p50_ms: u128,
pub p99_ms: u128,
pub max_ms: u128,
/// What the SERVER logged during this round, counted independently.
pub log_refused_server: usize,
pub log_refused_principal: usize,
/// Client-seconds of work divided by wall seconds: how many transfers were
/// genuinely in flight at once, on average, over the round.
///
/// This is the round's **non-vacuity floor**. `n` clients started together
/// is an intention; this is the measurement. If every client waited its turn
/// the figure is 1.0 however large `n` was, and a ramp built on a fixture
/// too small to keep two transfers alive at the same time reports exactly
/// that — while otherwise looking like a perfectly good concurrency result.
pub overlap_x: f64,
/// How many distinct principals the load arrived as.
pub principals: usize,
/// The admission gate's own `peak_in_flight`, when it said anything.
///
/// Only logged on a refusal, so it is `0` for every round under the ceiling.
/// Absent, not zero: the two must not be confused, which is why
/// [`PressureRound::render`] prints it only when it is non-zero.
pub peak_in_flight: u64,
}
impl PressureRound {
/// Do the client's account and the server's log tell the same story?
///
/// Not an equality: a refusal is logged before the client has finished
/// reading it, and `git` can also fail for its own reasons on the way. What
/// must hold is that the server never logged FEWER refusals than the
/// clients reported, because that would mean a client invented one.
pub fn agrees(&self) -> bool {
self.log_refused_principal + self.log_refused_server
>= self.refused_principal + self.refused_server
}
pub fn refused(&self) -> usize {
self.refused_principal + self.refused_server
}
pub fn render(&self) -> String {
format!(
" {:>4} concurrent /{:>2}p · {:>4} ok · {:>4} refused (principal {} / server {}) · \
{:>3} faults · wall {:>6} ms · p50 {:>5} ms · p99 {:>6} ms · overlap {:>5.1}x{} · \
server log {}+{} {}",
self.concurrency,
self.principals,
self.ok,
self.refused(),
self.refused_principal,
self.refused_server,
self.faults.len(),
self.wall_ms,
self.p50_ms,
self.p99_ms,
self.overlap_x,
if self.peak_in_flight > 0 {
format!(" · gate peak {}", self.peak_in_flight)
} else {
String::new()
},
self.log_refused_principal,
self.log_refused_server,
if self.agrees() { "" } else { "← DISAGREES" },
)
}
}
/// Build a repository big enough that transfers overlap, and push it.
///
/// Returns the number of objects it holds, so a caller can refuse a vacuous
/// pressure run the same way the object-graph comparison refuses a vacuous
/// match.
pub fn build_pressure_fixture(
server: &Instance,
repo: &str,
blobs: usize,
blob_bytes: usize,
) -> Result<usize, String> {
let ssh = server.probe().ssh_command();
let src = server.work.join("pressure-src");
let _ = std::fs::remove_dir_all(&src);
std::fs::create_dir_all(src.join("blobs")).map_err(|e| format!("{}: {e}", src.display()))?;
crate::git(&src, &ssh, &["init", "--quiet", "--object-format=sha1"])?;
crate::git(
&src,
&ssh,
&["config", "user.email", "pressure@gunnar.invalid"],
)?;
crate::git(&src, &ssh, &["config", "user.name", "gunnar pressure"])?;
crate::git(&src, &ssh, &["symbolic-ref", "HEAD", "refs/heads/main"])?;
let mut state = PRESSURE_SEED;
let mut next = || {
state = state
.wrapping_mul(6364136223846793005)
.wrapping_add(1442695040888963407);
(state >> 33) as u8
};
for i in 0..blobs {
let bytes: Vec<u8> = (0..blob_bytes).map(|_| next()).collect();
let path = src.join(format!("blobs/{i:06}.bin"));
std::fs::write(&path, &bytes).map_err(|e| format!("{}: {e}", path.display()))?;
}
crate::git(&src, &ssh, &["add", "blobs"])?;
crate::git(&src, &ssh, &["commit", "--quiet", "-m", "pressure fixture"])?;
let graph = Graph::read(&src)?;
if graph.objects.len() < blobs {
return Err(format!(
"the pressure fixture holds {} objects for {blobs} blobs; it did not get built",
graph.objects.len()
));
}
let url = server.url(repo);
crate::git(
&src,
&ssh,
&["push", "--quiet", &url, "refs/heads/main:refs/heads/main"],
)?;
Ok(graph.objects.len())
}
/// One concurrency step: `n` stock `git clone` processes started as close to
/// simultaneously as this box allows, plus `pushes` concurrent pushes.
///
/// Threads are `std::thread`, not a pool. The house multicore rule forbids rayon
/// and names gatling; neither is reachable from this detached crate, and
/// neither is the right tool for "hold `n` subprocesses open at once", which is
/// what this measures. The threads are blocked on `wait4` the entire time.
/// How a round should be driven. `Serial` is the red switch for the measurement
/// apparatus itself.
///
/// [`PressureRound::overlap_x`] is the claim that `n` clients were really
/// running at once, and a number nobody has watched go to 1.0 is a number
/// nobody has checked. The obvious red — shrink the fixture until transfers are
/// instant — **does not work**, and that was measured here on 2026-08-03: with a
/// single 16-byte blob, eight clients still reported 7.9x, because they overlap
/// in the SSH handshake whether or not they overlap in the transfer. That is
/// genuine concurrent load on the server and the figure is not lying; it simply
/// cannot be driven down by making the repository smaller.
///
/// So the red is to remove the concurrency instead of the work: run the clients
/// one after another and require the figure to collapse to ~1.
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
pub enum Drive {
/// Every client released together by a barrier.
Together,
/// One client at a time, each finished before the next starts.
Serial,
}
pub fn pressure_round(
server: &Instance,
repo: &str,
concurrency: usize,
pushes: usize,
drive: Drive,
) -> Result<PressureRound, String> {
pressure_round_as(server, repo, concurrency, pushes, drive, None)
}
/// [`pressure_round`], driving the load as at most `principals` identities.
///
/// `None` uses every key the instance was provisioned with. `Some(1)` is the
/// single-key appliance client — the case the per-principal gate used to cap at
/// one eighth of the advertised limit — and it needs to be selectable on an
/// instance that *also* has spare identities, so one server and one fixture can
/// carry both halves of the max-min measurement.
pub fn pressure_round_as(
server: &Instance,
repo: &str,
concurrency: usize,
pushes: usize,
drive: Drive,
principals: Option<usize>,
) -> Result<PressureRound, String> {
use std::sync::mpsc;
let url = server.url(repo);
// One `GIT_SSH_COMMAND` per client key. With a single key every client is
// the same principal, so the per-principal rule is the only one that can
// fire — which, before max-min fairness, meant a single-key client could
// never reach the server-wide limit at all.
let mut keys = server.client_keys();
if let Some(n) = principals {
if n == 0 || n > keys.len() {
return Err(format!(
"a pressure round was asked to drive {n} principals against an instance \
provisioned with {}; a round that silently used a different number of \
identities than it reports is a row about a load nobody ran",
keys.len()
));
}
keys.truncate(n);
}
let keyring: Vec<String> = keys
.into_iter()
.map(|k| {
let mut p = server.probe();
p.key = k;
p.ssh_command()
})
.collect();
let scratch = server.work.join(format!("pressure-{concurrency}"));
let _ = std::fs::remove_dir_all(&scratch);
std::fs::create_dir_all(&scratch).map_err(|e| format!("{}: {e}", scratch.display()))?;
let (server_before, principal_before) = refusals_in_log(&server.log);
// A gate all the clients wait on, so they start together rather than in
// spawn order. Without it a slow spawn loop staggers the load and the
// concurrency being reported is not the concurrency that happened.
// In `Serial` the barrier is a no-op of width 1 per waiter, so nothing
// waits for anything and the driver joins each client before starting the
// next.
let gate = std::sync::Arc::new(std::sync::Barrier::new(match drive {
Drive::Together => concurrency + pushes + 1,
Drive::Serial => 1,
}));
// In `Serial` the clients run inside the spawn loop, so the round's wall
// clock has to start before it. In `Together` it starts at the barrier,
// where the load actually begins; counting the spawn loop there would
// inflate the wall and deflate the overlap by exactly the time it took to
// create the threads.
let spawn_started = Instant::now();
let (tx, rx) = mpsc::channel::<(bool, u128, String)>();
let mut handles = Vec::with_capacity(concurrency + pushes);
for i in 0..concurrency {
let (url, tx, gate) = (url.clone(), tx.clone(), gate.clone());
let ssh = keyring[i % keyring.len()].clone();
let dest = scratch.join(format!("clone-{i}"));
handles.push(std::thread::spawn(move || {
gate.wait();
let started = Instant::now();
let out = Command::new("git")
.current_dir(dest.parent().unwrap())
.env("GIT_SSH_COMMAND", &ssh)
.env("GIT_TERMINAL_PROMPT", "0")
.env("GIT_CONFIG_GLOBAL", "/dev/null")
.env("GIT_CONFIG_SYSTEM", "/dev/null")
.args(["clone", "--quiet", &url, dest.to_str().unwrap()])
.output();
let ms = started.elapsed().as_millis();
let _ = match out {
Ok(o) if o.status.success() => tx.send((true, ms, String::new())),
Ok(o) => tx.send((
false,
ms,
String::from_utf8_lossy(&o.stderr).trim().to_string(),
)),
Err(e) => tx.send((false, ms, format!("spawning git: {e}"))),
};
}));
if drive == Drive::Serial {
if let Some(h) = handles.pop() {
let _ = h.join();
}
}
}
for i in 0..pushes {
let (url, tx, gate) = (url.clone(), tx.clone(), gate.clone());
let ssh = keyring[i % keyring.len()].clone();
let src = server.work.join("pressure-src").clone();
handles.push(std::thread::spawn(move || {
gate.wait();
let started = Instant::now();
// A distinct branch each, so the pushes do not race each other on
// one ref and report a lost-update as a server fault.
let refspec = format!("refs/heads/main:refs/heads/press-{i}");
let out = Command::new("git")
.current_dir(&src)
.env("GIT_SSH_COMMAND", &ssh)
.env("GIT_TERMINAL_PROMPT", "0")
.env("GIT_CONFIG_GLOBAL", "/dev/null")
.env("GIT_CONFIG_SYSTEM", "/dev/null")
.args(["push", "--quiet", &url, &refspec])
.output();
let ms = started.elapsed().as_millis();
let _ = match out {
Ok(o) if o.status.success() => tx.send((true, ms, String::new())),
Ok(o) => tx.send((
false,
ms,
String::from_utf8_lossy(&o.stderr).trim().to_string(),
)),
Err(e) => tx.send((false, ms, format!("spawning git: {e}"))),
};
}));
if drive == Drive::Serial {
if let Some(h) = handles.pop() {
let _ = h.join();
}
}
}
drop(tx);
let wall_started = if drive == Drive::Together {
gate.wait();
Instant::now()
} else {
spawn_started
};
let mut oks = 0usize;
let mut refused_principal = 0usize;
let mut refused_server = 0usize;
let mut faults: Vec<String> = Vec::new();
let mut times: Vec<u128> = Vec::new();
for (ok, ms, err) in rx {
times.push(ms);
if ok {
oks += 1;
} else if err.contains("per-principal limit") {
refused_principal += 1;
} else if err.contains("concurrent git transfers") {
refused_server += 1;
} else {
// Capped: 256 identical reset messages is not more information than
// four of them, and it is a much worse report to read.
if faults.len() < 4 {
faults.push(err.lines().next().unwrap_or("").to_string());
} else {
faults.push(String::new());
}
}
}
let wall_ms = wall_started.elapsed().as_millis();
for h in handles {
let _ = h.join();
}
times.sort_unstable();
let pick = |q: f64| -> u128 {
if times.is_empty() {
0
} else {
times[(((times.len() as f64 - 1.0) * q).round() as usize).min(times.len() - 1)]
}
};
let (server_after, principal_after) = refusals_in_log(&server.log);
let busy_ms: u128 = times.iter().sum();
Ok(PressureRound {
concurrency: concurrency + pushes,
attempted: concurrency + pushes,
ok: oks,
refused_principal,
refused_server,
faults: faults.into_iter().filter(|f| !f.is_empty()).collect(),
wall_ms,
p50_ms: pick(0.50),
p99_ms: pick(0.99),
max_ms: times.last().copied().unwrap_or(0),
log_refused_server: server_after.saturating_sub(server_before),
log_refused_principal: principal_after.saturating_sub(principal_before),
overlap_x: if wall_ms == 0 {
0.0
} else {
busy_ms as f64 / wall_ms as f64
},
peak_in_flight: peak_in_flight(&server.log),
principals: keyring.len(),
})
}