xtask/src/benchrun.rs
//! `cargo xtask bench <suite>` — the bench configuration, in Rust, not in your
//! shell history.
//!
//! # Why this exists
//!
//! Running gunnar's bencher correctly required typing four environment
//! variables, and getting any one of them wrong produced a *plausible* table
//! rather than an error. Measured on 2026-08-13, all four in one day:
//!
//! * `GUNNAR_BENCH_FORGE_BINARY` unset → the arm refuses (good) — but pointed at
//! a nine-day-old build it measures the wrong gunnar and says nothing;
//! * `GUNNAR_ENGINE` unset → the in-process arms open a **packfile** fixture
//! with the **znippy** engine, serve zero objects, and the arm reports
//! "the two servers delivered different object graphs" — which reads like a
//! server bug and is a configuration mistake;
//! * `--bin gunnar` rebuilt but not `gunnar-service` → a 35-hour-old subject;
//! * `NORNIR_BENCH_ONLY=gunnar.vs_forge_fetch` also matches
//! `…_fetch_gc`, quietly dragging a structurally-dead arm into the run.
//!
//! And a fifth, found on 2026-08-14 and the most expensive of them because this
//! file did not set it at all:
//!
//! * `GUNNAR_BENCH_GUNNAR_SERVER` unset → **seam 1 has no server**, so every
//! `vs_git_*` arm refuses before it starts. Three full sweeps reported all
//! five of them red on all five store columns, and the arm-level message —
//! *"the clone head-to-head driver exited Some(1)"* — named nothing, because
//! the refusal goes to the driver child's stderr and that is `Stdio::null()`
//! unless `GUNNAR_BENCH_H2H_VERBOSE=1`. `bench/src/engine.rs` has documented
//! the two seams side by side the whole time; this module set one of them.
//! See [`build_store_serve`].
//!
//! Every one of those is knowledge about *which arm needs what*. Knowledge like
//! that belongs in a table the compiler reads, not in a line somebody retypes.
//!
//! # The one thing this cannot remove
//!
//! `nornir-bench` and the forge arms take their configuration from the
//! environment — that is their ABI across the process boundary, and xtask does
//! not get to change it. What xtask *can* do is be the only thing that ever
//! writes those variables. **No operator types one.** They are set on the child
//! [`Command`] from [`SUITES`] below, which is the whole point.
//!
//! # The suites
//!
//! A suite is a set of arms that agree about their configuration. They have to
//! be separate runs because [`Engine`] is process-global: the packfile-fixture
//! arms need `gix` and the store-layout arms need `znippy`, and there is exactly
//! one switch. That is not a workaround, it is the measurement — see
//! [`Suite::why`] on each row.
use std::path::{Path, PathBuf};
use std::process::Command;
/// The storage engine the run selects, process-wide.
///
/// `GUNNAR_ENGINE` picks between the two implementations `gunnar-server`
/// compiles in. It is **global to the process**, which is why one suite cannot
/// hold arms that disagree about it.
/// **One gunnar-side store configuration — an `x` in `x1 vs x2 vs x3 vs y`.**
///
/// The engine is not one thing. znippy is a WRITER axis (`fast`/`safe`/`uring`)
/// and a LAYOUT axis (`packed`/`one-table`), and `gix` is a different engine
/// entirely. Five configurations, and until 2026-08-13 only the containerised
/// forge tests could hold more than one of them — every other test in gunnar's
/// bencher ran on exactly one store, so nothing outside the bake-off was ever a
/// comparison.
///
/// These are the same five columns `bench/examples/forge/mod.rs` runs in five
/// containers; here they are applied to the WHOLE bencher by running it once per
/// configuration.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Store {
/// writer `fast`, index `packed` — no fsync, no journal. The durability
/// ceiling, not a shipping option.
ZnippyFast,
/// writer `safe`, index `packed` — git's own contract.
ZnippySafe,
/// writer `uring`, index `packed` — same ordering, one `io_uring_enter`.
ZnippyUring,
/// writer `uring`, index `one-table` — the payload layout moved, writer
/// pinned. This is the LAYOUT axis; the three above are the WRITER axis.
ZnippyExplode,
/// A different engine: packfiles on disk.
Gix,
}
/// Every configuration, in the order the two axes were designed to be read.
pub const STORES: [Store; 5] = [
Store::ZnippyFast,
Store::ZnippySafe,
Store::ZnippyUring,
Store::ZnippyExplode,
Store::Gix,
];
impl Store {
/// The suite-name suffix and the column key the rows carry.
pub const fn key(self) -> &'static str {
match self {
Store::ZnippyFast => "znippy_fast",
Store::ZnippySafe => "znippy_safe",
Store::ZnippyUring => "znippy_uring",
Store::ZnippyExplode => "znippy_explode",
Store::Gix => "gix",
}
}
/// `(GUNNAR_ENGINE, ZNIPPY_GIT_WRITER, ZNIPPY_GIT_INDEX)`.
///
/// Set EXPLICITLY for every configuration including the defaults: "the
/// variables were absent and the server did whatever it does" is not a
/// statement a row can carry, and `parity.rs` reads these back off the
/// running server to prove the column is the store it claims.
pub const fn env(self) -> (&'static str, Option<(&'static str, &'static str)>) {
match self {
Store::ZnippyFast => ("znippy", Some(("fast", "packed"))),
Store::ZnippySafe => ("znippy", Some(("safe", "packed"))),
Store::ZnippyUring => ("znippy", Some(("uring", "packed"))),
Store::ZnippyExplode => ("znippy", Some(("uring", "one-table"))),
Store::Gix => ("gix", None),
}
}
/// Cargo features the gunnar binaries need for this store to be selectable.
pub const fn features(self) -> &'static str {
"http,systemd,engine-gix"
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum Engine {
/// The shipping default: znippy Arrow IPC. `GUNNAR_ENGINE` left unset, so
/// the run measures what an operator gets.
Znippy,
/// `gix` — packfiles on disk. Required by any arm whose fixture is a git
/// repository built by `git` itself, because znippy cannot read one.
Gix,
/// Both compiled in, and **nothing selected here**.
///
/// For the containerised forge arms: each column sets `GUNNAR_ENGINE` inside
/// its OWN container (`forge/mod.rs`, `GunnarStore::engine_name`), so a
/// value set on this process would be either ignored or, worse, a second
/// place that has to agree. The binary still needs both engines compiled in
/// or a column asking for `gix` gets a refusal.
Both,
}
impl Engine {
/// The value to put in the child's environment, or `None` for the default.
///
/// Deliberately `None` rather than `"znippy"` for the default: setting it
/// explicitly would make the row say a choice was made where the point is
/// that none was, and `parity.rs` reads this back off the container.
pub const fn env(self) -> Option<&'static str> {
match self {
Engine::Znippy | Engine::Both => None,
Engine::Gix => Some("gix"),
}
}
pub const fn label(self) -> &'static str {
match self {
Engine::Znippy => "znippy (default)",
Engine::Gix => "gix",
Engine::Both => "both, per-column",
}
}
/// Cargo features the gunnar binaries need for this engine to be
/// *selectable*. A binary built without `engine-gix` refuses `GUNNAR_ENGINE=gix`
/// rather than serving from the other store — measured 2026-08-13, and the
/// refusal is right, but it means the feature has to be on.
pub const fn features(self) -> &'static str {
match self {
Engine::Znippy => "http,systemd",
Engine::Gix | Engine::Both => "http,systemd,engine-gix",
}
}
}
/// One runnable set of arms plus everything it needs to be measured correctly.
pub struct Suite {
/// What you type after `cargo xtask bench`.
pub name: &'static str,
/// `NORNIR_BENCH_ONLY` prefixes. Empty means "everything not skipped".
pub only: &'static [&'static str],
/// `NORNIR_BENCH_SKIP` prefixes.
pub skip: &'static [&'static str],
pub engine: Engine,
/// `GUNNAR_BENCH_FORGE_ZNIPPY_ARMS` — the split-column list that turns one
/// gunnar bar into several. `None` runs a single `gunnar` column.
pub znippy_arms: Option<&'static str>,
/// Run `#[ignore]`d container/corpus/scale arms.
pub heavy: bool,
/// Why these arms are together and not in another suite. Printed on every
/// run, because a suite boundary that nobody can explain drifts.
pub why: &'static str,
}
/// Arms that are structurally dead and must not be dragged into a run.
///
/// Skipped BY NAME with a reason each, never by silently narrowing a prefix.
/// `NORNIR_BENCH_ONLY=gunnar.vs_forge_fetch` matching `…_fetch_gc` is exactly
/// how one of these got into a run that then discarded five good arms.
pub const DEAD_ARMS: &[(&str, &str)] = &[
(
"gunnar.vs_forge_fetch_gc",
"needs two `gunnar gc` passes on one repository, and the second always \
fails: `next_generation` targets `repository.g1.znippy` unconditionally \
and bails when it exists (znippy gc.rs:326/429)",
),
(
"gunnar.vs_forge_gc_orders",
"measures `gunnar gc --delta-order`, a flag deleted with the packfile \
engine on 2026-08-10; the arm's entire subject is gone",
),
(
"gunnar.fetkaos_monkey",
"the hang is FIXED and the arm is still held out, for a different \
reason. It hung twice on 2026-08-13 (41 and 54 minutes, container up, \
0.2% CPU — stalled, not slow), because every `git` child ran under an \
unbounded `Command::output()` and `thread::scope`'s join has no \
deadline: one parked child held the whole sweep. Both are bounded now \
— `OP_TIMEOUT` per operation, `STORM_DEADLINE` across the storm, plus \
ssh ConnectTimeout/ServerAlive — so a stall is a red arm naming the \
operation instead of a sweep that never ends. What is still missing is \
a BOX: the arm wants the linux.git corpus member, and it has never \
been run to completion, so re-admitting it here would be trading a \
known hang for an unknown one. Take it off this list in the same \
session you first run it green. It also leaks its container when \
killed — check `podman ps` before trusting the next run",
),
];
/// Every suite, and the reason each one is its own run.
pub const SUITES: &[Suite] = &[
Suite {
name: "forge",
only: &["gunnar.vs_forge_"],
skip: &[
"gunnar.vs_forge_fetch_gc",
"gunnar.vs_forge_gc_orders",
"gunnar.vs_forge_mixed",
],
engine: Engine::Znippy,
znippy_arms: None,
heavy: true,
why: "the containerised bake-off against gitea and Forgejo, one gunnar \
column, on the shipping engine. `mixed` is held out because it \
fails 8 of 8 clients (D7) and would be the only red in an \
otherwise clean run",
},
Suite {
name: "forge-mixed",
only: &["gunnar.vs_forge_mixed"],
skip: &[],
engine: Engine::Znippy,
znippy_arms: None,
heavy: true,
why: "D7 on its own: reads and writes at once on one repository. It is \
EXPECTED RED — `vs_forge_concurrent` passes at 19/32 cores while \
this fails every client — and it is run alone so that being red \
costs nothing else",
},
Suite {
name: "forge-variants",
only: &["gunnar.vs_forge_"],
skip: &[
"gunnar.vs_forge_fetch_gc",
"gunnar.vs_forge_gc_orders",
"gunnar.vs_forge_mixed",
],
engine: Engine::Both,
znippy_arms: Some("all"),
heavy: true,
why: "the x1..x5-vs-y run: znippy `fast`/`safe`/`uring`/`explode` plus \
`gix`, each in its own container, against gitea and Forgejo. Two \
axes held one at a time — writer with the index pinned `packed`, \
then layout with the writer pinned `uring`. Seven containers per \
arm, so it is more than twice the boot cost of `forge`",
},
// ── THE SAME ARM SET, ONCE PER ENGINE ───────────────────────────────────
//
// `inproc-gix` and `inproc-znippy` differ in ONE field: the engine. That is
// the whole point and it was wrong until 2026-08-13, when each suite ran a
// disjoint slice — three arms on gix, everything else on znippy — so no arm
// was ever measured on both and there was no comparison anywhere, only two
// half-measurements that could not be put side by side.
//
// The reason for the split was real and is not a reason to keep it: a few
// arms genuinely cannot run under one of the engines. A packfile fixture is
// unreadable to znippy, so `cpu_phase` serves zero objects under it; the
// store-layout arms measure znippy's own on-disk shape and have nothing to
// look at under gix. Those are FINDINGS and they belong on the row as a
// named red, not hidden by never running the arm on that engine at all.
//
// Run both. Compare what both answered. Read the reds as the boundary
// between the two stores, which is exactly what a two-engine bake-off is
// for.
Suite {
name: "inproc-znippy",
only: &[],
skip: INPROC_SKIP,
engine: Engine::Znippy,
znippy_arms: None,
heavy: true,
why: "every in-process arm on the SHIPPING engine. Same set as \
`inproc-gix`, so the two are comparable arm by arm; an arm that \
cannot work on this store goes red with its reason rather than \
being quietly withheld",
},
Suite {
name: "inproc-gix",
only: &[],
skip: INPROC_SKIP,
engine: Engine::Gix,
znippy_arms: None,
heavy: true,
why: "the identical set on `gix`. The pair is the two-engine \
comparison: where both are green the numbers stand side by side, \
and where one is red that red IS the boundary between the stores",
},
];
/// What neither in-process suite runs, and why — the forge arms (their own
/// suites), and the arms that are structurally dead or hang.
///
/// One list, shared by both suites, because the moment they skip different
/// things they stop being comparable and the comparison is the point.
pub const INPROC_SKIP: &[&str] = &[
"gunnar.vs_forge_",
// Hangs the whole sweep. See DEAD_ARMS.
"gunnar.fetkaos_monkey",
];
pub fn suite(name: &str) -> Option<&'static Suite> {
SUITES.iter().find(|s| s.name == name)
}
/// **Run the WHOLE bencher once per store configuration: `x1 … x5` vs `y`.**
///
/// The forge tests already did this in five containers. Nothing else did — every
/// other test in the bencher ran on exactly one store, so outside the bake-off
/// there was no comparison to read.
///
/// Five runs, five `bench_runs` rows, each stamped with the configuration it
/// measured. A test that cannot work on a given store goes RED on that run with
/// its reason, and that red is the finding: it marks the boundary between the
/// stores rather than being hidden by never running the test there.
pub fn run_every_store(root: &Path, _skip: &[&str], heavy: bool) -> Result<(), String> {
for (i, store) in STORES.iter().enumerate() {
eprintln!(
"\n═══ store {}/{}: {} ═══",
i + 1,
STORES.len(),
store.key()
);
// EVERY test except the forge ones (their columns are per-container and
// they get their own run below) and the monkey, which runs immediately
// after in its OWN process.
run_one_store(root, *store, &["gunnar.vs_forge_", HANGS], heavy)?;
// ── THE MONKEY RUNS. It is not excluded; it is ISOLATED ─────────────
//
// `fetkaos_monkey` throws a hundred thousand seeded-random operations at
// a disposable container holding two ~5 GB repositories — one single-pack
// with a bitmap, one multi-pack without, so the server must walk. It is a
// DEFECT FINDER, not a scoreboard: it found four wrong-object-graph bugs
// on 2026-08-04/05, and wrong answers are the class of bug nothing else
// here looks for. Leaving it out means the sweep cannot find them.
//
// It also hangs — 41 and 54 minutes on 2026-08-13, container up, process
// at 0.2% CPU. In one process with the others that hang costs all 47
// measurements. In its own process it costs only itself, and the
// watchdog below turns the hang into a bounded, reported failure instead
// of a stalled sweep.
eprintln!("── {} · the monkey, alone and watchdogged ──", store.key());
run_monkey(root, *store)?;
}
// ── AND THE `y` HALF, or the matrix has no rival to compare against ─────
//
// The five runs above are x1..x5 and NOTHING ELSE: the forge tests are the
// only ones that put gitea and Forgejo on the wire, and they were held out
// of every store run because their columns are per-container rather than
// per-process. Held out of the whole command as well, which is what I first
// shipped, that leaves five x columns and no y — a comparison with nothing
// on the other side.
//
// One run, all five gunnar columns in five containers, against both Go
// forges. That is the x1..x5-vs-y1,y2 row of the matrix.
eprintln!("\n═══ the rival half: all five columns vs gitea and Forgejo ═══");
run_forge_all_columns(root)?;
Ok(())
}
/// The containerised bake-off with every gunnar column at once.
///
/// `GUNNAR_BENCH_FORGE_ZNIPPY_ARMS=all` is the mechanism the forge tests already
/// had: seven containers per test — four znippy arms, `gix`, gitea, Forgejo.
/// Nothing here selects an engine on this process, because each column selects
/// its own inside its own container.
pub fn run_forge_all_columns(root: &Path) -> Result<(), String> {
build_binaries_with(root, "http,systemd,engine-gix")?;
let subject = bench_target_dir().join("release/gunnar");
let mut cmd = Command::new(nornir());
cmd.current_dir(root)
.arg("bench")
.arg("run")
.arg(crate::mashup::REPO)
.arg("--config")
.arg(root.join(crate::mashup::CONFIG))
.arg("--heavy")
.env("GUNNAR_BENCH_FORGE_BINARY", &subject)
.env("GUNNAR_BENCH_FORGE_ZNIPPY_ARMS", "all")
.env("NORNIR_BENCH_ONLY", "gunnar.vs_forge_")
.env("NORNIR_BENCH_SKIP", FORGE_SKIP.join(","))
.env_remove("GUNNAR_ENGINE")
.env_remove("ZNIPPY_GIT_WRITER")
.env_remove("ZNIPPY_GIT_INDEX");
let status = cmd
.status()
.map_err(|e| format!("spawn `{} bench run`: {e}", nornir()))?;
if !status.success() {
eprintln!("forge columns: at least one test went red. The run is still persisted.");
}
Ok(())
}
/// NOTHING is held out of the forge run.
///
/// `fetch_gc` and `gc_orders` cannot pass on any store — their subject was
/// deleted with the packfile engine — and `mixed` fails 8 of 8 clients (D7).
/// They run anyway. A red that is RECORDED on every column is data: it says the
/// failure is structural rather than a property of one store. Holding them out
/// produced a tidier run and a smaller truth.
/// The one test that hangs rather than failing. Run alone, under a watchdog.
pub const HANGS: &str = "gunnar.fetkaos_monkey";
/// How long the monkey gets before the watchdog calls it a hang.
///
/// Observed 41 and 54 minutes with the process at 0.2% CPU on 2026-08-13. 30
/// minutes is longer than any healthy run of it has ever taken and shorter than
/// either hang, so it separates the two.
pub const MONKEY_DEADLINE_SECS: u64 = 1800;
/// Run the monkey on one store, and KILL it if it stalls.
///
/// A pure-Rust watchdog rather than `timeout(1)`: this repository does not shell
/// out for things it can do itself, and a poll loop over `try_wait` is nine
/// lines.
pub fn run_monkey(root: &Path, store: Store) -> Result<(), String> {
let subject = bench_target_dir().join("release/gunnar");
let mut cmd = Command::new(nornir());
cmd.current_dir(root)
.arg("bench")
.arg("run")
.arg(crate::mashup::REPO)
.arg("--config")
.arg(root.join(crate::mashup::CONFIG))
.arg("--heavy")
.env("GUNNAR_BENCH_FORGE_BINARY", &subject)
.env("GUNNAR_BENCH_FIXTURES", fixture_root_for(store))
.env("NORNIR_BENCH_ONLY", HANGS)
.env_remove("NORNIR_BENCH_SKIP")
.env_remove("GUNNAR_BENCH_FORGE_ZNIPPY_ARMS");
let (engine, arms) = store.env();
cmd.env("GUNNAR_ENGINE", engine);
match arms {
Some((w, i)) => {
cmd.env("ZNIPPY_GIT_WRITER", w).env("ZNIPPY_GIT_INDEX", i);
}
None => {
cmd.env_remove("ZNIPPY_GIT_WRITER");
cmd.env_remove("ZNIPPY_GIT_INDEX");
}
}
let mut child = cmd.spawn().map_err(|e| format!("spawn the monkey: {e}"))?;
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(MONKEY_DEADLINE_SECS);
loop {
match child.try_wait() {
Ok(Some(_)) => return Ok(()),
Ok(None) => {}
Err(e) => return Err(format!("waiting on the monkey: {e}")),
}
if std::time::Instant::now() >= deadline {
let _ = child.kill();
let _ = child.wait();
eprintln!(
" ✗ {} · the monkey exceeded {}s and was killed — this is the HANG, \
recorded rather than allowed to stall the sweep",
store.key(),
MONKEY_DEADLINE_SECS
);
return Ok(());
}
std::thread::sleep(std::time::Duration::from_secs(5));
}
}
pub const FORGE_SKIP: &[&str] = &[];
/// One configuration, over every test that is not held out.
pub fn run_one_store(root: &Path, store: Store, skip: &[&str], heavy: bool) -> Result<(), String> {
build_binaries_with(root, store.features())?;
let subject = bench_target_dir().join("release/gunnar");
if !subject.exists() {
return Err(format!("{} missing after the build", subject.display()));
}
let store_serve = build_store_serve(root)?;
let mut cmd = Command::new(nornir());
cmd.current_dir(root)
.arg("bench")
.arg("run")
.arg(crate::mashup::REPO)
.arg("--config")
.arg(root.join(crate::mashup::CONFIG));
if heavy {
cmd.arg("--heavy");
}
cmd.env("GUNNAR_BENCH_FORGE_BINARY", &subject);
// Seam 1. Set here for the same reason seam 2 is: no operator types either
// of them, and the one that was never typed is the one five arms died on.
// See [`build_store_serve`].
cmd.env("GUNNAR_BENCH_GUNNAR_SERVER", &store_serve);
// Per-store fixture root: no store may be handed a repository another store
// wrote. See `fixture_root_for`.
cmd.env("GUNNAR_BENCH_FIXTURES", fixture_root_for(store));
let (engine, arms) = store.env();
cmd.env("GUNNAR_ENGINE", engine);
match arms {
Some((writer, index)) => {
cmd.env("ZNIPPY_GIT_WRITER", writer);
cmd.env("ZNIPPY_GIT_INDEX", index);
}
None => {
cmd.env_remove("ZNIPPY_GIT_WRITER");
cmd.env_remove("ZNIPPY_GIT_INDEX");
}
}
// One gunnar column per run: the split-container mechanism is the forge
// tests' own, and stacking it on top of a per-run store would put one
// configuration inside another.
cmd.env_remove("GUNNAR_BENCH_FORGE_ZNIPPY_ARMS");
cmd.env_remove("NORNIR_BENCH_ONLY");
if skip.is_empty() {
cmd.env_remove("NORNIR_BENCH_SKIP");
} else {
cmd.env("NORNIR_BENCH_SKIP", skip.join(","));
}
let status = cmd
.status()
.map_err(|e| format!("spawn `{} bench run`: {e}", nornir()))?;
if !status.success() {
eprintln!(
"store {}: at least one test went red. The run is still persisted.",
store.key()
);
}
Ok(())
}
fn build_binaries_with(root: &Path, features: &str) -> Result<(), String> {
let status = Command::new(cargo())
.current_dir(root)
.env("CARGO_TARGET_DIR", bench_target_dir())
.args([
"build",
"--release",
"-p",
"gunnar-cli",
"--features",
features,
"--bin",
"gunnar-cli",
])
.status()
.map_err(|e| format!("spawn cargo build: {e}"))?;
if !status.success() {
return Err(format!(
"building gunnar-cli with `--features {features}` failed"
));
}
bench_names()
}
/// **The file names the bench harness looks for** (`release/gunnar`, the git side, and
/// `release/gunnar-service`, the served side): since GUN-PRODUCTS (2026-10-30) both are
/// the one program `gunnar-cli` with the suite's features, so the build above lands it
/// under both names in the bench's own target directory.
fn bench_names() -> Result<(), String> {
let release = bench_target_dir().join("release");
let built = release.join("gunnar-cli");
for name in ["gunnar", "gunnar-service"] {
let at = release.join(name);
let _ = std::fs::remove_file(&at);
std::fs::copy(&built, &at)
.map_err(|e| format!("{} → {}: {e}", built.display(), at.display()))?;
}
Ok(())
}
/// **Seam 1's binary: the `ext::` server the `vs_git_*` arms clone from.**
///
/// # Why this is here, and what it cost to leave it out
///
/// `bench/src/engine.rs` names two seams and this module set exactly one of
/// them. `GUNNAR_BENCH_FORGE_BINARY` (seam 2, the containerised `vs_forge_*`
/// arms) was set on every child; `GUNNAR_BENCH_GUNNAR_SERVER` (seam 1) was set
/// on none. `bench/` links no storage engine, so with seam 1 unset
/// `engine::require_server()` refuses — correctly, and by design, because the
/// alternative is publishing a ratio table for a store that never ran — and
/// **every arm that serves gunnar's own column dies before it starts**:
///
/// Error: GUNNAR_BENCH_GUNNAR_SERVER is unset and there is no in-process
/// fallback any more.
///
/// MEASURED on the desktop 2026-08-14: that is the whole of `vs_git_clone`,
/// `vs_git_clone_sha256`, `vs_git_push`, `vs_git_fetch_divergent` and
/// `vs_git_clone_concurrent` being red on **all five** store columns of three
/// full sweeps. The arm-level message says only *"the clone head-to-head driver
/// exited Some(1)"* — the refusal is on the driver child's stderr, which is
/// `Stdio::null()` unless `GUNNAR_BENCH_H2H_VERBOSE=1`, so five arms failed for
/// a reason nothing in the run printed.
///
/// With it set, on the same tree and the same fixtures: `vs_git_clone`
/// `passed: true` (36 172 objects, `pack_ratio_x` 0.997),
/// `vs_git_clone_sha256`, `vs_git_push` and `vs_git_fetch_divergent` likewise.
/// `vs_git_clone_concurrent` stays red for its own reason and is not this.
///
/// # One binary, both engines, selected at run time
///
/// `crates/gunnar-server/examples/git-store-serve.rs` resolves its store through
/// `gunnar_server::storage::Engine::selected()`, i.e. the same `GUNNAR_ENGINE` /
/// `ZNIPPY_GIT_WRITER` / `ZNIPPY_GIT_INDEX` triple [`Store::env`] already puts
/// on the child. So the store column and the seam cannot disagree: there is one
/// place that decides, and this function does not get a vote. That is also why
/// there is no per-store copy of the binary and no path table to keep in step —
/// the shape `bench/two-engine.sh` documents at length as the thing that went
/// wrong in 2026-08-10's prebuilt pair.
fn build_store_serve(root: &Path) -> Result<PathBuf, String> {
let status = Command::new(cargo())
.current_dir(root)
.env("CARGO_TARGET_DIR", bench_target_dir())
.args([
"build",
"--release",
"-p",
"gunnar-server",
"--example",
"git-store-serve",
"--features",
// `engine-gix` ALONGSIDE znippy, which is `gunnar-server`'s own
// default. Naming `engine-znippy` here is not the way to keep it —
// `--features` does not turn defaults off, and spelling it out is
// how the recipe in `two-engine.sh` grew a variant that fails to
// resolve.
"engine-gix",
])
.status()
.map_err(|e| format!("spawn cargo build (git-store-serve): {e}"))?;
if !status.success() {
return Err("building the `git-store-serve` example failed".to_string());
}
let exe = bench_target_dir().join("release/examples/git-store-serve");
if !exe.is_file() {
return Err(format!(
"{} is missing after the build, so seam 1 would be unset and every `vs_git_*` arm \
would refuse with \"GUNNAR_BENCH_GUNNAR_SERVER is unset\"",
exe.display()
));
}
Ok(exe)
}
/// The gunnar binaries a suite measures, built with the features its engine
/// needs.
///
/// Both, always, and that is deliberate: on 2026-08-13 `--bin gunnar` was
/// rebuilt and `gunnar-service` was not, and three arms then refused with a
/// 35-hour staleness error. Building one and not the other is not a saving.
/// Where the bench arms look for gunnar's binaries.
///
/// `nornir_bench::runner` sets `CARGO_TARGET_DIR` to `$HOME/.cache/
/// nornir-bench-target` for the example it spawns, so every arm that resolves
/// `gunnar-service` by target dir resolves it THERE — not in the repository's
/// `target/`. Building into the repo and pointing the forge seam at the repo
/// therefore left `constellation` and `bare_lifecycle` measuring whatever copy
/// happened to be in the cache, which on 2026-08-13 was 116 minutes stale and
/// the arm's own guard caught it.
///
/// One directory, used for the build AND for every seam, so there is nothing to
/// keep in agreement.
fn bench_target_dir() -> PathBuf {
std::env::var_os("HOME")
.map(|h| Path::new(&h).join(".cache/nornir-bench-target"))
.unwrap_or_else(|| std::env::temp_dir().join("nornir-bench-target"))
}
fn build_binaries(root: &Path, engine: Engine) -> Result<(), String> {
let status = Command::new(cargo())
.current_dir(root)
.env("CARGO_TARGET_DIR", bench_target_dir())
.args([
"build",
"--release",
"-p",
"gunnar-cli",
"--features",
engine.features(),
"--bin",
"gunnar-cli",
])
.status()
.map_err(|e| format!("spawn cargo build: {e}"))?;
if !status.success() {
return Err(format!(
"building gunnar-cli with `--features {}` failed",
engine.features()
));
}
bench_names()
}
fn cargo() -> String {
std::env::var("CARGO").unwrap_or_else(|_| "cargo".to_string())
}
fn nornir() -> String {
std::env::var(crate::mashup::NORNIR_BIN_ENV).unwrap_or_else(|_| "nornir".to_string())
}
/// **Does this suite actually run `arm`?** — spelled the way the runner spells
/// it, once, so nothing here can hold a second opinion (one implementation, not two).
///
/// Both halves are prefix rules, and both were got wrong in the obvious
/// direction before this function existed:
///
/// * `only` empty means *no filter*, and `.any()` over an empty slice is
/// `false`, so the one suite that runs every arm read as the suite that runs
/// none;
/// * `skip` entries are prefixes (`nornir-bench` retains with
/// `!skip.iter().any(|p| id.starts_with(p))`), and they were compared with
/// `contains`, which is exact — so `gunnar.vs_forge_` did not register as
/// holding out `gunnar.vs_forge_fetch_gc`, and the run banner said nothing
/// was being held out when four arms were.
fn drops(s: &Suite, arm: &str) -> bool {
s.skip.iter().any(|p| arm.starts_with(p))
}
/// The other half: whether the suite's `only` filter selects `arm` at all.
fn selects(s: &Suite, arm: &str) -> bool {
s.only.is_empty() || s.only.iter().any(|p| arm.starts_with(p))
}
/// Run one suite: build its subject, then spawn the bencher with every variable
/// set from [`SUITES`].
pub fn run(root: &Path, s: &Suite) -> Result<(), String> {
eprintln!("bench suite `{}` — engine {}", s.name, s.engine.label());
eprintln!(" why: {}", s.why);
for (arm, reason) in DEAD_ARMS {
match (selects(s, arm), drops(s, arm)) {
(_, true) => eprintln!(" holding out {arm}: {reason}"),
// The banner says it too, not only `cargo test -p xtask`. A sweep
// is started by a person who then walks away for an hour, and
// being told at minute zero that a known-dead arm is in the run is
// worth more than being told afterwards by a test they were not
// running.
(true, false) => eprintln!(" WARNING: {arm} is in this run and is dead: {reason}"),
(false, false) => {}
}
}
build_binaries(root, s.engine)?;
let subject: PathBuf = bench_target_dir().join("release/gunnar");
if !subject.exists() {
return Err(format!(
"{} does not exist after the build",
subject.display()
));
}
let store_serve = build_store_serve(root)?;
let mut cmd = Command::new(nornir());
cmd.current_dir(root)
.arg("bench")
.arg("run")
.arg(crate::mashup::REPO)
.arg("--config")
.arg(root.join(crate::mashup::CONFIG));
if s.heavy {
cmd.arg("--heavy");
}
// ── EVERY environment variable this run needs, written HERE ─────────────
//
// Nobody types these. That is the module's reason for existing — and seam 1
// is the one nobody typed for three sweeps. See [`build_store_serve`].
cmd.env("GUNNAR_BENCH_FORGE_BINARY", &subject);
cmd.env("GUNNAR_BENCH_GUNNAR_SERVER", &store_serve);
match s.engine.env() {
Some(v) => {
cmd.env("GUNNAR_ENGINE", v);
}
// Removed rather than left inherited: a stray GUNNAR_ENGINE in the
// operator's shell would otherwise silently retarget the run, which is
// the exact class of mistake this file exists to end.
None => {
cmd.env_remove("GUNNAR_ENGINE");
}
}
match s.znippy_arms {
Some(v) => {
cmd.env("GUNNAR_BENCH_FORGE_ZNIPPY_ARMS", v);
}
None => {
cmd.env_remove("GUNNAR_BENCH_FORGE_ZNIPPY_ARMS");
}
}
if s.only.is_empty() {
cmd.env_remove("NORNIR_BENCH_ONLY");
} else {
cmd.env("NORNIR_BENCH_ONLY", s.only.join(","));
}
if s.skip.is_empty() {
cmd.env_remove("NORNIR_BENCH_SKIP");
} else {
cmd.env("NORNIR_BENCH_SKIP", s.skip.join(","));
}
let status = cmd.status().map_err(|e| {
format!(
"spawn `{} bench run`: {e} (on PATH, or {} set?)",
nornir(),
crate::mashup::NORNIR_BIN_ENV
)
})?;
// A red arm is NOT a failed run any more: since nornir 49b2c05 the envelope
// is persisted and the non-zero exit is the verdict on the arms, not on the
// measurement. Report it and let the caller decide.
if !status.success() {
eprintln!(
"bench suite `{}`: at least one arm went red. The run is still persisted — \
check `cargo xtask bench-landed`.",
s.name
);
}
Ok(())
}
#[cfg(test)]
mod tests {
use super::*;
/// Two suites may measure the same arm ONLY if their rows are
/// distinguishable by their column keys.
///
/// `forge` and `forge-variants` deliberately run the same arms — that is the
/// whole x1..x5-vs-y idea — and they stay separable because the split run
/// emits `znippy_*`/`gix_*` columns where the single run emits `gunnar_*`.
/// What must never happen is two suites emitting the SAME keys under
/// different engines: then only the timestamp says which store a row is
/// about, and the mashup picks by timestamp.
///
/// Red-first: give `forge-variants` `znippy_arms: None` and this fires.
#[test]
fn suites_sharing_arms_must_emit_distinguishable_columns() {
for a in SUITES {
for b in SUITES {
if a.name >= b.name {
continue;
}
let shares = a.only.iter().any(|x| b.only.contains(x));
if !shares {
continue;
}
assert_ne!(
(a.znippy_arms.is_some(), a.engine),
(b.znippy_arms.is_some(), b.engine),
"`{}` and `{}` measure the same arms and emit the same columns",
a.name,
b.name
);
}
}
}
/// A containerised suite must NOT set a process-global engine: the forge
/// columns each set their own inside their own container, and a second
/// place to say it is a second place to disagree.
#[test]
fn a_split_column_suite_selects_no_engine_of_its_own() {
for s in SUITES.iter().filter(|s| s.znippy_arms.is_some()) {
assert_eq!(
s.engine.env(),
None,
"suite `{}` splits columns AND pins GUNNAR_ENGINE",
s.name
);
assert!(
s.engine.features().contains("engine-gix"),
"suite `{}` has a `gix` column and builds without the feature",
s.name
);
}
}
/// The dead arms are held out of every suite that could otherwise match
/// them by prefix.
///
/// Red-first: drop `gunnar.vs_forge_fetch_gc` from `forge`'s skip list and
/// this fails — which is the run that lost five green arms twice.
///
/// # An empty `only` selects EVERYTHING, and this used to read it as nothing
///
/// `matched` was `s.only.iter().any(|p| arm.starts_with(p))`, and `any` on
/// an empty iterator is `false`. `inproc-znippy` has `only: &[]`, which the
/// runner means as *no filter, run every arm* — so the one suite that runs
/// the whole sweep was the one suite this test skipped. Deleting
/// `gunnar.fetkaos_monkey` from its skip list would have re-hung the sweep
/// with every test still green, which is the same blindness this test was
/// written to remove.
#[test]
fn every_dead_arm_is_held_out_of_the_suites_whose_prefix_matches_it() {
for (arm, _) in DEAD_ARMS {
for s in SUITES {
if selects(s, arm) {
assert!(
drops(s, arm),
"suite `{}` runs {arm} ({}) and does not hold it out",
s.name,
if s.only.is_empty() {
"it filters nothing, so it runs every arm"
} else {
"it matches by prefix"
}
);
}
}
}
}
/// A suite that selects `gix` must build with the feature that makes it
/// selectable, or the binary refuses at runtime.
#[test]
fn a_gix_suite_builds_a_binary_that_can_actually_select_gix() {
for s in SUITES.iter().filter(|s| s.engine == Engine::Gix) {
assert!(
s.engine.features().contains("engine-gix"),
"suite `{}` selects gix and builds without the feature",
s.name
);
}
assert_eq!(Engine::Znippy.env(), None, "the default states no choice");
}
/// ★ THE TWO-ENGINE INVARIANT. The in-process pair must differ in the
/// engine and in NOTHING else.
///
/// Red-first, and it was red in the field rather than in a test: until
/// 2026-08-13 `inproc-gix` ran three arms and `inproc-znippy` ran the rest,
/// so not one arm was measured on both stores and the pair could not be put
/// side by side. Give either suite a different `only`/`skip`/`heavy` and
/// this fires.
#[test]
fn the_in_process_pair_differs_only_in_the_engine() {
let z = suite("inproc-znippy").expect("znippy suite");
let g = suite("inproc-gix").expect("gix suite");
assert_ne!(z.engine, g.engine, "the pair must differ in the engine");
assert_eq!(z.only, g.only, "the pair must run the same arms");
assert_eq!(z.skip, g.skip, "the pair must hold out the same arms");
assert_eq!(z.heavy, g.heavy, "the pair must run at the same depth");
assert!(
z.znippy_arms.is_none() && g.znippy_arms.is_none(),
"the in-process arms have no container columns to split"
);
// And the shared hold-out list is the ONE list, not two that agree today.
assert!(
std::ptr::eq(z.skip, g.skip),
"both must use INPROC_SKIP itself"
);
}
#[test]
fn suites_are_addressable_by_name() {
for s in SUITES {
assert!(suite(s.name).is_some());
}
assert!(suite("nope").is_none());
}
}
#[cfg(test)]
mod store_tests {
use super::*;
/// ★ x1 … x5: five DISTINCT configurations, and every one states its choice.
///
/// Red-first: give two stores the same `(engine, writer, index)` and this
/// fires — which is the failure that would silently publish one store's
/// numbers under two names.
#[test]
fn every_store_is_a_distinct_and_fully_stated_configuration() {
// Two sets, not one: `Gix.env()` is `("gix", None)` and `Gix.key()` is
// `"gix"`, so a single set collides them and the guard fails on itself.
let mut envs = std::collections::BTreeSet::new();
let mut keys = std::collections::BTreeSet::new();
for s in STORES {
assert!(envs.insert(s.env()), "{} duplicates another store", s.key());
assert!(keys.insert(s.key()), "{} duplicates a key", s.key());
// Nothing is left to a default: a row must be able to say which
// store it measured.
let (engine, arms) = s.env();
assert!(!engine.is_empty());
if engine == "znippy" {
assert!(
arms.is_some(),
"{} leaves the znippy axes unstated",
s.key()
);
}
assert!(
s.features().contains("engine-gix"),
"{} builds a binary that cannot select gix",
s.key()
);
}
assert_eq!(STORES.len(), 5);
}
/// The two axes are held ONE AT A TIME, or the pair stops being a
/// comparison: writer varies with the index pinned `packed`, then the index
/// varies with the writer pinned `uring`.
#[test]
fn the_two_axes_move_one_at_a_time() {
let w: Vec<&str> = [Store::ZnippyFast, Store::ZnippySafe, Store::ZnippyUring]
.iter()
.map(|s| s.env().1.expect("znippy").1)
.collect();
assert_eq!(
w,
vec!["packed", "packed", "packed"],
"writer axis moved the index too"
);
let uring = Store::ZnippyUring.env().1.expect("znippy");
let explode = Store::ZnippyExplode.env().1.expect("znippy");
assert_eq!(uring.0, explode.0, "layout axis moved the writer too");
assert_ne!(uring.1, explode.1, "the layout axis does not move at all");
}
/// Every store's column key is one `mashup::MINE` knows, or the chart cannot
/// draw the run it just measured.
#[test]
fn every_store_key_is_one_the_reader_understands() {
for s in STORES {
assert!(
crate::mashup::MINE.contains(&s.key()),
"{} is not in mashup::MINE, so its rows would be unreadable",
s.key()
);
}
}
}
// ════════════════════════════════════════════════════════════════════════════
// `cargo xtask do-all` — the whole matrix, unattended
// ════════════════════════════════════════════════════════════════════════════
/// Container name prefixes the bencher creates.
///
/// A KILLED bench does not clean up after itself — measured three times on
/// 2026-08-13, once leaving a `fetkaos` container running 53 minutes after its
/// parent died, holding two ~5 GB repositories. A leftover container is not
/// merely untidy: the next run's numbers are taken on a box still serving the
/// last one, and nothing on the row says so.
pub const CONTAINER_PREFIXES: &[&str] = &[
"draupnir-gunnarforge-",
"fetkaos-",
"gunnar-population-",
"bakeoff-",
];
/// Remove every container this bencher is known to create, running or not.
///
/// Called BEFORE and AFTER every pass, because either end can be the dirty one:
/// before, in case a previous run was killed; after, so the next thing on this
/// box does not inherit ours.
pub fn cleanup_containers() -> usize {
let mut removed = 0usize;
for prefix in CONTAINER_PREFIXES {
let out = Command::new("podman")
.args(["ps", "-a", "--format", "{{.Names}}", "--filter"])
.arg(format!("name={prefix}"))
.output();
let Ok(out) = out else { continue };
for name in String::from_utf8_lossy(&out.stdout).lines() {
let name = name.trim();
if name.is_empty() {
continue;
}
if Command::new("podman")
.args(["rm", "-f", name])
.output()
.is_ok()
{
removed += 1;
}
}
}
if removed > 0 {
eprintln!(" cleanup: removed {removed} leftover container(s)");
}
removed
}
/// **Each store gets its OWN fixture root, and this is not tidiness.**
///
/// A gunnar repository on disk carries the layout of the engine that wrote it: a
/// `gunnar-git.znippy/` archive for the znippy arms, `objects/pack` for gix. The
/// fixtures are built once and reused, so running the same test on a second
/// store hands it a repository the FIRST store wrote — and the arms catch it:
///
/// ```text
/// real-corpus/nornir.git carries BOTH a gunnar-git.znippy/ store and packs in
/// objects/pack. That is two engines' layouts in one repository, and any single
/// number taken off it is one engine's bytes labelled with both engines' names.
/// ```
///
/// MEASURED 2026-08-13: with one shared root, **13 of 30 failures** on every
/// store were that message and nothing else — the same 13 tests, red identically
/// on `znippy_fast` and `znippy_safe`, which made the two passes look like proof
/// that the writer axis is behaviour-neutral. It was proof of a dirty fixture.
/// A shared fixture root does not just lose those 13 measurements, it
/// manufactures a false agreement between stores.
///
/// One root per store. They cost disk and a rebuild; a comparison between two
/// stores that were handed each other's repositories costs the whole point.
pub fn fixture_root_for(store: Store) -> PathBuf {
PathBuf::from(format!(
"{}/scratch/gunnar-bench-fixtures-{}",
std::env::var("HOME").unwrap_or_else(|_| "/root".to_string()),
store.key()
))
}
/// Scratch trees a killed run leaves behind, relative to the bench fixture root.
///
/// `conn_memory` refused to start on 2026-08-13 because
/// `scale-conn/served-push-1.git` carried BOTH a znippy store and a pack
/// directory — two engines' layouts in one repository, which the arm correctly
/// calls "one engine's bytes labelled with both engines' names". Running the
/// SAME test on five stores makes that certain rather than likely, so the
/// per-store fixtures are removed between passes.
///
/// The list was `scale-conn` and `forge-work` and that was two of the eight that
/// matter. `real-corpus` alone accounted for 11 of the 30 failures on every
/// store — nineteen repositories, written once by whichever engine ran first and
/// handed to every engine after it. `h2h-linear-*` accounted for two more.
pub const DIRTY_FIXTURES: &[&str] = &[
"scale-conn",
"forge-work",
// Written with the layout of whichever engine built them.
"real-corpus",
"scale-cpu",
"bench-bare",
// `engine_served`'s cache: a git mirror materialised into ONE engine's
// store, kept so the ingest is paid once per machine per store. It is
// derived from `real-corpus`, which is on this list, so leaving it behind
// would hand the next pass a store the previous engine wrote under a fresh
// mirror's name — the exact substitution the per-store roots exist to
// prevent.
"engine-served",
// The scratch directory `engine_store_shape` asks the engine to create so
// it can classify what appeared. Tiny, and removed on the way out, but a
// killed run leaves one and a stale one would be classified as this run's
// answer.
".engine-shape-probe",
];
/// Fixture directory PREFIXES that carry an engine's layout — matched against
/// the fixture root's entries, because these are named with their parameters
/// (`h2h-linear-sha1-2048c-1024f-16k`) rather than fixed.
pub const DIRTY_FIXTURE_PREFIXES: &[&str] = &["h2h-", "scale-", "diverge"];
/// Remove the fixture trees that cannot survive a change of store.
pub fn cleanup_fixtures(store: Store) {
let root = fixture_root_for(store);
let mut gone = 0usize;
for d in DIRTY_FIXTURES {
let p = root.join(d);
if p.exists() {
let _ = std::fs::remove_dir_all(&p);
gone += 1;
}
}
// Prefix-matched, because these carry their parameters in the name and a
// fixed list silently stops covering them the day a parameter changes —
// which is how `h2h-linear-sha1-2048c-1024f-16k` survived every sweep.
if let Ok(entries) = std::fs::read_dir(&root) {
for e in entries.flatten() {
let name = e.file_name().to_string_lossy().to_string();
if DIRTY_FIXTURE_PREFIXES.iter().any(|p| name.starts_with(p)) {
let _ = std::fs::remove_dir_all(e.path());
gone += 1;
}
}
}
let bare = crate::expand_home("~/scratch/gunnar-installtest/bench-bare");
let bare = bare.as_path();
if bare.exists() {
let _ = std::fs::remove_dir_all(bare);
gone += 1;
}
if gone > 0 {
eprintln!(
" cleanup: {gone} engine-written fixture tree(s) under {}",
root.display()
);
}
}
/// **Everything, once per store, unattended.**
///
/// Five stores × every test, the monkey isolated under its watchdog, then the
/// containerised bake-off with all five gunnar columns against gitea and
/// Forgejo. Containers and store-sensitive fixtures are cleaned between every
/// pass, so no run inherits the last one's state.
///
/// This is the whole point of the module: one command, no environment variable
/// typed by a person, no operator in the loop.
pub fn do_all(root: &Path) -> Result<(), String> {
eprintln!("╔══ cargo xtask do-all ══════════════════════════════════════╗");
eprintln!(
"║ {} stores × every test, + the bake-off vs gitea/Forgejo ║",
STORES.len()
);
eprintln!("╚════════════════════════════════════════════════════════════╝");
cleanup_containers();
for (i, store) in STORES.iter().enumerate() {
eprintln!(
"\n═══ store {}/{}: {} ═══",
i + 1,
STORES.len(),
store.key()
);
run_one_store(root, *store, &["gunnar.vs_forge_", HANGS], true)?;
cleanup_containers();
cleanup_fixtures(*store);
eprintln!("── {} · the monkey, alone and watchdogged ──", store.key());
run_monkey(root, *store)?;
cleanup_containers();
cleanup_fixtures(*store);
}
eprintln!("\n═══ the bake-off: all five gunnar columns vs gitea and Forgejo ═══");
run_forge_all_columns(root)?;
cleanup_containers();
eprintln!("\n═══ done. Regenerate the chart with `cargo xtask mashup`, ═══");
eprintln!("═══ and check both stores agree with `cargo xtask bench-landed`. ═══");
Ok(())
}