nordisk/gunnar
bench/examples/forge/mod.rs
//! **gunnar versus gitea versus Forgejo, over the one transport all three speak.**
//!
//! The `gunnar.vs_git_*` arms in the parent file compare gunnar against stock
//! `git upload-pack` on the far end of a pipe. That answers "is the pack
//! machinery competitive". It does not answer the question anyone actually asks,
//! which is "how does this compare to the forge we would otherwise deploy".
//! These arms are that comparison.
//!
//! # The shape
//!
//! ```text
//! stock git 2.53.0 on the host <- the SAME client, all engines
//! │
//! ▼
//! 127.0.0.1:<proxy> ── the counting TCP proxy, byte-for-byte the same code
//! │ in front of every engine, so it cancels out
//! ▼
//! 127.0.0.1:<published> ── podman's published port
//! │
//! ▼
//! ┌──────────────┬──────────────────────┬──────────────────────┐
//! │ gunnar │ gitea 1.24.3 │ Forgejo 12.0.4 │
//! │ gunnar-http │ Go smart-HTTP → │ Go smart-HTTP → │
//! │ + gunnar-wire│ git 2.49.1 upload-pack│ git 2.49.1 upload-pack│
//! └──────────────┴──────────────────────┴──────────────────────┘
//! every one of them a container, booted through draupnir (bollard →
//! podman REST), data on md1 by bind mount, CPU and memory read out of
//! its own cgroup
//! ```
//!
//! **Smart HTTP, not SSH and not `ext::`.** gitea and Forgejo do not have a
//! stdin/stdout `upload-pack` mode a harness can drive, and their SSH path goes
//! through their own `serv` shim. HTTP is the transport all three implement, so
//! HTTP is where the comparison is honest. That does mean these numbers are not
//! comparable to the `vs_git_*` rows, which have no HTTP framing in them at all.
//!
//! # What is measured, and why counted quantities lead
//!
//! the desktop is a shared box — load was 12.6–16.4 while this file was written — so
//! wall clock is the column that moves most with the host, so it is reported last.
//! What leads is what can be counted:
//!
//! | metric | where it comes from | what makes it comparable across engines |
//! |---|---|---|
//! | `wire_rx_b` / `wire_tx_b` | the counting proxy | the same proxy code in front of every engine |
//! | `pack_b` | `.pack` bytes in the resulting clone | a statement about what the SERVER produced |
//! | `objects` | `git count-objects -v` on the clone | equal across engines, or the run is red |
//! | `delta_pct` | `git verify-pack -v` | an undeltified pack is a correct pack, and only this sees it |
//! | `cpu_s` | the container's cgroup `cpu.stat` | counts every process the engine forked, which is the "wins by burning four cores" case |
//! | `rss_op_peak_mb` | the container's cgroup `memory.peak`, **reset immediately before the timed region** | what THIS operation made the box give up |
//! | `rss_op_anon_peak_mb` | sampled `memory.stat` `anon` across the region | the part no ceiling and no page cache can inflate |
//!
//! ## What the cgroup numbers do and do not say
//!
//! `cpu.stat` and `memory.peak` are read for the **whole container**, so for
//! gitea and Forgejo they include the s6 supervisor, the bundled `sshd` and
//! whatever housekeeping the web application does in the window. That is not a
//! flaw to be corrected away — it is what running that forge costs — but it is
//! not attributable to the git operation, so every arm also reports
//! `<engine>_idle_cpu_s`: the same cgroup measured over an idle window of the
//! same length, immediately before the operation. Subtract it before drawing a
//! conclusion, and note that the subtraction is the reader's, not the harness's.
//!
//! ### `memory.peak` **can** be reset on this kernel — the old claim was wrong
//!
//! This file used to say: *"`memory.peak` cannot be reset on this kernel
//! (writing `0` to it returns success and changes nothing; measured on the desktop,
//! 2026-08-04, kernel 7.0)"*, and therefore published `rss_peak_mb` as a
//! **lifetime** high-water mark — container boot and repository seeding
//! included — under column headings that named a single git operation. On
//! `vs_forge_push` that printed znippy **3242 MB** against gix **90 MB**, a 36×
//! headline of which almost all was seeding, while `rss_growth_mb` in the very
//! same row read 67.5 against 7.5, or 9.0×.
//!
//! The observation behind the old claim was real and the conclusion was wrong.
//! cgroup v2's `memory.peak` write handler (Linux ≥ 6.12) is a **per-open-file**
//! reset: the write registers a local watermark on *that* fd, and only reads
//! **through the same fd** see it. Re-open the file to read it back — the
//! natural thing to do — and you get the untouched global mark and conclude the
//! write did nothing. Re-measured on the laptop, kernel 7.0.0-29, 2026-08-11: same-fd
//! read after the write drops from a 39321 MB lifetime mark to the 24404 MB then
//! current, and a fresh open still reads 39321 MB.
//!
//! So the operation columns now lead with `rss_op_peak_mb`, reset immediately
//! before the timed region, which excludes seeding **by construction** rather
//! than by asking the reader to subtract. The lifetime number is still here,
//! renamed to `rss_lifetime_peak_mb` — it was never repaired, it was misnamed,
//! and a metric that changes meaning while keeping its name makes every
//! historical row uncomparable with nobody noticing.
//!
//! `rss_growth_mb` is unchanged and is deliberately **not** the headline: it is
//! end-minus-start, so a spike allocated and freed inside the window is
//! invisible to it. Measured against a 100 MB transient it reports **0.0 MB**.
//!
//! ### The ceiling echo, which the reset does not cure
//!
//! `memory.peak` counts page cache, and page cache is reclaimable — it expands
//! to fill whatever `--memory` the cgroup was given. One identical workload
//! (2000 MB of file data written then re-read, ~0 anonymous bytes) reports
//! 512.0 MB under a 512 MB ceiling, 1024.0 MB under a 1024 MB ceiling and
//! 2043.2 MB under a 3072 MB one. That is `vs_forge_clone`'s 8192 MB: the
//! `GUNNAR_BENCH_FORGE_MEM_MB` default read back as if it were a measurement.
//! Resetting the watermark does not help — the reset peak reads 512.0 and
//! 1024.0 too — so the arm also samples `memory.stat`'s `anon`, the charge the
//! kernel cannot reclaim, which stayed at 0.1 MB across all three ceilings.
//! `rss_op_peak_pct_of_ceiling` is emitted beside it so a reader can see a
//! pinned column for what it is instead of having to know.
//!
//! # What is NOT equalised, stated rather than pretended away
//!
//! 1. **gitea and Forgejo are products; gunnar is a git server.** They ship a
//! web UI, issues, pull requests, CI, an API and a user model, and every one
//! of those costs resident memory in the numbers below. This is a comparison
//! of four git operations, not of three products, and no row here says
//! anything about whether one of them is the right thing to deploy.
//! 2. **The base images differ.** gitea and Forgejo ship Alpine images with
//! their own bundled `git` (2.49.1 in both, measured); gunnar runs in the
//! suite's pinned `ubuntu:26.04` image. gunnar's server does not shell out to
//! `git` at all, so the bundled version is not a property of the gunnar
//! column — but the base image is a difference and it is stated.
//! 3. **The engines are asked for the same thing, not configured identically.**
//! gitea and Forgejo get `INSTALL_LOCK`, sqlite, SSH off and cron off;
//! gunnar gets `--http-allow-cleartext-credentials` (it refuses a token over
//! plain HTTP otherwise) and `--no-grpc`. Neither side is tuned.
//! 4. **gunnar's znippy znippy engine is not wired.** `pack_tier` returns an empty
//! vector, the fold has no call site in any server crate, and the control
//! plane prints that no znippy znippy engine is built. What is measured here is
//! gunnar on the filesystem, which is the only backend that survives a
//! restart today.
//!
//! # The bar: a server that never started must not look like a slow server
//!
//! That is the shape that has bitten this repo repeatedly, so it is asserted at
//! every point below and each one is reachable from a red switch
//! (`GUNNAR_BENCH_FORGE_RED=…`, comma-separated):
//!
//! | switch | what it breaks | which guard must fire |
//! |---|---|---|
//! | `nostart` | gives gitea a command that exits immediately | [`Forge::up`]'s readiness deadline, reporting `ContainerState::Exited` and the container's own log |
//! | `shallow` | clones one engine at `--depth 1` | the equal-objects assertion in [`assert_engines_agree`] |
//! | `noproxy` | points one engine's client at the published port, bypassing the meter | the wire-byte floor in [`assert_counted`] |
//! | `fatseed` | inflates **provisioning** by 1500 MB | none — this one is read, not caught. `rss_lifetime_peak_mb` must move (measured: **3.85×** on gitea) while the request is byte-identical (`wire_rx_b` unchanged to the byte), and `rss_op_anon_peak_mb` must not (measured: **1.00×** on gunnar, 796.25 → 794.73). See [`seed`] for what it does and does not hold constant. |
//! | `wrongengine` | gives gunnar's container the **other** storage engine from the one this run asked for — a container that boots, serves and answers every request | [`assert_storage_engine`], which reads the engine out of the server's own startup line. Nothing else in the run can see it: the caps are right, the git config is right, the byte counts are plausible, and the column is labelled with the engine that was *requested*. |
//! | `forgeengine` | puts `GUNNAR_ENGINE` on a Go forge, which has no storage engine to select and ignores it entirely | [`parity::assert_no_engine_var`]. The run stays correct in every measurable way; the defect is a variable on a container suggesting a choice that does not exist there. |
//! | `forgearms` | the same, one level down: puts `ZNIPPY_GIT_WRITER` and `ZNIPPY_GIT_INDEX` on a Go forge, which has no archive writer and no Arrow projection | [`parity::assert_no_engine_var`], once per key. |
//! | `wrongwriter` | gives a znippy column a **different writer arm** from the one its key names — a container that boots, reports the right storage engine, and serves every request | [`parity::assert_znippy_arms`], which reads `ArchiveWrite::name` out of the server's own line. The durability contract is the entire quantity that column exists to price, and nothing else in the run moves when it is wrong. |
//! | `wrongindex` | gives a znippy column a **different index layout** | [`parity::assert_znippy_arms`] again, on the other axis. `znippy_uring` and `znippy_explode` are the same writer and differ in nothing but this. |
//! | `concrefuse` | sets a low `GUNNAR_MAX_TRANSFERS` on the gunnar container so the concurrent ramp crosses a **binding** admission ceiling | the concurrent arm's own guard: `<engine>_<rung>_conc_refused` must be `> 0`, the gate's `refused_total` must agree with the clients turned away (`429`), and `conc_peak_in_flight` must equal the ceiling. This is the red that proves the in-flight/refusal column reads a real refusal, not a fabricated 0 — its load-independent half is also proven off-box by `gunnar_bench::admission_log`. Needs a podman rig to run the full arm. |
//!
//! # Which store served each gunnar column — engine, writer arm and layout
//!
//! `gunnar-server` can carry both storage engines at once and picks between them
//! at runtime from `GUNNAR_ENGINE` (`crates/gunnar-server/src/storage.rs`), and
//! its znippy engine picks a writer arm and an index layout from
//! `ZNIPPY_GIT_WRITER` / `ZNIPPY_GIT_INDEX` (`znippy-plugin-git`'s `arms.rs`).
//! So "gunnar" was never one column. **A container does not inherit the host
//! environment.** The `vs_git_*` arms in the parent file reach their server
//! through `git`'s `ext::` transport, which forks from the harness and therefore
//! carries the harness's variables; these arms reach theirs through podman,
//! which does not.
//!
//! Two separate defects were behind that, and only one of them was a harness
//! problem:
//!
//! * the engine variable never crossed the container boundary at all (fixed
//! 2026-08-11 — a two-engine run had differed on the five `ext::` arms and
//! reported znippy twice on the eight here, with nothing on the row saying
//! so);
//! * **the writer arm did not work even when it did cross.** `znippy_engine`
//! opened its stores with `GitStore::open_with`, a constructor znippy
//! documents as reading no environment — *"a caller that never asked for a
//! selector must not have its durability contract moved out from under it"* —
//! so `ZNIPPY_GIT_WRITER=fast` on a `gunnar serve` process selected nothing
//! and every push went through `SafeWriter`. That is fixed in the server, not
//! here: it now resolves `StoreConfig::from_env` once per process and opens
//! through `open_selected`.
//!
//! Four things close it, and they are deliberately separate:
//!
//! 1. [`gunnar_columns`] turns `GUNNAR_ENGINE` and
//! `GUNNAR_BENCH_FORGE_ZNIPPY_ARMS` into a list of [`GunnarStore`]s — one
//! per column — and [`plan_for`] puts each column's variables in *its*
//! container's environment through the same `with_env` path the allocation
//! already uses. The two Go forges are given none of them; the names mean
//! nothing there, and a variable on gitea's container would read as if it
//! had done something. [`parity::assert_no_engine_var`] checks that absence
//! rather than intending it.
//! 2. [`parity::assert_storage_engine`] **reads the engine back out of the
//! running server** — `gunnar::store`'s `store.engine_selected` line,
//! emitted from `ServerStores::open` before the first repository is opened.
//! 3. [`parity::assert_znippy_arms`] reads the **arms** back the same way, from
//! `store.znippy_arms_selected`, and compares against the *implementation*
//! names (`FastWriter`, `PackedPayload`) rather than against the environment
//! variable. That distinction is load-bearing: a server that ignores the
//! selector completely — which is what every build before 2026-08-11 was —
//! passes any check written against the request.
//! 4. Every gunnar column carries `<column>_storage_engine`, and every znippy
//! column also `<column>_writer_arm`, `<column>_writer_impl`,
//! `<column>_index_arm` and `<column>_index_impl`, so a number can never be
//! read a month later without knowing which store produced it.
//!
//! ## The seven columns, and the two axes they are laid out on
//!
//! `GUNNAR_BENCH_FORGE_ZNIPPY_ARMS=fast,safe,uring,explode,gix` draws seven
//! columns instead of three. It is **opt-in**: the default is the single
//! `gunnar` column this file has always had, because seven containers per arm is
//! more than twice the boot cost and every existing `*_over_gunnar_*` key would
//! change meaning under a default that split. See [`ZnippyArms`] for the table —
//! the writer axis is `fast`/`safe`/`uring` with the layout pinned at `packed`,
//! and the layout axis is `uring`/`explode` with the writer pinned at `uring`.
//!
//! # What still shells out to `podman`, and why — audited 2026-08-11
//!
//! The boot path never did, and two more call sites stopped: the janitor
//! ([`sweep_stale`]) and the environment read-back
//! ([`parity::container_env`]) now go through
//! `draupnir::container::ContainerControl`'s `list_containers` and
//! `container_env`, added to draupnir for these. `parity.rs` no longer spawns a
//! process at all. What remains is **three `podman unshare …` calls and one
//! `podman inspect`**, and they are listed rather than left to be rediscovered:
//!
//! | site | call | why it is not a draupnir call |
//! |---|---|---|
//! | [`Cgroup::of`] | `inspect --format {{.State.CgroupPath}}` | draupnir exposes no cgroup path. It *could*, and that would be the next one to lift; it is the only remaining read that a draupnir call would obviously subsume |
//! | [`Forge::pin_cpus`] | `unshare taskset -acp` | not container control: "run this in the engine's user namespace". draupnir models containers, not the userns its engine runs under |
//! | [`clear_dir`] | `unshare rm -rf` | same — removing root-owned files created inside the userns, from outside it |
//! | [`plan_for`] | `unshare chown -R 1000:1000` | same, and the image's own entrypoint requires it |
//!
//! The `unshare` three are not a gap in draupnir. They would be a new concept in
//! it — "borrow the engine's user namespace to run an arbitrary command" — which
//! is a shell by another name, and the reason to route containers through a
//! typed API is precisely to *not* have that.
//!
//! # Containers via draupnir, and not via `RealRunner`
//!
//! Every container here is booted through `draupnir::container::ContainerBoot`,
//! which drives podman's REST socket through bollard and has no shell fallback
//! at all: an unreachable socket is `Error::Backend`, never a pretend boot.
//! dwarves' `RealRunner` routes container steps to `DemoRunner`, which returns
//! `ok("would run …")` and reports success having executed nothing; a benchmark
//! built on that would produce a full table of numbers with no server behind it.
use std::collections::BTreeMap;
use std::io::{Read, Seek, Write};
use std::net::{Shutdown, TcpListener, TcpStream};
use std::path::{Path, PathBuf};
use std::process::Command;
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::sync::Arc;
use std::time::{Duration, Instant};
use anyhow::{bail, ensure, Context, Result};
use nornir_bench::api::Bencher;
use nornir_bench::BenchResult;
use serde_json::{json, Value};
use super::{
cached_repo, env_usize, fixture_root, h2h_carve, h2h_deltified_client, h2h_divergent_fixture,
h2h_git, h2h_linear_fixture, h2h_pack_bytes, preview, result, round,
};
/// The allocation every engine is given, and the read-back that proves it got
/// it. Kept in its own file so the equalising and the measuring stay separable.
mod parity;
/// The arms that put more than one client on the box at a time: the ramp, and
/// the mixed read-and-write workload. Everything above measures one client, and
/// one client cannot answer how many customers a box carries.
mod concurrent;
// ═══════════════════════════════════════════════════════════════════════════
// THE PINS
// ═══════════════════════════════════════════════════════════════════════════
/// gitea's image, pinned. A ratio is a statement about one specific build.
const GITEA_IMAGE: &str = "docker.io/gitea/gitea:1.24.3";
/// The version string `gitea --version` must print, asserted rather than assumed.
const GITEA_VERSION: &str = "1.24.3";
/// Forgejo's image, pinned. Pulled from `data.forgejo.org`, **never**
/// `codeberg.org`: Codeberg's quota for this account was exhausted on the
/// morning of 2026-08-04 and this suite must not touch it.
const FORGEJO_IMAGE: &str = "data.forgejo.org/forgejo/forgejo:12";
/// What `forgejo --version` must print.
const FORGEJO_VERSION: &str = "12.0.4";
/// The image gunnar's own server runs in: the suite's pinned toolchain image,
/// reused rather than twinned. gunnar's binary is bind-mounted in, which
/// works because that image is the desktop's own distribution and therefore the desktop's own
/// glibc.
const GUNNAR_IMAGE: &str = "localhost/gunnar-bench-git:2.53.0";
/// The `git` the **client** must be, on the host. Same client for every engine,
/// so this is the one version the whole comparison is relative to.
const CLIENT_GIT_VERSION: &str = "git version 2.53.0";
/// The port every engine listens on inside its own container.
const IN_CONTAINER_PORT: u16 = 3000;
/// The account the seeding and the authenticated operations use on gitea and
/// Forgejo. gunnar has no user model of this shape; it gets a principal and a
/// token, which is the same thing at the HTTP Basic seam.
const FORGE_USER: &str = "bench";
const FORGE_PASSWORD: &str = "benchpw-2026-08-04";
/// How long an engine gets to answer its own health endpoint before the arm is
/// red. Generous, because the desktop is shared; a deadline that expires is reported
/// with the container's state and log, never as a zero.
fn ready_budget() -> Duration {
Duration::from_secs(env_usize("GUNNAR_BENCH_FORGE_READY_S", 120) as u64)
}
/// Interleaved rounds per engine. Three by default: the median of three
/// interleaved rounds is the weakest claim that still survives one scheduling
/// hiccup on a box at load 12–218.
fn rounds() -> usize {
env_usize("GUNNAR_BENCH_FORGE_ROUNDS", if preview() { 1 } else { 3 })
}
/// The red switches. `GUNNAR_BENCH_FORGE_RED=nostart,shallow` drives several.
///
/// A guard nobody has watched fail is not a guard. These exist so each
/// failure can be produced on demand, in one command, next week as well as today.
fn red(which: &str) -> bool {
std::env::var("GUNNAR_BENCH_FORGE_RED")
.map(|v| v.split(',').any(|s| s.trim() == which))
.unwrap_or(false)
}
/// The variable that picks a storage engine, spelled once.
///
/// The same name `gunnar_server::storage::ENGINE_ENV` holds. It is a literal
/// here rather than an import because `bench/` is a detached crate that does
/// not depend on `gunnar-server` at all — it drives a *binary*, and the binary's
/// environment is the only contract between them.
const ENGINE_ENV: &str = "GUNNAR_ENGINE";
/// **The storage engine this run asked gunnar for**, resolved once.
///
/// The harness's own `GUNNAR_ENGINE`, lower-cased and trimmed, defaulting to
/// `znippy` — the same default `Engine::selected()` applies inside the server.
/// The default is spelled out here rather than left implicit because this string
/// is what the row is labelled with and what the read-back is compared against,
/// and "the row said nothing because the variable was unset" is the failure this
/// whole seam exists to remove.
///
/// An unrecognised name is *not* corrected here. It is passed through, the
/// server refuses it at startup naming what it does carry, and the arm reports
/// that refusal — which is a better answer than a harness that quietly
/// substitutes the default for a typo.
fn storage_engine() -> String {
std::env::var(ENGINE_ENV)
.ok()
.map(|v| v.trim().to_ascii_lowercase())
.filter(|v| !v.is_empty())
.unwrap_or_else(|| "znippy".to_string())
}
// ═══════════════════════════════════════════════════════════════════════════
// WHICH ZNIPPY, AND NOT JUST WHETHER ZNIPPY
// ═══════════════════════════════════════════════════════════════════════════
/// The two variables znippy's git plugin picks its arms from, spelled once.
///
/// `znippy_plugin_git::arms::{ENV_WRITER, ENV_INDEX}` hold the same names.
/// Literals here for the same reason [`ENGINE_ENV`] is one: `bench/` is a
/// detached crate that drives a *binary*, and the binary's environment is the
/// only contract between them.
///
/// The third arm — [`ZNIPPY_GC_ENV`] — is deliberately absent **from the
/// crossing**, and that part was always right: no column selects a gc arm. What
/// was wrong was reading that as a reason to leave it out of [`ZNIPPY_ENVS`]
/// too.
const ZNIPPY_WRITER_ENV: &str = "ZNIPPY_GIT_WRITER";
/// See [`ZNIPPY_WRITER_ENV`]. The payload layout behind the read stack's
/// `stree`; it cannot be switched off, only chosen between.
const ZNIPPY_INDEX_ENV: &str = "ZNIPPY_GIT_INDEX";
/// See [`ZNIPPY_WRITER_ENV`]. The third arm `StoreConfig::from_env` reads, and
/// the one no column selects — `new-generation` is the default and is also the
/// arm that keeps the old generation until the new one has been read back.
///
/// **Named here even though nothing crosses the boundary with it**, which is
/// the distinction that matters and that this file got wrong until 2026-08-11.
/// Not selecting an arm and not *asserting the absence* of its variable are two
/// different jobs. It belongs in [`ZNIPPY_ENVS`] — the must-be-absent list —
/// for exactly the reason the other two do: a Go forge that inherited a stray
/// `ZNIPPY_GIT_GC` from the harness's own environment would have gone unnoticed,
/// because the only list that looks was two entries long against a function
/// that reads three.
const ZNIPPY_GC_ENV: &str = "ZNIPPY_GIT_GC";
/// Every `ZNIPPY_*` variable a Go forge has to be **free** of, checked rather
/// than intended. See [`parity::assert_no_engine_var`].
///
/// # This list is not the crossing list, and conflating them is the trap
///
/// `[ZNIPPY_WRITER_ENV, ZNIPPY_INDEX_ENV]` until 2026-08-11. The two lists
/// coincided by accident — every variable a column *sets* is also one a Go
/// forge must not have — and the coincidence made a two-entry array look
/// complete when it was only complete for one of its two jobs.
/// `znippy_plugin_git::arms::StoreConfig::from_env` reads **three** keys in
/// three adjacent lines, and this asserted the absence of two of them.
///
/// The remaining gap is structural and is NOT closed by adding a third literal:
/// nothing here notices znippy growing a fourth key. A `pub const ALL_ENV` on
/// znippy's side, asserted against this list plus named exclusions, is what
/// would go red on the day it does. Recorded rather than done — znippy's `src`
/// is not edited from here.
const ZNIPPY_ENVS: [&str; 3] = [ZNIPPY_WRITER_ENV, ZNIPPY_INDEX_ENV, ZNIPPY_GC_ENV];
/// **One znippy column**: the two arms it names, and the two implementation
/// names the running server has to report back before the column is believed.
///
/// # Two axes, held one at a time
///
/// The four columns in [`ZNIPPY_COLUMNS`] are not four points on one scale.
/// They are a 2×2 read down two axes, and the design only says anything because
/// each axis moves with the other pinned:
///
/// | column | writer | index | what it isolates |
/// |---|---|---|---|
/// | `znippy_fast` | `fast` | `packed` | the durability contract, at its ceiling — no fsync, **no journal** |
/// | `znippy_safe` | `safe` | `packed` | git's own contract: blob fsync, journal row, journal fsync |
/// | `znippy_uring` | `uring` | `packed` | the same ordering, one `io_uring_enter` instead of four syscalls |
/// | `znippy_explode` | `uring` | `one-table` | the **payload layout**, writer held at `uring` |
///
/// So `fast`/`safe`/`uring` is the writer axis with the layout pinned at
/// `packed`, and `uring`/`explode` is the layout axis with the writer pinned at
/// `uring`. Move a second thing in either pair and the pair stops being a
/// comparison — which is the reason this is a table of fixed rows rather than
/// two environment variables a caller crosses at will.
///
/// # Why the implementation names are on the row
///
/// `writer` is what an operator types and `writer_impl` is what the built
/// object calls itself, and znippy keeps them deliberately distinct so that a
/// guard comparing them is comparing **the selector against applied output**
/// rather than a name against itself (`arms.rs`, `IndexArm::projection_name`).
/// The read-back in [`up_with`] compares against the `*_impl` pair, never
/// against `writer`/`index`.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
struct ZnippyArms {
/// The row key — and therefore the container name and the data directory,
/// which is what keeps four znippy stores from being one store.
key: &'static str,
/// The spelling `GUNNAR_BENCH_FORGE_ZNIPPY_ARMS` selects this column by.
token: &'static str,
/// `ZNIPPY_GIT_WRITER`.
writer: &'static str,
/// `ZNIPPY_GIT_INDEX`.
index: &'static str,
/// What `ArchiveWrite::name` answers for `writer`, out of the running
/// server.
writer_impl: &'static str,
/// What `ObjectIndex::name` answers for the projection `index` builds.
index_impl: &'static str,
}
/// The arms a single-column run gets: znippy's own shipping default.
///
/// `SafeWriter` + `OneTableFourColumns`, which is what `GitStore::open` builds
/// and therefore what every `vs_forge_*` row taken before 2026-08-11 was
/// measuring. Set **explicitly** rather than left unset, so that the default
/// column goes through the same read-back as the split ones: "the variables
/// were absent and the server did whatever it does" is not a statement a row
/// can carry.
const ZNIPPY_DEFAULT: ZnippyArms = ZnippyArms {
key: "gunnar",
token: "default",
writer: "safe",
index: "one-table",
writer_impl: "SafeWriter",
index_impl: "OneTableFourColumns",
};
/// The four znippy columns a split run draws. See [`ZnippyArms`] for the two
/// axes they are laid out on.
const ZNIPPY_COLUMNS: [ZnippyArms; 4] = [
ZnippyArms {
key: "znippy_fast",
token: "fast",
writer: "fast",
index: "packed",
writer_impl: "FastWriter",
index_impl: "PackedPayload",
},
ZnippyArms {
key: "znippy_safe",
token: "safe",
writer: "safe",
index: "packed",
writer_impl: "SafeWriter",
index_impl: "PackedPayload",
},
ZnippyArms {
key: "znippy_uring",
token: "uring",
writer: "uring",
index: "packed",
writer_impl: "UringWriter",
index_impl: "PackedPayload",
},
ZnippyArms {
key: "znippy_explode",
token: "explode",
writer: "uring",
index: "one-table",
writer_impl: "UringWriter",
index_impl: "OneTableFourColumns",
},
];
/// **What one gunnar column is made of.**
///
/// The gunnar family is not one column and not even two: it is a storage engine
/// *and*, on the znippy side, a writer arm and an index layout. Carrying that on
/// the [`Engine`] value rather than in a global is what lets several of them
/// stand side by side in one run — which is the whole point, because a
/// durability contract compared across two runs on a shared box is compared
/// against the box.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum GunnarStore {
/// `GUNNAR_ENGINE=znippy`, with the two `ZNIPPY_GIT_*` arms this column is.
Znippy(ZnippyArms),
/// Any other engine, named exactly as `GUNNAR_ENGINE` spells it.
///
/// `gix` in practice. The string is **passed through unchanged**, typos
/// included: the server refuses a name it does not carry and says what it
/// does carry, which is a better answer than a harness quietly substituting
/// the default. `key` is separate because this column is `gunnar` in a
/// single-column run and `gix` beside the znippy split.
Named {
key: &'static str,
engine: &'static str,
},
}
impl GunnarStore {
fn key(self) -> &'static str {
match self {
GunnarStore::Znippy(a) => a.key,
GunnarStore::Named { key, .. } => key,
}
}
/// The `GUNNAR_ENGINE` this column asks for.
fn engine_name(self) -> &'static str {
match self {
GunnarStore::Znippy(_) => "znippy",
GunnarStore::Named { engine, .. } => engine,
}
}
/// The znippy arms this column names, or `None` for a non-znippy store.
fn arms(self) -> Option<ZnippyArms> {
match self {
GunnarStore::Znippy(a) => Some(a),
GunnarStore::Named { .. } => None,
}
}
}
/// The single gunnar column a run gets when nobody asked for the split: keyed
/// `gunnar`, with its engine out of `GUNNAR_ENGINE`.
fn default_gunnar_store() -> GunnarStore {
let engine = storage_engine();
if engine == "znippy" {
GunnarStore::Znippy(ZNIPPY_DEFAULT)
} else {
GunnarStore::Named {
key: "gunnar",
engine: leak(&engine),
}
}
}
/// **Which gunnar columns this run measures.**
///
/// Unset — the default — is exactly what this file did before the split: ONE
/// gunnar column, keyed `gunnar`, whose store comes from `GUNNAR_ENGINE`. That
/// is deliberate and not timidity: seven containers per arm is more than twice
/// the boot cost of three, and every existing `gunnar_*` row key and every
/// `*_over_gunnar_*` ratio would change meaning under a default that silently
/// split.
///
/// `GUNNAR_BENCH_FORGE_ZNIPPY_ARMS` opts in, as a comma-separated list of
/// [`ZnippyArms::token`]s plus `gix`:
///
/// ```text
/// GUNNAR_BENCH_FORGE_ZNIPPY_ARMS=fast,safe,uring,explode,gix
/// GUNNAR_BENCH_FORGE_ZNIPPY_ARMS=all # the same five
/// GUNNAR_BENCH_FORGE_ZNIPPY_ARMS=fast,safe # the two cheap writer columns
/// ```
///
/// The order the tokens are written in is **not** the order the columns run in:
/// the table's order is, so that two runs whose lists were typed differently are
/// still comparable round by round.
///
/// ⚠ One letter from `GUNNAR_BENCH_FORGE_ZNIPPY_ARGS`, which is a different
/// thing entirely — the `gunnar serve` flags that would switch on the znippy-engine
/// column ([`znippy_serve_args`]). ARM**S** picks writer and index arms of the
/// znippy *engine*; ARG**S** wires a *tier* that is still not built. Both are
/// checked by [`assert_columns_consistent`], so a mistyped one is refused rather
/// than silently ignored.
///
/// An unrecognised token is a hard error rather than a dropped column, and it is
/// raised by [`assert_columns_consistent`] rather than here — a column set is
/// checked once per arm, and a `Vec` returned from a hundred call sites cannot
/// carry a diagnosis. Silently dropping it would produce a chart with a missing
/// bar, which is the failure this whole file is arranged against.
fn gunnar_columns() -> Vec<GunnarStore> {
let Some(raw) = column_tokens() else {
return vec![default_gunnar_store()];
};
let want = |t: &str| raw.iter().any(|w| w == t || w == "all");
let mut out: Vec<GunnarStore> = ZNIPPY_COLUMNS
.iter()
.filter(|a| want(a.token))
.map(|a| GunnarStore::Znippy(*a))
.collect();
if want("gix") {
out.push(GunnarStore::Named {
key: "gix",
engine: "gix",
});
}
// A list that named nothing this file knows is not a run with no gunnar
// column — `assert_columns_consistent` refuses it — but `selected()` is
// called from places that cannot fail, so it must not hand back an empty
// gunnar family and let a `_over_gunnar_` ratio divide by a column that
// does not exist.
if out.is_empty() {
out.push(default_gunnar_store());
}
out
}
/// The raw `GUNNAR_BENCH_FORGE_ZNIPPY_ARMS` tokens, or `None` when unset.
fn column_tokens() -> Option<Vec<String>> {
let raw = std::env::var("GUNNAR_BENCH_FORGE_ZNIPPY_ARMS").ok()?;
let v: Vec<String> = raw
.split(',')
.map(|s| s.trim().to_ascii_lowercase())
.filter(|s| !s.is_empty())
.collect();
if v.is_empty() {
None
} else {
Some(v)
}
}
// ═══════════════════════════════════════════════════════════════════════════
// THE SYSCALL FILTER, WHICH IS PART OF THE ALLOCATION
// ═══════════════════════════════════════════════════════════════════════════
/// The seccomp profile **every** container in this arm runs under: podman's own
/// default, plus the three `io_uring_*` syscalls.
///
/// # Why this file has to care about seccomp at all
///
/// podman's stock `/usr/share/containers/seccomp.json` allow-lists no
/// `io_uring_*` syscall, so `io_uring_setup` inside a stock container returns
/// **ENOSYS** — and `WriterArm::Uring` refuses to start rather than substitute a
/// `pwrite` path under an io_uring name, which is the right refusal and makes
/// the `znippy_uring` and `znippy_explode` columns unbootable. MEASURED on the desktop
/// 2026-08-11, kernel 7.0.0-29, one identical 12-line probe calling `syscall(425)`
/// three ways:
///
/// | where | `io_uring_setup` |
/// |---|---|
/// | on the host (`/proc/sys/kernel/io_uring_disabled` = 0) | fd 3 |
/// | in the container, podman's default profile | −1, errno 38 (ENOSYS) |
/// | in the container, `--security-opt seccomp=unconfined` | fd 3 |
///
/// So the kernel is willing and the container's filter is not, and there is no
/// way to discover that from inside the container: to the writer it is
/// indistinguishable from a kernel too old to have io_uring.
///
/// # Why `unconfined` is NOT what is used, and why it goes on all seven
///
/// A syscall filter costs something on every syscall, so a gunnar container
/// running `seccomp=unconfined` beside a gitea container running podman's
/// default would be a parity break **in gunnar's favour** — exactly the
/// direction this file refuses to let pass unremarked. Two things follow:
///
/// 1. the profile is podman's own with **three names added**, not the filter
/// switched off, so every other syscall is filtered exactly as it was; and
/// 2. it is applied to **every** container in the arm — gunnar, gitea and
/// Forgejo alike — so the columns still differ only in the engine.
///
/// A machine with no default profile to extend gets `None`, which applies
/// nothing to anybody: still uniform, and the uring columns then fail with
/// znippy's own refusal rather than being quietly served by a different writer.
fn seccomp_profile() -> Option<&'static Path> {
use std::sync::OnceLock;
static PROFILE: OnceLock<Option<PathBuf>> = OnceLock::new();
PROFILE
.get_or_init(|| build_seccomp_profile().ok())
.as_deref()
}
/// podman's default profile with `io_uring_setup`, `io_uring_enter` and
/// `io_uring_register` appended as one more `SCMP_ACT_ALLOW` block.
///
/// Written beside the fixtures rather than into the arm's work root: the daemon
/// reads it, `up` is called dozens of times per run, and a profile regenerated
/// per container would be three containers reading a file another thread is
/// rewriting.
fn build_seccomp_profile() -> Result<PathBuf> {
const BASE: &str = "/usr/share/containers/seccomp.json";
let text = std::fs::read_to_string(BASE)
.with_context(|| format!("reading podman's default seccomp profile at {BASE}"))?;
let mut json: Value = serde_json::from_str(&text)
.with_context(|| format!("{BASE} is not JSON, so it cannot be extended"))?;
let syscalls = json
.get_mut("syscalls")
.and_then(Value::as_array_mut)
.context("the default seccomp profile has no `syscalls` array to extend")?;
syscalls.push(json!({
"names": ["io_uring_setup", "io_uring_enter", "io_uring_register"],
"action": "SCMP_ACT_ALLOW",
}));
let out = fixture_root().join("forge-seccomp-io-uring.json");
if let Some(dir) = out.parent() {
std::fs::create_dir_all(dir)?;
}
// Written whole and then renamed, so a second `nornir-bench` process cannot
// hand the daemon a half-written profile.
let tmp = out.with_extension("json.tmp");
std::fs::write(&tmp, serde_json::to_vec_pretty(&json)?)
.with_context(|| format!("writing {}", tmp.display()))?;
std::fs::rename(&tmp, &out)?;
Ok(out)
}
/// The engine name a gunnar container is **actually handed**.
///
/// ── RED SWITCH: `wrongengine` ──────────────────────────────────────────────
///
/// Hand the container the *other* engine from the one this run asked for. The
/// container boots, passes every quota and git-config read-back, answers its
/// health probe and serves every request — and the column is labelled with the
/// engine that was requested. Nothing in the run can see it except
/// [`parity::assert_storage_engine`], which compares the server's own startup
/// line against what the column claims.
///
/// It flips znippy↔gix rather than passing something bogus on purpose. A bogus
/// name is refused by the server at startup, so it would be caught by the
/// boot deadline — a different guard, proving a different thing. This one has to
/// produce a *working server running the wrong store*, because that is the
/// failure a benchmark cannot see.
///
/// It needs a binary carrying **both** engines to prove the mismatch path; on a
/// single-engine build the server refuses to start instead and the boot deadline
/// is what reports it.
fn engine_for_container(store: GunnarStore) -> String {
let want = store.engine_name().to_string();
if !red("wrongengine") {
return want;
}
match want.as_str() {
"gix" => "znippy".to_string(),
_ => "gix".to_string(),
}
}
/// The writer arm a znippy container is **actually handed**.
///
/// ── RED SWITCH: `wrongwriter` ──────────────────────────────────────────────
///
/// The writer-axis twin of `wrongengine`, and it is a separate switch because it
/// is a separate blindness. Hand the column a *different* writer arm from the
/// one its key names. The container boots, the storage engine read-back is
/// correct, the quotas are correct, the byte counts are plausible, the git
/// config is right and every request is served — and the column labelled
/// `znippy_fast` is measuring a `SafeWriter` doing two device flushes per
/// append, which is the entire quantity the column exists to measure.
///
/// It swaps between two **working** arms rather than passing a typo. A typo is
/// refused at `StoreConfig::from_env`, so the container never serves and the
/// boot deadline reports it — a different guard proving a different thing. What
/// has to be produced here is a healthy server running the wrong writer.
fn writer_for_container(arms: ZnippyArms) -> &'static str {
if !red("wrongwriter") {
return arms.writer;
}
match arms.writer {
"safe" => "fast",
_ => "safe",
}
}
/// The index arm a znippy container is **actually handed**.
///
/// ── RED SWITCH: `wrongindex` ───────────────────────────────────────────────
///
/// The layout-axis twin of [`writer_for_container`]. `znippy_explode` and
/// `znippy_uring` differ in **nothing but this**, so a column whose layout
/// silently fell back while the row said `packed` is not a small mislabelling —
/// it is the entire content of that pair of columns, and there is no other
/// figure on the row that moves when it happens.
fn index_for_container(arms: ZnippyArms) -> &'static str {
if !red("wrongindex") {
return arms.index;
}
match arms.index {
"packed" => "one-table",
_ => "packed",
}
}
// ═══════════════════════════════════════════════════════════════════════════
// THE ENGINES
// ═══════════════════════════════════════════════════════════════════════════
/// Which server is behind the port.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum Engine {
/// gunnar, and **which** gunnar: the storage engine, plus the writer arm and
/// index layout when that engine is znippy. See [`GunnarStore`].
Gunnar(GunnarStore),
/// gunnar with the znippy znippy engine switched on.
///
/// **Not wired as of 2026-08-04, checked in this tree rather than taken on
/// report:** `RepackOptions::cold_tier` has no call site in any server
/// crate, the only implementation of the old cold-storage trait was in the deleted engine crate, that
/// crate is in the root manifest's `exclude` list, `Cargo.lock` holds zero
/// `znippy` rows, `gunnar serve --help` and `gunnar gc --help` offer no
/// znippy-engine switch, and the control plane answers
/// `"fs (gix loose objects + packs); no znippy znippy engine is built"`
/// (`gunnar-cli/src/server/control_plane.rs:903`).
///
/// So this column cannot be measured yet, and the arms say so on the row
/// (`znippy_wired = 0`) rather than quietly drawing three bars where four
/// were asked for. The slot is here so the lane wiring the tier turns it on
/// with one environment variable and gets every metric the other columns
/// already carry — see [`znippy_serve_args`].
GunnarZnippy,
Gitea,
Forgejo,
}
impl Engine {
/// The columns this run measures, in the order every arm interleaves them.
///
/// Three by default — one gunnar column and the two Go forges — and up to
/// seven when [`gunnar_columns`] has been asked for the split. The gunnar
/// columns lead so `<baseline>_drift_x` prices host movement across the
/// widest window the arm spans, and they are in [`ZNIPPY_COLUMNS`] order
/// rather than in the order the tokens were typed.
fn selected() -> Vec<Engine> {
let mut v: Vec<Engine> = gunnar_columns().into_iter().map(Engine::Gunnar).collect();
if znippy_serve_args().is_some() {
v.push(Engine::GunnarZnippy);
}
v.push(Engine::Gitea);
v.push(Engine::Forgejo);
v
}
/// **The column every ratio is taken against**: the first gunnar column.
///
/// `gunnar` in a default run, so `gitea_over_gunnar_small_wall` is spelled
/// exactly as it always was and a year of rows stays comparable. In a split
/// run it is the first znippy column, and the ratio keys say so
/// (`gitea_over_znippy_fast_small_wall`) rather than keeping a `gunnar` in
/// the name that no column answers to — a denominator that silently
/// resolves to nothing yields `ratio(x, 0.0) == 0.0`, which is shaped
/// exactly like a free engine.
fn baseline_key() -> &'static str {
Engine::selected()
.into_iter()
.find(|e| e.is_gunnar_family())
.map(|e| e.key())
.unwrap_or("gunnar")
}
fn key(self) -> &'static str {
match self {
Engine::Gunnar(s) => s.key(),
Engine::GunnarZnippy => "gunnar_znippy",
Engine::Gitea => "gitea",
Engine::Forgejo => "forgejo",
}
}
/// The gunnar store behind this column, or `None` for a Go forge.
///
/// The znippy-engine slot rides whatever the default column's store is: it is
/// the same binary and the same provisioning, differing only in the
/// `gunnar serve` flags [`znippy_serve_args`] supplies.
fn gunnar_store(self) -> Option<GunnarStore> {
match self {
Engine::Gunnar(s) => Some(s),
Engine::GunnarZnippy => Some(default_gunnar_store()),
Engine::Gitea | Engine::Forgejo => None,
}
}
/// The znippy arms this column claims, or `None` when it is not a znippy
/// column. What the read-back in [`up_with`] is compared against.
fn znippy_arms(self) -> Option<ZnippyArms> {
self.gunnar_store().and_then(GunnarStore::arms)
}
fn image(self) -> &'static str {
match self {
Engine::Gunnar(_) | Engine::GunnarZnippy => GUNNAR_IMAGE,
Engine::Gitea => GITEA_IMAGE,
Engine::Forgejo => FORGEJO_IMAGE,
}
}
/// `true` for the two Go forges, which share a provisioning path because
/// Forgejo is a gitea fork and its admin CLI, its API and its `GITEA__`
/// environment override scheme are all still gitea's.
fn is_gitea_family(self) -> bool {
matches!(self, Engine::Gitea | Engine::Forgejo)
}
/// The in-container binary name. Forgejo renamed gitea's; everything else
/// about the two invocations is identical.
fn admin_binary(self) -> &'static str {
match self {
Engine::Forgejo => "forgejo",
_ => "gitea",
}
}
/// The environment-variable prefix the image reads its `app.ini` overrides
/// from. Forgejo accepts `FORGEJO__`; gitea only `GITEA__`.
fn env_prefix(self) -> &'static str {
match self {
Engine::Forgejo => "FORGEJO__",
_ => "GITEA__",
}
}
fn expected_version(self) -> Option<&'static str> {
match self {
Engine::Gitea => Some(GITEA_VERSION),
Engine::Forgejo => Some(FORGEJO_VERSION),
Engine::Gunnar(_) | Engine::GunnarZnippy => None,
}
}
/// The gunnar columns, which share a binary, a provisioning path and a
/// URL space and differ only in the object backend underneath — and, since
/// 2026-08-11, in which arms of that backend were built.
fn is_gunnar_family(self) -> bool {
matches!(self, Engine::Gunnar(_) | Engine::GunnarZnippy)
}
}
// ═══════════════════════════════════════════════════════════════════════════
// THE COUNTING PROXY
// ═══════════════════════════════════════════════════════════════════════════
/// A byte-counting TCP relay: `client → proxy → engine`.
///
/// Byte-for-byte the same code in front of every engine, so its cost (one extra
/// hop, two pumping threads per connection, one memcpy of the stream) is present
/// in every column and cancels out of every ratio. Counting on only one side
/// would be worse than not counting at all.
///
/// It exists because the alternative — asking each server what it sent — is not
/// a comparison. gunnar can report its own byte counts; `git upload-pack` behind
/// gitea's Go handler cannot, and a metric derived from one side's self-report
/// would be a statement about the reporting.
struct Meter {
port: u16,
rx: Arc<AtomicU64>,
tx: Arc<AtomicU64>,
conns: Arc<AtomicU64>,
stop: Arc<AtomicBool>,
}
impl Meter {
/// Bind a loopback port and start relaying to `upstream`.
fn start(upstream: u16) -> Result<Meter> {
let listener = TcpListener::bind("127.0.0.1:0").context("binding the counting proxy")?;
let port = listener.local_addr()?.port();
let rx = Arc::new(AtomicU64::new(0));
let tx = Arc::new(AtomicU64::new(0));
let conns = Arc::new(AtomicU64::new(0));
let stop = Arc::new(AtomicBool::new(false));
{
let (rx, tx, conns, stop) = (
Arc::clone(&rx),
Arc::clone(&tx),
Arc::clone(&conns),
Arc::clone(&stop),
);
std::thread::spawn(move || {
for stream in listener.incoming() {
if stop.load(Ordering::Relaxed) {
break;
}
let Ok(client) = stream else { continue };
conns.fetch_add(1, Ordering::Relaxed);
let (rx, tx) = (Arc::clone(&rx), Arc::clone(&tx));
std::thread::spawn(move || {
let _ = relay(client, upstream, rx, tx);
});
}
});
}
Ok(Meter {
port,
rx,
tx,
conns,
stop,
})
}
/// Zero the counters. Called immediately before a timed operation, so the
/// seeding push and the warm-up round are not in the reading.
fn reset(&self) {
self.rx.store(0, Ordering::Relaxed);
self.tx.store(0, Ordering::Relaxed);
self.conns.store(0, Ordering::Relaxed);
}
/// `(server→client, client→server, connections)`.
fn read(&self) -> (u64, u64, u64) {
(
self.rx.load(Ordering::Relaxed),
self.tx.load(Ordering::Relaxed),
self.conns.load(Ordering::Relaxed),
)
}
}
impl Drop for Meter {
fn drop(&mut self) {
self.stop.store(true, Ordering::Relaxed);
// Poke the acceptor so it observes the flag and exits rather than
// sitting in `accept` for the life of the process.
let _ = TcpStream::connect(("127.0.0.1", self.port));
}
}
/// One relayed connection. Both directions are pumped and both are counted; the
/// half-closes are propagated, because git's smart-HTTP client relies on the
/// end of a request body being visible to the server.
fn relay(client: TcpStream, upstream: u16, rx: Arc<AtomicU64>, tx: Arc<AtomicU64>) -> Result<()> {
let server = TcpStream::connect(("127.0.0.1", upstream))?;
client.set_nodelay(true).ok();
server.set_nodelay(true).ok();
let c_read = client.try_clone()?;
let s_write = server.try_clone()?;
let up = std::thread::spawn(move || pump(c_read, s_write, tx));
pump(server, client, rx)?;
let _ = up.join();
Ok(())
}
fn pump(mut from: TcpStream, mut to: TcpStream, count: Arc<AtomicU64>) -> Result<()> {
let mut buf = vec![0u8; 64 * 1024];
loop {
let n = match from.read(&mut buf) {
Ok(0) | Err(_) => break,
Ok(n) => n,
};
if to.write_all(&buf[..n]).is_err() {
break;
}
count.fetch_add(n as u64, Ordering::Relaxed);
}
let _ = to.shutdown(Shutdown::Write);
Ok(())
}
// ═══════════════════════════════════════════════════════════════════════════
// THE CGROUP SAMPLER
// ═══════════════════════════════════════════════════════════════════════════
/// CPU seconds and memory of a whole container, out of cgroup v2.
///
/// The container-level counterpart of the `wait4`/`RUSAGE_BOTH` reading the
/// `vs_git_*` meter takes: it counts **every** process the engine forked, which
/// is the point. gitea serves a clone by forking `git upload-pack`, which forks
/// `git pack-objects`; a per-process reading would miss the work and a server
/// that wins wall clock by burning four cores would look free.
#[derive(Debug, Clone, Copy, Default)]
struct CgroupSample {
cpu_s: f64,
/// `memory.peak` through a **fresh** open: the cgroup's LIFETIME high-water
/// mark, container boot and repository seeding included. A real number about
/// a real cost, and not attributable to any one operation — for that, see
/// [`Cgroup::watch_peak`].
lifetime_peak_b: u64,
/// `memory.stat`'s `anon` — anonymous pages, the part of the charge the
/// kernel **cannot** reclaim under pressure.
anon_b: u64,
/// `memory.stat`'s `file` — page cache, which is reclaimable and therefore
/// expands to fill whatever ceiling the cgroup was given.
file_b: u64,
}
struct Cgroup {
dir: PathBuf,
/// `memory.max` in bytes, or `u64::MAX` for an unlimited cgroup. Kept so a
/// memory figure can be reported as a fraction of its own ceiling: a peak
/// pinned at 100% of the ceiling is a `--memory` flag being read back, not a
/// measurement of what the workload wanted.
ceiling_b: u64,
}
impl Cgroup {
/// Resolve the cgroup of a running container from its podman-reported path.
///
/// **A container whose cgroup cannot be found is red**, not a zero. A
/// silently absent cgroup would report `cpu_s: 0.0` for every round, which
/// is shaped exactly like an engine that did no work.
fn of(container: &str) -> Result<Cgroup> {
let out = Command::new("podman")
.args(["inspect", container, "--format", "{{.State.CgroupPath}}"])
.output()
.context("asking podman for a container's cgroup path")?;
ensure!(
out.status.success(),
"podman inspect {container} exited {:?}: {}",
out.status.code(),
String::from_utf8_lossy(&out.stderr).trim()
);
let rel = String::from_utf8_lossy(&out.stdout).trim().to_string();
ensure!(
!rel.is_empty(),
"podman reported an empty cgroup path for {container}; every CPU and memory figure \
this arm would print is a zero that looks exactly like an idle server"
);
let dir = PathBuf::from("/sys/fs/cgroup").join(rel.trim_start_matches('/'));
ensure!(
dir.join("cpu.stat").exists(),
"{} has no cpu.stat, so the container's CPU cannot be counted",
dir.display()
);
let ceiling_b = std::fs::read_to_string(dir.join("memory.max"))
.ok()
.map(|s| s.trim().to_string())
.map_or(u64::MAX, |s| s.parse::<u64>().unwrap_or(u64::MAX));
Ok(Cgroup { dir, ceiling_b })
}
/// One whole-number file out of the cgroup directory, or zero.
fn read_u64(&self, name: &str) -> u64 {
std::fs::read_to_string(self.dir.join(name))
.ok()
.and_then(|s| s.trim().parse::<u64>().ok())
.unwrap_or(0)
}
/// One field out of `memory.stat`, or zero.
///
/// The trailing-space match is load-bearing: a bare prefix test for `anon`
/// also matches `anon_thp`, which is a different and much smaller number.
fn stat_field(&self, field: &str) -> u64 {
std::fs::read_to_string(self.dir.join("memory.stat"))
.ok()
.and_then(|text| {
text.lines()
.find_map(|l| l.strip_prefix(field)?.strip_prefix(' '))
.and_then(|v| v.trim().parse::<u64>().ok())
})
.unwrap_or(0)
}
fn sample(&self) -> Result<CgroupSample> {
ensure!(
self.dir.as_os_str().len() > 1,
"a cgroup sample was taken before the container's cgroup was resolved; every CPU \
figure from here would be a zero that looks like an idle server"
);
let stat = std::fs::read_to_string(self.dir.join("cpu.stat"))
.with_context(|| format!("reading {}/cpu.stat", self.dir.display()))?;
let usec = stat
.lines()
.find_map(|l| l.strip_prefix("usage_usec "))
.and_then(|v| v.trim().parse::<f64>().ok())
.context("cpu.stat carries no usage_usec line")?;
Ok(CgroupSample {
cpu_s: usec / 1e6,
lifetime_peak_b: self.read_u64("memory.peak"),
anon_b: self.stat_field("anon"),
file_b: self.stat_field("file"),
})
}
/// Reset this cgroup's `memory.peak` and watch it across **one operation**.
///
/// # The fd is the whole mechanism
///
/// cgroup v2's `memory.peak` write handler (Linux ≥ 6.12) is a
/// **per-open-file** reset: the write registers a local watermark on *that*
/// `struct kernfs_open_file`, and only reads **through the same fd** see it.
/// A fresh `open()` still returns the global lifetime mark, and closing the
/// fd throws the local watermark away.
///
/// This is why this arm spent months asserting that `memory.peak` "cannot be
/// reset on this kernel — writing `0` returns success and changes nothing".
/// It changes plenty; just not for anyone who re-opens the file to read it
/// back, which is the natural thing to do and produces exactly the evidence
/// that talked us out of the fix. Measured on the laptop, kernel 7.0.0-29,
/// 2026-08-11: after the write, the same fd reads 24404 MB — the cgroup's
/// then-current charge — while a fresh open still reads the 39321 MB
/// lifetime mark. A subsequent 300 MB transient moved the same-fd number to
/// 24703 and it **stayed** there after the free, which is what makes it a
/// peak and not a gauge.
///
/// # Why a peak and not `rss_growth_mb`
///
/// `rss_growth_mb` is end-minus-start, so a spike allocated and freed inside
/// the window is invisible to it. Measured with the `fatseed` red switch's
/// probe: a 100 MB transient reports **0.0 MB** of growth. The reset peak
/// reports 101.9 MB.
fn watch_peak(&self) -> Result<PeakWatch> {
let path = self.dir.join("memory.peak");
let mut file = std::fs::OpenOptions::new()
.read(true)
.write(true)
.open(&path)
.with_context(|| {
format!(
"opening {} read-write to reset the operation's high-water mark. Without \
the reset every memory column would carry the container's LIFETIME peak — \
boot and repository seeding included — under a heading that names one \
operation.",
path.display()
)
})?;
// Read BEFORE the write, so the floor is not itself a product of the
// reset. This is the charge the operation inherited and did not cause.
let floor_b = self.read_u64("memory.current");
file.write_all(b"0").with_context(|| {
format!(
"writing to {} to reset the high-water mark. A kernel without the cgroup v2 \
`memory.peak` write handler (before 6.12) refuses this, and so does a cgroup \
the harness does not own.",
path.display()
)
})?;
let mut watch = PeakWatch { file, floor_b };
// ── The reset must be PROVED to have taken ─────────────────────────
// A silent no-op here is the worst outcome on the menu: every row would
// publish the container's lifetime peak under a per-operation heading
// and look exactly like a correct measurement, which is the defect this
// code exists to end rather than to reproduce. Immediately after a reset
// the watermark IS the cgroup's current charge, so a read-back well
// above `memory.current` can only be a stale lifetime mark.
const SLACK_B: u64 = 64 << 20;
let back = watch.read()?;
ensure!(
back <= floor_b.saturating_add(SLACK_B),
"resetting {} did not take: the same fd reads back {} MB against a memory.current \
of {} MB. The per-open-file reset semantics every per-operation memory column here \
depends on are not what this kernel does, and publishing the number anyway would \
attribute container boot and repository seeding to one git operation.",
path.display(),
back / 1048576,
floor_b / 1048576
);
Ok(watch)
}
}
/// A reset high-water mark on one open fd: what a **single operation** peaked at.
///
/// Held open for the whole timed region. Dropping it closes the fd, and closing
/// the fd is what discards the local watermark — so this must outlive the
/// operation it is measuring.
struct PeakWatch {
file: std::fs::File,
/// `memory.current` at the instant of the reset.
floor_b: u64,
}
impl PeakWatch {
/// The high-water mark since the reset, through the fd that did the reset.
///
/// The `rewind` is load-bearing. `memory.peak` is a seq_file: a read at a
/// non-zero offset returns the tail of the previous read instead of asking
/// the kernel again, so a second `read` without seeking back reports an
/// empty string rather than a fresh number.
fn read(&mut self) -> Result<u64> {
self.file
.rewind()
.context("rewinding the reset memory.peak fd before re-reading it")?;
let mut buf = String::new();
self.file
.read_to_string(&mut buf)
.context("reading the reset memory.peak")?;
buf.trim()
.parse::<u64>()
.with_context(|| format!("memory.peak read back as {buf:?}, which is not a number"))
}
}
/// A **sampled** high-water mark of the cgroup's anonymous memory.
///
/// # Why sample, when `memory.peak` is exact and free
///
/// Because `memory.peak` counts page cache, and page cache is reclaimable: it
/// expands to fill whatever ceiling the cgroup was given, so the number tracks
/// `--memory` rather than the workload. Measured on the laptop, 2026-08-11 — one
/// identical workload (a container writes then re-reads 2000 MB of file data,
/// allocating essentially no anonymous memory) under three ceilings:
///
/// | `--memory` | `memory.peak` | `memory.stat` anon |
/// |---|---|---|
/// | 512 MB | **512.0 MB** — 100% of the ceiling | 0.1 MB |
/// | 1024 MB | **1024.0 MB** — 100% of the ceiling | 0.1 MB |
/// | 3072 MB | 2043.2 MB | 0.1 MB |
///
/// That is `vs_forge_clone`'s 8192 MB, and note that **resetting the watermark
/// does not cure it** — the reset peak reads 512.0 and 1024.0 too. A reset fixes
/// *attribution*; it cannot help when the thing filling the mark is cache that
/// inflates to whatever room it is given. The `anon` column is flat across all
/// three, because anonymous pages are the ones the kernel cannot reclaim.
///
/// The kernel maintains no `anon` high-water mark, so the only way to get one is
/// to poll. This figure is therefore **sampled**, is named and documented as
/// sampled, and reports its own sample count beside it: a sampler that never ran
/// yields a zero peak, and a zero peak is shaped exactly like a free operation.
struct AnonWatch {
stop: Arc<AtomicBool>,
peak_b: Arc<AtomicU64>,
samples: Arc<AtomicU64>,
handle: Option<std::thread::JoinHandle<()>>,
floor_b: u64,
}
/// How often the anon sampler looks. One small read of `memory.stat`; at a push
/// measured in seconds this is a few hundred samples and no measurable load.
const ANON_SAMPLE_PERIOD: Duration = Duration::from_millis(5);
impl AnonWatch {
fn start(cgroup: &Cgroup) -> AnonWatch {
let floor_b = cgroup.stat_field("anon");
let stop = Arc::new(AtomicBool::new(false));
let peak_b = Arc::new(AtomicU64::new(floor_b));
let samples = Arc::new(AtomicU64::new(0));
let dir = cgroup.dir.clone();
let (s, p, n) = (stop.clone(), peak_b.clone(), samples.clone());
let handle = std::thread::spawn(move || {
let probe = Cgroup {
dir,
ceiling_b: u64::MAX,
};
while !s.load(Ordering::Relaxed) {
p.fetch_max(probe.stat_field("anon"), Ordering::Relaxed);
n.fetch_add(1, Ordering::Relaxed);
std::thread::sleep(ANON_SAMPLE_PERIOD);
}
// One last look after the stop flag, so an operation shorter than
// the sampling period is still sampled at its end rather than only
// at its start.
p.fetch_max(probe.stat_field("anon"), Ordering::Relaxed);
n.fetch_add(1, Ordering::Relaxed);
});
AnonWatch {
stop,
peak_b,
samples,
handle: Some(handle),
floor_b,
}
}
/// Stop sampling. Returns `(peak, floor, samples taken)`.
fn finish(mut self) -> (u64, u64, u64) {
self.stop.store(true, Ordering::Relaxed);
if let Some(h) = self.handle.take() {
let _ = h.join();
}
(
self.peak_b.load(Ordering::Relaxed),
self.floor_b,
self.samples.load(Ordering::Relaxed),
)
}
}
/// Everything the memory columns of one timed operation are built from.
///
/// Several numbers rather than one, because the single number this arm used to
/// publish answered a different question than the one its column heading asked.
/// They are all kept, and each is named for what it actually is.
#[derive(Debug, Clone, Copy, Default)]
struct MemReading {
/// `memory.peak` through a fresh open after the operation: the container's
/// **lifetime** high-water mark, boot and repository seeding included. Real,
/// worth knowing, and not attributable to the operation. Published as
/// `rss_lifetime_peak_mb` — which is what it was always measuring, under the
/// name `rss_peak_mb`.
lifetime_peak_b: u64,
/// The movement of that lifetime mark across the operation. Blind to a spike
/// allocated and freed inside the window. Published unchanged as
/// `rss_growth_mb`, because rows already exist against that name and it
/// still means exactly what it meant.
growth_b: u64,
/// `memory.peak` reset immediately before the operation and read after it
/// through the same fd: the most the whole cgroup was charged **at any
/// instant during the operation**. Exact, kernel-maintained, spike-visible.
/// Published as `rss_op_peak_mb`.
op_peak_b: u64,
/// `memory.current` at the reset — the charge the operation inherited.
op_floor_b: u64,
/// Sampled high-water mark of `memory.stat`'s `anon` across the operation:
/// the figure that does not echo the ceiling. Published as
/// `rss_op_anon_peak_mb`. See [`AnonWatch`] for why this one is sampled.
anon_peak_b: u64,
/// `anon` at the start of the operation.
anon_floor_b: u64,
/// How many times the anon sampler looked. Asserted on rather than trusted.
anon_samples: u64,
/// `memory.stat`'s `anon` and `file` at the END of the operation.
///
/// The split is carried because a single total cannot answer the question a
/// large memory figure always provokes — *is that the engine holding data,
/// or is it page cache the kernel will hand straight back?* Those have
/// opposite implications for whether a ceiling is a problem, and the arm was
/// previously unable to tell them apart at all.
end_anon_b: u64,
end_file_b: u64,
/// `memory.max`, so a peak can be priced against its own ceiling.
ceiling_b: u64,
}
impl MemReading {
/// Anonymous memory the operation **added** at its high-water mark.
fn anon_growth_b(&self) -> u64 {
self.anon_peak_b.saturating_sub(self.anon_floor_b)
}
/// Total charge the operation added at its high-water mark.
///
/// The spike-visible counterpart of `rss_growth_mb`: same "what did this
/// operation add" question, but measured against a peak rather than against
/// the end state, so memory that was allocated and released inside the
/// window still counts.
fn op_growth_b(&self) -> u64 {
self.op_peak_b.saturating_sub(self.op_floor_b)
}
/// The operation's total peak as a percentage of the cgroup's own ceiling.
///
/// Emitted rather than inferred, because a reader cannot otherwise tell
/// "this workload wanted 8192 MB" from "this cgroup was given 8192 MB and
/// the page cache took all of it". Near 100 means the latter.
fn ceiling_pct(&self) -> f64 {
if self.ceiling_b == 0 || self.ceiling_b == u64::MAX {
0.0
} else {
100.0 * self.op_peak_b as f64 / self.ceiling_b as f64
}
}
}
// ═══════════════════════════════════════════════════════════════════════════
// A RUNNING ENGINE
// ═══════════════════════════════════════════════════════════════════════════
/// One engine, up, provisioned, and metered.
struct Forge {
engine: Engine,
/// The draupnir handle. Dropping `Forge` stops and removes the container.
machine: draupnir::Machine,
boot: draupnir::container::ContainerBoot,
/// The counting proxy the client is pointed at.
meter: Meter,
/// The published host port, behind the proxy. Only the `noproxy` red switch
/// hands this to a client.
published: u16,
cgroup: Cgroup,
/// The version string the engine reported about itself.
version: String,
/// The `git` the engine has inside it (empty for gunnar, which has none).
inner_git: String,
/// The HTTP Basic password an authenticated operation presents: the account
/// password on the Go forges, the minted token on gunnar.
secret: String,
/// Whether [`Forge::down`] has already run.
stopped: bool,
/// What this engine was **actually** running under: the caps the kernel
/// applied, the pack settings read back out of the container, and the
/// runtime thread count podman records. Populated at the end of [`up`].
applied: parity::Applied,
/// The arm's scratch root and the gunnar binary, so [`Forge::gc`] can bring
/// the engine back up over the SAME store. gunnar's store lock is
/// cross-process and exclusive while `serve` runs (measured again on
/// 2026-08-04: `gunnar gc` against a live data root refuses with
/// "the store is locked by another writer"), so maintaining a gunnar
/// repository means stopping the server. That is an operational property
/// worth stating, and it is why this field exists.
root: PathBuf,
binary: PathBuf,
/// What the LAST `gunnar gc` on this engine reported about itself. Empty for
/// the Go forges, which have no equivalent telemetry.
last_gc: RepackTelemetry,
}
/// The fields `gunnar gc` prints about what it actually did.
///
/// Every one of these is READ off the repacker's own log line. A benchmark that
/// concludes "the encoder did not help" from an unchanged pack size cannot tell
/// that from "the encoder never ran", and those are opposite findings.
#[derive(Debug, Clone, Default)]
struct RepackTelemetry {
/// `true` when a new pack was installed.
rewrote: bool,
/// `true` when a re-encode was attempted and lost to the existing deltas.
declined: bool,
delta_fraction_before: f64,
delta_fraction_after: f64,
/// Bytes the re-encode produced, when it ran and lost.
recoded_bytes: f64,
/// Bytes copying the existing deltas produced — what was kept.
copy_bytes: f64,
secs: f64,
/// Whether a reachability bitmap is present AFTER the maintenance — the
/// state the rows measured next are served from. `1` yes, `0` no, **`-1`
/// this store's layout cannot be asked** (see [`Forge::bitmap_column`]);
/// the default below is `-1` and not `0` for that reason — a telemetry
/// struct that was never filled in must not read as a measured absence.
bitmap_after: f64,
}
/// Pull [`RepackTelemetry`] out of `gunnar gc`'s own output.
///
/// **Strips ANSI first.** The tracing formatter writes
/// `\x1b[3mrecoded_bytes\x1b[0m\x1b[2m=\x1b[0m2888804`, so `recoded_bytes=`
/// is not a contiguous string and a grep for it finds nothing — which reads
/// exactly like a field that was never emitted.
fn parse_repack_telemetry(stderr: &str, stdout: &str, secs: f64) -> RepackTelemetry {
// The same ESC-sequence strip the admission-log reader needs, shared rather
// than twinned — one place to be right about `\x1b[3m…=\x1b[0m`.
let clean = gunnar_bench::admission_log::strip_ansi(&format!("{stderr}{stdout}"));
let num = |key: &str| -> f64 {
clean
.split(&format!("{key}="))
.nth(1)
.and_then(|rest| {
let end = rest
.find(|c: char| !(c.is_ascii_digit() || c == '.' || c == '-'))
.unwrap_or(rest.len());
rest[..end].parse::<f64>().ok()
})
.unwrap_or(0.0)
};
let flag = |key: &str| -> bool {
clean
.split(&format!("{key}="))
.nth(1)
.map(|r| r.starts_with("true"))
.unwrap_or(false)
};
RepackTelemetry {
rewrote: flag("rewrote"),
declined: flag("reencode_declined"),
delta_fraction_before: num("delta_fraction_before"),
delta_fraction_after: num("delta_fraction_after"),
recoded_bytes: num("recoded_bytes"),
copy_bytes: num("copy_bytes"),
secs,
// Not measured here — `gc_inner` fills it in after the maintenance.
// `-1` ("unanswered") and not `0` ("measured, and there is none"),
// because a struct that never reached the filling-in step must not
// publish a fact it did not establish.
bitmap_after: -1.0,
}
}
impl Forge {
/// The URL stock `git` is given for `slug`, through the meter.
fn url(&self, slug: &str, auth: bool) -> String {
let port = if red("noproxy") && self.engine == Engine::Gitea {
// ── RED SWITCH: `noproxy` ──────────────────────────────────────
// Bypass the meter for one engine. Its wire columns collapse to
// zero while every other column stays perfectly plausible, which is
// the exact shape of a measurement apparatus that silently stopped
// measuring. `assert_counted` must catch it.
self.published
} else {
self.meter.port
};
let path = self.repo_path(slug);
if auth {
format!(
"http://{FORGE_USER}:{}@127.0.0.1:{port}/{path}",
self.secret
)
} else {
format!("http://127.0.0.1:{port}/{path}")
}
}
/// Where a repository lives in the engine's own URL space. The Go forges put
/// it under the owning user; gunnar under a namespace.
fn repo_path(&self, slug: &str) -> String {
if self.engine.is_gunnar_family() {
format!("bench/{slug}.git")
} else {
format!("{FORGE_USER}/{slug}.git")
}
}
/// Run a command inside the container through draupnir's exec, failing on a
/// non-zero exit with the command's own stderr.
///
/// **draupnir's `exec` does not fail on a non-zero exit code**, it returns
/// one — so this is where the check has to be. An `ExecOutcome` that is
/// discarded is a provisioning step that silently did nothing, and the
/// benchmark that follows it measures an empty repository.
fn exec(&self, argv: &[&str]) -> Result<String> {
use draupnir::container::ContainerControl;
let outcome = self
.boot
.exec(&self.machine, argv)
.map_err(|e| anyhow::anyhow!("exec {argv:?} in {}: {e}", self.engine.key()))?;
let stdout = outcome.stdout.join("\n");
let stderr = outcome.stderr.join("\n");
ensure!(
outcome.exit_code == Some(0),
"`{}` in the {} container exited {:?}\n stdout: {stdout}\n stderr: {stderr}",
argv.join(" "),
self.engine.key(),
outcome.exit_code
);
Ok(stdout)
}
}
impl Forge {
/// Stop and remove the container, explicitly.
///
/// **Not left to `Drop` alone.** The first `nostart` red run leaked two
/// containers (`Up 2 minutes` and `Exited (3)` still listed a minute after
/// the arm failed), so teardown is called at a named point and `Drop` is
/// only the backstop for a panic. A benchmark that leaves servers running
/// changes the next run's numbers.
fn down(&mut self) {
use draupnir::container::ContainerControl;
if !self.stopped {
self.stopped = true;
self.boot.stop(&self.machine);
// draupnir's `stop` swallows its errors (it returns `()`), so the
// removal is CONFIRMED rather than assumed: a container that is
// still there is a container the next arm will collide with.
//
// This used to confirm by shelling `podman rm -f -i`, which is not a
// confirmation at all — it is a second removal whose own failure was
// discarded with `let _`. `list_containers` asks instead, and says
// so when the answer is "still there".
match self.boot.list_containers(&self.machine.id) {
Ok(left) if left.is_empty() => {}
Ok(left) => eprintln!(
"bench: the {} container did not go away after stop ({left:?}); the next \
arm may collide with it",
self.engine.key()
),
Err(err) => eprintln!(
"bench: could not confirm the {} container was removed ({err})",
self.engine.key()
),
}
}
}
}
/// What one server's maintenance actually did, counted on both sides of it.
///
/// **A `gc` that silently no-ops is the single most likely failure of the
/// maintained-state arm**, and it would look exactly like a real result: the
/// label says `gc`'d, the numbers are the un-`gc`'d ones. So the store is read
/// before and after and the arm refuses a maintenance step that moved nothing.
///
/// 🔴 The reading used to be `read_dir(objects/pack)`, which a znippy repository
/// does not have, so both sides came back 0 and *"before AND after it would read
/// as a gc that changed nothing"* — the arm's own words, and it fired. It is
/// [`gunnar_bench::store_bytes`] now: the same counter the `real_repo_clone` arm
/// uses, which is what keeps the three arms' definition of "stored bytes" one
/// definition.
#[derive(Debug, Clone, Copy)]
struct GcReport {
before: gunnar_bench::store_bytes::StoreBytes,
after: gunnar_bench::store_bytes::StoreBytes,
/// **Did the engine actually receive the aggressive request this report was
/// taken under?**
///
/// `true` for every default pass (nothing was asked for) and for the Go
/// forges under `--aggressive`, which have the flag. `false` for gunnar
/// under `--aggressive`, which does NOT: the packfile engine was amputated
/// on 2026-08-10 and `GcArgs` is now `path` / `--prune` / `--prune-grace`
/// and nothing else, deliberately — "a flag that parses and is ignored is
/// worse than one that is not there"
/// (`crates/gunnar-server/src/admin.rs:615`).
///
/// The arm kept passing `--aggressive` to `gunnar gc` until 2026-08-13 and
/// clap rejected it with exit 2, which killed `gunnar.vs_forge_fetch_gc`
/// outright — measured on this box the same day. Dropping the flag is the
/// fix; dropping it SILENTLY would have been the disease, because gunnar's
/// `gcx` columns would then be its `gc` columns under a heading that says
/// "recompute every delta". So the answer travels on the report and lands
/// on the row as `<engine>_..._gcx_aggressive_available`.
aggressive_available: bool,
}
impl GcReport {
fn packs_before(&self) -> u64 {
self.before.containers
}
fn packs_after(&self) -> u64 {
self.after.containers
}
fn bytes_before(&self) -> u64 {
self.before.payload
}
fn bytes_after(&self) -> u64 {
self.after.payload
}
/// **Not `bytes_before != bytes_after`, and the difference is the whole
/// point of the type.**
///
/// `git gc` rewrites `objects/pack`, so on the git shape the payload and the
/// pack count both move. gunnar's `gc` seals and compacts *the archive* and
/// leaves `objects.pack` byte-for-byte identical — MEASURED on the laptop
/// 2026-08-10: payload `1 246 466 -> 1 246 466`, with a
/// `repository.g1.znippy` of `1 257 574` appearing beside it and `gunnar gc`
/// itself reporting `1 284 150 -> 1 257 574`. A `moved()` that only watched
/// the payload would call that a no-op and refuse a `gc` that did a great
/// deal of work. [`StoreBytes::moved`] watches the sealed generations and
/// the total as well.
fn moved(&self) -> bool {
self.before.moved(&self.after)
}
}
/// The columns that say **which definition** the `pack_b_*` figures were taken
/// under, and what moved that the payload figure cannot show.
///
/// Numbers and not prose, for the reason `H2hCorpus` is: of every string this
/// harness emits, only `name` reaches a `bench_runs` row, so a condition a
/// reader needs in order to reproduce or distrust a figure has to be a float.
///
/// * `*_store_shape` — 1 = git (`objects/pack/*.pack`), 2 = znippy
/// (`gunnar-git.znippy/objects.pack`). **A 0 in a `pack_b_*` column means
/// something completely different depending on this**, and until this column
/// existed there was no way to tell from the row.
/// * `*_sealed_b_after` — the sealed znippy generations a `gc` produced. Always
/// 0 on the git shape. This is where gunnar's maintenance shows up; the
/// payload figure beside it is expected NOT to move on that engine.
/// * `*_store_total_b_*` — every byte of the store, so the two engines' very
/// different index overheads are visible rather than folded into the
/// comparable number or hidden from it.
fn gc_shape_metrics(
engine: &str,
tag: &str,
r: &GcReport,
) -> Vec<(&'static str, serde_json::Value)> {
use gunnar_bench::store_bytes::StoreShape;
let shape = |s: StoreShape| match s {
StoreShape::Git => 1.0,
StoreShape::Znippy => 2.0,
};
vec![
(
leak(&format!("{engine}_{tag}_store_shape")),
json!(shape(r.after.shape)),
),
(
leak(&format!("{engine}_{tag}_sealed_b_before")),
json!(r.before.sealed as f64),
),
(
leak(&format!("{engine}_{tag}_sealed_b_after")),
json!(r.after.sealed as f64),
),
(
leak(&format!("{engine}_{tag}_store_total_b_before")),
json!(r.before.total as f64),
),
(
leak(&format!("{engine}_{tag}_store_total_b_after")),
json!(r.after.total as f64),
),
]
}
impl Forge {
/// Where a repository's bare git directory is, as the HOST sees it.
///
/// gunnar keeps a plain bare repository per repo under
/// `<data-root>/repos/<ns>/<name>.git`, bind-mounted from the arm's scratch
/// root, so the host can count its packs and `fsck` it without an exec. The
/// Go forges' stores are inside the container's user namespace and are
/// reached through [`Forge::exec`] instead.
fn host_repo_dir(&self, slug: &str) -> PathBuf {
self.root
.join(format!("{}-data", self.engine.key()))
.join("data")
.join("repos")
.join("bench")
.join(format!("{slug}.git"))
}
/// Every host pid inside this container's cgroup, the whole subtree.
///
/// podman puts the payload in a `container/` child of the scope, so the
/// scope's own `cgroup.procs` is empty and reading only that would pin
/// nothing while reporting success.
fn container_pids(&self) -> Result<Vec<i32>> {
fn walk(dir: &Path, out: &mut Vec<i32>) {
if let Ok(text) = std::fs::read_to_string(dir.join("cgroup.procs")) {
out.extend(text.lines().filter_map(|l| l.trim().parse::<i32>().ok()));
}
let Ok(entries) = std::fs::read_dir(dir) else {
return;
};
for e in entries.flatten() {
if e.file_type().map(|t| t.is_dir()).unwrap_or(false) {
walk(&e.path(), out);
}
}
}
let mut pids = Vec::new();
walk(&self.cgroup.dir, &mut pids);
pids.sort_unstable();
pids.dedup();
ensure!(
!pids.is_empty(),
"{}: no process in the container's cgroup at {} — an affinity that is set on nothing \
reports success and changes nothing",
self.engine.key(),
self.cgroup.dir.display()
);
Ok(pids)
}
/// **Confine this engine to a set of CPUs, and read the confinement back.**
///
/// A cgroup `cpu.max` quota of 8 across 32 CPUs is not the same allocation
/// as 8 dedicated CPUs, and the difference is not a detail: a request gunnar
/// serves is three threads that synchronise continuously, and under quota
/// without a cpuset the scheduler has ~3N runnable threads and 32 places to
/// put them, so every hand-off crosses a core and a cross-core hand-off is a
/// futex. Which of the two a benchmark used is therefore part of the
/// measurement, and this arm reports both rather than picking one.
///
/// `taskset` and not `cpuset.cpus`: the cpuset controller is NOT delegated
/// to the rootless user's cgroup tree on this box — `podman update
/// --cpuset-cpus` fails with "controller `cpuset` is not available" and no
/// `cpuset.cpus` file exists (measured, the desktop, 2026-08-04). Affinity needs no
/// delegation. `podman unshare` rather than a bare `taskset` because the
/// container's processes belong to a mapped subuid, and rather than an
/// in-container `taskset` because gitea's and Forgejo's Alpine images do not
/// ship one and gunnar's image does — one mechanism for all three, which is
/// the only way the two columns mean the same thing.
///
/// Returns the affinity the kernel actually applied, read out of
/// `/proc/<pid>/status`. An affinity that was asked for and not applied
/// looks exactly like one that was.
fn pin_cpus(&self, range: &str) -> Result<String> {
let pids = self.container_pids()?;
for pid in &pids {
let out = Command::new("podman")
.args(["unshare", "taskset", "-acp", range, &pid.to_string()])
.output()
.context("podman unshare taskset")?;
ensure!(
out.status.success(),
"{}: pinning pid {pid} to {range} exited {:?}: {}",
self.engine.key(),
out.status.code(),
String::from_utf8_lossy(&out.stderr).trim()
);
}
let mut applied: Option<String> = None;
for pid in &pids {
let status = std::fs::read_to_string(format!("/proc/{pid}/status"))
.with_context(|| format!("reading /proc/{pid}/status back"))?;
let list = status
.lines()
.find_map(|l| l.strip_prefix("Cpus_allowed_list:"))
.map(|v| v.trim().to_string())
.context("no Cpus_allowed_list in /proc/<pid>/status")?;
ensure!(
list == range,
"{}: pid {pid} was pinned to {range} and reports {list} — an affinity that did \
not apply reads exactly like one that did, and this column would then be the \
unpinned column under a name that says otherwise",
self.engine.key()
);
applied = Some(list);
}
applied.context("no pid reported its affinity")
}
/// **Drain the container's log and read the admission gate's refusal
/// counters out of it**.
///
/// `drain_logs_split` *empties* the draupnir buffer (it is a drain, not a
/// snapshot — see the note in [`up`]), so calling this once at the end of a
/// batch reads exactly the lines that batch produced: the previous batch's
/// call took everything before it. The counters are the gate's own —
/// `refused_total` and `peak_in_flight` off the `*.transfer.refused` lines,
/// which are `gunnar_ssh::AdmissionStats::refused()` and `peak_in_flight` —
/// parsed by [`gunnar_bench::admission_log`], the same reader its `#[test]`
/// proves against a live gate.
///
/// Only gunnar has an admission gate; the Go forges never write these lines,
/// so this reads `(0, 0)` for them by construction — which is the honest
/// comparison point, not a missing measurement.
fn drain_admission(&self) -> gunnar_bench::admission_log::AdmissionRefusals {
use draupnir::container::ContainerControl;
let (mut out, mut err) = self.boot.drain_logs_split(&self.machine);
// Both streams: the tracing WARN goes to stderr, but a formatter or a
// redirect could put it on stdout, and a reader that watched only one
// would silently read 0 the day it moved — the exact blindness this
// column exists to remove.
out.append(&mut err);
gunnar_bench::admission_log::scan_refusals(&out)
}
/// Run a shell command inside a Go forge's container **as the account that
/// owns its store**.
///
/// draupnir's exec lands as root. git refuses to operate on a repository
/// owned by somebody else, and a repack run as root would leave root-owned
/// pack files behind in a store the forge serves as `git` — so the command
/// is dropped to that account rather than the ownership check silenced.
fn exec_as_git(&self, command: &str) -> Result<String> {
self.exec(&["su", "-s", "/bin/sh", "git", "-c", command])
}
/// The path the same repository has INSIDE a gitea or Forgejo container.
fn container_repo_dir(&self, slug: &str) -> String {
format!("/data/git/repositories/{FORGE_USER}/{slug}.git")
}
/// **Does this engine's copy of `slug` carry a reachability bitmap?**
///
/// It matters because it decides which code path is being measured on BOTH
/// sides. `git clone --bare` writes no `.bitmap`, and `h2h_carve` builds
/// the divergent fixture with exactly that — so the fetch arm's corpus had
/// none, and neither did any server seeded from it, because `index-pack` on
/// the receiving end of a push does not write one either. A fetch measured
/// against a bitmap-less server is a measurement of a configuration nobody
/// runs: the fetch-bytes lane isolated it on 2026-08-04 and found stock
/// git's own pack grows 292 336 -> 341 475 bytes when the server it is
/// talking to has the bitmap a repacked repository actually has.
///
/// So it is recorded per engine per repository rather than assumed, and it
/// goes on the row. An arm whose row says `bitmap = 0` is not wrong, it is
/// measuring the state a push leaves — but it is not the shipping path and
/// the row has to say which.
/// `Some(true)` / `Some(false)` when the question can be asked of this
/// store, and **`None` when it cannot** — which is not the same as "no".
///
/// 🔴 The same defect the pack counter had, on the same day, one metric
/// over. This looked for `objects/pack/*.bitmap`, so a znippy repository —
/// which has no `objects/` directory and keeps its reachability bitmaps as
/// a reserved Arrow section *inside* a sealed `repository*.znippy` — always
/// answered `false`, and `false` on this column means "measured, and there
/// is none", which is a claim about the code path the row beside it
/// measures. `bench/` cannot read an Arrow section (it links no engine, see
/// `gunnar_bench::engine`), so the honest answer is that it does not know,
/// and the row says `-1` rather than `0`.
fn has_bitmap(&self, slug: &str) -> Result<Option<bool>> {
if self.engine.is_gitea_family() {
let dir = self.container_repo_dir(slug);
let out = self.exec(&[
"sh",
"-c",
&format!(
"for f in {dir}/objects/pack/*.bitmap; do [ -e \"$f\" ] && echo yes; done"
),
])?;
return Ok(Some(out.contains("yes")));
}
let repo = self.host_repo_dir(slug);
if gunnar_bench::store_bytes::of(&repo)?.shape
== gunnar_bench::store_bytes::StoreShape::Znippy
{
return Ok(None);
}
let dir = repo.join("objects").join("pack");
let Ok(entries) = std::fs::read_dir(&dir) else {
return Ok(Some(false));
};
Ok(Some(entries.flatten().any(|e| {
e.path().extension().and_then(|x| x.to_str()) == Some("bitmap")
})))
}
/// [`Forge::has_bitmap`] as the number that reaches a row: `1` yes, `0` no,
/// **`-1` unanswerable on this store shape**. Three values, because two
/// cannot carry three answers and the third is the one that used to be
/// silently folded into "no".
fn bitmap_column(&self, slug: &str) -> f64 {
match self.has_bitmap(slug) {
Ok(Some(true)) => 1.0,
Ok(Some(false)) => 0.0,
_ => -1.0,
}
}
/// **What one repository's object store holds**, whatever shape it is.
///
/// 🔴 This was `(pack files, total pack bytes)` off `objects/pack`, and it
/// was blind on exactly one of the three engines it is asked about. gunnar
/// on its default engine keeps its bytes in
/// `gunnar-git.znippy/objects.pack` and writes no `objects/` directory at
/// all, so the counter came back `(0, 0)` — the reading the floor in
/// [`Forge::gc_inner`] refuses, and rightly: *"before AND after it would
/// read as a gc that changed nothing"*. It is
/// [`gunnar_bench::store_bytes`] now, shared with `gunnar.real_repo_clone`,
/// so all three arms measure one quantity under one definition.
///
/// The two collectors below are the same classifier over two ways of
/// listing files, and that is deliberate: gitea's and Forgejo's stores are
/// inside a container's user namespace and gunnar's is bind-mounted to the
/// host, and two hand-written counters is two chances for one of them to
/// mean something slightly different. This function once shipped exactly
/// that bug — the first version used `find … -printf '%s\n'`, these are
/// Alpine images, busybox `find` has no `-printf`, the pipeline's exit
/// status was `cat`'s and therefore 0, and both Go forges reported 0 packs
/// and 0 bytes on both sides of their own `gc`. The listing command is
/// POSIX for that reason and its parser treats a malformed line as an
/// error rather than skipping it.
fn store_bytes(&self, slug: &str) -> Result<gunnar_bench::store_bytes::StoreBytes> {
if self.engine.is_gitea_family() {
let dir = self.container_repo_dir(slug);
let out = self.exec(&[
"sh",
"-c",
&gunnar_bench::store_bytes::listing_command(&dir),
])?;
let files = gunnar_bench::store_bytes::parse_listing(&out).with_context(|| {
format!("listing {dir} inside the {} container", self.engine.key())
})?;
return gunnar_bench::store_bytes::classify(
&format!("{}:{dir}", self.engine.key()),
&files,
);
}
gunnar_bench::store_bytes::of(&self.host_repo_dir(slug))
}
/// `git fsck --full --strict` on one repository — **on what the engine
/// SERVES, not on how it stores it.**
///
/// Maintenance that corrupts a store while making it smaller would be the
/// worst possible way to win this arm, and nothing else here would see it.
///
/// # Why this stopped being `fsck <the store directory>`
///
/// It was, and on znippy it could only ever fail:
///
/// ```text
/// fsck of diverge on znippy_fast: git [-C .../repos/bench/diverge.git
/// fsck --full --strict] exited 128: fatal: not a git repository
/// ```
///
/// A znippy repository is an Arrow-IPC archive. There is no `objects/`, no
/// `refs/`, nothing for `git fsck` to walk — the check was asking the
/// filesystem a question only the gix arm can answer. It cost **four**
/// separate measurements on 2026-08-11: `vs_forge_fetch` passed every time
/// and `vs_forge_fetch_gc` shares its name prefix, so `nornir bench run`
/// discarded the whole receipt each time and the passing arm's numbers were
/// never persisted.
///
/// The same class of defect this file already fixed once for `GcReport`,
/// whose reading *"used to be `read_dir(objects/pack)`, which a znippy
/// repository does not have, so both sides came back 0"*.
///
/// # What it asks instead
///
/// It clones the repository out of the **running server** and fscks the
/// clone. That is engine-neutral by construction — every engine has to
/// produce a real packfile on the wire, whatever it keeps on disk — and it
/// is a *stronger* check than the original: a store that fscks clean but
/// cannot serve a valid pack now fails, and that is exactly the shape of
/// the two silent under-serving defects found in this engine today.
///
/// The server is up at this point: [`Forge::gc`] stops it to take the store
/// lock and `up_with(…, keep_data: true)` has already brought it back.
fn fsck(&self, slug: &str) -> Result<()> {
if self.engine.is_gitea_family() {
self.exec_as_git(&format!(
"git -C {} fsck --full --strict",
self.container_repo_dir(slug)
))
.with_context(|| format!("fsck of {slug} on {}", self.engine.key()))?;
return Ok(());
}
// A bare mirror, so tags and every branch come across and the fsck sees
// the whole ref namespace rather than one branch's reachable set.
let into = self
.root
.join(format!("fsck-{}-{slug}", self.engine.key()))
.join("clone.git");
let _ = std::fs::remove_dir_all(&into);
if let Some(parent) = into.parent() {
std::fs::create_dir_all(parent)?;
}
let url = self.url(slug, true);
let out = Command::new("git")
.args(["clone", "--quiet", "--bare", "--mirror", &url])
.arg(&into)
.env("GIT_TERMINAL_PROMPT", "0")
.output()
.with_context(|| format!("cloning {slug} out of {} to fsck it", self.engine.key()))?;
ensure!(
out.status.success(),
"{}: could not clone {slug} back to fsck it ({:?}): {}. The store may be intact and \
unservable — which is the failure this check exists to catch, not a reason to skip \
it.",
self.engine.key(),
out.status.code(),
String::from_utf8_lossy(&out.stderr).trim()
);
h2h_git(
&["-C", &into.to_string_lossy(), "fsck", "--full", "--strict"],
None,
)
.with_context(|| format!("fsck of the clone {} served for {slug}", self.engine.key()))?;
let _ = std::fs::remove_dir_all(into.parent().unwrap_or(&into));
Ok(())
}
/// **Maintain this engine's copy of `slugs`, the way this engine does it.**
///
/// gitea and Forgejo get `git gc` inside their own container, which is what
/// their own cron runs. gunnar gets `gunnar gc`, which cannot run against a
/// live data root — its store lock is cross-process and exclusive while
/// `serve` is up — so the server is stopped, the repositories are
/// maintained, and the server is brought back up over the same store. That
/// stop is not an artefact of this harness; it is what maintaining a gunnar
/// repository costs today, and it is recorded in the conditions line.
///
/// Returns what moved, per slug, so the caller can refuse a no-op.
fn gc(&mut self, slugs: &[&str]) -> Result<BTreeMap<String, GcReport>> {
self.gc_with(slugs, false)
}
/// [`Forge::gc`], optionally the expensive half.
///
/// `aggressive` is `git gc --aggressive` on the Go forges and `gunnar gc
/// --aggressive` on gunnar — the same name on both sides because it is the
/// same idea: recompute every delta instead of copying the ones that are
/// there. It exists because the DEFAULT `gc` correctly declines to do
/// anything on these fixtures — they are `repack -adfq`'d before they are
/// pushed, so they are already well deltified, `DeltaPolicy::Auto`
/// re-encodes only below 20% and `gc` is monotone so a re-encode that does
/// not shrink the pack is not installed. Measuring only the default `gc`
/// therefore measures an engine declining, on all three, and never reaches
/// the encoder at all.
fn gc_with(&mut self, slugs: &[&str], aggressive: bool) -> Result<BTreeMap<String, GcReport>> {
self.gc_ordered(slugs, aggressive, None)
}
/// [`Forge::gc_with`] under a named delta ordering, capturing what the
/// repacker said it did.
///
/// **The policy that fired is READ, not inferred from the output size.**
/// `gunnar gc` at `RUST_LOG=info` emits `delta_fraction_before`,
/// `delta_fraction_after`, `reencode_declined`, and — when a re-encode was
/// attempted and lost — `recoded_bytes` and `copy_bytes`. A pack that came
/// out the same size can mean the encoder never ran, or that it ran and was
/// beaten; those are opposite findings and only these fields tell them
/// apart. Measured on the `diverge` fixture, 2026-08-04: `Auto` declines
/// because the pack is already 78.8% deltified, and `--aggressive` runs the
/// encoder and loses 1.687x to 3.589x depending on the ordering.
///
/// The ANSI escapes have to be stripped first: the fields come out as
/// `\x1b[3mrecoded_bytes\x1b[0m\x1b[2m=\x1b[0m2888804`, so a grep for
/// `recoded_bytes=` finds nothing and reads exactly like a field that was
/// never emitted. That cost one round of this probe.
fn gc_ordered(
&mut self,
slugs: &[&str],
aggressive: bool,
order: Option<&str>,
) -> Result<BTreeMap<String, GcReport>> {
self.gc_inner(slugs, aggressive, order)
}
fn gc_inner(
&mut self,
slugs: &[&str],
aggressive: bool,
order: Option<&str>,
) -> Result<BTreeMap<String, GcReport>> {
let mut out: BTreeMap<String, GcReport> = BTreeMap::new();
let mut before: BTreeMap<String, gunnar_bench::store_bytes::StoreBytes> = BTreeMap::new();
let mut refs_before: BTreeMap<String, Vec<(String, String)>> = BTreeMap::new();
for slug in slugs {
let stats = self.store_bytes(slug)?;
// A repository that has just been seeded by a push and is being
// served has a pack. Zero here is never the repository, it is the
// instrument looking somewhere else — and a zero on both sides of
// the maintenance is indistinguishable from a `gc` that did
// nothing, which is the one reading this arm must not get wrong.
//
// The counter is store-shaped now, so this floor has changed
// meaning: it used to catch "the instrument is pointed at
// `objects/pack` and this engine has none", which is what it
// actually fired on, and the counter refuses that case itself with
// the directory listing attached. What is left for the floor is the
// reading it was written for — a repository that really is empty.
ensure!(
stats.containers > 0 && stats.payload > 0,
"{}/{slug}: the store counter read {} container(s) and {} payload byte(s) off a \
repository that has been seeded and is being served. Before AND after, that \
reads as a gc that changed nothing.\n {}",
self.engine.key(),
stats.containers,
stats.payload,
stats.describe()
);
before.insert((*slug).to_string(), stats);
refs_before.insert((*slug).to_string(), ls_remote_refs(self, slug)?);
}
// Whether the engine this handle drives can be ASKED for the expensive
// pass at all. Only meaningful when one was requested; a default pass
// asks for nothing and every engine can do that.
let aggressive_available = !aggressive || self.engine.is_gitea_family();
if self.engine.is_gitea_family() {
for slug in slugs {
// `--aggressive` when the caller asks for it, and then on
// every engine — comparing a hand-tuned repack on one side
// against a default one on the other is a measurement of the
// flags.
//
// As the `git` user, not as root. draupnir's exec lands as
// root, the store belongs to uid 1000 inside the container's
// namespace, and git 2.49 refuses a repository it does not own
// — "detected dubious ownership", measured 2026-08-04. Adding
// `safe.directory` instead would have run the repack as root
// and left root-owned pack files in a store the forge then
// serves as `git`, which is a slower way to the same failure.
self.exec_as_git(&format!(
"git -C {} gc --quiet{}",
self.container_repo_dir(slug),
if aggressive { " --aggressive" } else { "" }
))
.with_context(|| format!("git gc of {slug} on {}", self.engine.key()))?;
}
} else {
let data = self
.root
.join(format!("{}-data", self.engine.key()))
.join("data");
// The lock. Stop first, and confirm the container is gone before
// taking it — a `gunnar gc` that races the shutdown fails with the
// lock message and would look like a broken harness rather than
// the ordering mistake it is.
self.down();
for slug in slugs {
let mut cmd = Command::new(&self.binary);
cmd.env("RUST_LOG", "info");
cmd.arg("--data-root").arg(&data).arg("gc");
// ── NO `--aggressive`, NO `--delta-order`, AND THAT IS THE
// MEASUREMENT ───────────────────────────────────────────
//
// Both flags were passed here and both are gone from the verb.
// `gunnar gc` is `path` / `--prune` / `--prune-grace`
// (`gunnar-common/src/cli.rs`, `GcArgs`); window, depth, delta
// policy and delta order went out with the packfile engine on
// 2026-08-10, because the archive engine's `maintain` takes
// none of them. clap answers an unknown flag with exit 2, so
// this arm did not measure a worse gunnar — it did not run at
// all: `gunnar.vs_forge_fetch_gc ✗ failed: … unexpected
// argument '--aggressive' found`, on every invocation from the
// amputation until 2026-08-13.
//
// So the request cannot be forwarded, and the honest thing is
// not to pretend it was. It is recorded on the report and
// emitted as a column: gunnar has ONE maintenance verb, and its
// `gcx` figures are that one verb, not a second, expensive one.
//
// `--delta-order` is NOT treated the same way, and the asymmetry
// is deliberate. `--aggressive` was one knob on an arm whose
// subject is the fetch; dropping it costs that arm a column and
// it says so. An ORDERING is the entire subject of
// `gunnar.vs_forge_gc_orders`, and the Go forges have no
// ordering flag either — so silently ignoring it would run one
// identical `gc` four times and emit four columns labelled
// `size` / `nh1` / `nh2` / `pw` that differ in nothing but their
// name. That is a full table of numbers about a flag that does
// not exist, which is worse than the clap error it replaces.
ensure!(
order.is_none(),
"gunnar gc has no --delta-order since the packfile engine was amputated \
(2026-08-10): `GcArgs` is path/--prune/--prune-grace and the archive \
engine's `maintain` takes no window, depth, policy or ordering. \
`gunnar.vs_forge_gc_orders` measures exactly that flag, so its subject is \
gone — the arm has to be retired or the ordering restored, and reporting \
four identical columns under four ordering names is neither. Requested: \
{:?}",
order
);
let started = Instant::now();
let done = cmd
.arg(format!("bench/{slug}"))
.output()
.context("running gunnar gc")?;
let secs = started.elapsed().as_secs_f64();
self.last_gc = parse_repack_telemetry(
&String::from_utf8_lossy(&done.stderr),
&String::from_utf8_lossy(&done.stdout),
secs,
);
ensure!(
done.status.success(),
"gunnar gc bench/{slug} exited {:?}: {}{}",
done.status.code(),
String::from_utf8_lossy(&done.stdout).trim(),
String::from_utf8_lossy(&done.stderr).trim()
);
}
let engine = self.engine;
let (root, binary) = (self.root.clone(), self.binary.clone());
// The restart replaces the whole handle, and the telemetry the gc
// just produced lives on it. Carried across explicitly: losing it
// here would leave every ordering reporting the default's zeroes.
let telemetry = self.last_gc.clone();
let fresh = up_with(engine, &root, &binary, true)?;
*self = fresh;
self.last_gc = telemetry;
}
for slug in slugs {
// The bitmap AFTER the maintenance, which is the state the rows
// that follow are measured against. `with_engines_mut` records it
// once after seeding, and that is the right moment for an arm that
// does no maintenance and the WRONG one for an arm that does:
// measured 2026-08-04, a push leaves gunnar's store with `idx` and
// `pack` only and `gunnar gc` adds the `.bitmap` even on the
// "already consolidated, nothing to do" path. So the gc arms record
// it again here, under their own key.
self.last_gc.bitmap_after = self.bitmap_column(slug);
let after = self.store_bytes(slug)?;
let b = before[*slug];
ensure!(
b.shape == after.shape,
"{}/{slug}: the store was a {} store before maintenance and a {} store after. \
A before/after across two layouts is two measurements, not one.\n before: \
{}\n after: {}",
self.engine.key(),
b.shape,
after.shape,
b.describe(),
after.describe()
);
out.insert(
(*slug).to_string(),
GcReport {
before: b,
after,
aggressive_available,
},
);
// The maintenance must not have changed WHAT is served. `fsck`
// catches a corrupted store; this catches an intact store that is
// no longer the same repository — a pruned ref, a dropped tag —
// which `fsck` is perfectly happy with and which would make the
// measured fetch a fetch of something else.
let now = ls_remote_refs(self, slug)?;
let was = &refs_before[*slug];
ensure!(
&now == was,
"{}/{slug}: maintenance changed the advertisement from {} ref(s) to {} ref(s) — an intact store that is no longer the same repository, which `fsck` passes and which makes every byte below a measurement of a different fetch",
self.engine.key(),
was.len(),
now.len()
);
self.fsck(slug)?;
}
Ok(out)
}
}
impl Drop for Forge {
fn drop(&mut self) {
self.down();
}
}
/// Delete a directory that may be owned by a container's mapped uid, and
/// **check it is gone**.
///
/// `std::fs::remove_dir_all` cannot touch what gitea's entrypoint created: the
/// files belong to uid 1000 *inside the container's user namespace*, which is a
/// subuid on the host. Discarding that failure with `let _ =` is how a run
/// reused the previous run's sqlite database and failed with `CreateUser: user
/// already exists` — a stale state that, one branch over, would have been a
/// pre-seeded repository and a benchmark of nothing.
fn clear_dir(dir: &Path) -> Result<()> {
if dir.exists() {
let _ = std::fs::remove_dir_all(dir);
}
if dir.exists() {
let out = Command::new("podman")
.args(["unshare", "rm", "-rf"])
.arg(dir)
.output()
.context("podman unshare rm on a container-owned directory")?;
ensure!(
out.status.success(),
"podman unshare rm -rf {} exited {:?}: {}",
dir.display(),
out.status.code(),
String::from_utf8_lossy(&out.stderr).trim()
);
}
ensure!(
!dir.exists(),
"{} survived every attempt to remove it; an engine started on top of the previous run's \
state is measuring that state and not this run's fixture",
dir.display()
);
Ok(())
}
/// Remove containers this suite left behind — **only the ones whose owning
/// process is gone.**
///
/// Every container is named `…-forge-<engine>-<pid>` after the process that
/// booted it, and that suffix is load-bearing rather than decorative. A sweep
/// that removed every `draupnir-gunnar-forge-` container would remove another
/// another job's live ones, and on 2026-08-04 it did: two lanes ran these arms at the
/// same time out of two worktrees, each swept the other's servers out from
/// under it, and both runs died mid-arm. The in-process mutex serialises arms
/// inside one binary and says nothing about a second binary.
///
/// So a container is removed only when `/proc/<pid>` is gone, or when the pid is
/// this process's own (a leftover from earlier in this run).
///
/// **Through draupnir, not `podman`** — since 2026-08-11.
///
/// This function used to say *"draupnir has no list-containers call, so this is
/// `podman` directly — the boot path is what has to be zero-shell, not the
/// janitor"*. The first half was true and is now false: `list_containers` and
/// `container_env` were added to `draupnir::container::ContainerControl` for
/// exactly these two call sites. The second half was the weaker argument. A
/// janitor that shells out reintroduces, in the sweep, precisely the failures
/// the boot path refuses — `podman` not on `PATH`, a different engine, a
/// permission error — and it reintroduces them **silently**, because a failed
/// `podman ps` and a genuinely empty box both produce empty stdout and this
/// function's `let Ok(out) = out else { return }` treated them identically. A
/// sweep that cannot look is not a sweep that found nothing.
///
/// The name filter is draupnir's, and it is a real prefix match. podman's own
/// `--filter name=` is a **substring** match, so the old spelling would also
/// have matched `x-draupnir-gunnarforge-…`; nothing was named that, so it never
/// bit here, but it is the same class of bug as the 2026-08-04 collision above.
fn sweep_stale() {
use draupnir::container::{ContainerBoot, ContainerControl};
use draupnir::{Kind, Machine, PowerState};
let boot = ContainerBoot::new();
// An unreachable socket is an error, not an empty list — so it is logged
// rather than read as "nothing to sweep".
let found = match boot.list_containers("draupnir-gunnarforge-") {
Ok(found) => found,
Err(err) => {
eprintln!("bench: could not list containers to sweep ({err}); leaving them alone");
return;
}
};
let me = std::process::id();
for (name, _) in found {
let owner: Option<u32> = name.rsplit('-').next().and_then(|p| p.parse().ok());
let mine = owner == Some(me);
// No parseable pid means it predates this naming and cannot belong to a
// live run of this code; a pid with no `/proc` entry is a dead owner.
let orphaned = owner.is_none_or(|pid| !Path::new(&format!("/proc/{pid}")).exists());
if !(mine || orphaned) {
continue;
}
// `stop` is stop-then-remove and idempotent. The handle is rebuilt from
// the name because a sweep by definition has no `Machine` — the process
// that booted it is gone.
boot.stop(&Machine {
id: name.clone(),
spec_name: name,
kind: Kind::Container,
power: PowerState::On,
});
}
}
/// The engine-specific half of bringing a container up.
struct UpPlan {
env: BTreeMap<String, String>,
cmd: Vec<String>,
binds: Vec<String>,
/// Secret for authenticated HTTP. Minted before the container starts for
/// gunnar (its store takes a cross-process exclusive lock, so the control
/// plane cannot be written while the server holds it) and after, by the
/// admin CLI, for the Go forges.
secret: Option<String>,
}
/// Start an engine, wait for it to actually serve, and check which one it is.
///
/// Everything that can silently produce a plausible zero is a hard error here:
/// a container that exited, a health endpoint that never answered, a cgroup that
/// cannot be found, a version that is not the pinned one.
fn up(engine: Engine, root: &Path, gunnar_bin: &Path) -> Result<Forge> {
up_with(engine, root, gunnar_bin, false)
}
/// [`up`], optionally over a store that is already there.
///
/// `keep_data` exists for exactly one caller: [`Forge::gc`], which has to stop
/// gunnar to take its lock and then bring it back up over the repositories it
/// just maintained. Clearing the directory there would turn a maintenance step
/// into an empty server, and an empty server is very fast.
fn up_with(engine: Engine, root: &Path, gunnar_bin: &Path, keep_data: bool) -> Result<Forge> {
use draupnir::container::{ContainerBoot, ContainerControl, ContainerState, Readiness};
use draupnir::{BootSpec, NetMode};
let data = root.join(format!("{}-data", engine.key()));
if !keep_data {
clear_dir(&data)?;
}
std::fs::create_dir_all(&data)?;
let published = gunnar_installtest::free_port().map_err(|e| anyhow::anyhow!(e))?;
let plan = plan_for(engine, &data, gunnar_bin, gunnar_installtest::free_port)?;
let _ = keep_data;
let alloc = parity::allocation();
// `gunnarforge`, deliberately not `gunnar-forge`. A concurrent lane running
// an older build of this file sweeps every container matching
// `draupnir-gunnar-forge-` regardless of who owns it, and killed two of this
// suite's runs mid-arm on 2026-08-04 ("Recv failure: Connection reset by
// peer" at seeding; "can only create exec sessions on running containers").
// The sweep here is pid-aware and spares theirs; this name is what spares
// mine from theirs. It stops mattering once both sides carry the pid check,
// and costs nothing until then.
let name = format!("gunnarforge-{}-{}", engine.key(), std::process::id());
let mut spec = BootSpec::container(&name, engine.image())
.with_port_map(published, IN_CONTAINER_PORT)
// `Bridge`, not `NetMode::None`: the whole arm is a client on the host
// talking to a server in the container, and `--network none` would make
// the published port unreachable. draupnir's `NetMode` cannot express
// the forward itself — `PortMap` does, and that is the container
// backend's path; a KVM backend would need tunnr's `hostfwd` instead.
.with_net(NetMode::Default)
.with_cmd(plan.cmd.clone());
for (k, v) in &plan.env {
spec = spec.with_env(k, v);
}
// ── THE ALLOCATION ─────────────────────────────────────────────────────
// The same CPU quota and the same memory limit on every engine, and the
// thread counts set explicitly on top of them: a `--cpus 8` container still
// reports 32 to `nproc`, so a quota on its own gives an engine thirty-two
// threads inside eight cores' worth of budget rather than eight threads.
// See `parity.rs`; all of it is read back below, none of it is assumed.
//
// ── RED SWITCH: `nocap` ────────────────────────────────────────────────
// Boot one engine with no caps at all. Every other column stays plausible
// while that engine quietly has the whole 32-core box; `assert_quota` must
// refuse the `cpu.max = max` it then reads.
if !(red("nocap") && engine == Engine::Forgejo) {
spec = spec.with_cpus(alloc.cpus).with_mem_limit_mb(alloc.mem_mb);
}
// ── AND THE SAME SYSCALL FILTER ON ALL OF THEM ─────────────────────────
// Unconditional and engine-blind, which is the whole point: a filter that
// one column ran under and another did not would be a difference between
// the columns that has nothing to do with the engines. See
// [`seccomp_profile`] for what it is and why the io_uring columns cannot
// boot without it.
if let Some(profile) = seccomp_profile() {
spec = spec.with_security_opt(format!("seccomp={}", profile.display()));
}
let alloc_env = if engine.is_gitea_family() {
alloc.forge_env(engine.env_prefix())
} else {
alloc.gunnar_env()
};
for (k, v) in &alloc_env {
spec = spec.with_env(k, v);
}
let boot = ContainerBoot::new();
let machine = boot
.boot_with_binds(&spec, &plan.binds)
.map_err(|e| anyhow::anyhow!("booting the {} container: {e}", engine.key()))?;
// The handle is built HERE, one statement after the boot, so that every
// `?` below tears the container down on its way out. Building it after
// readiness — which is what the first version did — leaked the container
// of any engine that failed to become ready, which is precisely the path
// the `nostart` red switch exercises.
let mut forge = Forge {
engine,
root: root.to_path_buf(),
binary: gunnar_bin.to_path_buf(),
machine,
boot,
meter: Meter::start(published)?,
published,
cgroup: Cgroup {
dir: PathBuf::new(),
ceiling_b: u64::MAX,
},
version: String::new(),
inner_git: String::new(),
secret: plan.secret.clone().unwrap_or_default(),
stopped: false,
last_gc: RepackTelemetry::default(),
// Replaced wholesale at the end of `up`, once the container is up far
// enough to be asked. The placeholder carries a zero quota, which
// `assert_quotas_agree` would refuse if it ever reached the report.
applied: parity::Applied {
engine: leak(engine.key()),
quota: parity::AppliedQuota {
cpus: 0.0,
mem_b: 0,
},
git_config: BTreeMap::new(),
runtime_threads: String::new(),
visible_cpus: String::new(),
pack_threads: String::new(),
storage_engine: String::new(),
arms: parity::AppliedArms::default(),
},
};
let (boot, machine) = (&forge.boot, &forge.machine);
// ── THE CONTAINER'S LOG, ACCUMULATED ───────────────────────────────────
//
// `drain_logs_split` **empties** the buffer it reads (draupnir
// `container.rs:1422`: `std::mem::take`), so it is a drain and not a
// snapshot, and two readers of it are one reader plus a silent loss. The
// engine read-back below has to poll — the startup line arrives when the
// server gets to it — so it drains repeatedly, and everything it drains is
// kept here for the readiness failure path, which used to do its own single
// drain and would otherwise report an empty log for a container that had
// spoken.
let mut log_out: Vec<String> = Vec::new();
let mut log_err: Vec<String> = Vec::new();
let pump = |out: &mut Vec<String>, err: &mut Vec<String>| {
let (mut o, mut e) = boot.drain_logs_split(machine);
out.append(&mut o);
err.append(&mut e);
};
// ── THE BAR ────────────────────────────────────────────────────────────
// A server that never started must not look like a slow server. Readiness
// is the engine answering a real HTTP request, not a TCP accept: podman
// publishes the host port the instant the container is created, so a
// successful `connect` says nothing at all about the process inside.
boot.wait_ready(machine, &Readiness::Running, Duration::from_secs(30))
.map_err(|e| {
anyhow::anyhow!("the {} container never reached Running: {e}", engine.key())
})?;
// ── WHICH STORAGE ENGINE OPENED THE STORE ──────────────────────────────
//
// Before readiness, deliberately. `ServerStores::open` logs the engine
// *before* the first repository is opened, so this is the earliest honest
// answer, and asking here means a container running the wrong store is
// reported as the wrong store rather than as whatever it goes on to do with
// it. See `parity::assert_storage_engine` for why the passed variable is not
// an acceptable substitute for the served one.
let storage_engine = if let Some(store) = engine.gunnar_store() {
let want = store.engine_name().to_string();
let deadline = Instant::now() + ready_budget();
let got = loop {
pump(&mut log_out, &mut log_err);
let seen = parity::storage_engine_in_log(&log_out)
.or_else(|| parity::storage_engine_in_log(&log_err));
if let Some(name) = seen {
break name;
}
if Instant::now() >= deadline {
let state = boot.container_state(machine);
bail!(
"the {} container never said which storage engine it opened within {:?}. \
`ServerStores::open` logs `store.engine_selected` on target `gunnar::store` \
at info before it opens the first repository, so no such line means the \
server did not get that far, the log filter is not the default, or the \
event was renamed. Refusing rather than labelling this column with the \
engine that was REQUESTED, which is the mislabelling this read-back exists \
to stop.\n container state: {state:?}\n stdout:\n {}\n stderr:\n {}",
engine.key(),
ready_budget(),
tail_of(&log_out),
tail_of(&log_err),
);
}
std::thread::sleep(Duration::from_millis(100));
};
parity::assert_storage_engine(engine.key(), &want, &got)?
} else {
// The Go forges are given NONE of gunnar's engine variables, and that
// is checked here rather than left to `plan_for`'s control flow. Three
// keys, not one: the storage engine and both znippy arms. Each of them
// is a variable that a Go forge ignores completely and that would read,
// on a row and in a diff, as though it had selected something.
for key in std::iter::once(ENGINE_ENV).chain(ZNIPPY_ENVS) {
parity::assert_no_engine_var(engine.key(), &forge.machine.id, key)?;
}
String::new()
};
// What "ready" means, per engine, and **with a credential where one is
// needed**. gunnar's repositories are private by default, so an
// unauthenticated advertisement request answers 401 forever: the first run
// of the `nostart` red probe failed on exactly that and reported a server
// that had in fact started perfectly. A readiness check that can never
// succeed is the same class of defect as one that can never fail.
let (health, probe_auth) = match engine {
// The znippy-engine slot is the same binary and the same URL space, so it
// takes the same probe. It used to fall through to `/api/healthz`,
// which gunnar does not serve — a column that could never become ready.
Engine::Gunnar(_) | Engine::GunnarZnippy => (
format!(
"http://127.0.0.1:{published}/bench/{}/info/refs?service=git-upload-pack",
GUNNAR_PROBE_REPO
),
plan.secret.clone().map(|t| format!("{FORGE_USER}:{t}")),
),
_ => (format!("http://127.0.0.1:{published}/api/healthz"), None),
};
let deadline = Instant::now() + ready_budget();
loop {
let last = match http_status(&health, probe_auth.as_deref()) {
Ok(code) if code == 200 => break,
Ok(code) => format!("HTTP {code}"),
Err(err) => err.to_string(),
};
if Instant::now() >= deadline {
// The three facts that tell "never started" apart from "slow":
// what the container is doing now, what it said, and what we asked.
// The log is the ACCUMULATED one: the engine read-back above drains
// the same buffer, and a fresh drain here would report an empty log
// for a container whose lines had already been taken.
let state = boot.container_state(machine);
pump(&mut log_out, &mut log_err);
bail!(
"the {} container never served {health} within {:?} — last answer: {last}\n \
container state: {state:?}\n stdout:\n {}\n stderr:\n {}\n\n \
This is a server that did not start, and it is reported as a hard failure \
precisely so it can never be read as a slow one.",
engine.key(),
ready_budget(),
tail_of(&log_out),
tail_of(&log_err),
);
}
std::thread::sleep(Duration::from_millis(250));
}
ensure!(
matches!(boot.container_state(machine), ContainerState::Running),
"the {} container answered its health check and is not Running",
engine.key()
);
// ── WHICH ZNIPPY ARMS ACTUALLY GOT BUILT ───────────────────────────────
//
// **After** readiness, and that is not laziness. The engine line above is
// emitted by `ServerStores::open` before anything is served, but the arms
// are a property of a *store*, and gunnar opens no store until a request
// asks for a repository. The readiness probe is that request — it fetches
// `bench/probe.git`'s advertisement, which cannot be answered without the
// refs — so the earliest point this line can exist is here.
//
// What is compared is the pair of **implementation** names the server
// reported (`FastWriter`, `PackedPayload`) against the pair this column
// claims. Never against `ZNIPPY_GIT_WRITER` and never against
// `ZnippyArms::writer`: those are the selector, and znippy keeps the
// operator spelling and the built object's own name distinct precisely so
// that a guard can tell "the arm was selected" from "the arm was
// requested". A server that ignored the selector entirely — which is what
// every gunnar-server build before 2026-08-11 was — passes any check
// written against the request and fails this one.
let applied_arms = if let Some(want) = engine.znippy_arms() {
let deadline = Instant::now() + ready_budget();
let (writer, index) = loop {
pump(&mut log_out, &mut log_err);
let seen = parity::znippy_arms_in_log(&log_out)
.or_else(|| parity::znippy_arms_in_log(&log_err));
if let Some(pair) = seen {
break pair;
}
if Instant::now() >= deadline {
let state = boot.container_state(machine);
bail!(
"the {} container never said which znippy arms it built within {:?}. \
`znippy_engine::open_shared` logs `store.znippy_arms_selected` on target \
`gunnar::store` at info as it opens each store, so no such line means the \
server opened no store, the log filter is not the default, or the event \
was renamed. Refusing rather than labelling this column with the writer \
arm that was REQUESTED — a server built before 2026-08-11 ignored the \
request completely and would look identical here.\n container state: \
{state:?}\n stdout:\n {}\n stderr:\n {}",
engine.key(),
ready_budget(),
tail_of(&log_out),
tail_of(&log_err),
);
}
std::thread::sleep(Duration::from_millis(100));
};
parity::assert_znippy_arms(engine.key(), &want, &writer, &index)?
} else {
parity::AppliedArms::default()
};
let cgroup = Cgroup::of(&forge.machine.id)?;
forge.cgroup = cgroup;
// ── identity, asserted rather than assumed ─────────────────────────────
if engine.is_gitea_family() {
let bin = engine.admin_binary();
forge.version = forge.exec(&[bin, "--version"])?.trim().to_string();
forge.inner_git = forge.exec(&["git", "--version"])?.trim().to_string();
let want = engine
.expected_version()
.expect("a Go forge has a pinned version");
ensure!(
forge.version.contains(want),
"the {} container reports {:?}, which does not carry the pinned {want:?}. A byte \
ratio is a statement about one specific build.",
engine.key(),
forge.version
);
// The admin account, and the API credential every later step uses.
// `su`, because draupnir's exec has no user field and gitea refuses to
// run as root.
forge.exec(&[
"/bin/sh",
"-c",
&format!(
"su git -c '{bin} admin user create --username {FORGE_USER} \
--password {FORGE_PASSWORD} --email {FORGE_USER}@gunnar.invalid --admin \
--must-change-password=false'"
),
])?;
forge.secret = FORGE_PASSWORD.to_string();
} else {
forge.version =
String::from_utf8_lossy(&Command::new(gunnar_bin).arg("--version").output()?.stdout)
.trim()
.to_string();
}
// ── WHAT THE ENGINE ACTUALLY GOT ───────────────────────────────────────
// Not what was requested. `cpu.max` and `memory.max` come from the kernel,
// the pack settings from `git config --get` run inside the container against
// the very file the engine's `upload-pack` reads, and the runtime thread
// count from podman's own record of the environment the process started
// with. Every one of these can differ from the request without anything
// else in the run noticing.
let quota = parity::assert_quota(engine.key(), &forge.cgroup.dir, &alloc)?;
// `getconf _NPROCESSORS_ONLN`, not `nproc`. The first version of this asked
// `nproc` and got 8 from gunnar's ubuntu image and 32 from the two Alpine
// ones under the identical `--cpus 8` quota — which reads exactly like the
// engines being given different machines. They were not: GNU coreutils'
// `nproc` reads the cgroup quota and busybox's does not, so the column was a
// property of each base image's coreutils. `sysconf(_SC_NPROCESSORS_ONLN)`
// is what `git`'s thread auto-detect and Go's `runtime.NumCPU()` actually
// read, it answers 32 in all three containers, and that is the number that
// makes setting `pack.threads` and `GOMAXPROCS` necessary.
let visible_cpus = forge
.exec(&["getconf", "_NPROCESSORS_ONLN"])
.map(|s| s.trim().to_string())
.unwrap_or_default();
let (git_config, pack_threads) = if engine.is_gitea_family() {
let cfg = parity::assert_git_config_applied(engine.key(), &alloc, |argv| forge.exec(argv))?;
let threads = cfg.get("pack.threads").cloned().unwrap_or_default();
(cfg, threads)
} else {
// gunnar's serving pack path is `thread_limit: Some(1)` by
// construction — guarded by `xtask/tests/pack_pipeline.rs` — and no
// configuration changes it. Reported as the 1 it is, beside the Go
// forges' N, rather than left out because it is not a knob.
(
BTreeMap::new(),
"1 (by construction, not configurable)".to_string(),
)
};
let runtime_key = if engine.is_gitea_family() {
"GOMAXPROCS"
} else {
"TOKIO_WORKER_THREADS"
};
let runtime_threads = parity::container_env(&forge.machine.id, runtime_key)?;
ensure!(
runtime_threads == alloc.threads.to_string(),
"the {} container carries {runtime_key}={runtime_threads:?}, not the {} every engine was \
given. The runtime would size its own pool from the machine instead of the allocation.",
engine.key(),
alloc.threads
);
forge.applied = parity::Applied {
engine: leak(engine.key()),
quota,
git_config,
runtime_threads,
visible_cpus,
pack_threads,
storage_engine,
arms: applied_arms,
};
Ok(forge)
}
/// The last dozen lines of a captured stream, indented for an error message.
///
/// Twelve because a stack trace or a panic message is the thing worth seeing and
/// a chatty boot would otherwise push it off the top; by reference because the
/// two callers both hold an accumulating buffer they are not finished with.
fn tail_of(lines: &[String]) -> String {
lines
.iter()
.rev()
.take(12)
.rev()
.cloned()
.collect::<Vec<_>>()
.join("\n ")
}
/// The repository gunnar's readiness probe asks for: created during
/// provisioning, empty, and never measured.
const GUNNAR_PROBE_REPO: &str = "probe.git";
/// Build the engine-specific boot plan, doing whatever provisioning has to
/// happen **before** the process starts.
fn plan_for(
engine: Engine,
data: &Path,
gunnar_bin: &Path,
_free_port: fn() -> std::result::Result<u16, String>,
) -> Result<UpPlan> {
if engine.is_gitea_family() {
// The image's entrypoint chowns /data and drops to uid 1000; under
// rootless podman the bind-mounted host directory has to be owned by
// the *mapped* uid, which `podman unshare chown` is the sanctioned way
// to arrange. Doing it with a host chown would need root and would be
// wrong anyway.
let out = Command::new("podman")
.args(["unshare", "chown", "-R", "1000:1000"])
.arg(data)
.output()
.context("podman unshare chown on the forge data directory")?;
ensure!(
out.status.success(),
"podman unshare chown on {} exited {:?}: {}",
data.display(),
out.status.code(),
String::from_utf8_lossy(&out.stderr).trim()
);
let p = engine.env_prefix();
let mut env = BTreeMap::new();
let mut set = |k: &str, v: &str| {
env.insert(format!("{p}{k}"), v.to_string());
};
set("security__INSTALL_LOCK", "true");
set("database__DB_TYPE", "sqlite3");
set("server__DISABLE_SSH", "true");
set("service__DISABLE_REGISTRATION", "true");
set("log__LEVEL", "warn");
// Housekeeping off, so the cgroup CPU reading is as close to the git
// operation as this shape allows. The residue that remains is what
// `<engine>_idle_cpu_s` measures.
set("cron__ENABLED", "false");
set("cron.update_checker__ENABLED", "false");
set("indexer__REPO_INDEXER_ENABLED", "false");
env.insert("USER_UID".into(), "1000".into());
env.insert("USER_GID".into(), "1000".into());
// ── RED SWITCH: `forgeengine` ──────────────────────────────────────
// Put gunnar's storage-engine variable on a Go forge, which has no
// storage engine to select and will ignore it completely. Everything
// about the run stays correct — gitea serves, the numbers are real —
// and the only thing wrong is a variable on a row suggesting a choice
// was made where none exists. `assert_no_storage_engine` must refuse
// it; nothing else in the run looks at a container's environment for a
// key it did not expect to find.
if red("forgeengine") {
env.insert(ENGINE_ENV.to_string(), storage_engine());
}
// ── RED SWITCH: `forgearms` ────────────────────────────────────────
// The same defect one level down, and a separate switch because it is
// a separate variable: put znippy's writer and index arms on a Go
// forge, which has no archive, no journal and no Arrow projection.
// gitea serves perfectly and every number stays real; what is wrong is
// three variables on a container implying that a durability contract
// and a payload layout were chosen there.
// `assert_no_engine_var` must refuse each of them.
if red("forgearms") {
env.insert(ZNIPPY_WRITER_ENV.to_string(), ZNIPPY_DEFAULT.writer.into());
env.insert(ZNIPPY_INDEX_ENV.to_string(), ZNIPPY_DEFAULT.index.into());
}
// ── RED SWITCH: `nostart` ──────────────────────────────────────────
// Replace gitea's entrypoint with a command that exits at once. The
// container is created and started successfully; nothing ever serves.
// The readiness deadline must report that, with the container's state
// and its log, rather than letting the arm proceed to a table of zeros.
let cmd = if red("nostart") && engine == Engine::Gitea {
vec![
"/bin/sh".to_string(),
"-c".to_string(),
"echo forge-red-nostart; exit 3".to_string(),
]
} else {
Vec::new()
};
return Ok(UpPlan {
env,
cmd,
binds: vec![format!("{}:/data", data.display())],
secret: None,
});
}
// ── gunnar ─────────────────────────────────────────────────────────────
//
// Every repository, principal, grant and token is created **before** the
// server starts, and that is not a stylistic choice: `gunnar serve` holds a
// cross-process exclusive lock on the control repository's store, so
// `gunnar repo create` against a running server fails with "the store is
// locked by another writer". It is a real operational property — offline
// administration, one writer — and it means provisioning owns the window
// before boot.
let control = data.join("data");
let already = control.join("control.git").exists();
std::fs::create_dir_all(&control)?;
let key = data.join("id_ed25519");
gunnar_installtest::keygen(&key).map_err(|e| anyhow::anyhow!(e))?;
// Which gunnar this column is. Every environment decision below reads it,
// and it is resolved once so that the provisioning half and the serving
// half cannot disagree about which store they are talking about.
let store = engine
.gunnar_store()
.expect("the gitea family returned above");
let admin = |verbs: &[&str]| -> Result<String> {
let mut cmd = Command::new(gunnar_bin);
// Provisioning runs on the HOST, where it would inherit the harness's
// own `GUNNAR_ENGINE` and `ZNIPPY_GIT_*` anyway — set explicitly so
// the store is *created* under the arms this column names rather than
// under whatever the operator's shell happens to hold, and so they stay
// the **requested** arms when a red switch is feeding the container
// different ones.
//
// The writer arm matters here and not only in the server: a store
// seeded by `SafeWriter` carries a journal that `FastWriter` will not
// append to, and a reopen on the fast arm would then diff the store
// format's `indexed` bit against a log nothing is writing (`arms.rs`,
// `WriterArm::journal`). Same arms both sides, or the column measures a
// store neither arm built.
cmd.env(ENGINE_ENV, store.engine_name());
if let Some(a) = store.arms() {
cmd.env(ZNIPPY_WRITER_ENV, a.writer)
.env(ZNIPPY_INDEX_ENV, a.index);
} else {
// A gix column must not carry them: they would mean nothing to it,
// and a variable that means nothing still reads like a choice.
cmd.env_remove(ZNIPPY_WRITER_ENV)
.env_remove(ZNIPPY_INDEX_ENV);
}
let out = cmd
.arg("--data-root")
.arg(&control)
.args(verbs)
.output()
.with_context(|| format!("running gunnar {}", verbs.join(" ")))?;
ensure!(
out.status.success(),
"gunnar {} exited {:?}: {}",
verbs.join(" "),
out.status.code(),
String::from_utf8_lossy(&out.stderr).trim()
);
Ok(String::from_utf8_lossy(&out.stdout).into_owned())
};
// A SECOND boot over an existing store re-uses everything: the namespace,
// the principal, the key and the repositories are already there, and
// `repo create` over an existing repository is an error rather than a
// no-op. Only the token is minted again, because the first one is not
// recoverable from the store.
if !already {
admin(&["namespace", "create", "bench"])?;
admin(&["principal", "add", FORGE_USER])?;
admin(&["key", "add", FORGE_USER, &format!("@{}.pub", key.display())])?;
// The readiness probe's repository, plus every repository any arm will
// ask for. Creating them all up front is what the exclusive lock forces.
for slug in std::iter::once(GUNNAR_PROBE_REPO.trim_end_matches(".git").to_string())
.chain(all_slugs())
{
admin(&["repo", "create", &format!("bench/{slug}"), "--hash", "sha1"])?;
admin(&[
"grant",
FORGE_USER,
"write",
"--repo",
&format!("bench/{slug}"),
])?;
}
}
let minted = admin(&["token", "add", FORGE_USER])?;
let token = minted
.lines()
.find(|l| l.trim_start().starts_with("gunnar_pat_"))
.map(|l| l.trim().to_string())
.context("`gunnar token add` printed no gunnar_pat_ line, so nothing can authenticate")?;
// ── THE STORAGE ENGINE, ACROSS THE CONTAINER BOUNDARY ──────────────────
//
// The one thing the host environment does NOT carry across. `gunnar serve`
// reads `GUNNAR_ENGINE` from its own process environment; podman gives the
// container the environment in the boot spec and nothing else, so a run
// launched `GUNNAR_ENGINE=gix …` reached the five `ext::` arms (which fork
// from the harness) and never reached these eight. Both engines would have
// been reported as znippy and the row would not have said which.
//
// It goes in `UpPlan::env`, which `up_with` feeds through the same
// `spec.with_env` loop the allocation block uses — one path into a
// container's environment, not two.
//
// Only gunnar gets it: the gitea family's `plan_for` arm returns above,
// and `assert_no_engine_var` checks that rather than trusting the control
// flow. `GUNNAR_ENGINE` means nothing to a Go forge, and setting it on one
// would read on a row as though it had selected something.
let mut env = BTreeMap::new();
env.insert(ENGINE_ENV.to_string(), engine_for_container(store));
// ── THE ZNIPPY ARMS, ACROSS THE SAME BOUNDARY ──────────────────────────
//
// Exactly the same crossing, one level down, and it is the same variable
// path on purpose: `UpPlan::env` → `up_with`'s `spec.with_env`
// loop. One door into a container's environment, not three.
//
// These are read inside the server by `StoreConfig::from_env`, which
// `znippy_engine::arms` calls once per process — and which, until
// 2026-08-11, the server did not call at all: it built its stores through
// `GitStore::open_with`, a constructor documented as reading no
// environment, so `ZNIPPY_GIT_WRITER` was a variable that could be set and
// could not do anything. A column named for a writer arm on that binary
// would have measured `SafeWriter` four times.
//
// Nothing is set for a non-znippy column. `gix` has no writer arm and no
// Arrow projection, and a variable on its container would put a choice on
// its row that the column never made.
if let Some(a) = store.arms() {
env.insert(
ZNIPPY_WRITER_ENV.to_string(),
writer_for_container(a).to_string(),
);
env.insert(
ZNIPPY_INDEX_ENV.to_string(),
index_for_container(a).to_string(),
);
}
// ── THE BINDING CEILING (`concrefuse`) ─────────────────────────────────
//
// The concurrent arm's refusal column can only read a non-zero refusal if
// the gate actually BINDS, and the shipped default share does not: since
// the max-min-fair rework `DEFAULT_MAX_TRANSFERS_PER_PRINCIPAL` is a share,
// not a wall, so 64 clients on one key get 64 slots and nothing is refused
// (the gate was never asked to bind). `GUNNAR_MAX_TRANSFERS`
// (`gunnar_server::budget::MAX_TRANSFERS_ENV`) is the server-wide wall; set
// low, the ramp crosses it and the gate sheds — which is the red the
// column exists to make reachable. `GUNNAR_BENCH_FORGE_CONC_CEILING` tunes
// it; the default 3 refuses at rungs of 4 and 16. This is the ONLY thing
// `concrefuse` changes on the container — the guard-side handling that
// turns a `429` into data instead of a fault is in `clone_batch`.
if red("concrefuse") {
let ceiling = env_usize("GUNNAR_BENCH_FORGE_CONC_CEILING", 3).max(1);
env.insert("GUNNAR_MAX_TRANSFERS".to_string(), ceiling.to_string());
}
// ── THE DIAGNOSTIC DOOR ────────────────────────────────────────────────
//
// `GUNNAR_BENCH_FORGE_EXTRA_ENV="K=V,K2=V2"` puts arbitrary variables into
// the gunnar container, through the same one door as everything above.
//
// This exists because of a defect that produced FIVE wrong conclusions on
// 2026-08-11 alone, four of them mine: podman passes nothing implicitly, so
// a host variable set to probe a knob simply does not arrive, the server
// runs on its default, and the probe reports "no effect" — a null result
// shaped exactly like a real one. `ZNIPPY_GIT_REDB_CACHE_BYTES` was
// "falsified" that way; `GUNNAR_ENGINE` reached neither container seam;
// `GUNNAR_BENCH_H2H_SAME_SIDE` and `..._SHALLOW_GIT` had it before any of
// us arrived, recorded in `nornir-bench.rs` as "both red runs reported a
// cheerful green".
//
// The pattern in every case was the same: to test a knob you first had to
// EDIT THIS FILE and rebuild, so the cheap thing to do was to set it on the
// host and believe the answer. A door that is already open is the fix; the
// alternative is trusting five people in a row to remember.
//
// gunnar only, like `GUNNAR_ENGINE` above — `assert_no_engine_var` checks
// the Go forges are clean rather than trusting the control flow. It is a
// DIAGNOSTIC and not a column axis: anything worth persisting on a row
// belongs in `ZNIPPY_COLUMNS` where the table can name it.
if let Ok(extra) = std::env::var("GUNNAR_BENCH_FORGE_EXTRA_ENV") {
for pair in extra.split(',').filter(|s| !s.trim().is_empty()) {
let (k, v) = pair.split_once('=').with_context(|| {
format!(
"GUNNAR_BENCH_FORGE_EXTRA_ENV entry {pair:?} has no `=`; the format is \
`KEY=VALUE,KEY2=VALUE2`. Refusing rather than silently dropping it — a \
dropped variable is the exact failure this switch exists to end"
)
})?;
env.insert(k.trim().to_string(), v.trim().to_string());
}
}
Ok(UpPlan {
env,
cmd: vec![
gunnar_bin.display().to_string(),
"--data-root".into(),
control.display().to_string(),
"serve".into(),
"--http-bind".into(),
format!("0.0.0.0:{IN_CONTAINER_PORT}"),
// gunnar refuses to read a credential off a plain connection
// unless a proxy attests TLS. There is no proxy here and the whole
// comparison is plaintext HTTP on loopback, so the switch is set —
// explicitly, and it is a difference from the default worth stating.
"--http-allow-cleartext-credentials".into(),
"--no-grpc".into(),
// The SSH listener still starts; it is not published and no arm
// touches it. Bound inside the container's own namespace.
"--ssh-bind".into(),
"127.0.0.1:2222".into(),
]
.into_iter()
.chain(match engine {
// The znippy-engine column is the same binary, the same provisioning and
// the same URL space; the only difference is what the object store
// sits on, which is what these arguments select.
Engine::GunnarZnippy => znippy_serve_args().unwrap_or_default(),
_ => Vec::new(),
})
.collect(),
binds: vec![
format!("{0}:{0}", data.display()),
format!(
"{0}:{0}:ro",
gunnar_bin
.parent()
.expect("a binary has a directory")
.display()
),
],
secret: Some(token),
})
}
/// Every repository slug any arm serves.
///
/// gunnar needs **all** of them before its server starts, because `gunnar serve`
/// holds a cross-process exclusive lock on the control repository: `repo create`
/// against a running server fails with "the store is locked by another writer".
/// The Go forges create theirs over the API on demand, which is a real
/// difference in operational shape and is why this list exists at all.
///
/// The push arm needs one receiver per round plus a warm-up, so the `push*`
/// slugs are generated rather than listed; a missing one shows up as
/// `repository not found` at seeding time, which is how this was found.
fn all_slugs() -> Vec<String> {
let mut v: Vec<String> = [
"small", "large", "real", "diverge",
// The scale corpora and the two load arms' repositories. gunnar's
// repositories have to exist before `serve` starts — the store lock is
// exclusive while it serves — so a slug missing from this list
// does not produce a smaller run, it produces `repository not found`
// halfway through seeding. Measured on 2026-08-04: adding an arm and
// not its slug here cost one twenty-minute preview.
"year", "xl", "conc", "mix",
// One repository per delta ordering: a `gc` mutates the store, so
// ordering B applied after ordering A would be re-encoding A's output.
"ord-size", "ord-nh1", "ord-nh2", "ord-pw",
]
.iter()
.map(|s| s.to_string())
.collect();
// Generous: the arm builds `rounds() + 1`, and over-provisioning an empty
// repository costs nothing while under-provisioning is a hard failure
// halfway through a twenty-minute run.
for i in 0..(rounds() + 4) {
v.push(format!("push{i}"));
}
v
}
/// A one-shot HTTP GET that reports only the status code.
///
/// Hand-rolled rather than pulled in as a dependency: the bench crate has no
/// HTTP client and this needs one request, no redirects and no TLS.
fn http_status(url: &str, userpass: Option<&str>) -> Result<u16> {
let rest = url.strip_prefix("http://").context("not an http:// url")?;
let (authority, path) = match rest.find('/') {
Some(i) => (&rest[..i], &rest[i..]),
None => (rest, "/"),
};
let mut stream = TcpStream::connect(authority)?;
stream.set_read_timeout(Some(Duration::from_secs(10)))?;
let auth = match userpass {
Some(up) => format!(
"Authorization: Basic {}\r\n",
base64_lite::encode(up.as_bytes())
),
None => String::new(),
};
write!(
stream,
"GET {path} HTTP/1.0\r\nHost: {authority}\r\n{auth}Connection: close\r\n\r\n"
)?;
stream.flush()?;
let mut head = [0u8; 64];
let mut filled = 0usize;
while filled < head.len() {
match stream.read(&mut head[filled..]) {
Ok(0) => break,
Ok(n) => {
filled += n;
if head[..filled].contains(&b'\n') {
break;
}
}
Err(err) => return Err(err.into()),
}
}
let line = String::from_utf8_lossy(&head[..filled]);
let code = line
.split_whitespace()
.nth(1)
.and_then(|c| c.parse::<u16>().ok())
.with_context(|| format!("no status code in {line:?}"))?;
Ok(code)
}
// ═══════════════════════════════════════════════════════════════════════════
// SEEDING: the same fixture, put there the same way, on every engine
// ═══════════════════════════════════════════════════════════════════════════
/// Create `slug` on the engine and push `fixture` into it with **stock git**.
///
/// The push is by the same client for every engine, so what lands is each
/// engine's own receive path applied to the same bytes — and what a later clone
/// reads out is that engine's storage, repacked or not, exactly as it chose to
/// leave it. Nothing is repacked afterwards by the harness: a forge that leaves
/// a thin, badly-deltified pack on disk should be seen doing it.
fn seed(forge: &Forge, slug: &str, fixture: &Path) -> Result<()> {
if forge.engine.is_gitea_family() {
let body = format!(r#"{{"name":"{slug}","auto_init":false,"private":false}}"#);
let code = http_post_json(
&format!("http://127.0.0.1:{}/api/v1/user/repos", forge.meter.port),
&body,
&format!("{FORGE_USER}:{FORGE_PASSWORD}"),
)?;
ensure!(
code == 201,
"creating {slug} on {} returned HTTP {code}, not 201",
forge.engine.key()
);
}
// `--mirror` so tags and every branch land, and so the ref namespace on the
// server is the fixture's own rather than a subset a later clone would then
// "agree" with vacuously.
let url = forge.url(slug, true);
let out = Command::new("git")
.args([
"-C",
&fixture.to_string_lossy(),
"push",
"--quiet",
"--mirror",
&url,
])
.output()
.context("seeding push")?;
// ── AND WHAT THE SERVER SAID, NOT ONLY WHAT THE CLIENT SAW ─────────────
//
// `git push` reports "Could not read from remote repository" for a server
// that died mid-push and for a URL that was never right, in the same words.
// This path used to carry the client's stderr alone, so a crashed engine was
// indistinguishable from a typo — and on 2026-08-11 that cost an hour on the
// `znippy_uring` column: the container was speaking the whole time and
// nothing here was listening. Draining is safe at this point because boot's
// own accumulation has already been consumed by the readiness path above.
if !out.status.success() {
use draupnir::container::ContainerControl;
let (o, e) = forge.boot.drain_logs_split(&forge.machine);
// The WHOLE log goes to a file beside the arm's work root, and only its
// tail into the message. Twelve lines is the right size for a human
// reading a failure; it is the wrong size for the failure where the
// interesting event is a thread dying two hundred lines earlier.
let dump = fixture_root().join(format!("forge-seed-failure-{}.log", forge.engine.key()));
let whole = format!(
"── stdout ──\n{}\n── stderr ──\n{}\n",
o.join("\n"),
e.join("\n")
);
let dumped = match std::fs::write(&dump, &whole) {
Ok(()) => format!(
" full log ({} + {} lines): {}",
o.len(),
e.len(),
dump.display()
),
Err(err) => format!(
" (the full log could not be written to {}: {err})",
dump.display()
),
};
let inside = if o.is_empty() && e.is_empty() {
"the container said NOTHING — no stdout, no stderr. A server that \
logged nothing at all usually did not survive to log."
.to_string()
} else {
format!(
"stdout:\n {}\n stderr:\n {}\n{dumped}",
tail_of(&o),
tail_of(&e)
)
};
anyhow::bail!(
"seeding {} on {} failed ({:?}): {}\n ── what the container itself said ──\n {inside}",
slug,
forge.engine.key(),
out.status.code(),
String::from_utf8_lossy(&out.stderr).trim(),
);
}
// ── RED SWITCH: `fatseed` ──────────────────────────────────────────────
// Inflate PROVISIONING, leave the request identical, and see which memory
// columns move.
//
// This switch exists because the defect it hunts is invisible: a
// wrong-but-plausible memory number looks exactly like a right one, which
// is how a container's lifetime peak reached a published headline reading
// 36× under a column headed `push`. Measured on the laptop, 2026-08-11, 1500 MB:
//
// metric gunnar gitea verdict
// wire_rx_b 1449 → 1449 2164 → 2164 the request is IDENTICAL
// rss_lifetime_peak_mb 3299 → 4051 469 → 1805 3.85× — pure seeding. RED.
// rss_growth_mb 67.8 → 0.00 6.18 → 0.00 collapses to zero. RED.
// rss_op_anon_peak_mb 796.3 → 794.7 142.2 → 134.2 1.00× / 0.94×. GREEN.
//
// `rss_growth_mb` is the striking one: with a fat seed the lifetime mark is
// already above anything the push can reach, so end-minus-start is exactly
// **0.00 MB** for all three engines on a push that really happened. The
// fallback metric is not merely blind to transients, it is itself a
// function of how much seeding preceded it.
//
// ── WHAT THIS SWITCH DOES NOT HOLD CONSTANT, stated rather than glossed ──
// Pushing 1500 MB through the page cache EVICTS the seeded repository's
// cache, so the operation that follows runs colder. That is visible and it
// is real: `cpu_net_s` moves 2.2–2.6× and `rss_op_peak_mb` moves 0.66×,
// because a total peak includes the inherited cache the operation started
// with and that inheritance genuinely changed. Those columns are not wrong
// here; they are reporting a different operation. The clean invariance
// claim belongs to the anon columns and to the standalone probe, where the
// workload is a tmpfs write with no cache coupling and the reset peak reads
// 101.9 MB under both a 500 MB and a 2000 MB seed.
//
// The inflation is page cache from a large write, then freed — the same
// mechanism by which the real seeding push inflates the mark, and gone
// before any timed region opens. `dd` and `rm` are all this needs, which
// both the Alpine forges and gunnar's ubuntu image have.
if red("fatseed") {
let mb = std::env::var("GUNNAR_BENCH_FORGE_RED_FATSEED_MB")
.ok()
.and_then(|v| v.parse::<u64>().ok())
.unwrap_or(1500);
forge.exec(&[
"sh",
"-c",
&format!(
"dd if=/dev/zero of=/tmp/forge-red-fatseed bs=1M count={mb} 2>/dev/null; \
rm -f /tmp/forge-red-fatseed"
),
])?;
}
Ok(())
}
/// A one-shot HTTP POST of a JSON body with Basic auth, returning the status.
fn http_post_json(url: &str, body: &str, userpass: &str) -> Result<u16> {
use base64_lite::encode;
let rest = url.strip_prefix("http://").context("not an http:// url")?;
let (authority, path) = match rest.find('/') {
Some(i) => (&rest[..i], &rest[i..]),
None => (rest, "/"),
};
let mut stream = TcpStream::connect(authority)?;
stream.set_read_timeout(Some(Duration::from_secs(30)))?;
let auth = encode(userpass.as_bytes());
write!(
stream,
"POST {path} HTTP/1.0\r\nHost: {authority}\r\nAuthorization: Basic {auth}\r\n\
Content-Type: application/json\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
body.len()
)?;
stream.flush()?;
let mut buf = Vec::new();
stream.read_to_end(&mut buf)?;
let text = String::from_utf8_lossy(&buf);
let line = text.lines().next().unwrap_or_default();
line.split_whitespace()
.nth(1)
.and_then(|c| c.parse::<u16>().ok())
.with_context(|| format!("no status code in {line:?}"))
}
/// The smallest base64 that will do. Pulling a crate in for twelve lines would
/// be the twin the one-implementation rule forbids in the other direction.
mod base64_lite {
const A: &[u8; 64] = b"ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/";
pub fn encode(input: &[u8]) -> String {
let mut out = String::with_capacity(input.len().div_ceil(3) * 4);
for chunk in input.chunks(3) {
let b = [
chunk[0],
*chunk.get(1).unwrap_or(&0),
*chunk.get(2).unwrap_or(&0),
];
let n = (u32::from(b[0]) << 16) | (u32::from(b[1]) << 8) | u32::from(b[2]);
out.push(A[(n >> 18) as usize & 63] as char);
out.push(A[(n >> 12) as usize & 63] as char);
out.push(if chunk.len() > 1 {
A[(n >> 6) as usize & 63] as char
} else {
'='
});
out.push(if chunk.len() > 2 {
A[n as usize & 63] as char
} else {
'='
});
}
out
}
}
// ═══════════════════════════════════════════════════════════════════════════
// THE OPERATIONS
// ═══════════════════════════════════════════════════════════════════════════
/// One measured operation against one engine.
///
/// **A batch, not necessarily a single client.** The single-client arms produce
/// a `Run` with `clients: 1`; the ramped arms produce one `Run` per batch of N
/// concurrent clients, and every column below is then the batch's — `wall_ms`
/// is the batch's span, `cpu_s` is what the container burned over it, and the
/// percentiles are over the N clients' own latencies. One struct rather than
/// two, so `assert_counted`, `assert_engines_agree` and `metrics_of` are the
/// same code for both, and a ramped row cannot dodge a guard a
/// single-client row passes.
#[derive(Debug, Clone, Default)]
struct Run {
engine: &'static str,
fixture: &'static str,
wall_ms: f64,
wire_rx: u64,
wire_tx: u64,
conns: u64,
cpu_s: f64,
idle_cpu_s: f64,
/// Every memory figure this run produced. One struct rather than two loose
/// `u64`s so that a call site cannot pick up the lifetime peak while meaning
/// the operation's — which is the mistake that put a 36× seeding cost in a
/// column headed `push`.
mem: MemReading,
/// Bytes of `.pack` in the artefact the operation produced (a clone or a
/// fetch's target); zero for `ls-remote`, which produces none.
pack_b: u64,
/// The **unique object set** the operation left behind (a clone: the whole
/// clone; a fetch: the objects the repository gained). This is the field
/// every engine must agree on, and it is deliberately not a packed-object
/// count: `index-pack --fix-thin` copies bases the client already held
/// *loose* into the new pack, so two engines that transferred exactly the
/// same history can honestly land different numbers of PACKED objects.
/// Measured on 2026-08-04: gitea 58 packed, gunnar 54, same fetch.
objects: u64,
/// Packed objects the operation added — the denominator `delta_pct` is over,
/// and reported in its own right because the gap above is a real difference
/// between the engines and hiding it would be the flattering choice.
packed: u64,
deltas: u64,
/// The tip `refs/heads/main` resolved to, so two engines agreeing on an
/// object count but not on history cannot pass.
tip: String,
/// Concurrent clients in this batch. 1 for the single-client arms.
clients: u64,
/// Per-client latency percentiles over the batch, ms. A mean hides the
/// queue: at saturation the median client is still fast and the 99th is
/// where the box actually is.
p50_ms: f64,
p95_ms: f64,
p99_ms: f64,
/// The admission gate's **refused-at-capacity count** observed in this
/// batch's log window (`gunnar_bench::admission_log`), read off the
/// server's own `*.transfer.refused` lines — the same `refused_total`
/// field the SSH/HTTP/gRPC refusals log, which is
/// `gunnar_ssh::AdmissionStats::refused()`. `0` for the Go forges (no gate)
/// and for a gunnar batch that stayed under the ceiling. **A refusal here
/// is data, not a fault** — it is only non-zero when the gate was made to
/// bind (a low `GUNNAR_MAX_TRANSFERS`, e.g. the `concrefuse` red switch);
/// the default share lets N clients on one key have N slots, so an
/// ordinary ramp reads 0.
refused: u64,
/// The gate's **peak concurrent transfers** as it reported them on a
/// refusal line (`peak_in_flight`, ==
/// `gunnar_ssh::AdmissionStats::peak_in_flight`). `0` when no refusal line
/// was written in this window (the peak is only surfaced on a refusal); at
/// a binding ceiling it settles on the ceiling, which is the number that
/// says the gate actually bound rather than merely being configured.
peak_in_flight: u64,
}
impl Run {
/// The concurrency block of a **one-client** operation.
///
/// The batch is that client, so every percentile is its own latency. Filled
/// in rather than left zero so that a single-client row and a ramped row
/// come out of the warehouse through the same columns — a zero here would
/// read as "the percentile was not measured", which is a different claim.
fn solo(wall_ms: f64) -> Run {
Run {
clients: 1,
p50_ms: wall_ms,
p95_ms: wall_ms,
p99_ms: wall_ms,
..Run::default()
}
}
/// Completed operations per second.
///
/// For a batch of one this is the reciprocal of one latency and says
/// nothing about saturation; for a ramped batch it is the throughput that
/// N concurrent clients actually got out of the server.
fn ops_sec(&self) -> f64 {
if self.wall_ms > 0.0 {
self.clients as f64 / (self.wall_ms / 1000.0)
} else {
0.0
}
}
}
/// Wrap a timed operation in the cgroup and meter sampling both sides need.
///
/// The **idle window** is taken first, over a fixed short interval, so a reader
/// can price the forge's background noise before subtracting nothing at all from
/// gunnar (whose container runs one process) and something real from gitea's
/// (which runs an s6 supervisor, an sshd and a web application).
/// ## The memory reading is per-operation, and that is the point
///
/// `memory.peak` is reset on a held-open fd immediately before the timed region
/// and read back through that same fd after it, so `op_peak_b` excludes the
/// container's boot and the repository seeding **by construction** rather than
/// by subtraction. The lifetime mark is still taken and still published, under a
/// name that says what it is.
fn measured<T>(
forge: &Forge,
idle: Duration,
op: impl FnOnce() -> Result<T>,
) -> Result<(T, f64, f64, u64, u64, u64, MemReading, f64)> {
let idle_a = forge.cgroup.sample()?;
std::thread::sleep(idle);
let idle_b = forge.cgroup.sample()?;
let idle_cpu_rate = (idle_b.cpu_s - idle_a.cpu_s) / idle.as_secs_f64().max(1e-9);
forge.meter.reset();
let before = forge.cgroup.sample()?;
// Taken as late as possible. Everything charged to this cgroup before this
// line — container boot, repository seeding, the previous round — is now
// outside the operation's peak, and no reader has to subtract anything.
let mut peak = forge.cgroup.watch_peak()?;
let anon = AnonWatch::start(&forge.cgroup);
let started = Instant::now();
let value = op()?;
let wall_ms = started.elapsed().as_secs_f64() * 1000.0;
let op_peak_b = peak.read()?;
let (anon_peak_b, anon_floor_b, anon_samples) = anon.finish();
let after = forge.cgroup.sample()?;
let (rx, tx, conns) = forge.meter.read();
Ok((
value,
wall_ms,
after.cpu_s - before.cpu_s,
rx,
tx,
conns,
MemReading {
lifetime_peak_b: after.lifetime_peak_b,
growth_b: after.lifetime_peak_b.saturating_sub(before.lifetime_peak_b),
op_peak_b,
op_floor_b: peak.floor_b,
anon_peak_b,
anon_floor_b,
anon_samples,
end_anon_b: after.anon_b,
end_file_b: after.file_b,
ceiling_b: forge.cgroup.ceiling_b,
},
// Scaled to the operation's own duration: what the background would
// have burned in the same window.
idle_cpu_rate * (wall_ms / 1000.0),
))
}
/// The extra `gunnar serve` arguments that switch the znippy znippy engine on, or
/// `None` when nobody has named any.
///
/// **This is the whole wiring contract.** The lane that lands the znippy engine sets
/// `GUNNAR_BENCH_FORGE_ZNIPPY_ARGS` to whatever `gunnar serve` needs, and a
/// fourth column appears carrying every metric the other three already carry.
/// Nothing else in this file has to change.
///
/// Asking for the column without naming the arguments is a **hard error**, not a
/// three-bar chart with a missing fourth: see [`znippy_demanded`].
fn znippy_serve_args() -> Option<Vec<String>> {
let raw = std::env::var("GUNNAR_BENCH_FORGE_ZNIPPY_ARGS").ok()?;
let args: Vec<String> = raw.split_whitespace().map(str::to_string).collect();
if args.is_empty() {
None
} else {
Some(args)
}
}
/// `GUNNAR_BENCH_FORGE_ZNIPPY=1` — "I want the znippy-engine column".
fn znippy_demanded() -> bool {
std::env::var("GUNNAR_BENCH_FORGE_ZNIPPY")
.map(|v| v != "0")
.unwrap_or(false)
}
/// Refuse to run a four-bar comparison with three bars in it.
///
/// The failure this guards is the one that has bitten this repo repeatedly: an
/// absent subject that comes out as a tidy chart nobody reads twice. Asking for
/// the znippy column and silently getting the filesystem column relabelled, or
/// getting three bars where four were requested, are both that failure.
fn assert_columns_consistent() -> Result<()> {
// ── the gunnar column set, before anything boots ───────────────────────
// An unrecognised token is refused here rather than dropped in
// `gunnar_columns`, which cannot carry a diagnosis. A dropped column is a
// chart with a missing bar and a run that took less time than it should
// have, and neither says anything about which arm was meant.
if let Some(tokens) = column_tokens() {
let known: Vec<&str> = ZNIPPY_COLUMNS
.iter()
.map(|a| a.token)
.chain(["gix", "all"])
.collect();
for t in &tokens {
ensure!(
known.contains(&t.as_str()),
"GUNNAR_BENCH_FORGE_ZNIPPY_ARMS names {t:?}, which is not a gunnar column. \
Accepted: {}. A token this file does not know would silently drop a column, \
and a comparison with a bar missing is the failure this whole arm is arranged \
against.",
known.join(", ")
);
}
// Two columns that resolve to the same key would share a container
// name and a data directory, which is one store measured twice under
// two headings. The table cannot express it today; the check is here so
// that a future row added to it cannot.
let mut keys: Vec<&str> = gunnar_columns().iter().map(|c| c.key()).collect();
keys.sort_unstable();
let before = keys.len();
keys.dedup();
ensure!(
keys.len() == before,
"two gunnar columns resolved to the same row key ({}). They would share a container \
name and a data directory, so one store would be measured twice under two headings.",
keys.join(", ")
);
}
ensure!(
!(znippy_demanded() && znippy_serve_args().is_none()),
"GUNNAR_BENCH_FORGE_ZNIPPY asks for the znippy-engine column and \
GUNNAR_BENCH_FORGE_ZNIPPY_ARGS does not say how to switch it on.\n\
\n\
Checked in this tree on 2026-08-04, and every one of these has to change \
before the column can exist:\n \
* `RepackOptions::cold_tier` has no call site in any server crate\n \
* the only implementation of the old cold-storage trait was the deleted engine crate, which the root \
manifest `exclude`s\n \
* `Cargo.lock` holds zero `znippy` rows\n \
* `gunnar serve --help` offers no znippy-engine switch\n \
* the control plane answers \"no znippy znippy engine is built\"\n\
\n\
Set GUNNAR_BENCH_FORGE_ZNIPPY_ARGS='<gunnar serve flags>' once it is wired. \
Until then this arm measures gunnar on the filesystem and says so in \
`znippy_wired`."
);
Ok(())
}
/// The idle window every arm prices its background noise over.
fn idle_window() -> Duration {
Duration::from_millis(env_usize("GUNNAR_BENCH_FORGE_IDLE_MS", 400) as u64)
}
/// A bare clone, through the meter, with the resulting repository inspected.
fn clone_once(forge: &Forge, slug: &str, fixture_label: &'static str, work: &Path) -> Result<Run> {
let target = work.join(format!("clone-{}-{slug}", forge.engine.key()));
let _ = std::fs::remove_dir_all(&target);
let url = forge.url(slug, true);
let shallow = red("shallow") && forge.engine == Engine::Gitea;
let (_, wall_ms, cpu_s, rx, tx, conns, mem, idle_cpu_s) =
measured(forge, idle_window(), || {
let mut c = Command::new("git");
c.args(["clone", "-q", "--bare"]);
// ── RED SWITCH: `shallow` ──────────────────────────────────────
// A depth-1 clone is a valid, fsck-clean repository with a
// completely different object set and a wonderfully small pack.
// That is the exact shape of a bug that would flatter one engine,
// and `assert_engines_agree` must catch it.
if shallow {
c.args(["--depth", "1"]);
}
let out = c
.arg(&url)
.arg(&target)
.env("GIT_TERMINAL_PROMPT", "0")
.output()
.context("clone")?;
ensure!(
out.status.success(),
"the {} clone of {slug} failed ({:?}): {}",
forge.engine.key(),
out.status.code(),
String::from_utf8_lossy(&out.stderr).trim()
);
Ok(())
})?;
let pack_b = h2h_pack_bytes(&target)?;
let objects = unique_objects(&target)?;
let (packed, deltas) = h2h_deltified_client(&target)?;
let tip = tip_of(&target)?;
let _ = std::fs::remove_dir_all(&target);
Ok(Run {
engine: leak(forge.engine.key()),
fixture: fixture_label,
wall_ms,
wire_rx: rx,
wire_tx: tx,
conns,
cpu_s,
idle_cpu_s,
mem,
pack_b,
objects,
packed,
deltas,
tip,
..Run::solo(wall_ms)
})
}
/// A fetch into a genuinely divergent client, so the server has to negotiate.
///
/// A fast-forward fetch measures nothing a clone does not: with no shared base
/// to find, the server never walks for a cut point. The client here shares a
/// long history with the server and then goes its own way, which is the case
/// that makes a `have`/`ACK` round trip real work.
fn fetch_once(
forge: &Forge,
slug: &str,
client_src: &Path,
fixture_label: &'static str,
work: &Path,
) -> Result<Run> {
let target = work.join(format!("fetch-{}-{slug}", forge.engine.key()));
let _ = std::fs::remove_dir_all(&target);
// A fetch mutates the client, so every round gets a fresh copy of it.
copy_repo(client_src, &target)?;
let before_objects = unique_objects(&target)?;
// The client already holds a pack — it is a copy of a real repository — so
// the pack bytes on disk afterwards are NOT what the fetch transferred.
// The first green run caught this the honest way: `wire_rx >= pack_b` fired
// with 15 899 bytes on the wire against a 33 112-byte pack, which is not a
// meter that stopped counting but a denominator that was never the fetch's.
let before_pack = h2h_pack_bytes(&target)?;
let (before_packed, before_deltas) = h2h_deltified_client(&target)?;
let url = forge.url(slug, true);
let (_, wall_ms, cpu_s, rx, tx, conns, mem, idle_cpu_s) =
measured(forge, idle_window(), || {
let out = Command::new("git")
// `transfer.unpackLimit=1` forces the answer to land as a PACK.
// git explodes a small fetch into loose objects below the
// default limit of 100, and then `pack_b` is zero — which the
// arm's own floor caught, on the preview fixture, exactly as it
// should have. Set identically for every engine, so it is a
// property of the harness and not of any server.
.args(["-c", "transfer.unpackLimit=1"])
.args(["-C", &target.to_string_lossy(), "fetch", "--quiet", &url])
.arg("+refs/heads/*:refs/remotes/origin/*")
.env("GIT_TERMINAL_PROMPT", "0")
.output()
.context("fetch")?;
ensure!(
out.status.success(),
"the {} fetch of {slug} failed ({:?}): {}",
forge.engine.key(),
out.status.code(),
String::from_utf8_lossy(&out.stderr).trim()
);
Ok(())
})?;
let after_objects = unique_objects(&target)?;
ensure!(
after_objects > before_objects,
"the {} fetch of {slug} added no objects ({before_objects} → {after_objects}); a fetch \
that transferred nothing is not a measurement of a fetch",
forge.engine.key()
);
let pack_b = h2h_pack_bytes(&target)?.saturating_sub(before_pack);
ensure!(
pack_b > 0,
"the {} fetch of {slug} left the pack directory the same size it found it; the arm would \
then report a zero that looks exactly like a free transfer",
forge.engine.key()
);
// The delta share of what the FETCH brought, not of the client's whole
// object store: the pre-existing pack is subtracted out of both halves.
let (after_packed, after_deltas) = h2h_deltified_client(&target)?;
let deltas = after_deltas.saturating_sub(before_deltas);
let packed_added = after_packed.saturating_sub(before_packed);
ensure!(
packed_added > 0,
"the {} fetch of {slug} added no PACKED objects, so its delta share would be a ratio \
over zero",
forge.engine.key()
);
let tip = h2h_git(
&[
"-C",
&target.to_string_lossy(),
"rev-parse",
"refs/remotes/origin/main",
],
None,
)
.map(|s| s.trim().to_string())
.unwrap_or_default();
let _ = std::fs::remove_dir_all(&target);
Ok(Run {
engine: leak(forge.engine.key()),
fixture: fixture_label,
wall_ms,
wire_rx: rx,
wire_tx: tx,
conns,
cpu_s,
idle_cpu_s,
mem,
pack_b,
objects: after_objects - before_objects,
packed: packed_added,
deltas,
tip,
..Run::solo(wall_ms)
})
}
The file goes on; the console shows all of it.