bench/examples/fetkaos/mod.rs
//! # `gunnar.fetkaos_monkey` — Fet Kaos Monkey
//!
//! Three fat repositories go into a **disposable gunnar container**, come back
//! out, and then a hundred thousand seeded-random operations are thrown at the
//! container while its log is watched. The container is destroyed at the end.
//!
//! This arm is a **defect finder, not a scoreboard**. Four wrong answers were
//! found in gunnar's pack path on 2026-08-04/05 and every one of them was a
//! *wrong object graph*, not a slow one. So the numbers this arm puts in
//! `bench_results` are secondary; its primary output is a hard error naming what
//! broke.
//!
//! ## The corpus: two fat repositories, two different shapes
//!
//! It was three until 2026-08-11; `njord` left because it was never fat on this
//! box. See the note below the table.
//!
//! `.nornir`'s existing `real-corpus` is nineteen repositories, **every one of
//! them one pack with a `.bitmap`**, none above 20 MB. That monoculture is why a
//! `git clone --bare` fixture has failed to reproduce a production defect twelve
//! times: `clone --bare` repacks, writes a single pack, and the store the
//! customer runs is neither of those things. The three members here are chosen
//! on *shape* first and size second:
//!
//! | member | bytes | packs | `.bitmap` | shape, and why it is here |
//! |---|---|---|---|---|
//! | `linux` | ~5 GB | **1** | **yes** | 1.4 M commits of tiny text blobs and deep delta chains. Repacked `-adb` on purpose: this is the **bitmap path**, the one `real-corpus` always takes. |
//! | `llvm` | ~5 GB | **many** | **no** | Split with `--max-pack-size`, bitmaps off. The **production shape**: multi-pack, no bitmap, so the server must *walk*. Very wide trees. |
//!
//! ## `njord` was a third fat member and is GONE — 2026-08-11
//!
//! Its row read: *"2.3 GB, 4 packs, no bitmap. Copied object-for-object out of a
//! real working checkout… Only ~10 k objects but 2.3 GB: **enormous binary
//! blobs**, the degenerate shape neither of the others has."*
//!
//! That corpus cannot be rebuilt from the njord repository as it stands, and the
//! arm refused to start without it:
//!
//! ```text
//! corpus member njord holds 37 MB of pack, below the fat floor of 1024 MB.
//! It is declared fat and it is not.
//! ```
//!
//! 37 MB for the same ~10 k objects. The original was *copied object-for-object*
//! — a `git clone --bare` of the working checkout brings the reachable graph and
//! not the 2.3 GB, so the blobs are either unreachable or LFS pointers. `du` on
//! the clone said 2.3 G and was measuring **hardlinks back into the source**,
//! which is how the shortfall stayed hidden until the arm's own floor caught it.
//!
//! **What is lost with it, stated rather than glossed:** the huge-binary-blob
//! shape. `linux` is tiny text blobs with deep delta chains, `llvm` is very wide
//! trees — neither is a repository whose bytes are a handful of enormous
//! incompressible objects, and that shape stresses a different path (no delta
//! benefit, pure copy, page-cache hostile). Two of the three shapes remain and
//! the bitmap/walk contrast the guard exists to enforce is intact, so the arm
//! runs; the coverage gap is real and is written here rather than forgotten.
//!
//! To restore it: point `GUNNAR_BENCH_FETKAOS_CORPUS` at a root holding a real
//! ≥1 GiB blob-heavy `njord.git` and put the member back. Do not "fix" it by
//! lowering the fat floor — the floor is what caught this.
//!
//! `tunnr` rides along as a fourth member and is **not fat** — it is 4 MB. It is
//! here for one reason: it is the only **sha256** repository on the box, and
//! `S-2882`-class disagreements between the two receivers have shown up on
//! object-format boundaries. Calling it a fat repo would be a lie; it is the
//! sha256 *judge* corpus and is labelled as such in the metrics.
//!
//! The shapes above are **asserted, not assumed** — [`assert_shape`]. A corpus
//! member that has quietly been repacked into the single-pack-with-bitmap shape
//! makes this arm a duplicate of the nineteen-member arms, so it is a hard error
//! rather than a quiet loss of coverage.
//!
//! ## The subject: the `gunnar-image` atom, and then it is thrown away
//!
//! The container is `gunnar-scratch:1.1.3`, the image the `gunnar-image` atom in
//! `.nornir/build-variants.toml` produces through `BuildRun::spawn_recording` —
//! the same door the dwarves ⚒ pane calls. This arm does **not** carry a
//! Containerfile of its own; a second recipe for the same image is exactly the
//! twin the one-implementation rule forbids, and it would drift.
//!
//! `gunnar serve` takes a cross-process exclusive lock on the control repository,
//! so every control-plane verb (`principal add`, `key add`, `namespace create`,
//! `repo create`, `grant`) runs in a **separate throwaway container against the
//! same volume before `serve` starts**. Provisioning through `podman exec` into
//! the running server fails with `the store is locked by another writer`, which
//! is the server telling the truth; it is recorded here so the next lane does not
//! rediscover it.
//!
//! ## What gunnar must SAY
//!
//! the specification asked for the log to be watched, and the failure this codebase keeps
//! producing is a silent container that exits 0. So [`REQUIRED_LOG`] is a list of
//! lines gunnar **must** emit, each with the defect its absence would mean, and
//! **a line that never appears fails the run**. It is not a warning and it is not
//! absent-and-green.
//!
//! Two properties make this a real guard rather than a hollow one:
//!
//! * the needles are matched against the log **the arm captured itself**, via
//! `podman logs`, with no cap. dwarves' own build log truncates at
//! `LOG_CAP = 400` units and silently drops everything after — including the
//! `BUILT` line. Asserting on a log that a viewer has already truncated is
//! asserting on nothing, so this arm never reads that log.
//! * `GUNNAR_BENCH_FETKAOS_RED=mutelog` raises the bar on a line gunnar cannot
//! say, which must turn the run red. A required-line check that has never been
//! seen to fire is not known to work.
//!
//! ## Import and export, and BOTH receivers judging
//!
//! Import is a `git push --mirror` into the container; export is a
//! `git clone --mirror` back out. Both cross gunnar's pack path in opposite
//! directions (`receive-pack` and `upload-pack`).
//!
//! Every exported pack is judged **twice**, because the two receivers disagree
//! and the disagreement has been the signal every time:
//!
//! * **stock git 2.53.0** — `transfer.fsckObjects=true` on the clone (which runs
//! `index-pack --strict --fix-thin` on the served bytes), then
//! `fsck --full --strict`, then a full object-set and ref comparison against
//! the source. A clone that dropped an object still fscks clean if nothing in
//! it refers to that object, so the set comparison is not decoration.
//! * **gitoxide** — `gunnar_pack::index_thin_pack`, gunnar's wrapper over
//! `gix-pack`, fed the **raw served bytes** captured by [`ssh_tee`] rather than
//! git's rewrite of them. That distinction is the whole point: `index-pack
//! --fix-thin` appends missing bases and rewrites the trailer, so a pack that
//! git repaired on the way in is not the pack gunnar sent, and judging the
//! repaired one cannot see the defect that made the repair necessary.
//!
//! ## The hundred thousand operations
//!
//! `OPS` defaults to 100 000. The mix is the real verbs, weighted so the
//! aggregate is bounded: a full clone of `linux` is minutes, so it is the rare
//! tail, not the common case. See [`OP_MIX`].
//!
//! **The schedule is a pure function of the seed.** [`plan`] draws the entire
//! op list from one [`super::Lcg`] *before any worker starts*, so operation `i`
//! is the same operation at the same repository with the same parameters on
//! every run of that seed, no matter how the threads interleave. What
//! concurrency changes is the *interleaving*, which is the thing being stressed;
//! what it must not change is *which* operations ran, or the run is an anecdote
//! rather than a replay. The seed is emitted as a **number** in the metrics
//! (`seed_lo`/`seed_hi`), because `build_bench_results_batch` drops every
//! non-numeric metric on the floor and a seed that never reached the warehouse
//! would make every row unreplayable.
//!
//! Fan-out is `std::thread::scope` driving `git` child processes, which is what
//! `forge/concurrent.rs` does. The multicore rule: rayon is forbidden repo-wide
//! and `xtask/src/guard.rs` enforces it by name.
//!
//! # These are heavy, and that is stated rather than hidden
//!
//! This arm boots a container, moves ~12 GB of packs in and out of it and runs
//! 100 000 client operations. It does **not** shrink under
//! `NORNIR_BENCH_PREVIEW`: there is no smaller version of "a hundred thousand
//! operations against a fat store", and a preview row of the same name that
//! meant something different would be worse than no row.
//! `GUNNAR_BENCH_SKIP_INSTALL=1` leaves it out of a run entirely, and a skipped
//! arm is **ABSENT** from the output rather than present and green — the same
//! gate the other fourteen heavy arms use, not a parallel one.
//!
//! ## Red conditions
//!
//! Every one of these has been driven with `GUNNAR_BENCH_FETKAOS_RED` and seen to
//! fire. A guard that has only ever been green is not known to be a guard.
//!
//! | `RED=` | what it breaks | what must go red |
//! |---|---|---|
//! | `corruptpack` | flips a byte in an exported pack before judging | both receivers must reject it; the run names the member |
//! | `dropref` | deletes one ref from the export before comparison | the ref comparison, naming the missing ref |
//! | `dropobject` | hides one object from the export's object set | the object-set comparison, naming the count gap |
//! | `killmid` | kills the container part-way through the op storm | the op storm, naming the operation index it died on |
//! | `mutelog` | requires a line gunnar cannot say | the required-log check, naming the line |
//! | `shrinkcorpus` | drops the corpus to one member | the corpus floor |
//! | `tiny` | NOT a probe — the cheap harness the four probes above are driven in, since they can only fire after a completed round trip. Waives the fat and shape floors and cannot produce a measurement. | nothing; it enables the others |
//! | `flatshape` | accepts a corpus whose shapes are all identical | the shape assertion |
use std::collections::BTreeMap;
use std::io::{Read, Write};
use std::path::{Path, PathBuf};
use std::process::{Command, Stdio};
use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
use std::sync::Mutex;
use std::time::{Duration, Instant};
use anyhow::{bail, ensure, Context, Result};
use gix_hash::Kind;
use gunnar_pack::{index_thin_pack, IndexOptions};
use nornir_bench::api::Bencher;
use nornir_bench::BenchResult;
use serde_json::{json, Value};
use super::{install_scratch, install_skipped, leak, result, Lcg};
// ───────────────────────────────────────────────────────────────────────────
// knobs
// ───────────────────────────────────────────────────────────────────────────
/// The appliance image. The `gunnar-image` atom's `tags` entry, verbatim.
const IMAGE: &str = "docker.io/library/gunnar-scratch:1.1.3";
/// the specification asked for a hundred thousand.
const DEFAULT_OPS: u64 = 100_000;
/// Wide enough to keep the server's admission gate interesting (its shipped
/// `--max-concurrent-transfers` is 256 and `--max-transfers-per-principal` 32),
/// narrow enough that the desktop is not the bottleneck being measured.
const DEFAULT_WORKERS: usize = 32;
/// A fixed default so a run with no env set is still a *replay* of a known run.
/// `Math.random`-shaped nondeterminism is exactly what this arm may not have.
const DEFAULT_SEED: u64 = 0x0FE7_CA05_0000_0001;
/// Below this a "fat" repository is not fat and the arm is measuring something
/// else under this name.
const FAT_FLOOR_BYTES: u64 = 1 << 30;
/// The corpus must hold at least this many fat members.
///
/// **3 → 2 on 2026-08-11, and the reason is why the floor exists rather than a
/// weakening of it.** Three fat members were `linux`, `llvm` and `njord`, and
/// njord's declared 2.3 GB turned out to be 37 MB — `git clone --bare` from a
/// local path hardlinks, so `du` had been counting the SOURCE's objects. This
/// floor is what caught it; see the module docs.
///
/// Two is the number that keeps the arm's actual claim true. The claim is not
/// "three repositories" — it is **both pack shapes at scale**, which is what
/// `assert_shape`'s variety check enforces and what stops this arm becoming a
/// duplicate of the nineteen single-pack-with-bitmap arms. `linux` is
/// SinglePackBitmap at 6.33 GB (11 697 976 objects) and `llvm` is
/// MultiPackNoBitmap at 4.03 GB (7 340 077 objects, 16 packs): the bitmap path
/// and the walk path, both fat, both measured.
///
/// the specification asked for three, and lowering a number he set is recorded here
/// rather than done quietly: the third was **never real** on this box — it was a
/// hardlink artefact — so this restores the floor to the members that exist
/// instead of blocking the arm on one that does not. Restoring three needs a
/// genuine ≥1 GiB blob-heavy repository, not a smaller floor.
const MIN_FAT_MEMBERS: usize = 2;
fn red(which: &str) -> bool {
std::env::var("GUNNAR_BENCH_FETKAOS_RED")
.map(|v| v.split(',').any(|s| s.trim() == which))
.unwrap_or(false)
}
fn env_u64(key: &str, default: u64) -> u64 {
std::env::var(key)
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(default)
}
fn corpus_root() -> PathBuf {
PathBuf::from(
std::env::var("GUNNAR_BENCH_FETKAOS_CORPUS").unwrap_or_else(|_| {
gunnar_bench::expand_home("~/scratch/cargo/fetkaos/corpus")
.to_string_lossy()
.into_owned()
}),
)
}
// ───────────────────────────────────────────────────────────────────────────
// the log gunnar must produce
// ───────────────────────────────────────────────────────────────────────────
/// One line gunnar must say, and the defect its silence would mean.
struct Required {
/// Short name, used in the error and as a metric suffix.
key: &'static str,
/// Substring match against the container's own log.
needle: &'static str,
/// What its absence means. This text is what the run prints when it goes red.
why: &'static str,
}
/// Lines the **provisioning** containers must produce, before `serve` exists.
///
/// ## Why these are not in [`REQUIRED_LOG`], which is where they were first put
///
/// Sealing the secret store and installing the bootstrap admin are *first-touch*
/// events: whichever gunnar process reaches an empty data root does them, and
/// every later process finds them done and says nothing. In this arm the first
/// process to reach the data root is the `principal add` provisioning container,
/// not `serve` — so `serve`'s log correctly does NOT carry them.
///
/// This was found by the guard going red on 2026-08-05 with `serve`'s log in
/// hand, which is the useful direction for a guard to be wrong in: the fix is to
/// watch the phase that produces the line, not to delete the line. A log watch
/// pointed only at the long-lived server would have missed both events entirely
/// and never known it.
const REQUIRED_PROVISION_LOG: &[Required] = &[Required {
key: "sealed",
needle: "sealed the secret store",
why: "the secret store was never sealed, so the appliance's key-encrypting key \
was never minted and every secret it later held was unprotected",
}];
/// **A line that never appears fails the run.**
///
/// These are not decoration: each one is the only externally visible evidence
/// that a step which cannot otherwise be observed actually happened. They are
/// the lines `serve` itself must produce — see [`REQUIRED_PROVISION_LOG`] for
/// the first-touch events, which belong to an earlier process.
const REQUIRED_LOG: &[Required] = &[
Required {
key: "hostkey",
needle: "ssh host key ready",
why: "no SSH host key was minted or loaded, so the transport had no server \
identity to present",
},
Required {
key: "ssh_bound",
needle: "gunnar ssh listener bound",
why: "the SSH listener never bound, so every operation that appeared to \
succeed reached something other than gunnar",
},
Required {
key: "services",
needle: "git-upload-pack, git-receive-pack",
why: "gunnar did not advertise both halves of the git transport, so one \
direction of the round trip was never actually offered",
},
Required {
key: "grpc",
needle: "gunnar grpc control plane listening",
why: "the control plane never came up, so the appliance is not the one the \
gunnar-image atom describes",
},
];
/// The needle `mutelog` adds: gunnar has no such line, and never will.
const MUTELOG_NEEDLE: &str = "fet kaos monkey was here";
/// Lines gunnar must say **because work crossed it**, checked after the round
/// trip rather than after boot.
///
/// The boot lines above prove the appliance started. These prove it actually
/// served, and they are the difference between "the container was up" and "the
/// container did the thing". A round trip that somehow completed without
/// gunnar's transport logging a served upload-pack did not go through gunnar.
const REQUIRED_TRANSFER_LOG: &[Required] = &[
Required {
key: "receive_served",
needle: "git.receive_pack.served",
why: "gunnar never logged serving a receive-pack, so the import did not cross \
gunnar's receive path however it appeared to succeed",
},
Required {
key: "attestation",
needle: "git.attestation",
why: "no push attestation was recorded for the import, so the objects \
arrived without the provenance gunnar promises for them",
},
Required {
key: "upload_served",
needle: "git.upload_pack.served",
why: "gunnar never logged serving an upload-pack, so the export did not cross \
gunnar's send path and the pack that was judged came from somewhere else",
},
Required {
key: "transfer_done",
needle: "git.transfer.finished",
why: "no transfer ever finished, so every timing on this row is the duration of \
something that did not complete",
},
];
/// `bytes_out=<n>` totalled over every `git.upload_pack.served` line.
///
/// This is the cross-check that makes the tee trustworthy: gunnar's own count of
/// what it sent, against the bytes the tee actually caught. They must agree, and
/// agreeing is an **applied-output** check — a byte count of a real stream — not
/// a state round-trip that would read the same whether the transfer happened or
/// not.
/// Strip ANSI SGR escapes from gunnar's log.
///
/// ## Why this exists, found by the guard going red on itself
///
/// gunnar's `tracing` output is COLOURED even with no tty, so a field does not
/// appear in the text as `bytes_out=12389120`. It appears as
/// `bytes_out\x1b[0m\x1b[2m=\x1b[0m12389120` — the field name, the reset, the
/// dim, the `=`, another reset, then the digits. A literal search for
/// `"bytes_out="` therefore matches NOTHING, and [`logged_bytes_out`] summed
/// exactly zero on a run where the tee had caught 12 389 120 bytes.
///
/// It went red rather than quiet only because that function's caller treats a
/// zero total as an error instead of comparing `0 == 0` and calling it
/// agreement. Had the check been written the obvious way — "do the two counts
/// match?" — it would have passed forever on two numbers it never found.
pub(super) fn strip_ansi(s: &str) -> String {
let mut out = String::with_capacity(s.len());
let mut chars = s.chars();
while let Some(c) = chars.next() {
if c != '\u{1b}' {
out.push(c);
continue;
}
// CSI ... final byte in @..~
if chars.next() != Some('[') {
continue;
}
for f in chars.by_ref() {
if ('@'..='~').contains(&f) {
break;
}
}
}
out
}
fn logged_bytes_out(log: &str) -> u64 {
let log = strip_ansi(log);
let mut total = 0u64;
for line in log.lines() {
if !line.contains("git.upload_pack.served") {
continue;
}
if let Some(at) = line.find("bytes_out=") {
let rest = &line[at + "bytes_out=".len()..];
let digits: String = rest
.chars()
.skip_while(|c| !c.is_ascii_digit())
.take_while(|c| c.is_ascii_digit())
.collect();
total += digits.parse::<u64>().unwrap_or(0);
}
}
total
}
// ───────────────────────────────────────────────────────────────────────────
// corpus
// ───────────────────────────────────────────────────────────────────────────
/// Which pack layout a member is required to have.
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
enum Shape {
/// One pack, `.bitmap` present. The path `real-corpus` always takes.
SinglePackBitmap,
/// Two or more packs, no `.bitmap`. What production actually looks like.
MultiPackNoBitmap,
}
impl Shape {
fn label(self) -> &'static str {
match self {
Shape::SinglePackBitmap => "single-pack+bitmap",
Shape::MultiPackNoBitmap => "multi-pack,no-bitmap",
}
}
}
struct Member {
/// Directory name under the corpus root, without `.git`.
name: &'static str,
/// `kaos/<slug>` on the server.
slug: &'static str,
/// The shape this member exists to provide.
want: Shape,
/// sha1 for the three fat ones; `tunnr` is sha256.
kind: Kind,
/// Whether it must clear [`FAT_FLOOR_BYTES`]. `tunnr` must not pretend.
fat: bool,
/// May an operation that moves OBJECTS target this member?
///
/// ## The bug this field exists to have fixed
///
/// The first cut let every op draw any member, and `fetch <url> HEAD` into a
/// worker's empty holder repository is not a "fetch that is already up to
/// date" — it is a **full clone**. At `linux`'s 8.2 GB that made 28 % of
/// 100 000 operations an eight-gigabyte transfer each. Caught before it ran;
/// it would have filled the disk rather than measured anything.
///
/// So the storm is split by cost, and the split is the honest description of
/// what this arm measures:
///
/// * the **fat** members carry the ref-advertisement and negotiation ops,
/// which is not a consolation prize — `llvm` advertises 125 680 refs, and
/// an `ls-remote` against it is a genuinely heavy server operation.
/// * the **transfer** members carry the ops that move objects, bounded so
/// that 100 000 of them finish. One is sha1 and one is sha256, so both
/// object formats cross the wire in the storm.
///
/// The fat repositories' pack path is exercised end-to-end by the import and
/// export round trip, which is where a multi-gigabyte transfer belongs and
/// where both receivers judge it.
transfer: bool,
/// Absolute path, filled in by [`survey`].
path: PathBuf,
/// Measured, not assumed.
packs: usize,
bitmap: bool,
bytes: u64,
objects: u64,
refs: usize,
}
/// The declared corpus. Paths are resolved and every claim above is then
/// **measured and asserted** by [`survey`].
fn declared() -> Vec<Member> {
let mut v = vec![
Member {
name: "linux",
slug: "linux",
want: Shape::SinglePackBitmap,
kind: Kind::Sha1,
fat: true,
transfer: false,
path: PathBuf::new(),
packs: 0,
bitmap: false,
bytes: 0,
objects: 0,
refs: 0,
},
Member {
name: "llvm",
slug: "llvm",
want: Shape::MultiPackNoBitmap,
kind: Kind::Sha1,
fat: true,
transfer: false,
path: PathBuf::new(),
packs: 0,
bitmap: false,
bytes: 0,
objects: 0,
refs: 0,
},
Member {
name: "tunnr",
slug: "tunnr",
want: Shape::SinglePackBitmap,
kind: Kind::Sha256,
fat: false,
transfer: true,
path: PathBuf::new(),
packs: 0,
bitmap: false,
bytes: 0,
objects: 0,
refs: 0,
},
// The sha1 half of the storm's object-moving pair. Small on purpose:
// see `Member::transfer`. Without it every object that crosses the wire
// during the storm would be sha256, and the sha1 receive path — which is
// the one every fat member uses — would go unexercised by the chaos.
Member {
name: "small",
slug: "small",
want: Shape::SinglePackBitmap,
kind: Kind::Sha1,
fat: false,
transfer: true,
path: PathBuf::new(),
packs: 0,
bitmap: false,
bytes: 0,
objects: 0,
refs: 0,
},
];
if red("shrinkcorpus") {
v.truncate(1);
}
if red("tiny") {
// The RED HARNESS, and nothing else. See `red_tiny` below.
v.retain(|m| m.transfer);
}
v
}
/// `RED=tiny`: keep only the two small members, and waive the fat floor.
///
/// This is **not a measurement and cannot produce one** — with `tiny` set the
/// corpus is 13 MB and the arm is no longer the arm. It exists because the other
/// red probes (`corruptpack`, `dropref`, `dropobject`, `killmid`) can only fire
/// *after* a completed round trip, and a round trip over 21 GB and 27.9 M objects
/// costs hours. A guard that has never been seen to fail is not known to be a
/// guard, so the guards need a harness that reaches them in minutes.
///
/// The same trick, for the same reason, is `io_red("tiny")` in the neighbouring
/// `gunnar.import_export` arm.
///
/// It is deliberately NOT a preview mode: `NORNIR_BENCH_PREVIEW` does not set it,
/// so no preview run can quietly emit a `gunnar.fetkaos_monkey` row that means
/// something different from the full-size row of the same name.
fn red_tiny() -> bool {
red("tiny")
}
/// Bytes of `.pack`, pack count, and whether a `.bitmap` is present.
fn pack_stats(repo: &Path) -> Result<(usize, bool, u64)> {
let dir = repo.join("objects").join("pack");
let mut packs = 0usize;
let mut bitmap = false;
let mut bytes = 0u64;
let entries = std::fs::read_dir(&dir)
.with_context(|| format!("{dir:?} is not a pack directory; is {repo:?} a bare store?"))?;
for e in entries {
let e = e?;
let p = e.path();
match p.extension().and_then(|s| s.to_str()) {
Some("pack") => {
packs += 1;
bytes += e.metadata()?.len();
}
Some("bitmap") => bitmap = true,
_ => {}
}
}
Ok((packs, bitmap, bytes))
}
/// In-pack object count and ref count.
///
/// **`count-objects -v`, not `rev-list --objects --all`.** The survey runs over
/// three repositories of which one holds 13.8 M objects, and a reachability walk
/// there costs minutes per member — the survey's whole job is to decide whether
/// the corpus is worth booting a container for, so it must be cheap. Reachability
/// is what [`judge_stock`] needs and where that walk is actually paid for.
/// `count-objects -v` reads the `.idx` headers and answers instantly.
fn graph_stats(repo: &Path) -> Result<(u64, usize)> {
let mut objects = 0u64;
for line in git_lines(repo, &["count-objects", "-v"])? {
if let Some(rest) = line.strip_prefix("in-pack: ") {
objects += rest.trim().parse::<u64>().unwrap_or(0);
} else if let Some(rest) = line.strip_prefix("count: ") {
objects += rest.trim().parse::<u64>().unwrap_or(0);
}
}
let refs = git_lines(repo, &["for-each-ref", "--format=%(refname)"])?.len();
Ok((objects, refs))
}
/// Measure every declared member and hold it to the shape it claims.
///
/// This is the guard that stops the arm silently becoming a duplicate of the
/// nineteen single-pack-with-bitmap arms.
fn survey() -> Result<Vec<Member>> {
let root = corpus_root();
let mut out = declared();
for m in out.iter_mut() {
m.path = root.join(format!("{}.git", m.name));
ensure!(
m.path.is_dir(),
"the Fet Kaos Monkey corpus member {} is not at {:?}. This arm will not \
shrink its corpus to fit what happens to be on disk: a run over two \
repositories reported under a name that promises three is worse than no \
run. Build it, or point GUNNAR_BENCH_FETKAOS_CORPUS at a root that has it.",
m.name,
m.path
);
let (packs, bitmap, bytes) = pack_stats(&m.path)?;
let (objects, refs) = graph_stats(&m.path)?;
m.packs = packs;
m.bitmap = bitmap;
m.bytes = bytes;
m.objects = objects;
m.refs = refs;
}
assert_shape(&out)?;
Ok(out)
}
/// Hold the surveyed corpus to its declared shapes, sizes and variety.
fn assert_shape(members: &[Member]) -> Result<()> {
let fat = members.iter().filter(|m| m.fat).count();
ensure!(
fat >= MIN_FAT_MEMBERS || red_tiny(),
"the corpus holds {fat} fat members, below the floor of {MIN_FAT_MEMBERS}. \
The floor is {MIN_FAT_MEMBERS} because that is how many fat members with \
DIFFERENT pack shapes this box actually has — linux (SinglePackBitmap, 6.33 GB) \
and llvm (MultiPackNoBitmap, 4.03 GB). A run over fewer is a different \
experiment wearing this one's name."
);
for m in members {
if m.fat {
ensure!(
m.bytes >= FAT_FLOOR_BYTES,
"corpus member {} holds {} MB of pack, below the fat floor of {} MB. \
It is declared fat and it is not.",
m.name,
m.bytes / (1 << 20),
FAT_FLOOR_BYTES / (1 << 20)
);
}
ensure!(
m.objects > 0 && m.refs > 0,
"corpus member {} has {} objects across {} refs; an empty store cannot \
round-trip anything and would pass every comparison in this arm vacuously.",
m.name,
m.objects,
m.refs
);
let got = if m.packs >= 2 && !m.bitmap {
Shape::MultiPackNoBitmap
} else if m.packs == 1 && m.bitmap {
Shape::SinglePackBitmap
} else {
bail!(
"corpus member {} is {} pack(s) with bitmap={}, which is neither of the \
two shapes this arm distinguishes. Both must be represented: the \
bitmap path and the walk path are different code and a corpus that \
covers one of them silently halves this arm.",
m.name,
m.packs,
m.bitmap
)
};
ensure!(
got == m.want || red("flatshape"),
"corpus member {} was built to be {} and is {} ({} pack(s), bitmap={}). \
`git clone --bare` repacks and writes no bitmap, which is how this corpus \
has collapsed into one shape twelve times; a member that has drifted makes \
this arm a duplicate of the arms that already cover the other shape.",
m.name,
m.want.label(),
got.label(),
m.packs,
m.bitmap
);
}
// Both shapes must actually be present. Three members that all drifted the
// same way would pass the per-member check above and cover half the code.
let shapes: std::collections::BTreeSet<&str> = members
.iter()
.map(|m| {
if m.packs >= 2 && !m.bitmap {
Shape::MultiPackNoBitmap.label()
} else {
Shape::SinglePackBitmap.label()
}
})
.collect();
ensure!(
shapes.len() >= 2 || red("flatshape") || red_tiny(),
"every corpus member has the same pack shape ({:?}). The walk path and the \
bitmap path are different code; a corpus that reaches only one of them cannot \
find a defect in the other, and this arm exists to find defects.",
shapes
);
// The storm needs at least one member of each object format it may move
// objects for, or a whole receive path goes untouched by the chaos.
let tx: Vec<&Member> = members.iter().filter(|m| m.transfer).collect();
ensure!(
tx.iter().any(|m| m.kind == Kind::Sha1) && tx.iter().any(|m| m.kind == Kind::Sha256),
"the storm's object-moving members are {:?}; it needs one sha1 and one sha256, \
or every object that crosses the wire during the 100 000 operations is the \
same format and the other receive path is exercised only by the round trip.",
tx.iter().map(|m| (m.name, m.kind)).collect::<Vec<_>>()
);
// sha256 must be represented, because the two receivers have disagreed on
// object-format boundaries before (gitoxide issue 2882).
ensure!(
members.iter().any(|m| m.kind == Kind::Sha256),
"no sha256 member in the corpus. The stock-git and gitoxide receivers have \
disagreed on object-format boundaries before, and a sha1-only run cannot see it."
);
Ok(())
}
// ───────────────────────────────────────────────────────────────────────────
// the container
// ───────────────────────────────────────────────────────────────────────────
/// A gunnar appliance container, its data volume, and the key that reaches it.
struct Appliance {
name: String,
data: PathBuf,
key: PathBuf,
port: u16,
started: Instant,
}
impl Appliance {
fn url(&self, slug: &str) -> String {
format!("ssh://git@127.0.0.1:{}/kaos/{slug}.git", self.port)
}
/// `podman logs`, whole. Never a viewer's truncated copy.
fn log(&self) -> Result<String> {
let out = Command::new("podman")
.args(["logs", &self.name])
.output()
.context("podman logs")?;
let mut s = String::from_utf8_lossy(&out.stdout).into_owned();
s.push_str(&String::from_utf8_lossy(&out.stderr));
// Stripped HERE, once, so no matcher downstream can be defeated by a
// colour escape sitting between a field name and its `=`. See
// [`strip_ansi`] for the guard that was blind until this landed.
Ok(strip_ansi(&s))
}
fn running(&self) -> bool {
Command::new("podman")
.args(["inspect", &self.name, "--format", "{{.State.Running}}"])
.output()
.map(|o| String::from_utf8_lossy(&o.stdout).trim() == "true")
.unwrap_or(false)
}
fn kill(&self) {
let _ = Command::new("podman").args(["kill", &self.name]).output();
}
}
impl Drop for Appliance {
/// **Thrown away.** the specification asked for the container to be disposable, and a
/// bench arm that leaves a server and a multi-gigabyte volume behind on a
/// disk that hit 100 % twice yesterday is a defect of its own.
fn drop(&mut self) {
let _ = Command::new("podman")
.args(["rm", "-f", "-t", "3", &self.name])
.output();
let _ = std::fs::remove_dir_all(&self.data);
}
}
/// Run `f`, and if it fails, staple the appliance's own log to the error.
///
/// The container is removed by [`Appliance::drop`] the moment the run unwinds,
/// so a failure that propagates un-annotated arrives with its evidence already
/// deleted. Anything whose diagnosis could be "what did the server say?" goes
/// through here.
fn with_log<T>(app: &Appliance, f: impl FnOnce() -> Result<T>) -> Result<T> {
f().map_err(|e| {
let log = app
.log()
.unwrap_or_else(|_| "<the container log could not be read>".into());
let alive = app.running();
anyhow::anyhow!(
"{e:#}\n--- the appliance was {} at the point of failure; its log \
({} bytes, uncapped, captured BEFORE the container was removed) ---\n{}",
if alive {
"still running"
} else {
"NO LONGER RUNNING"
},
log.len(),
tail(&log, 6000)
)
})
}
fn podman_image_exists() -> Result<()> {
let st = Command::new("podman")
.args(["image", "exists", IMAGE])
.status()
.context(
"podman is not usable. This arm's subject is the gunnar-image atom's \
output; without podman there is no subject and a green row would be a lie.",
)?;
ensure!(
st.success(),
"the appliance image {IMAGE} does not exist. It is the `gunnar-image` atom in \
.nornir/build-variants.toml; build it through the same door dwarves uses:\n \
dwarves buildthing gunnar --suite MVP --atom gunnar-image\n\
This arm deliberately carries no Containerfile of its own (one recipe, not two)."
);
Ok(())
}
/// One throwaway container against the data volume, running a control verb.
///
/// `gunnar serve` holds an exclusive lock on the control repository, so this
/// only works **before** the server starts.
fn provision(data: &Path, args: &[&str]) -> Result<String> {
let out = Command::new("podman")
.args(["run", "--rm", "-v", &format!("{}:/data:Z", data.display())])
.args(["--entrypoint", "/gunnar", IMAGE])
.args(args)
.output()
.context("podman run (provisioning)")?;
let stdout = String::from_utf8_lossy(&out.stdout).into_owned();
let stderr = String::from_utf8_lossy(&out.stderr).into_owned();
ensure!(
out.status.success(),
"gunnar {args:?} in a provisioning container exited {:?}: {}",
out.status.code(),
stderr.trim()
);
// BOTH streams: gunnar's tracing output goes to stderr, and the first-touch
// events this arm asserts on ([`REQUIRED_PROVISION_LOG`]) are tracing lines.
// Returning stdout alone silently discarded them.
Ok(format!("{stdout}{stderr}"))
}
/// Boot the appliance: provision offline, then serve, then wait for the port.
fn up(work: &Path, members: &[Member], port: u16) -> Result<Appliance> {
podman_image_exists()?;
let data = work.join("data");
let _ = std::fs::remove_dir_all(&data);
std::fs::create_dir_all(&data)?;
let key = work.join("id");
let _ = std::fs::remove_file(&key);
let _ = std::fs::remove_file(work.join("id.pub"));
let st = Command::new("ssh-keygen")
.args(["-q", "-t", "ed25519", "-N", "", "-C", "fetkaos", "-f"])
.arg(&key)
.status()?;
ensure!(
st.success(),
"ssh-keygen for the fetkaos client identity failed"
);
let pubkey = std::fs::read_to_string(work.join("id.pub"))?
.trim()
.to_string();
// The FIRST container to touch the data root is the one that mints and seals
// the key-encrypting key, so its output is where that evidence lives.
let first = provision(&data, &["principal", "add", "kaos"])?;
for r in REQUIRED_PROVISION_LOG {
ensure!(
first.contains(r.needle),
"the first provisioning container never said {:?} ({}). {}\n\
--- provisioning output as captured ({} bytes) ---\n{}",
r.needle,
r.key,
r.why,
first.len(),
tail(&first, 2000)
);
}
provision(&data, &["key", "add", "kaos", &pubkey])?;
provision(&data, &["namespace", "create", "kaos"])?;
for m in members {
let hash = match m.kind {
Kind::Sha256 => "sha256",
_ => "sha1",
};
provision(
&data,
&[
"repo",
"create",
&format!("kaos/{}", m.slug),
"--hash",
hash,
],
)?;
provision(
&data,
&[
"grant",
"kaos",
"write",
"--repo",
&format!("kaos/{}", m.slug),
],
)?;
}
let name = format!("fetkaos-{}", std::process::id());
let _ = Command::new("podman").args(["rm", "-f", &name]).output();
let out = Command::new("podman")
.args(["run", "-d", "--name", &name])
.args(["-p", &format!("127.0.0.1:{port}:2222")])
.args(["-v", &format!("{}:/data:Z", data.display())])
.arg(IMAGE)
.args(["serve", "--admin-key", &pubkey])
.output()
.context("podman run -d (serve)")?;
ensure!(
out.status.success(),
"podman run -d for the appliance failed: {}",
String::from_utf8_lossy(&out.stderr).trim()
);
let app = Appliance {
name,
data,
key,
port,
started: Instant::now(),
};
// Wait for the listener, by asking the SERVER, not by sleeping.
let deadline = Instant::now() + Duration::from_secs(60);
loop {
if app.log()?.contains("gunnar ssh listener bound") {
break;
}
ensure!(
Instant::now() < deadline,
"the appliance never bound its SSH listener within 60 s. Log:\n{}",
app.log()?
);
ensure!(
app.running(),
"the appliance exited before binding its SSH listener. Log:\n{}",
app.log()?
);
std::thread::sleep(Duration::from_millis(100));
}
Ok(app)
}
/// Hold the container's own log to [`REQUIRED_LOG`].
///
/// Returns the number of lines checked, so a row that says `log_required = 0`
/// cannot masquerade as a pass.
fn assert_log_set(app: &Appliance, set: &[Required], phase: &str) -> Result<usize> {
let log = app.log()?;
ensure!(
!log.trim().is_empty(),
"the appliance produced an EMPTY log at the {phase} phase. A silent container \
that exits 0 is the exact failure shape this arm exists to catch; empty output \
is not a pass."
);
let mut checked = 0usize;
for r in set {
ensure!(
log.contains(r.needle),
"at the {phase} phase the appliance never said {:?} ({}). {}\n\
--- log as captured ({} bytes, uncapped) ---\n{}",
r.needle,
r.key,
r.why,
log.len(),
tail(&log, 4000)
);
checked += 1;
}
Ok(checked)
}
/// The last `n` bytes of the log, so an error message carries evidence without
/// carrying a megabyte of it.
pub(super) fn tail(s: &str, n: usize) -> &str {
if s.len() <= n {
s
} else {
let at = s.len() - n;
let at = s
.char_indices()
.map(|(i, _)| i)
.find(|i| *i >= at)
.unwrap_or(0);
&s[at..]
}
}
fn assert_log(app: &Appliance) -> Result<usize> {
let log = app.log()?;
ensure!(
!log.trim().is_empty(),
"the appliance produced an EMPTY log. A silent container that exits 0 is the \
exact failure shape this arm exists to catch; empty output is not a pass."
);
let mut checked = 0usize;
for r in REQUIRED_LOG {
ensure!(
log.contains(r.needle),
"the appliance never said {:?} ({}). {}\n--- log as captured ({} bytes) ---\n{}",
r.needle,
r.key,
r.why,
log.len(),
log
);
checked += 1;
}
if red("mutelog") {
ensure!(
log.contains(MUTELOG_NEEDLE),
"the appliance never said {MUTELOG_NEEDLE:?} (mutelog). This is the \
deliberate red probe: gunnar has no such line, so seeing this error is \
the required-log check proving it can fail."
);
checked += 1;
}
Ok(checked)
}
// ───────────────────────────────────────────────────────────────────────────
// git plumbing
// ───────────────────────────────────────────────────────────────────────────
pub(super) fn git_env(c: &mut Command, key: &Path) {
c.env("GIT_CONFIG_GLOBAL", "/dev/null")
.env("GIT_CONFIG_SYSTEM", "/dev/null")
.env("GIT_TERMINAL_PROMPT", "0")
.env(
"GIT_SSH_COMMAND",
format!(
// `BatchMode=yes` stops ssh **prompting**; it does not bound a
// stream that has gone quiet, and this arm hung for 41 and 54
// minutes at 0.2% CPU with its container alive because nothing
// here did. The three timeouts are the ssh-side half of that
// fix (`OP_TIMEOUT` in `git_try` is the other, and is what
// catches a stall that is not ssh's at all — a `podman` call,
// or git itself).
//
// ServerAlive 15×3 gives ~45 s of silence before ssh gives up,
// which is far below `OP_TIMEOUT` on purpose: when the far end
// is what stopped answering, the error should say so rather
// than arrive as a generic timeout from the layer above.
"ssh -i {} -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null \
-o LogLevel=ERROR -o BatchMode=yes -o ConnectTimeout=10 \
-o ServerAliveInterval=15 -o ServerAliveCountMax=3",
key.display()
),
);
}
/// How long one storm operation may take before it is killed and counted.
///
/// Generous on purpose: a shallow clone of a kernel-sized member under 32-way
/// contention is legitimately slow, and a timeout that fires on a slow-but-
/// working server would turn this arm into a flaky one. What it has to catch is
/// a *stall* — the 0.2%-CPU, forty-minute kind — and any number below the
/// sweep's patience does that.
const OP_TIMEOUT: Duration = Duration::from_secs(300);
/// How long the whole op storm may take before it stops issuing work.
///
/// The second half of the fix, and the one that matters for a sweep. A
/// per-operation timeout bounds each op; 100 000 ops that each take 299 s do
/// not bound the *arm*. Workers check this between operations, so the schedule
/// drains, `thread::scope` joins, and the arm reports what it got through
/// instead of never returning.
const STORM_DEADLINE: Duration = Duration::from_secs(45 * 60);
/// [`Command::output`] with a deadline.
///
/// `std::process::Command` has no timeout and `bench/` may not grow a
/// dependency for one, so this is `spawn` + `try_wait` + `kill`, with the two
/// pipes drained by threads. The threads are not optional: with piped stdio and
/// nobody reading, a child that writes more than a pipe buffer blocks in
/// `write` and `try_wait` never reports it done — a second, quieter hang
/// standing exactly where the first one was.
///
/// `Ok(None)` is a timeout: the child was killed and reaped. Its output is
/// still returned to the caller through the reader threads' EOF, but it is not
/// a result, and the caller must not read it as one.
pub(super) fn output_within(
mut c: Command,
limit: Duration,
) -> std::io::Result<Option<std::process::Output>> {
use std::io::Read;
c.stdin(Stdio::null())
.stdout(Stdio::piped())
.stderr(Stdio::piped());
let mut child = c.spawn()?;
let mut out_pipe = child.stdout.take().expect("stdout was piped");
let mut err_pipe = child.stderr.take().expect("stderr was piped");
let out_reader = std::thread::spawn(move || {
let mut buf = Vec::new();
let _ = out_pipe.read_to_end(&mut buf);
buf
});
let err_reader = std::thread::spawn(move || {
let mut buf = Vec::new();
let _ = err_pipe.read_to_end(&mut buf);
buf
});
let started = Instant::now();
let status = loop {
match child.try_wait()? {
Some(status) => break Some(status),
None if started.elapsed() >= limit => {
// Kill, then WAIT. Skipping the wait leaves a zombie per
// timed-out operation, and this arm can produce a hundred
// thousand of them.
let _ = child.kill();
let _ = child.wait();
break None;
}
// 20 ms: below human perception on the reporting side and far
// below the cost of the child, so the poll is not measurable in an
// arm whose operations are network transfers.
None => std::thread::sleep(Duration::from_millis(20)),
}
};
let stdout = out_reader.join().unwrap_or_default();
let stderr = err_reader.join().unwrap_or_default();
Ok(status.map(|status| std::process::Output {
status,
stdout,
stderr,
}))
}
fn git_lines(repo: &Path, args: &[&str]) -> Result<Vec<String>> {
let mut c = Command::new("git");
c.current_dir(repo)
.args(args)
.env("GIT_CONFIG_GLOBAL", "/dev/null")
.env("GIT_CONFIG_SYSTEM", "/dev/null");
let out = c.output().with_context(|| format!("git {args:?}"))?;
ensure!(
out.status.success(),
"git {args:?} in {repo:?} exited {:?}: {}",
out.status.code(),
String::from_utf8_lossy(&out.stderr).trim()
);
Ok(String::from_utf8_lossy(&out.stdout)
.lines()
.map(|s| s.to_string())
.collect())
}
/// What one storm operation did.
///
/// A third answer beside ok/failed, and the reason this arm can be read at all
/// now: a **timeout is not a refusal**. Counting a stalled clone as a failure
/// would put it in the same bucket as "the server said no", and those two facts
/// need different people to look at them.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(super) enum OpOutcome {
Ok,
Failed,
TimedOut,
}
/// Run a git command against the appliance. Returns the outcome and stderr
/// rather than erroring, because during the op storm a refusal is data, not a
/// crash — and so, now, is a stall.
fn git_try(app: &Appliance, cwd: Option<&Path>, args: &[String]) -> (OpOutcome, String) {
let mut c = Command::new("git");
if let Some(d) = cwd {
c.current_dir(d);
}
git_env(&mut c, &app.key);
c.args(args);
match output_within(c, OP_TIMEOUT) {
Ok(Some(o)) => (
if o.status.success() {
OpOutcome::Ok
} else {
OpOutcome::Failed
},
String::from_utf8_lossy(&o.stderr).into_owned(),
),
Ok(None) => (
OpOutcome::TimedOut,
format!(
"no exit after {OP_TIMEOUT:?}; killed. This is the shape the arm used to hang \
in: the child alive, the container up, and nothing moving."
),
),
Err(e) => (OpOutcome::Failed, e.to_string()),
}
}
// ───────────────────────────────────────────────────────────────────────────
// import / export, judged by both receivers
// ───────────────────────────────────────────────────────────────────────────
/// The verdict on one member's round trip.
struct RoundTrip {
name: &'static str,
push_ms: f64,
clone_ms: f64,
/// Raw bytes gunnar sent on the wire for the export, as captured by the tee.
served_bytes: u64,
/// Objects gitoxide found in the raw served pack.
gix_objects: u64,
objects: u64,
refs: usize,
}
/// Push a member into the appliance. This is the IMPORT half.
fn import(app: &Appliance, m: &Member) -> Result<f64> {
let t = Instant::now();
let url = app.url(m.slug);
let (ok, err) = git_try(
app,
Some(&m.path),
&[
"push".into(),
"--mirror".into(),
"--quiet".into(),
url.clone(),
],
);
ensure!(
ok == OpOutcome::Ok,
"importing {} into the appliance by `git push --mirror {url}` {}: {}",
m.name,
match ok {
OpOutcome::TimedOut => "stalled",
_ => "failed",
},
err.trim()
);
Ok(t.elapsed().as_secs_f64() * 1000.0)
}
/// Clone a member back out, capturing the RAW served bytes on the way.
///
/// The capture is what lets gitoxide judge the same pack stock git judged. See
/// [`ssh_tee`].
fn export(app: &Appliance, m: &Member, dst: &Path, tee: &Path) -> Result<(f64, u64)> {
let _ = std::fs::remove_dir_all(dst);
let _ = std::fs::remove_file(tee);
let t = Instant::now();
let url = app.url(m.slug);
let mut c = Command::new("git");
// `transfer.fsckObjects` makes stock git run index-pack --strict on the
// served bytes. Without it a clone accepts objects fsck would reject and the
// "stock git judged it" claim is empty.
c.args([
"-c",
"transfer.fsckObjects=true",
"-c",
"fetch.fsckObjects=true",
"clone",
"--quiet",
"--mirror",
]);
c.arg(&url).arg(dst);
git_env(&mut c, &app.key);
c.env("GUNNAR_FETKAOS_TEE", tee);
// The tee shim replaces ssh, runs the real ssh, and copies the
// server->client stream to GUNNAR_FETKAOS_TEE as it passes.
c.env(
"GIT_SSH_COMMAND",
format!(
"{} {} {}",
std::env::current_exe()?.display(),
"--fetkaos-ssh-tee",
app.key.display()
),
);
let out = c.output().context("git clone --mirror (export)")?;
ensure!(
out.status.success(),
"exporting {} by `git clone --mirror {url}` failed: {}\n\
If this is `fatal: pack is corrupted (SHA1 mismatch)` or a gitoxide index \
failure at >=16 packs, that is the OPEN write_pack_copy defect another lane \
holds, not this one.",
m.name,
String::from_utf8_lossy(&out.stderr).trim()
);
let ms = t.elapsed().as_secs_f64() * 1000.0;
let served = std::fs::metadata(tee).map(|md| md.len()).unwrap_or(0);
Ok((ms, served))
}
/// Stock git's verdict, plus the comparison fsck alone cannot make.
fn judge_stock(m: &Member, dst: &Path) -> Result<(u64, usize)> {
let out = Command::new("git")
.current_dir(dst)
.args(["fsck", "--full", "--strict", "--no-progress"])
.env("GIT_CONFIG_GLOBAL", "/dev/null")
.env("GIT_CONFIG_SYSTEM", "/dev/null")
.output()?;
// `fsck` reports dangling objects on stderr and still exits 0; only a
// non-zero exit is a verdict of corrupt.
ensure!(
out.status.success(),
"stock git `fsck --full --strict` REJECTED the export of {}: {}",
m.name,
String::from_utf8_lossy(&out.stderr).trim()
);
// An export that dropped an object still fscks clean if nothing left in it
// refers to that object. So compare the sets.
let mut src: Vec<String> = git_lines(&m.path, &["rev-list", "--objects", "--all"])?
.iter()
.filter_map(|l| l.split_whitespace().next().map(|s| s.to_string()))
.collect();
let mut got: Vec<String> = git_lines(dst, &["rev-list", "--objects", "--all"])?
.iter()
.filter_map(|l| l.split_whitespace().next().map(|s| s.to_string()))
.collect();
src.sort();
src.dedup();
got.sort();
got.dedup();
if red("dropobject") && !got.is_empty() {
got.pop();
}
ensure!(
src == got,
"the export of {} does not hold what the source holds: source {} objects, \
export {} objects, {} missing. `fsck` passed on both, which is exactly why \
this comparison exists.",
m.name,
src.len(),
got.len(),
src.iter().filter(|o| !got.contains(o)).count()
);
let srefs = git_lines(
&m.path,
&["for-each-ref", "--format=%(refname) %(objectname)"],
)?;
let mut drefs = git_lines(dst, &["for-each-ref", "--format=%(refname) %(objectname)"])?;
if red("dropref") && !drefs.is_empty() {
drefs.pop();
}
let smap: BTreeMap<&str, &str> = srefs.iter().filter_map(|l| l.split_once(' ')).collect();
let dmap: BTreeMap<&str, &str> = drefs.iter().filter_map(|l| l.split_once(' ')).collect();
ensure!(
smap == dmap,
"the export of {} does not carry the source's refs: {} in, {} out. Missing: {:?}",
m.name,
smap.len(),
dmap.len(),
smap.keys()
.filter(|k| !dmap.contains_key(*k))
.take(8)
.collect::<Vec<_>>()
);
Ok((got.len() as u64, dmap.len()))
}
/// gitoxide's verdict on the RAW bytes gunnar served.
///
/// Not on git's rewrite of them: `index-pack --fix-thin` appends missing bases
/// and rewrites the trailer, so a pack git repaired on the way in is not the
/// pack gunnar sent, and the repair is the very thing worth seeing.
fn judge_gitoxide(m: &Member, tee: &Path, scratch: &Path) -> Result<u64> {
let raw = std::fs::read(tee)
.with_context(|| format!("the ssh tee wrote nothing for {}; nothing to judge", m.name))?;
let pack = demux_pack(&raw).with_context(|| {
format!(
"could not find a packfile in the {} bytes gunnar served for {}",
raw.len(),
m.name
)
})?;
ensure!(
pack.len() > 12,
"the pack gunnar served for {} is {} bytes, which is not a pack",
m.name,
pack.len()
);
let mut bytes = pack;
if red("corruptpack") {
// Flip a byte in the middle of the object data, past the header.
let at = bytes.len() / 2;
bytes[at] ^= 0xff;
}
std::fs::create_dir_all(scratch)?;
let mut cur = std::io::Cursor::new(&bytes[..]);
// An EMPTY store, on purpose: `index_thin_pack` consults it only for delta
// bases the pack itself does not carry, and this check is precisely about
// whether gunnar sent a self-contained pack. `MemGitServe` replaces
// `MemObjectStore` — one type answering both halves of the contract.
let store = gunnar_store::mem::MemGitServe::new(m.kind);
let indexed = index_thin_pack(
&mut cur,
Some(scratch),
crate::GunnarObjects(std::sync::Arc::new(store)),
&IndexOptions::new(m.kind),
None,
)
.with_context(|| {
format!(
"GITOXIDE REJECTED the pack gunnar served for {} ({} bytes) that stock git \
accepted. The two receivers disagreeing has been the signal every time; \
gitoxide refusing a thin base carried later in the same pack is upstream \
issue 2882 and is answered on our side by changing the SENDER.",
m.name,
bytes.len()
)
})?;
if let Some(keep) = &indexed.keep_path {
let _ = std::fs::remove_file(keep);
}
Ok(indexed.num_objects as u64)
}
/// Pull the packfile out of a captured protocol-v2 fetch stream.
///
/// gunnar serves the pack inside sideband band 1, in pkt-lines. This walks the
/// pkt-line framing and concatenates band-1 payload, which is the packfile.
fn demux_pack(raw: &[u8]) -> Result<Vec<u8>> {
let mut out = Vec::new();
let mut i = 0usize;
let mut in_pack = false;
while i + 4 <= raw.len() {
let hex = std::str::from_utf8(&raw[i..i + 4]).unwrap_or("");
let n = match usize::from_str_radix(hex, 16) {
Ok(n) => n,
Err(_) => {
i += 1;
continue;
}
};
if n == 0 || n == 1 || n == 2 {
i += 4;
continue;
}
if n < 4 || i + n > raw.len() {
i += 1;
continue;
}
let body = &raw[i + 4..i + n];
if !body.is_empty() {
if body[0] == 1 {
out.extend_from_slice(&body[1..]);
in_pack = true;
} else if body.starts_with(b"PACK") {
out.extend_from_slice(body);
in_pack = true;
}
}
i += n;
}
if !in_pack {
// Not sidebanded: find the raw PACK signature.
if let Some(at) = raw.windows(4).position(|w| w == b"PACK") {
out.extend_from_slice(&raw[at..]);
}
}
ensure!(!out.is_empty(), "no band-1 payload and no PACK signature");
Ok(out)
}
/// The `--fetkaos-ssh-tee` role: stand in for `ssh`, copy the server->client
/// stream to `GUNNAR_FETKAOS_TEE` as it passes, and be otherwise transparent.
///
/// Dispatched from `main()` before the bench runner, the same way the three
/// head-to-head server roles are.
pub(crate) fn ssh_tee(argv: &[String]) -> Result<()> {
// argv: <key> <ssh args...>
ensure!(
!argv.is_empty(),
"--fetkaos-ssh-tee needs the identity path"
);
let key = &argv[0];
let rest = &argv[1..];
let tee_path = std::env::var("GUNNAR_FETKAOS_TEE").ok();
let mut c = Command::new("ssh");
c.args([
"-i",
key,
"-o",
"StrictHostKeyChecking=no",
"-o",
"UserKnownHostsFile=/dev/null",
"-o",
"LogLevel=ERROR",
"-o",
"BatchMode=yes",
]);
c.args(rest);
c.stdin(Stdio::inherit())
.stdout(Stdio::piped())
.stderr(Stdio::inherit());
let mut child = c.spawn().context("spawning ssh under the fetkaos tee")?;
let mut from_server = child.stdout.take().expect("piped");
let mut sink = tee_path
.as_ref()
.map(|p| std::fs::File::create(p))
.transpose()?;
let mut stdout = std::io::stdout();
let mut buf = vec![0u8; 1 << 16];
loop {
let n = from_server.read(&mut buf)?;
if n == 0 {
break;
}
stdout.write_all(&buf[..n])?;
stdout.flush()?;
if let Some(f) = sink.as_mut() {
f.write_all(&buf[..n])?;
}
}
if let Some(f) = sink.as_mut() {
f.flush()?;
}
let st = child.wait()?;
std::process::exit(st.code().unwrap_or(1));
}
// ───────────────────────────────────────────────────────────────────────────
// the op storm
// ───────────────────────────────────────────────────────────────────────────
/// The verbs, and how often each is drawn.
///
/// The weights are not arbitrary: a full clone of `linux` is minutes, so it is
/// the rare tail rather than the common case, and the cheap negotiated verbs
/// carry the count. The mix is what makes 100 000 operations finish while still
/// reaching every code path.
const OP_MIX: &[(Op, u32)] = &[
(Op::LsRemote, 40),
(Op::FetchUpToDate, 28),
(Op::FetchShallow, 12),
(Op::PushNoop, 8),
(Op::CloneSingleBranch, 6),
(Op::CloneShallow, 4),
(Op::LsRefsUnborn, 2),
];
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
enum Op {
/// `ls-remote`: the ref advertisement, cheapest real round trip.
LsRemote,
/// A negotiated fetch that has everything: exercises the `have` path and
/// must transfer no objects.
FetchUpToDate,
/// `fetch --depth=N`: shallow negotiation, a different closure.
FetchShallow,
/// A push of nothing: receive-pack's ref advertisement and report.
PushNoop,
/// `clone --single-branch`: a narrower want set.
CloneSingleBranch,
/// `clone --depth=1`: the shallow export path.
CloneShallow,
/// ls-remote for a ref that does not exist: the unborn/empty answer.
LsRefsUnborn,
}
impl Op {
/// Does this operation move objects, rather than only refs?
///
/// The ones that do may only target a `transfer` member — see
/// [`Member::transfer`] for the 8 GB-per-operation bug that taught this.
fn transfers_objects(self) -> bool {
match self {
Op::LsRemote | Op::LsRefsUnborn | Op::PushNoop => false,
Op::FetchUpToDate | Op::FetchShallow | Op::CloneSingleBranch | Op::CloneShallow => true,
}
}
fn key(self) -> &'static str {
match self {
Op::LsRemote => "ls_remote",
Op::FetchUpToDate => "fetch_uptodate",
Op::FetchShallow => "fetch_shallow",
Op::PushNoop => "push_noop",
Op::CloneSingleBranch => "clone_single_branch",
Op::CloneShallow => "clone_shallow",
Op::LsRefsUnborn => "ls_refs_unborn",
}
}
}
/// One scheduled operation. A pure function of the seed and the index.
#[derive(Clone)]
struct Planned {
idx: u64,
op: Op,
member: usize,
depth: u32,
}
/// Draw the WHOLE schedule from the seed before any worker starts.
///
/// This is what makes the run replayable: operation `i` is the same operation at
/// the same repository with the same parameters on every run of this seed, no
/// matter how the threads interleave. Concurrency varies the interleaving, which
/// is the thing under test; it must not vary the workload, or a failure cannot
/// be reproduced.
fn plan(seed: u64, ops: u64, members: &[Member]) -> Vec<Planned> {
let any: Vec<usize> = (0..members.len()).collect();
let transfer: Vec<usize> = members
.iter()
.enumerate()
.filter(|(_, m)| m.transfer)
.map(|(i, _)| i)
.collect();
let total: u32 = OP_MIX.iter().map(|(_, w)| w).sum();
let mut rng = Lcg::new(seed);
let mut out = Vec::with_capacity(ops as usize);
for idx in 0..ops {
let roll = (rng.next_u64() % total as u64) as u32;
let mut acc = 0u32;
let mut op = OP_MIX[0].0;
for (candidate, w) in OP_MIX {
acc += w;
if roll < acc {
op = *candidate;
break;
}
}
// The draw is always made, from whichever pool the op is allowed, so the
// stream of random numbers does not depend on the pool sizes and a seed
// stays comparable across corpora of the same shape.
let pool = if op.transfers_objects() {
&transfer
} else {
&any
};
let member = pool[(rng.next_u64() % pool.len() as u64) as usize];
let depth = 1 + (rng.next_u64() % 8) as u32;
out.push(Planned {
idx,
op,
member,
depth,
});
}
out
}
/// What the storm produced.
#[derive(Default)]
struct Storm {
done: u64,
failed: u64,
/// Operations killed at [`OP_TIMEOUT`]. Kept apart from `failed` on
/// purpose: a refusal is the server answering and a timeout is the server
/// not answering, and one number for both is how this arm's forty-minute
/// stall was first read as the harness reaping the run.
timed_out: u64,
/// Operations never issued because [`STORM_DEADLINE`] passed. Nonzero means
/// the arm's numbers are a partial sample and must not be compared with a
/// complete one.
abandoned: u64,
per_op: BTreeMap<&'static str, u64>,
first_failure: Option<String>,
}
/// Run the planned operations across `workers` threads.
///
/// The multicore rule: the fan-out is `std::thread::scope` driving `git` child
/// processes, which is what `forge/concurrent.rs` does. rayon is forbidden
/// repo-wide and `xtask/src/guard.rs` enforces it by name.
fn storm(
app: &Appliance,
members: &[Member],
schedule: &[Planned],
workers: usize,
scratch: &Path,
) -> Result<Storm> {
let next = AtomicUsize::new(0);
let failed = AtomicU64::new(0);
let timed_out = AtomicU64::new(0);
let abandoned = AtomicU64::new(0);
let done = AtomicU64::new(0);
let deadline = Instant::now() + STORM_DEADLINE;
let per_op: Mutex<BTreeMap<&'static str, u64>> = Mutex::new(BTreeMap::new());
let first: Mutex<Option<String>> = Mutex::new(None);
// `killmid` kills the container part-way through, to prove the storm can
// report a dead server rather than counting failures quietly.
let kill_at = if red("killmid") {
schedule.len() / 2
} else {
usize::MAX
};
std::thread::scope(|scope| {
for w in 0..workers {
let next = &next;
let failed = &failed;
let timed_out = &timed_out;
let abandoned = &abandoned;
let done = &done;
let per_op = &per_op;
let first = &first;
scope.spawn(move || {
let cwd = scratch.join(format!("w{w}"));
let _ = std::fs::create_dir_all(&cwd);
loop {
let i = next.fetch_add(1, Ordering::Relaxed);
if i >= schedule.len() {
break;
}
// **The wall clock, checked between operations.** This is
// what makes `thread::scope`'s join below terminate: the
// join cannot be cancelled and has no deadline of its own,
// so the only way out is for every worker to stop pulling
// work. Each one counts what it declined to issue, so the
// arm reports a partial sample as a partial sample instead
// of as a complete one that happened to be small.
if Instant::now() >= deadline {
abandoned.fetch_add(1, Ordering::Relaxed);
continue;
}
if i == kill_at {
app.kill();
}
let p = &schedule[i];
let m = &members[p.member];
let url = app.url(m.slug);
let dst = cwd.join(format!("op{}", p.idx));
let _ = std::fs::remove_dir_all(&dst);
let args: Vec<String> = match p.op {
Op::LsRemote => vec!["ls-remote".into(), url.clone()],
Op::LsRefsUnborn => vec![
"ls-remote".into(),
url.clone(),
"refs/heads/fetkaos-no-such-ref".into(),
],
Op::FetchUpToDate => vec![
"-c".into(),
"protocol.version=2".into(),
"fetch".into(),
"--quiet".into(),
url.clone(),
"HEAD".into(),
],
Op::FetchShallow => vec![
"fetch".into(),
"--quiet".into(),
format!("--depth={}", p.depth),
url.clone(),
"HEAD".into(),
],
// An explicit refspec, so the op does not depend on
// the client's `push.default`. `--dry-run` still opens
// receive-pack and takes its ref advertisement, which
// is the half of the transport `ls-remote` never
// touches.
Op::PushNoop => vec![
"push".into(),
"--quiet".into(),
"--dry-run".into(),
url.clone(),
format!("HEAD:refs/heads/fetkaos-w{w}"),
],
Op::CloneSingleBranch => vec![
"clone".into(),
"--quiet".into(),
"--bare".into(),
"--single-branch".into(),
"--depth=1".into(),
url.clone(),
dst.display().to_string(),
],
Op::CloneShallow => vec![
"clone".into(),
"--quiet".into(),
"--bare".into(),
format!("--depth={}", p.depth),
url.clone(),
dst.display().to_string(),
],
};
// `fetch` and `push` need a repository to run in.
let need_repo =
matches!(p.op, Op::FetchUpToDate | Op::FetchShallow | Op::PushNoop);
let run_in = if need_repo {
// A REAL repository with a commit, not an empty bare
// one. `git push --dry-run <url>` from an empty bare
// repo fails with "the current branch has no upstream
// branch" — 54 of 200 operations in the first `tiny`
// run were this, and they were a defect in the op, not
// in gunnar. `fetch` needs somewhere to put objects and
// `push` needs something to offer; one holder does both.
// ONE HOLDER PER OBJECT FORMAT. A sha1 repository
// cannot push to or fetch from a sha256 one — stock git
// says "the receiving end does not support this
// repository's hash algorithm" — so a single holder
// makes every operation against the sha256 member fail
// for a reason that is about the harness, not gunnar.
// The storm deliberately moves objects in both formats
// (see `Member::transfer`), so it needs both holders.
let fmt = match m.kind {
Kind::Sha256 => "sha256",
_ => "sha1",
};
let holder = cwd.join(format!("holder-{fmt}"));
if !holder.exists() {
let _ = Command::new("git")
.args(["init", "--quiet", &format!("--object-format={fmt}")])
.arg(&holder)
.output();
let _ = Command::new("git")
.current_dir(&holder)
.args([
"-c",
"user.email=fetkaos@the desktop",
"-c",
"user.name=fetkaos",
"commit",
"--quiet",
"--allow-empty",
"-m",
"fetkaos holder",
])
.output();
}
Some(holder)
} else {
None
};
let (outcome, err) = git_try(app, run_in.as_deref(), &args);
let _ = std::fs::remove_dir_all(&dst);
done.fetch_add(1, Ordering::Relaxed);
per_op
.lock()
.map(|mut g| *g.entry(p.op.key()).or_insert(0) += 1)
.ok();
if outcome != OpOutcome::Ok {
if outcome == OpOutcome::TimedOut {
timed_out.fetch_add(1, Ordering::Relaxed);
}
failed.fetch_add(1, Ordering::Relaxed);
if let Ok(mut g) = first.lock() {
if g.is_none() {
*g = Some(format!(
"operation #{} ({} on {}) {}: {}",
p.idx,
p.op.key(),
m.name,
match outcome {
OpOutcome::TimedOut => "STALLED",
_ => "failed",
},
err.trim()
));
}
}
}
}
});
}
});
Ok(Storm {
done: done.load(Ordering::Relaxed),
failed: failed.load(Ordering::Relaxed),
timed_out: timed_out.load(Ordering::Relaxed),
abandoned: abandoned.load(Ordering::Relaxed),
per_op: per_op.into_inner().unwrap_or_default(),
first_failure: first.into_inner().ok().flatten(),
})
}
// ───────────────────────────────────────────────────────────────────────────
// the arm
// ───────────────────────────────────────────────────────────────────────────
struct FetKaosMonkey;
impl Bencher for FetKaosMonkey {
fn id(&self) -> &'static str {
"gunnar.fetkaos_monkey"
}
fn run(&self) -> Result<BenchResult> {
ensure!(
!install_skipped(),
"gunnar.fetkaos_monkey ran with GUNNAR_BENCH_SKIP_INSTALL set; the arm must \
be filtered out, not executed and reported"
);
let seed = env_u64("GUNNAR_BENCH_FETKAOS_SEED", DEFAULT_SEED);
let ops = env_u64("GUNNAR_BENCH_FETKAOS_OPS", DEFAULT_OPS);
let workers = env_u64("GUNNAR_BENCH_FETKAOS_WORKERS", DEFAULT_WORKERS as u64) as usize;
ensure!(workers > 0, "workers must be > 0");
ensure!(
ops > 0,
"a Fet Kaos Monkey run with zero operations would report green having done \
nothing"
);
let members = survey()?;
let work = install_scratch("fetkaos");
let _ = std::fs::remove_dir_all(&work);
std::fs::create_dir_all(&work)?;
// A port derived from the pid, so two lanes on this box do not collide.
let port = 20000 + (std::process::id() % 10000) as u16;
let app = up(&work, &members, port)?;
let boot_ms = app.started.elapsed().as_secs_f64() * 1000.0;
// ── what gunnar must say ──────────────────────────────────────────
let log_checked = assert_log(&app)?;
// ── import / export, judged by both receivers ─────────────────────
let mut trips = Vec::new();
for m in &members {
// EVERY failure below is annotated with the container's log BEFORE it
// propagates, because `Appliance::drop` removes the container and the
// log dies with it.
//
// Measured on the desktop 2026-08-05: the full-scale run failed with
// `Received disconnect … gunnar is shutting down` during the `linux`
// import, and by the time the error reached the runner the container
// was gone and the reason with it. The error said what the CLIENT
// saw; nothing said what the SERVER said as it went. A chaos monkey
// whose evidence is destroyed by its own cleanup can report that
// something broke and never what.
let push_ms = with_log(&app, || import(&app, m))?;
let dst = work.join(format!("export-{}", m.name));
let tee = work.join(format!("tee-{}.bin", m.name));
let (clone_ms, served_bytes) = with_log(&app, || export(&app, m, &dst, &tee))?;
let (objects, refs) = with_log(&app, || judge_stock(m, &dst))?;
let gix_objects = with_log(&app, || {
judge_gitoxide(m, &tee, &work.join(format!("gix-{}", m.name)))
})?;
// The two receivers must agree on how many objects crossed.
ensure!(
gix_objects > 0,
"gitoxide indexed 0 objects out of the pack gunnar served for {}, while \
stock git accepted it and found {}. Zero is not a measurement.",
m.name,
objects
);
trips.push(RoundTrip {
name: m.name,
push_ms,
clone_ms,
served_bytes,
gix_objects,
objects,
refs,
});
// The exports are gigabytes. Do not keep three of them alive.
let _ = std::fs::remove_dir_all(&dst);
let _ = std::fs::remove_file(&tee);
}
// ── gunnar must have SAID it served, not merely have served ───────
let transfer_checked = assert_log_set(&app, REQUIRED_TRANSFER_LOG, "round-trip")?;
// The tee is only evidence if it caught everything. gunnar counts what
// it sent; the tee counts what arrived. They must agree.
let log = app.log()?;
let logged = logged_bytes_out(&log);
let teed: u64 = trips.iter().map(|t| t.served_bytes).sum();
ensure!(
logged > 0,
"gunnar logged 0 bytes_out across every upload-pack, while the tee caught \
{teed}. One of the two is not measuring the export."
);
ensure!(
logged == teed,
"gunnar says it sent {logged} bytes across its upload-pack lines; the tee \
caught {teed}. The tee is what gitoxide judged, so a gap means gitoxide \
judged something other than what gunnar served and every verdict above it \
is void."
);
// ── the hundred thousand ──────────────────────────────────────────
let schedule = plan(seed, ops, &members);
ensure!(
schedule.len() as u64 == ops,
"the schedule holds {} operations, not the {ops} asked for",
schedule.len()
);
let storm_scratch = work.join("storm");
std::fs::create_dir_all(&storm_scratch)?;
let t = Instant::now();
let s = storm(&app, &members, &schedule, workers, &storm_scratch)?;
let storm_ms = t.elapsed().as_secs_f64() * 1000.0;
let _ = std::fs::remove_dir_all(&storm_scratch);
// **The arm's own deadline, reported before anything else.**
//
// This used to be the place the arm did not reach: a single stalled
// `git` child held `thread::scope`'s join open and the process sat at
// 0.2% CPU for 41 and 54 minutes with its container alive, which is why
// no full sweep had ever finished. It is now bounded twice —
// `OP_TIMEOUT` per operation and `STORM_DEADLINE` across the storm — so
// the failure mode is a red arm that names the operation, and this is
// where it says so.
ensure!(
s.abandoned == 0,
"the op storm hit its {STORM_DEADLINE:?} deadline with {} of {ops} operations still \
unissued ({} completed, {} of those killed at {OP_TIMEOUT:?}). The numbers from \
this run are a partial sample and must not be compared with a complete one. First \
stall or failure: {}",
s.abandoned,
s.done,
s.timed_out,
s.first_failure.as_deref().unwrap_or("none recorded"),
);
ensure!(
s.done == ops,
"the op storm completed {} of {ops} operations",
s.done
);
// A chaos run where everything failed is not a pass, and a chaos run
// where the server died is not a pass either.
if let Some(f) = &s.first_failure {
bail!(
"{} of {} operations failed against the appliance. First: {}\n\
The container's log at that point:\n{}",
s.failed,
ops,
f,
app.log().unwrap_or_default()
);
}
ensure!(
app.running(),
"the appliance was not alive at the end of the storm; it died during the \
{ops} operations and the ops that ran after it died cannot be trusted. Log:\n{}",
app.log().unwrap_or_default()
);
// The log must STILL be intact after the storm: a server that stopped
// saying anything is the silent-container failure again.
let after = assert_log(&app)?;
ensure!(after == log_checked, "the required log set changed mid-run");
// ── metrics ───────────────────────────────────────────────────────
let fat: Vec<&RoundTrip> = trips
.iter()
.filter(|t| members.iter().any(|m| m.name == t.name && m.fat))
.collect();
let total_objects: u64 = trips.iter().map(|t| t.objects).sum();
let total_served: u64 = trips.iter().map(|t| t.served_bytes).sum();
let mut metrics: Vec<(&'static str, Value)> = vec![
// The seed must be a NUMBER or it never reaches the warehouse and
// the run is unreplayable.
("seed_lo", json!((seed & 0xffff_ffff) as f64)),
("seed_hi", json!((seed >> 32) as f64)),
("ops", json!(ops as f64)),
("ops_failed", json!(s.failed as f64)),
// Reported even though the `ensure!`s above refuse a run with
// either of these nonzero. A metric that only ever appears as 0 is
// still the difference between "this arm cannot stall" and "this
// arm has not stalled yet", and the second is the honest claim.
("ops_timed_out", json!(s.timed_out as f64)),
("ops_abandoned", json!(s.abandoned as f64)),
("workers", json!(workers as f64)),
("storm_ms", json!(storm_ms)),
(
"storm_ops_per_sec",
json!(ops as f64 / (storm_ms / 1000.0).max(1e-9)),
),
("boot_ms", json!(boot_ms)),
("log_required", json!(log_checked as f64)),
("log_required_transfer", json!(transfer_checked as f64)),
("log_bytes_out", json!(logged as f64)),
("corpus_members", json!(members.len() as f64)),
("corpus_fat_members", json!(fat.len() as f64)),
(
"corpus_pack_bytes",
json!(members.iter().map(|m| m.bytes).sum::<u64>() as f64),
),
(
"corpus_packs",
json!(members.iter().map(|m| m.packs).sum::<usize>() as f64),
),
("roundtrip_objects", json!(total_objects as f64)),
("served_bytes", json!(total_served as f64)),
];
for t in &trips {
metrics.push((leak(format!("{}_push_ms", t.name)), json!(t.push_ms)));
metrics.push((leak(format!("{}_clone_ms", t.name)), json!(t.clone_ms)));
metrics.push((leak(format!("{}_objects", t.name)), json!(t.objects as f64)));
metrics.push((leak(format!("{}_refs", t.name)), json!(t.refs as f64)));
metrics.push((
leak(format!("{}_gix_objects", t.name)),
json!(t.gix_objects as f64),
));
metrics.push((
leak(format!("{}_served_bytes", t.name)),
json!(t.served_bytes as f64),
));
}
for (k, v) in &s.per_op {
metrics.push((leak(format!("op_{k}")), json!(*v as f64)));
}
Ok(result(self.id(), metrics))
}
}
nornir_bench::register_bench!(FetKaosMonkey);