bench/examples/opstorm/mod.rs
//! # `gunnar.opstorm` — the whole-surface Operation Storm
//!
//! `fetkaos` (its sibling arm) storms exactly ONE surface: the pack/object
//! graph, through the git client, against a disposable gunnar container. The
//! specification asked for the other thing — massive sequence testing, not one
//! function at a time but whole OPERATIONS: a massive seeded SEQUENCE,
//! not one code path. This arm widens the alphabet to gunnar's whole externally
//! reachable surface and interleaves it under one schedule:
//!
//! * **git transport** — `ls-remote`, shallow/full `clone`, shallow `fetch`,
//! branch `push`, branch `push :delete` — over real SSH against a running
//! `gunnar serve`.
//! * **the gRPC control plane** — `repo {create,list,get,delete}`,
//! `namespace create`, `user {add,remove}`, `grant`, `replication list`,
//! and the authenticated reads `health`/`stats`/`storage` — every one a real
//! `gunnar control … --at <grpc>` call, signed with a fresh single-use
//! `sshsig` per call, against the SAME live server the git ops hit.
//! * **replication / abdication** — a single-node `replication abdicate`
//! (freeze → empty-roster demote) and `replication promote`, proving the role
//! flip actually stops and restarts client writes.
//!
//! ## The determinism contract is fetkaos's, unchanged
//!
//! **The whole schedule is a pure function of the seed.** [`plan`] draws every
//! operation — its kind, its target repository, its depth, its per-op nonce —
//! from one [`super::Lcg`] *before any worker starts*, so operation `i` is the
//! same operation with the same parameters on every run of that seed, no matter
//! how the 16 workers interleave. Concurrency varies the *interleaving*, which
//! is the thing under test; it must not vary *which* operations ran, or a
//! failure is an anecdote instead of a replay. The seed reaches the warehouse as
//! **numbers** (`seed_lo`/`seed_hi`) because `build_bench_results_batch` drops
//! every non-numeric metric, and a failure names the **seed + op index** so it
//! replays from the seed alone.
//!
//! The multicore rule (gatling, never rayon): the fan-out is
//! `std::thread::scope` driving child processes, the same shape `fetkaos` and
//! `forge/concurrent.rs` use. rayon is forbidden repo-wide.
//!
//! One implementation, not two: this arm does **not** re-implement the chaos
//! harness. The process-with-deadline runner [`super::fetkaos::output_within`],
//! the three-way [`super::fetkaos::OpOutcome`], the git-over-ssh environment
//! [`super::fetkaos::git_env`] and the log helpers `strip_ansi`/`tail` are
//! `pub(super)` in the fetkaos arm and imported here rather than twinned. The
//! seed generator, the metric envelope and the install scratch dir come from the
//! harness (`super::{Lcg, install_scratch, result, leak}`).
//!
//! ## The subject: a `gunnar-solo` binary, run as a real process
//!
//! `fetkaos` boots the `gunnar-scratch` container because its subject is the
//! image atom. This arm's subject is the **control plane**, which is reachable
//! online only against a live `gunnar serve` over gRPC (the offline throwaway-
//! container verbs take an exclusive lock and cannot run while `serve` holds the
//! data root). So it drives the binary directly, exactly as
//! `<ops>/checks/abdication-drill.sh` and `failover-drill.sh` do, reusing their
//! proven `serve --data-root … --ssh-bind … --grpc-bind … --admin-key @…` shape
//! (reuse — that shell drill already proves the replication mechanism end to
//! end; this arm's contribution is interleaving the control plane INTO a seeded
//! git-path storm and proving the guards fire). The binary is
//! `gunnar-solo` (server + client + control in one), located via
//! `GUNNAR_OPSTORM_CLI`. The data root and the server process are thrown away on
//! drop.
//!
//! ## It is a defect finder, not a scoreboard
//!
//! Its primary output is a hard error naming WHAT broke at WHICH seed + op
//! index, not a throughput number. Every successful clone is judged by an
//! **applied** `git fsck` of the object graph it produced (measured, not
//! asserted — a torn clone can exit 0). Every churn op reads its own applied
//! state back (a deleted repo must be gone; a removed user must be gone). The
//! control plane's liveness is checked *throughout* the git storm — its own
//! runtime, separate from the SSH listener, is what must keep it answered.
//!
//! ## Red conditions — each has been SEEN to fire, naming the op index
//!
//! | `GUNNAR_BENCH_OPSTORM_RED=` | what it breaks | what must go red |
//! |---|---|---|
//! | `killmid` | SIGKILLs the server at the schedule midpoint | the storm, naming the op index the server died on |
//! | `corruptclone` | flips a byte in every cloned pack before fsck | the clone judge, naming the op index whose graph failed fsck |
//! | `deafcontrol` | routes the control-liveness probe at the SSH port, not the gRPC control runtime | the liveness detector, naming the op index control did not answer (the control-plane-starvation failure mode) |
//! | `freezeaccepts` | asserts the freeze-refusal check against a NON-abdicated primary | the abdicate phase — a primary ACCEPTS the write the check calls refused |
//! | `tiny` | NOT a probe — the small harness the four probes are driven in (few ops, few arenas), since they need a running server first | nothing; it makes the reds reachable in a smoke |
//!
//! ## External-target mode — storm a running appliance as a client
//!
//! By default this arm spawns and OWNS its `gunnar serve` ([`up`]). Set
//! `GUNNAR_OPSTORM_TARGET_SSH=<host:port>` **and**
//! `GUNNAR_OPSTORM_TARGET_GRPC=<host:port>` (plus
//! `GUNNAR_OPSTORM_TARGET_ADMIN_KEY=<private key>` and, if the account is not
//! `admin`, `GUNNAR_OPSTORM_TARGET_ACCOUNT=<account>`) and it instead ATTACHES
//! ([`attach`]) and drives that already-running gunnar over the network as a
//! client — the shape used to storm a real git.example.test stand-in KVM. Only
//! the constructor branches; the whole alphabet, the `ctl()`/git helpers, the
//! seeded schedule, the fsck-clean clone judge and the liveness checks are
//! identical. It cleans up after itself (purges its arenas) on the shared box.
//!
//! Two probes need to OWN the server and so are **N/A** against an external
//! PID-1 appliance — `killmid` (SIGKILL its child) and `corruptclone` (the
//! owning-server corruption path). In external mode they SKIP LOUDLY
//! (`PROBE killmid: N/A against an external appliance …`), never a silent green.
//! The separate-control-runtime log line is likewise not readable across
//! the wire, so that one guard is skipped honestly rather than faked. Every
//! other probe (`deafcontrol`, `freezeaccepts`) and the whole positive alphabet
//! run unchanged.
//!
//! ## TODO(next): the primary+STANDBY pair storm
//!
//! v1 storms one node whole-surface. The abdicate probe here is single-node
//! (freeze → refuse → promote), which proves the role flip but NOT the
//! **abdicate-while-a-push-is-mid-flight to a real standby that auto-promotes**
//! case — the highest-value replication defect surface. That needs a SECOND
//! `gunnar serve`, a `--kind machine` deploy key enrolled on the standby, and
//! the abdicate flush window (`GUNNAR_ABDICATE_FLUSH_SECS`). It is deliberately
//! left out of v1 because a two-server rig doubles the arm's boot cost and the
//! mechanism is already proven end-to-end by `<ops>/checks/abdication-drill.sh`
//! (reuse it rather than twin it); the arm's delta there is interleaving the
//! abdicate under the seeded schedule against a concurrent mid-flight push,
//! which is the heavier follow-up. Honest partial over fake-complete.
use std::collections::BTreeMap;
use std::os::unix::fs::PermissionsExt;
use std::path::{Path, PathBuf};
use std::process::{Child, Command, Stdio};
use std::sync::atomic::{AtomicU64, AtomicUsize, Ordering};
use std::sync::Mutex;
use std::time::{Duration, Instant};
use anyhow::{bail, ensure, Context, Result};
use nornir_bench::api::Bencher;
use nornir_bench::BenchResult;
use serde_json::{json, Value};
// The chaos harness is fetkaos's, not twinned. These are `pub(super)`
// there and shared here.
use super::fetkaos::{git_env, output_within, strip_ansi, tail, OpOutcome};
use super::{install_scratch, install_skipped, leak, result, Lcg};
// ───────────────────────────────────────────────────────────────────────────
// knobs
// ───────────────────────────────────────────────────────────────────────────
/// the specification asked for a massive sequence. This is the default full-run count; a
/// smoke passes `GUNNAR_OPSTORM_OPS` a small number to prove the reds red-first
/// without a full storm on a loaded box.
const DEFAULT_OPS: u64 = 20_000;
/// Wide enough that the admission gate and the control runtime split are actually
/// contended, narrow enough that the desktop is not the bottleneck being measured.
const DEFAULT_WORKERS: usize = 16;
/// A fixed default so a run with no env set is still a *replay* of a known run.
const DEFAULT_SEED: u64 = 0x0057_0A05_0000_0001;
/// Arena repositories: pre-created, seeded, and the target of the read-mostly
/// git ops. Two shapes are enough here (this arm's subject is the control
/// plane, not the pack corpus — that is fetkaos's job).
const DEFAULT_ARENAS: usize = 4;
/// One control call may take this long before it is killed and counted. The
/// control plane's cheap verbs are milliseconds; a call that takes 30 s under
/// the storm is a stall — exactly what the control plane's own runtime prevents.
const CTL_TIMEOUT: Duration = Duration::from_secs(30);
/// One git op may take this long. Generous: a full clone of a seeded arena
/// under 16-way contention is legitimately slow; what this must catch is a
/// *stall*, not slowness.
const GIT_TIMEOUT: Duration = Duration::from_secs(120);
/// The whole storm's wall-clock ceiling. 16 workers that each stall do not bound
/// the arm; this does, and a nonzero `abandoned` count makes the row honest
/// about being a partial sample.
const STORM_DEADLINE: Duration = Duration::from_secs(20 * 60);
/// The gunnar log line that proves the control plane came up on its OWN runtime.
const CONTROL_PLANE_NEEDLE: &str = "grpc control plane listening on its own runtime";
/// A standby refuses a client push with this. The abdicate phase asserts it.
const STANDBY_REFUSAL: &str = "this box is a standby";
fn red(which: &str) -> bool {
std::env::var("GUNNAR_BENCH_OPSTORM_RED")
.map(|v| v.split(',').any(|s| s.trim() == which))
.unwrap_or(false)
}
fn env_u64(key: &str, default: u64) -> u64 {
std::env::var(key)
.ok()
.and_then(|v| v.parse().ok())
.unwrap_or(default)
}
/// `gunnar-solo`: the one binary with `serve`, `control` and a git client all
/// three. Located, never built here (fetkaos does not build its image either —
/// the subject is required, and its absence is a hard error, not a green row).
fn locate_cli() -> Result<PathBuf> {
if let Ok(p) = std::env::var("GUNNAR_OPSTORM_CLI") {
let pb = PathBuf::from(&p);
ensure!(
pb.exists(),
"GUNNAR_OPSTORM_CLI={p:?} does not exist. Point it at a `gunnar-solo` binary."
);
return Ok(pb);
}
// One convenience fallback, off the bench's own target dir, before the hard
// error — so a `CARGO_TARGET_DIR=… cargo run` finds the sibling build.
if let Ok(t) = std::env::var("CARGO_TARGET_DIR") {
let pb = PathBuf::from(t).join("release/gunnar-solo");
if pb.exists() {
return Ok(pb);
}
}
bail!(
"no `gunnar-solo` binary. This arm's subject is the CONTROL PLANE, reachable online \
only against a live `gunnar serve`, so it drives the binary directly (as \
`<ops>/checks/abdication-drill.sh does). Build it and point GUNNAR_OPSTORM_CLI at it:\n \
cargo build --release -p gunnar-cli --bin gunnar-solo\n \
GUNNAR_OPSTORM_CLI=<target>/release/gunnar-solo nornir bench run gunnar.opstorm\n\
`gunnar-solo` is the server+client+control binary; `gunnar` (client-only) and a \
`--features server` appliance build each carry only half and cannot both serve and \
drive the control plane."
)
}
fn opstorm_scratch() -> PathBuf {
install_scratch("opstorm")
}
// ───────────────────────────────────────────────────────────────────────────
// the server, run as a real process and thrown away
// ───────────────────────────────────────────────────────────────────────────
/// A `gunnar serve` the arm owns, OR an external appliance it was merely given.
///
/// The two are the same subject to every operation in the alphabet — the
/// `ctl()`/git helpers do not care which — so they share one struct and branch
/// only on `external` where ownership actually matters: the two owning-probes
/// (`killmid`/`corruptclone`), the on-disk log, and the teardown: the storm
/// is not forked, only the constructor is (`up` vs `attach`).
struct Server {
cli: PathBuf,
/// The data root — `Some` for a self-spawned server (removed on `Drop`),
/// `None` for an external appliance this arm does not own.
data: Option<PathBuf>,
/// Admin PRIVATE key path; its public half is enrolled as the account
/// principal (`admin` for a self-spawn, the bootstrapped account for an
/// external target).
key: PathBuf,
/// The git-transport (SSH) host and port. `127.0.0.1` for a self-spawn,
/// whatever `GUNNAR_OPSTORM_TARGET_SSH` names for an external target.
ssh_host: String,
ssh_port: u16,
/// The control-plane (gRPC) host and port.
grpc_host: String,
grpc_port: u16,
/// The account whose namespace the arena/churn repos live in, and the git
/// principal that reaches it. `admin` for a self-spawn.
account: String,
/// The captured server log — `Some` only for a self-spawned server (an
/// external PID-1 appliance's log is not reachable from the host).
log: Option<PathBuf>,
/// `Some` for a self-spawned server (held for `kill` (killmid) and the
/// disposable-teardown `Drop`); `None` for an external target this arm must
/// neither kill nor tear down.
child: Option<Mutex<Child>>,
/// True when the subject is an external appliance given to the arm, not a
/// server this arm spawned and owns.
external: bool,
started: Instant,
}
impl Server {
fn grpc_addr(&self) -> String {
format!("{}:{}", self.grpc_host, self.grpc_port)
}
fn ssh_addr(&self) -> String {
format!("{}:{}", self.ssh_host, self.ssh_port)
}
/// `ssh://<account>@<host>:<ssh>/<account>/<repo>` — the git URL, the
/// account's own namespace.
fn url(&self, repo: &str) -> String {
format!(
"ssh://{}@{}:{}/{}/{repo}",
self.account, self.ssh_host, self.ssh_port, self.account
)
}
/// The captured server log, whole and ANSI-stripped (the tracing output is
/// coloured even with no tty — the trap `fetkaos::strip_ansi` documents).
/// Empty for an external target, whose log the host cannot read.
fn log_text(&self) -> String {
self.log
.as_ref()
.map(|l| strip_ansi(&std::fs::read_to_string(l).unwrap_or_default()))
.unwrap_or_default()
}
/// SIGKILL the server and reap it — a no-op for an external target the arm
/// does not own. Explicit child handle, never `pkill -f` (which would match
/// this very process — the recorded trap).
fn kill(&self) {
if let Some(child) = &self.child {
if let Ok(mut g) = child.lock() {
let _ = g.kill();
let _ = g.wait();
}
}
}
/// Alive == the control plane still answers the unauthenticated `status`.
/// An applied liveness check, not a `/proc` existence read that a zombie
/// would still satisfy.
fn alive(&self) -> bool {
let (o, out) = ctl_status(self, &self.grpc_addr());
o == OpOutcome::Ok && out.contains("control plane")
}
}
impl Drop for Server {
fn drop(&mut self) {
// Never kill or delete an external appliance the arm was merely given.
if self.external {
return;
}
self.kill();
if let Some(data) = &self.data {
let _ = std::fs::remove_dir_all(data);
}
}
}
/// Run `gunnar control <args> -i <admin> --at <grpc>`, returning the outcome and
/// the combined output with the `signing as` noise stripped. A refusal is DATA
/// during the storm, not a crash, so this never errors.
fn ctl(server: &Server, args: &[&str]) -> (OpOutcome, String) {
let mut c = Command::new(&server.cli);
c.arg("control")
.args(args)
.arg("-i")
.arg(&server.key)
.args(["--at", &server.grpc_addr()]);
run_captured(c, CTL_TIMEOUT)
}
/// The UNAUTHENTICATED `control status --at <addr>` liveness probe. Its own
/// helper because it takes an explicit address (the `deafcontrol` red points it
/// at the SSH port to prove the detector can report DOWN) and carries no `-i`.
fn ctl_status(server: &Server, at: &str) -> (OpOutcome, String) {
let mut c = Command::new(&server.cli);
c.args(["control", "status", "--at", at]);
run_captured(c, CTL_TIMEOUT)
}
/// [`output_within`] + outcome classification + `signing as` stripping. Both
/// control helpers route through here (one writer, one implementation).
fn run_captured(mut c: Command, limit: Duration) -> (OpOutcome, String) {
c.env("GIT_TERMINAL_PROMPT", "0");
match output_within(c, limit) {
Ok(Some(o)) => {
let mut s = String::from_utf8_lossy(&o.stdout).into_owned();
s.push_str(&String::from_utf8_lossy(&o.stderr));
let s = strip_ansi(&s)
.lines()
.filter(|l| !l.trim_start().starts_with("gunnar: signing as"))
.collect::<Vec<_>>()
.join("\n");
(
if o.status.success() {
OpOutcome::Ok
} else {
OpOutcome::Failed
},
s,
)
}
Ok(None) => (
OpOutcome::TimedOut,
format!("no exit after {limit:?}; killed"),
),
Err(e) => (OpOutcome::Failed, e.to_string()),
}
}
/// Run a git command against the server. Reuses `fetkaos::git_env` for the
/// hardened SSH command (batch-mode, connect + keepalive timeouts).
///
/// **Honours gunnar's own retryable signal.** Under concurrent read+write a
/// clone/fetch of a repository that a `receive-pack` is mid-transaction on is
/// answered with `the repository is unavailable; try again` — gunnar EXPLICITLY
/// asking the client to retry (a coarse repo-level lock; found on this arm's
/// first real run, op#22 `clone_full` racing pushes to the same arena — the
/// coarse repo-level lock). A client that treats an explicit "try again" as a hard defect
/// is the one in the wrong, and it would mask every injected red behind normal
/// contention. So this retries a bounded number of times on THAT sentence only;
/// every other failure is returned immediately and unretried.
fn git_try(server: &Server, cwd: Option<&Path>, args: &[&str]) -> (OpOutcome, String) {
let mut last = (OpOutcome::Failed, String::new());
for attempt in 0..RETRYABLE_ATTEMPTS {
let mut c = Command::new("git");
if let Some(d) = cwd {
c.current_dir(d);
}
git_env(&mut c, &server.key);
c.args(args);
let (o, out) = run_captured(c, GIT_TIMEOUT);
if o == OpOutcome::Ok || !out.contains("try again") {
return (o, out);
}
last = (o, out);
std::thread::sleep(Duration::from_millis(80 * (attempt as u64 + 1)));
}
last
}
/// How many times a git op honours gunnar's explicit `try again` before it is a
/// real failure. Four is enough to clear a receive-pack's ref transaction under
/// this arm's contention without hiding a server that is actually stuck.
const RETRYABLE_ATTEMPTS: usize = 4;
/// stock `git fsck --full --strict` over a cloned graph — the APPLIED-output
/// judge, measured not asserted. `corruptclone` flips a byte before this runs, and this is what
/// must then reject it.
fn fsck_ok(repo: &Path) -> (OpOutcome, String) {
let mut c = Command::new("git");
c.current_dir(repo)
.args(["fsck", "--full", "--strict", "--no-progress"])
.env("GIT_CONFIG_GLOBAL", "/dev/null")
.env("GIT_CONFIG_SYSTEM", "/dev/null");
run_captured(c, GIT_TIMEOUT)
}
/// Flip one byte in the object store of a cloned bare repo — the `corruptclone`
/// injection. Tries a `.pack` first, then a LOOSE object (a small clone lands
/// under `fetch.unpackLimit` and is stored loose, with no pack at all — the trap
/// that made this probe a silent no-op on its first run). Returns whether it
/// found ANYTHING to corrupt; the caller turns `false` into a hard failure, so
/// a probe that injected nothing is never a green.
fn corrupt_clone(repo: &Path) -> bool {
// A pack, if the clone kept one.
if let Ok(entries) = std::fs::read_dir(repo.join("objects/pack")) {
for e in entries.flatten() {
let p = e.path();
if p.extension().and_then(|s| s.to_str()) == Some("pack") {
if flip_byte(&p) {
return true;
}
}
}
}
// Otherwise a loose object: objects/<2 hex>/<38+ hex>. Flip a byte in the
// zlib stream past its 2-byte header — fsck recomputes the oid and rejects.
if let Ok(dirs) = std::fs::read_dir(repo.join("objects")) {
for d in dirs.flatten() {
let n = d.file_name().to_string_lossy().into_owned();
if n.len() != 2 || !n.chars().all(|c| c.is_ascii_hexdigit()) {
continue;
}
if let Ok(objs) = std::fs::read_dir(d.path()) {
for o in objs.flatten() {
if flip_byte(&o.path()) {
return true;
}
}
}
}
}
false
}
/// Flip one byte in the middle of a file, making it writable first — git writes
/// clone packs and loose objects mode 0444, so a bare `write` silently fails
/// (the trap that made corruptclone report "nothing to corrupt").
fn flip_byte(p: &Path) -> bool {
let Ok(mut bytes) = std::fs::read(p) else {
return false;
};
if bytes.len() <= 4 {
return false;
}
let at = bytes.len() / 2;
bytes[at] ^= 0xff;
let _ = std::fs::set_permissions(p, std::fs::Permissions::from_mode(0o644));
std::fs::write(p, &bytes).is_ok()
}
// ───────────────────────────────────────────────────────────────────────────
// boot
// ───────────────────────────────────────────────────────────────────────────
/// Boot one `gunnar serve`, wait until BOTH transports actually speak, and
/// create the `admin` namespace the arena/churn repos live in.
fn up(cli: &Path, work: &Path, ssh_port: u16, grpc_port: u16) -> Result<Server> {
let data = work.join("data");
let _ = std::fs::remove_dir_all(&data);
std::fs::create_dir_all(&data)?;
let key = work.join("admin");
let _ = std::fs::remove_file(&key);
let _ = std::fs::remove_file(work.join("admin.pub"));
let st = Command::new("ssh-keygen")
.args(["-q", "-t", "ed25519", "-N", "", "-C", "admin", "-f"])
.arg(&key)
.status()
.context("ssh-keygen for the opstorm admin identity")?;
ensure!(st.success(), "ssh-keygen failed");
let pubkey_arg = format!("@{}", work.join("admin.pub").display());
let log = work.join("serve.log");
let logf = std::fs::File::create(&log)?;
let child = Command::new(cli)
.args(["serve", "--data-root"])
.arg(&data)
.args(["--ssh-bind", &format!("127.0.0.1:{ssh_port}")])
.args(["--grpc-bind", &format!("127.0.0.1:{grpc_port}")])
.args(["--admin-key", &pubkey_arg])
.stdin(Stdio::null())
.stdout(logf.try_clone()?)
.stderr(logf)
.spawn()
.context("spawning gunnar serve")?;
let server = Server {
cli: cli.to_path_buf(),
data: Some(data),
key,
ssh_host: "127.0.0.1".into(),
ssh_port,
grpc_host: "127.0.0.1".into(),
grpc_port,
account: "admin".into(),
log: Some(log),
child: Some(Mutex::new(child)),
external: false,
started: Instant::now(),
};
// Wait until the CONTROL plane answers AND the git transport speaks — ask
// the server, do not sleep. (The drills' hollow-probe lesson: a bound port
// is not a speaking server.)
wait_transports(&server)?;
// Prove the separate-runtime line is actually in the log — the whole
// point of the liveness checks below is that a control-plane stall cannot
// reach the git listener, and that guarantee is this line.
ensure!(
server.log_text().contains(CONTROL_PLANE_NEEDLE),
"the server never logged the separate-runtime line {CONTROL_PLANE_NEEDLE:?}; the \
control-plane-under-git-load guarantee this arm leans on is not in this build. Log:\n{}",
tail(&server.log_text(), 4000)
);
// The admin account's namespace does not exist until asked for (measured:
// `repo create admin/…` returns "namespace admin not found" on a fresh box).
let (o, out) = ctl(&server, &["namespace", "create", &server.account]);
ensure!(
o == OpOutcome::Ok,
"creating the admin namespace failed: {out}"
);
Ok(server)
}
/// Attach to an EXTERNAL, already-running gunnar appliance instead of spawning
/// one. Same subject to the whole alphabet — only the constructor differs from
/// [`up`]. The `GUNNAR_OPSTORM_TARGET_*` env names it:
///
/// * `GUNNAR_OPSTORM_TARGET_SSH=<host:port>` — the git (SSH) endpoint,
/// * `GUNNAR_OPSTORM_TARGET_GRPC=<host:port>` — the control (gRPC) endpoint,
/// * `GUNNAR_OPSTORM_TARGET_ADMIN_KEY=<path>` — the admin PRIVATE key whose
/// public half is enrolled on the appliance as the account principal,
/// * `GUNNAR_OPSTORM_TARGET_ACCOUNT=<account>` — the account/namespace the
/// arenas live in (default `admin`).
///
/// The arm does not own the appliance, so `external` gates off the two
/// owning-probes (`killmid`/`corruptclone`), the teardown, and the log
/// needle (a PID-1 appliance's log is not reachable from the host — the guard
/// is skipped HONESTLY, not faked).
fn attach(cli: &Path, ssh: &str, grpc: &str, key: PathBuf, account: String) -> Result<Server> {
ensure!(
key.exists(),
"GUNNAR_OPSTORM_TARGET_ADMIN_KEY={key:?} does not exist. Point it at the PRIVATE key whose \
public half is enrolled on the appliance as the {account:?} principal."
);
let (ssh_host, ssh_port) =
split_hostport(ssh).context("GUNNAR_OPSTORM_TARGET_SSH must be host:port")?;
let (grpc_host, grpc_port) =
split_hostport(grpc).context("GUNNAR_OPSTORM_TARGET_GRPC must be host:port")?;
let server = Server {
cli: cli.to_path_buf(),
data: None,
key,
ssh_host,
ssh_port,
grpc_host,
grpc_port,
account,
log: None,
child: None,
external: true,
started: Instant::now(),
};
eprintln!(
"opstorm: EXTERNAL target — git ssh {} · control grpc {} · account {:?}. This arm does not \
own the server: killmid/corruptclone are N/A and the separate-runtime log line \
cannot be read across the wire.",
server.ssh_addr(),
server.grpc_addr(),
server.account
);
wait_transports(&server)?;
// The account namespace must exist for `<account>/arenaN`. A freshly
// bootstrapped principal usually already owns it; create it if not, and
// tolerate an already-exists refusal on a shared box (seed_arenas fails
// loudly if the namespace is genuinely absent).
let (o, out) = ctl(&server, &["namespace", "create", &server.account]);
if o != OpOutcome::Ok {
eprintln!(
"opstorm: namespace create {:?} not applied ({}); assuming it already exists.",
server.account,
out.trim()
);
}
Ok(server)
}
/// `host:port` → `(host, port)`. An empty host means loopback (`:2222`).
fn split_hostport(s: &str) -> Result<(String, u16)> {
let (h, p) = s
.rsplit_once(':')
.with_context(|| format!("expected host:port, got {s:?}"))?;
let port: u16 = p.parse().with_context(|| format!("bad port in {s:?}"))?;
let host = if h.is_empty() {
"127.0.0.1".to_string()
} else {
h.to_string()
};
Ok((host, port))
}
/// Wait until BOTH transports actually speak — ask the server, do not sleep.
/// Shared by the self-spawn [`up`] and the external [`attach`] (a bound port
/// is not a speaking server — the drills' hollow-probe lesson).
fn wait_transports(server: &Server) -> Result<()> {
let deadline = Instant::now() + Duration::from_secs(60);
loop {
let (o, out) = ctl_status(server, &server.grpc_addr());
let grpc_up = o == OpOutcome::Ok && out.contains("control plane");
// git speaks == an ls-remote of a missing repo gets gunnar's own
// "repository not found"/"not found", proof the russh handshake + auth +
// git request completed, needing no repo to exist yet.
let (_go, gout) = git_try(server, None, &["ls-remote", &server.url("__probe__")]);
let git_up = gout.contains("repository not found") || gout.contains("not found");
if grpc_up && git_up {
return Ok(());
}
ensure!(
Instant::now() < deadline,
"the server never brought BOTH transports up within 60 s (grpc_up={grpc_up}, \
git_up={git_up}). Server log:\n{}",
tail(&server.log_text(), 4000)
);
std::thread::sleep(Duration::from_millis(150));
}
}
/// Create the arena repositories and seed each with a small real history, then
/// prove each is clonable and fsck-clean. This is the "control created it → git
/// serves it" consistency, checked serially before the concurrent storm.
fn seed_arenas(server: &Server, work: &Path, arenas: usize, seed: u64) -> Result<()> {
// One seed source repo, a handful of commits of Lcg-drawn content.
let src = work.join("seed-src");
let _ = std::fs::remove_dir_all(&src);
git_init(&src, "sha1")?;
let mut rng = Lcg::new(seed ^ 0xA5A5_1234);
for i in 0..5 {
let blob = src.join(format!("f{i}"));
let len = 2048 + (rng.next_u64() % 4096) as usize;
std::fs::write(&blob, rng.text(len))?;
run_ok(
Command::new("git").current_dir(&src).args([
"-c",
"user.email=opstorm@the desktop",
"-c",
"user.name=opstorm",
"add",
"-A",
]),
"git add seed",
)?;
run_ok(
Command::new("git").current_dir(&src).args([
"-c",
"user.email=opstorm@the desktop",
"-c",
"user.name=opstorm",
"commit",
"-q",
"-m",
&format!("seed {i}"),
]),
"git commit seed",
)?;
}
for a in 0..arenas {
let name = format!("arena{a}");
let (o, out) = ctl(
server,
&[
"repo",
"create",
&format!("{}/{name}", server.account),
"--hash",
"sha1",
],
);
ensure!(o == OpOutcome::Ok, "creating arena {name} failed: {out}");
let (o, out) = git_try(
server,
Some(&src),
&["push", "-q", &server.url(&name), "HEAD:refs/heads/main"],
);
ensure!(
o == OpOutcome::Ok,
"seeding arena {name} by push failed: {out}"
);
// Verify: clone it back, fsck it, confirm main is what we pushed.
let dst = work.join(format!("verify-{name}"));
let _ = std::fs::remove_dir_all(&dst);
let (o, out) = git_try(
server,
None,
&[
"clone",
"-q",
"--bare",
&server.url(&name),
&dst.display().to_string(),
],
);
ensure!(
o == OpOutcome::Ok,
"verify-clone of arena {name} failed: {out}"
);
let (fo, fout) = fsck_ok(&dst);
ensure!(
fo == OpOutcome::Ok,
"seeded arena {name} did NOT fsck clean straight after creation: {fout}"
);
let _ = std::fs::remove_dir_all(&dst);
}
Ok(())
}
fn git_init(dir: &Path, fmt: &str) -> Result<()> {
std::fs::create_dir_all(dir)?;
run_ok(
Command::new("git")
.args(["init", "-q", &format!("--object-format={fmt}")])
.arg(dir),
"git init",
)
}
fn run_ok(c: &mut Command, what: &str) -> Result<()> {
let out = c
.env("GIT_CONFIG_GLOBAL", "/dev/null")
.env("GIT_CONFIG_SYSTEM", "/dev/null")
.output()
.with_context(|| what.to_string())?;
ensure!(
out.status.success(),
"{what} failed: {}",
String::from_utf8_lossy(&out.stderr).trim()
);
Ok(())
}
// ───────────────────────────────────────────────────────────────────────────
// the operation alphabet
// ───────────────────────────────────────────────────────────────────────────
/// The mix. Read-mostly so 20 000 ops finish, but every mutating verb is
/// present and every one is contended by 16 workers.
const OP_MIX: &[(Op, u32)] = &[
// git transport
(Op::LsRemote, 16),
(Op::FetchShallow, 8),
(Op::CloneShallow, 6),
(Op::CloneFull, 2),
(Op::PushBranch, 6),
(Op::BranchChurn, 4),
// control plane — reads
(Op::CtlStatus, 12),
(Op::CtlRepoList, 8),
(Op::CtlRepoGet, 6),
(Op::CtlHealth, 5),
(Op::CtlStats, 5),
(Op::CtlStorage, 3),
(Op::CtlReplList, 3),
// control plane — mutating churn (self-contained + applied read-back)
(Op::RepoChurn, 6),
(Op::UserChurn, 4),
(Op::DeleteUnderClone, 2),
];
#[derive(Clone, Copy, PartialEq, Eq, Debug)]
enum Op {
LsRemote,
FetchShallow,
CloneShallow,
CloneFull,
PushBranch,
/// push a fresh branch, then delete it, then confirm ls-remote no longer
/// lists it — a ref write + ref delete with an applied read-back.
BranchChurn,
/// the UNAUTHENTICATED liveness probe; `deafcontrol` misroutes it.
CtlStatus,
CtlRepoList,
CtlRepoGet,
CtlHealth,
CtlStats,
CtlStorage,
CtlReplList,
/// create a private churn repo, delete+purge it, confirm it is gone.
RepoChurn,
/// add a user with a key, remove it, confirm the roster no longer holds it.
UserChurn,
/// create + seed a repo, then clone it and delete it CONCURRENTLY, and
/// require the clone to be either fsck-clean or a clean failure — never a
/// torn graph accepted as success.
DeleteUnderClone,
}
impl Op {
fn key(self) -> &'static str {
match self {
Op::LsRemote => "ls_remote",
Op::FetchShallow => "fetch_shallow",
Op::CloneShallow => "clone_shallow",
Op::CloneFull => "clone_full",
Op::PushBranch => "push_branch",
Op::BranchChurn => "branch_churn",
Op::CtlStatus => "ctl_status",
Op::CtlRepoList => "ctl_repo_list",
Op::CtlRepoGet => "ctl_repo_get",
Op::CtlHealth => "ctl_health",
Op::CtlStats => "ctl_stats",
Op::CtlStorage => "ctl_storage",
Op::CtlReplList => "ctl_repl_list",
Op::RepoChurn => "repo_churn",
Op::UserChurn => "user_churn",
Op::DeleteUnderClone => "delete_under_clone",
}
}
/// The control-liveness op the `deafcontrol`/`killmid` reds show as a
/// liveness failure rather than a plain op failure.
fn is_liveness(self) -> bool {
self == Op::CtlStatus
}
/// Does this op PUSH to its arena? Push targets are the WRITE arenas; every
/// other arena-touching op reads, and reads target the READ arenas. See
/// [`arena_split`]. (The churn ops carry an arena only for an incidental
/// grant / repo-get and are happy on a read arena.)
fn writes_arena(self) -> bool {
matches!(self, Op::PushBranch | Op::BranchChurn)
}
}
/// One scheduled operation — a pure function of the seed and the index.
#[derive(Clone)]
struct Planned {
idx: u64,
op: Op,
arena: usize,
depth: u32,
}
/// READ arenas vs WRITE arenas — the split that keeps this a CORRECTNESS
/// defect-finder rather than a fairness one.
///
/// Found on the first real run: a `clone` of an arena that a
/// steady `push` stream is hammering is answered `the repository is unavailable;
/// try again` on nearly every attempt — gunnar's coarse repo-level write lock
/// starving the reader. That is INTENDED ("try again") and a fairness smell, not
/// a torn graph, so an arm that reds on it never reaches its injected
/// correctness reds. So — exactly as `fetkaos` splits its corpus into ref-only
/// and object-moving members — clones/fetches target the READ arenas (never
/// pushed to during the storm) and pushes target the WRITE arenas. Cross-arena
/// concurrency (clone A while push B) still storms both paths at once; what is
/// removed is only the same-repo reader-vs-writer collision, which is realistic
/// (real clients clone one repo while others push another) and whose fairness
/// finding is recorded rather than allowed to mask corruption.
fn arena_split(arenas: usize) -> (usize, usize) {
let read_n = (arenas / 2).max(1);
let write_start = if arenas > read_n { read_n } else { 0 };
(read_n, write_start)
}
/// Draw the WHOLE schedule from the seed before any worker starts.
fn plan(seed: u64, ops: u64, arenas: usize) -> Vec<Planned> {
let total: u32 = OP_MIX.iter().map(|(_, w)| w).sum();
let (read_n, write_start) = arena_split(arenas);
let write_n = arenas - write_start;
let mut rng = Lcg::new(seed);
let mut out = Vec::with_capacity(ops as usize);
for idx in 0..ops {
let roll = (rng.next_u64() % total as u64) as u32;
let mut acc = 0u32;
let mut op = OP_MIX[0].0;
for (candidate, w) in OP_MIX {
acc += w;
if roll < acc {
op = *candidate;
break;
}
}
// Two draws, ALWAYS made, so the RNG stream does not depend on the op
// kind and a seed stays comparable across arena counts of the same shape.
let read_pick = (rng.next_u64() % read_n as u64) as usize;
let write_pick = write_start + (rng.next_u64() % write_n as u64) as usize;
let arena = if op.writes_arena() {
write_pick
} else {
read_pick
};
let depth = 1 + (rng.next_u64() % 8) as u32;
out.push(Planned {
idx,
op,
arena,
depth,
});
}
out
}
// ───────────────────────────────────────────────────────────────────────────
// the storm
// ───────────────────────────────────────────────────────────────────────────
#[derive(Default)]
struct StormOut {
done: u64,
failed: u64,
timed_out: u64,
abandoned: u64,
/// control-liveness probes issued, and how many did not answer. A nonzero
/// `liveness_failed` is that guarantee broken (or `deafcontrol`/
/// `killmid` proving the detector can report it).
liveness_checks: u64,
liveness_failed: u64,
per_op: BTreeMap<&'static str, u64>,
first_failure: Option<String>,
}
/// Run the planned operations across `workers` threads. The multicore rule
/// (gatling, never rayon): `thread::scope` driving child processes.
fn storm(
server: &Server,
schedule: &[Planned],
workers: usize,
scratch: &Path,
seed: u64,
) -> StormOut {
let next = AtomicUsize::new(0);
let done = AtomicU64::new(0);
let failed = AtomicU64::new(0);
let timed_out = AtomicU64::new(0);
let abandoned = AtomicU64::new(0);
let liveness_checks = AtomicU64::new(0);
let liveness_failed = AtomicU64::new(0);
let per_op: Mutex<BTreeMap<&'static str, u64>> = Mutex::new(BTreeMap::new());
let first: Mutex<Option<String>> = Mutex::new(None);
let deadline = Instant::now() + STORM_DEADLINE;
// `killmid` SIGKILLs the server the arm OWNS. It cannot fire against an
// external appliance (run() already printed the loud N/A); off it stays.
let kill_at = if red("killmid") && !server.external {
schedule.len() / 2
} else {
usize::MAX
};
let record_fail = |p: &Planned, kind: &str, detail: &str| {
if let Ok(mut g) = first.lock() {
if g.is_none() {
*g = Some(format!(
"seed=0x{seed:016x} operation #{} ({} on arena{}) {kind}: {}",
p.idx,
p.op.key(),
p.arena,
detail.trim()
));
}
}
};
std::thread::scope(|scope| {
for w in 0..workers {
let (next, done, failed, timed_out, abandoned) =
(&next, &done, &failed, &timed_out, &abandoned);
let (liveness_checks, liveness_failed) = (&liveness_checks, &liveness_failed);
let (per_op, record_fail) = (&per_op, &record_fail);
scope.spawn(move || {
let cwd = scratch.join(format!("w{w}"));
let _ = std::fs::create_dir_all(&cwd);
loop {
let i = next.fetch_add(1, Ordering::Relaxed);
if i >= schedule.len() {
break;
}
if Instant::now() >= deadline {
abandoned.fetch_add(1, Ordering::Relaxed);
continue;
}
if i == kill_at {
server.kill();
}
let p = &schedule[i];
let (outcome, detail) =
run_one(server, p, w, &cwd, liveness_checks, liveness_failed);
done.fetch_add(1, Ordering::Relaxed);
if let Ok(mut g) = per_op.lock() {
*g.entry(p.op.key()).or_insert(0) += 1;
}
match outcome {
OpOutcome::Ok => {}
OpOutcome::TimedOut => {
timed_out.fetch_add(1, Ordering::Relaxed);
failed.fetch_add(1, Ordering::Relaxed);
let kind = if p.op.is_liveness() {
"LIVENESS STALLED (control did not answer under load)"
} else {
"STALLED"
};
record_fail(p, kind, &detail);
}
OpOutcome::Failed => {
failed.fetch_add(1, Ordering::Relaxed);
let kind = if p.op.is_liveness() {
"LIVENESS FAILED (control did not answer)"
} else {
"failed"
};
record_fail(p, kind, &detail);
}
}
}
});
}
});
StormOut {
done: done.load(Ordering::Relaxed),
failed: failed.load(Ordering::Relaxed),
timed_out: timed_out.load(Ordering::Relaxed),
abandoned: abandoned.load(Ordering::Relaxed),
liveness_checks: liveness_checks.load(Ordering::Relaxed),
liveness_failed: liveness_failed.load(Ordering::Relaxed),
per_op: per_op.into_inner().unwrap_or_default(),
first_failure: first.into_inner().ok().flatten(),
}
}
/// Execute one operation and return its outcome. Every branch either returns a
/// real op outcome, or an injected-defect verdict (the reds) that the storm
/// reports as a failure naming this op's index.
fn run_one(
server: &Server,
p: &Planned,
w: usize,
cwd: &Path,
liveness_checks: &AtomicU64,
liveness_failed: &AtomicU64,
) -> (OpOutcome, String) {
let arena = format!("arena{}", p.arena);
match p.op {
Op::LsRemote => git_try(server, None, &["ls-remote", &server.url(&arena)]),
Op::FetchShallow => {
let holder = worker_holder(cwd);
git_try(
server,
Some(&holder),
&[
"fetch",
"-q",
&format!("--depth={}", p.depth),
&server.url(&arena),
"HEAD",
],
)
}
Op::CloneShallow | Op::CloneFull => {
let dst = cwd.join(format!("clone-{}", p.idx));
let _ = std::fs::remove_dir_all(&dst);
// `unpackLimit=1` keeps a `.pack` even for a tiny clone, so the
// `corruptclone` injection has a stable target (and so does a real
// pack-path defect). Without it a small clone unpacks to loose
// objects — the trap that made corruptclone a silent no-op.
let mut args = vec![
"-c".to_string(),
"fetch.unpackLimit=1".into(),
"-c".into(),
"transfer.unpackLimit=1".into(),
"clone".into(),
"-q".into(),
"--bare".into(),
];
if p.op == Op::CloneShallow {
args.push(format!("--depth={}", p.depth));
}
args.push(server.url(&arena));
args.push(dst.display().to_string());
let argref: Vec<&str> = args.iter().map(|s| s.as_str()).collect();
let (o, out) = git_try(server, None, &argref);
let verdict = if o == OpOutcome::Ok {
// Judge the APPLIED graph, not the exit code. `corruptclone`
// flips a byte here so the judge is proven able to reject — and a
// probe that found nothing to corrupt is a hard failure, never a
// green. Gated off against an external appliance (run() printed
// the loud N/A): the arm does not corrupt a box it does not own.
if red("corruptclone") && !server.external && !corrupt_clone(&dst) {
(
OpOutcome::Failed,
"corruptclone found no object to corrupt in the clone".into(),
)
} else {
let (fo, fout) = fsck_ok(&dst);
if fo == OpOutcome::Ok {
(OpOutcome::Ok, String::new())
} else {
(
OpOutcome::Failed,
format!("a clone that git accepted did NOT fsck clean: {fout}"),
)
}
}
} else {
(o, out)
};
let _ = std::fs::remove_dir_all(&dst);
verdict
}
Op::PushBranch => {
let holder = worker_holder(cwd);
git_try(
server,
Some(&holder),
&[
"push",
"-q",
&server.url(&arena),
&format!("HEAD:refs/heads/opstorm-w{w}-{}", p.idx),
],
)
}
Op::BranchChurn => {
let holder = worker_holder(cwd);
let branch = format!("refs/heads/churn-w{w}-{}", p.idx);
let (o, out) = git_try(
server,
Some(&holder),
&["push", "-q", &server.url(&arena), &format!("HEAD:{branch}")],
);
if o != OpOutcome::Ok {
return (o, out);
}
let (o, out) = git_try(
server,
Some(&holder),
&["push", "-q", &server.url(&arena), &format!(":{branch}")],
);
if o != OpOutcome::Ok {
return (o, format!("deleting the just-pushed branch failed: {out}"));
}
// Applied read-back: the branch must be gone.
let (_lo, lout) = git_try(server, None, &["ls-remote", &server.url(&arena), &branch]);
if lout.trim().is_empty() {
(OpOutcome::Ok, String::new())
} else {
(
OpOutcome::Failed,
format!("deleted branch {branch} is STILL advertised: {lout}"),
)
}
}
Op::CtlStatus => {
liveness_checks.fetch_add(1, Ordering::Relaxed);
// `deafcontrol`: point the liveness probe at the SSH port, where a
// gRPC control call cannot be answered — proving the detector reports
// DOWN. In green mode it hits the control runtime, whose own
// thread pool keeps answering even under the git storm.
let at = if red("deafcontrol") {
server.ssh_addr()
} else {
server.grpc_addr()
};
let (o, out) = ctl_status(server, &at);
if o == OpOutcome::Ok && out.contains("control plane") {
(OpOutcome::Ok, String::new())
} else {
liveness_failed.fetch_add(1, Ordering::Relaxed);
(
if o == OpOutcome::TimedOut {
OpOutcome::TimedOut
} else {
OpOutcome::Failed
},
format!("control status at {at} did not answer 'control plane': {out}"),
)
}
}
Op::CtlRepoList => ctl(server, &["repo", "list"]),
Op::CtlRepoGet => ctl(
server,
&["repo", "get", &format!("{}/{arena}", server.account)],
),
Op::CtlHealth => ctl(server, &["health"]),
Op::CtlStats => ctl(server, &["stats"]),
Op::CtlStorage => ctl(server, &["storage"]),
Op::CtlReplList => ctl(server, &["replication", "list"]),
Op::RepoChurn => {
let name = format!("{}/churn-{}", server.account, p.idx);
let hash = if p.idx % 2 == 0 { "sha1" } else { "sha256" };
let (o, out) = ctl(server, &["repo", "create", &name, "--hash", hash]);
if o != OpOutcome::Ok {
return (o, format!("churn create {name} failed: {out}"));
}
let (o, out) = ctl(server, &["repo", "delete", &name, "--yes", "--purge"]);
if o != OpOutcome::Ok {
return (o, format!("churn delete {name} failed: {out}"));
}
// Applied read-back: a `repo get` on the purged name must NOT succeed.
let (go, _gout) = ctl(server, &["repo", "get", &name]);
if go == OpOutcome::Ok {
(
OpOutcome::Failed,
format!("purged repo {name} still answers `repo get`"),
)
} else {
(OpOutcome::Ok, String::new())
}
}
Op::UserChurn => {
let name = format!("opstorm-u-{}", p.idx);
let holder = worker_holder(cwd);
let ukey = holder.join(format!("uk-{}", p.idx));
let _ = std::fs::remove_file(&ukey);
let _ = std::fs::remove_file(holder.join(format!("uk-{}.pub", p.idx)));
if !Command::new("ssh-keygen")
.args(["-q", "-t", "ed25519", "-N", "", "-C", &name, "-f"])
.arg(&ukey)
.status()
.map(|s| s.success())
.unwrap_or(false)
{
return (OpOutcome::Failed, "ssh-keygen for churn user failed".into());
}
let keyarg = format!("@{}", holder.join(format!("uk-{}.pub", p.idx)).display());
let (o, out) = ctl(
server,
&["user", "add", &name, "--key", &keyarg, "--kind", "user"],
);
if o != OpOutcome::Ok {
return (o, format!("churn user add {name} failed: {out}"));
}
// A grant on an arena, then remove the whole user.
let _ = ctl(
server,
&[
"grant",
&name,
"read",
"--repo",
&format!("{}/{arena}", server.account),
],
);
let (o, out) = ctl(server, &["user", "remove", &name]);
if o != OpOutcome::Ok {
return (o, format!("churn user remove {name} failed: {out}"));
}
// `user remove` takes the principal, its keys and its grants, but NOT
// the account namespace named after them (measured: a re-`user add`
// of the same name then fails `namespace <name> already exists`).
// This op's contract is to leave NO trace, and the seeded-schedule
// determinism contract says the SAME seed must replay against a box
// that already ran it — both of which the orphan namespace breaks. So
// delete it too, making the op idempotent by construction (this is
// also what lets external mode re-storm a shared box cleanly).
let _ = ctl(server, &["namespace", "delete", &name]);
// Applied read-back: the roster must not still list it.
let (_lo, lout) = ctl(server, &["user", "list"]);
let still = lout.lines().any(|l| {
l.split_whitespace()
.next()
.map(|first| first == name)
.unwrap_or(false)
});
if still {
(
OpOutcome::Failed,
format!("removed user {name} is STILL in `user list`"),
)
} else {
(OpOutcome::Ok, String::new())
}
}
Op::DeleteUnderClone => delete_under_clone(server, p, cwd),
}
}
/// A per-worker holder repo with one commit, created lazily — `fetch`/`push`
/// need a repository to run in.
fn worker_holder(cwd: &Path) -> PathBuf {
let holder = cwd.join("holder");
if !holder.join("HEAD").exists() {
let _ = git_init(&holder, "sha1");
let _ = Command::new("git")
.current_dir(&holder)
.args([
"-c",
"user.email=opstorm@the desktop",
"-c",
"user.name=opstorm",
"commit",
"-q",
"--allow-empty",
"-m",
"opstorm holder",
])
.env("GIT_CONFIG_GLOBAL", "/dev/null")
.output();
}
holder
}
/// Create + seed a repo, then clone and delete it CONCURRENTLY. The invariant:
/// the clone is either fsck-clean or a CLEAN failure — a delete racing a read
/// must never leave the client a torn graph it accepted.
fn delete_under_clone(server: &Server, p: &Planned, cwd: &Path) -> (OpOutcome, String) {
let name = format!("duc-{}", p.idx);
let full = format!("{}/{name}", server.account);
let (o, out) = ctl(server, &["repo", "create", &full, "--hash", "sha1"]);
if o != OpOutcome::Ok {
return (o, format!("duc create {full} failed: {out}"));
}
let holder = worker_holder(cwd);
let (o, out) = git_try(
server,
Some(&holder),
&["push", "-q", &server.url(&name), "HEAD:refs/heads/main"],
);
if o != OpOutcome::Ok {
return (o, format!("duc seed push {full} failed: {out}"));
}
let dst = cwd.join(format!("duc-clone-{}", p.idx));
let _ = std::fs::remove_dir_all(&dst);
// Race a clone against a delete. The clone in one thread, the delete in the
// main thread of this op, joined.
let clone_verdict = std::thread::scope(|s| {
let h = s.spawn(|| {
let url = server.url(&name);
git_try(
server,
None,
&["clone", "-q", "--bare", &url, &dst.display().to_string()],
)
});
// Delete concurrently.
let (_do, _dout) = ctl(server, &["repo", "delete", &full, "--yes", "--purge"]);
h.join()
.unwrap_or((OpOutcome::Failed, "clone thread panicked".into()))
});
let verdict = match clone_verdict {
(OpOutcome::Ok, _) => {
// If it succeeded, the graph it produced MUST be clean.
let (fo, fout) = fsck_ok(&dst);
if fo == OpOutcome::Ok {
(OpOutcome::Ok, String::new())
} else {
(
OpOutcome::Failed,
format!("clone of {full} won the race but produced a TORN graph fsck rejected: {fout}"),
)
}
}
// A clean failure (the delete won) is fine — the invariant is "never a
// torn accepted graph", not "the clone always wins".
_ => (OpOutcome::Ok, String::new()),
};
let _ = std::fs::remove_dir_all(&dst);
verdict
}
/// Throw away the repos this storm created on an external box it does not own —
/// the arenas and the abdicate phase's leftover branches inside them. The churn
/// ops (`RepoChurn`/`UserChurn`/`DeleteUnderClone`) already self-clean, so this
/// only has to purge the arenas. Best-effort: a purge that does not land is
/// logged, not fatal — the run's verdict already stands, and a self-spawned
/// server's whole data root is deleted on `Drop` regardless.
fn cleanup_external(server: &Server, arenas: usize) {
for a in 0..arenas {
let name = format!("{}/arena{a}", server.account);
let (o, out) = ctl(server, &["repo", "delete", &name, "--yes", "--purge"]);
if o != OpOutcome::Ok {
eprintln!("opstorm cleanup: could not purge {name}: {}", out.trim());
}
}
}
// ───────────────────────────────────────────────────────────────────────────
// the abdicate phase — single-node freeze → refuse → promote
// ───────────────────────────────────────────────────────────────────────────
/// Prove the role flip actually stops and restarts client writes. Not in the
/// random storm (a frozen box would poison every later push); a bounded phase.
///
/// `freezeaccepts` red: assert the freeze-refusal BEFORE abdicating — a primary
/// accepts the write, so the "refused" assertion fires, proving the detector is
/// not hollow.
fn abdicate_phase(server: &Server, work: &Path) -> Result<()> {
let src = work.join("seed-src");
let arena = "arena0";
let push_once = |branch: &str| -> (OpOutcome, String) {
git_try(
server,
Some(&src),
&[
"push",
"-q",
&server.url(arena),
&format!("HEAD:refs/heads/{branch}"),
],
)
};
// Preflight: the box is a primary and DOES take writes (so the refusal check
// that follows means something).
let (o, out) = push_once("abdicate-preflight");
ensure!(
o == OpOutcome::Ok,
"preflight push to the primary failed, before any abdication: {out}"
);
if red("freezeaccepts") {
// The hollow-probe guard: assert refusal against a box we did NOT
// abdicate. It accepts, so this fires — proving the refusal detector can
// tell accepted from refused.
let (o, out) = push_once("freezeaccepts-red");
ensure!(
o != OpOutcome::Ok && out.contains(STANDBY_REFUSAL),
"RED `freezeaccepts` PROVEN: a client push to a NON-abdicated PRIMARY was ACCEPTED \
(outcome={o:?}), yet the abdicate probe treats this same check as meaning 'refused'. \
The freeze-refusal detector is therefore able to tell the two apart — which is what \
this red exists to show — and asserting 'refused' here correctly fails."
);
// Unreachable in red mode: the ensure! above always fires first.
}
// Abdicate → the box freezes to a read-only standby with an empty roster.
let (o, out) = ctl(server, &["replication", "abdicate"]);
ensure!(o == OpOutcome::Ok, "replication abdicate failed: {out}");
// A client push is now REFUSED with the standby sentence.
let (o, out) = push_once("post-abdicate");
ensure!(
o != OpOutcome::Ok && out.contains(STANDBY_REFUSAL),
"after `replication abdicate` the frozen box STILL accepted a client push (outcome={o:?}, \
out={out:?}). The freeze is hollow — a client mis-pointed at an abdicated box would take \
a write the new primary never sees."
);
// Promote → it is a writable primary again, and the same push now lands.
let (o, out) = ctl(server, &["replication", "promote"]);
ensure!(o == OpOutcome::Ok, "replication promote failed: {out}");
let (o, out) = push_once("post-promote");
ensure!(
o == OpOutcome::Ok,
"after `replication promote` the box did NOT accept a client push: {out}"
);
Ok(())
}
// ───────────────────────────────────────────────────────────────────────────
// the arm
// ───────────────────────────────────────────────────────────────────────────
struct OpStorm;
impl Bencher for OpStorm {
fn id(&self) -> &'static str {
"gunnar.opstorm"
}
fn run(&self) -> Result<BenchResult> {
ensure!(
!install_skipped(),
"gunnar.opstorm ran with GUNNAR_BENCH_SKIP_INSTALL set; the arm must be filtered out, \
not executed and reported"
);
let seed = env_u64("GUNNAR_OPSTORM_SEED", DEFAULT_SEED);
let ops = env_u64("GUNNAR_OPSTORM_OPS", DEFAULT_OPS);
let workers = env_u64("GUNNAR_OPSTORM_WORKERS", DEFAULT_WORKERS as u64) as usize;
let arenas = env_u64("GUNNAR_OPSTORM_ARENAS", DEFAULT_ARENAS as u64) as usize;
ensure!(
ops > 0,
"an opstorm with zero operations would report green having done nothing"
);
ensure!(workers > 0, "workers must be > 0");
ensure!(arenas > 0, "arenas must be > 0");
// `tiny`: the small harness the reds are reachable in. It is NOT a
// preview mode — it shrinks the corpus, so it may never emit a full-size
// row under the same name.
let arenas = if red("tiny") { arenas.min(2) } else { arenas };
let cli = locate_cli()?;
let work = opstorm_scratch();
let _ = std::fs::remove_dir_all(&work);
std::fs::create_dir_all(&work)?;
// The subject: an external appliance if `GUNNAR_OPSTORM_TARGET_SSH` and
// `_GRPC` are both given (attach — drive a running gunnar as a client
// over the network), otherwise a `gunnar serve` this arm spawns and owns
// (up). Only the constructor branches; the whole storm below is
// identical.
let external_target = std::env::var("GUNNAR_OPSTORM_TARGET_SSH")
.ok()
.zip(std::env::var("GUNNAR_OPSTORM_TARGET_GRPC").ok());
let server = if let Some((ssh, grpc)) = external_target {
let key = std::env::var("GUNNAR_OPSTORM_TARGET_ADMIN_KEY")
.map(PathBuf::from)
.map_err(|_| anyhow::anyhow!(
"GUNNAR_OPSTORM_TARGET_SSH/_GRPC are set but GUNNAR_OPSTORM_TARGET_ADMIN_KEY is \
not — external mode needs the admin PRIVATE key the storm signs its control \
calls with (public half enrolled on the appliance as the account principal)."
))?;
let account =
std::env::var("GUNNAR_OPSTORM_TARGET_ACCOUNT").unwrap_or_else(|_| "admin".into());
// The two owning-probes cannot fire against a remote PID-1 appliance
// — skip them LOUDLY: an honest N/A, never a silent green.
for probe in ["killmid", "corruptclone"] {
if red(probe) {
eprintln!(
"PROBE {probe}: N/A against an external appliance — needs to own the server"
);
}
}
attach(&cli, &ssh, &grpc, key, account)?
} else {
// Ports derived from the pid so two lanes on this box do not collide.
let base = 20000 + (std::process::id() % 8000) as u16;
up(&cli, &work, base, base + 10000)?
};
let boot_ms = server.started.elapsed().as_secs_f64() * 1000.0;
seed_arenas(&server, &work, arenas, seed)?;
let schedule = plan(seed, ops, arenas);
ensure!(
schedule.len() as u64 == ops,
"the schedule holds {} operations, not the {ops} asked for",
schedule.len()
);
let storm_scratch = work.join("storm");
std::fs::create_dir_all(&storm_scratch)?;
let t = Instant::now();
let s = storm(&server, &schedule, workers, &storm_scratch, seed);
let storm_ms = t.elapsed().as_secs_f64() * 1000.0;
let _ = std::fs::remove_dir_all(&storm_scratch);
// The arm's own deadline, reported before anything else.
ensure!(
s.abandoned == 0,
"the op storm hit its {STORM_DEADLINE:?} deadline with {} of {ops} operations unissued \
({} completed). Partial sample; do not compare with a complete run. First failure: {}",
s.abandoned,
s.done,
s.first_failure.as_deref().unwrap_or("none recorded"),
);
ensure!(
s.done == ops,
"the op storm completed {} of {ops} operations",
s.done
);
if let Some(f) = &s.first_failure {
bail!(
"{} of {} operations failed against the running server. First:\n {}\n\
--- server log tail ---\n{}",
s.failed,
ops,
f,
tail(&server.log_text(), 4000)
);
}
ensure!(
s.liveness_failed == 0,
"the control plane failed {} of {} liveness probes DURING the git storm — the \
separate-runtime guarantee did not hold. First: {}",
s.liveness_failed,
s.liveness_checks,
s.first_failure.as_deref().unwrap_or("(subsumed above)")
);
ensure!(
server.alive(),
"the server was not answering its control plane at the end of the storm; it died \
during the {ops} operations. Log:\n{}",
tail(&server.log_text(), 4000)
);
// Post-storm integrity: every arena still clonable + fsck-clean.
for a in 0..arenas {
let name = format!("arena{a}");
let dst = work.join(format!("post-{name}"));
let _ = std::fs::remove_dir_all(&dst);
let (o, out) = git_try(
&server,
None,
&[
"clone",
"-q",
"--bare",
&server.url(&name),
&dst.display().to_string(),
],
);
ensure!(
o == OpOutcome::Ok,
"post-storm clone of {name} failed: {out}"
);
let (fo, fout) = fsck_ok(&dst);
ensure!(
fo == OpOutcome::Ok,
"post-storm {name} did not fsck clean: {fout}"
);
let _ = std::fs::remove_dir_all(&dst);
}
// The replication / abdication phase.
abdicate_phase(&server, &work)?;
// On a shared external box, throw away everything the storm created that
// does not clean itself (the arenas + the abdicate phase's leftover
// branches inside them). The churn ops already self-clean.
if server.external {
cleanup_external(&server, arenas);
}
// ── metrics ───────────────────────────────────────────────────────
let cli_meta = std::fs::metadata(&cli).ok();
let mut metrics: Vec<(&'static str, Value)> = vec![
("seed_lo", json!((seed & 0xffff_ffff) as f64)),
("seed_hi", json!((seed >> 32) as f64)),
("ops", json!(ops as f64)),
("ops_failed", json!(s.failed as f64)),
("ops_timed_out", json!(s.timed_out as f64)),
("ops_abandoned", json!(s.abandoned as f64)),
("liveness_checks", json!(s.liveness_checks as f64)),
("liveness_failed", json!(s.liveness_failed as f64)),
("workers", json!(workers as f64)),
("arenas", json!(arenas as f64)),
("storm_ms", json!(storm_ms)),
(
"storm_ops_per_sec",
json!(ops as f64 / (storm_ms / 1000.0).max(1e-9)),
),
("boot_ms", json!(boot_ms)),
// 1.0 when the storm drove an EXTERNAL appliance (attach mode), 0.0
// when it spawned and owned its own `gunnar serve`.
("external", json!(if server.external { 1.0 } else { 0.0 })),
// Proven, not asserted: the abdicate phase ran green.
("abdicate_freeze_verified", json!(1.0)),
(
"cli_bytes",
json!(cli_meta.as_ref().map(|m| m.len()).unwrap_or(0) as f64),
),
];
for (k, v) in &s.per_op {
metrics.push((leak(format!("op_{k}")), json!(*v as f64)));
}
Ok(result(self.id(), metrics))
}
}
nornir_bench::register_bench!(OpStorm);