xtask/src/benchland.rs
//! `cargo xtask bench-landed` — did the measurement reach BOTH stores?
//!
//! Ported from gunnar's `xtask/src/benchland.rs` (2026-09-20). The reasoning
//! below is gunnar's, restated for holger; the code is holger's own and parses
//! with `serde_json`, which this xtask already depends on, rather than carrying
//! a second hand-rolled JSON reader across the repo boundary.
//!
//! # The two artefacts, and why they must not be able to disagree
//!
//! One `nornir bench run holger` produces one `BenchRun` value and writes it
//! twice, from one producer, in one process:
//!
//! * into the iceberg `bench_runs` table — the record everything queries;
//! * into `bench_history.jsonl` at the repo root — one JSON object per run, a
//! durable receipt that needs no catalog, no server and no Iceberg reader.
//! It is TRACKED IN GIT (owner's ruling, 2026-09-20), because it is the
//! backup the doc mashup renders from when nornir/mimir cannot be opened on
//! the machine building the docs: a measurement that exists only in a
//! warehouse nobody can open is a page that says NOT MEASURED. dwarves
//! already tracks its own `bench_history.jsonl`; gunnar gitignores its one on
//! the grounds that committing it makes every measurement a merge conflict.
//! That objection is real and is answered rather than ignored — the file is
//! append-only JSONL and `.gitattributes` gives it `merge=union`, so two runs
//! on two branches both survive a merge instead of colliding.
//!
//! Two writes of one value can only stay honest if something checks. On
//! 2026-08-12, in gunnar, ten runs landed in the receipt and **none** of them in
//! any warehouse, and nothing anywhere went red: they were taken through
//! `dwarves bench run`, whose forge door writes the receipt and then submits to
//! nornir over `Bench.Submit` **only if a server endpoint is configured**. With
//! none configured it printed `NOT SUBMITTED — no nornir server configured`,
//! reported no failure, and exited 0. The measurement was real, the receipt was
//! written, the row never existed, and the exit status said the run was fine.
//! Check output, never an exit code.
//!
//! holger's version of that defect was larger and quieter still: until today no
//! holger measurement had ever been taken through a persisting door at all. All
//! 15 arms in `xtask/examples/nornir-bench.rs` were run by invoking the example
//! directly and redirecting stdout to a file, so `nornir bench history-show
//! holger` answered `no bench runs recorded for 'holger'` while bench output
//! sat in `bench-results/`.
//!
//! # What this asserts
//!
//! **The newest receipt has a warehouse row.** That is the invariant a person
//! actually depends on — *the run I just did is in both places* — and it is the
//! one an every-run guard can hold without going permanently red over history
//! it cannot retroactively fix.
//!
//! The agreement over the WHOLE receipt is reported as a ratio rather than a
//! raw count, because "1 of 47 receipts is missing" and "10 of 47 are missing"
//! are different situations and a bare pass/fail hides which one you are in.
//!
//! # Matching a receipt to a row
//!
//! By timestamp truncated to the second, plus the run's arm-name set. The two
//! stores do not agree on sub-second precision — the receipt keeps nanoseconds
//! (`…T19:18:53.941849574+00:00`) and the row comes back with microseconds
//! (`…T19:18:53.941849+00:00`) — so an equality on the full string reports
//! every run as missing, which is a guard that is red for a reason that has
//! nothing to do with the defect. The arm-name set is carried as well so that
//! two runs inside one second cannot be matched to each other by accident.
//!
//! # What it cannot assert, and says so instead of guessing
//!
//! In CENTRAL mode (`NORNIR_SERVER` set) `nornir bench run` delegates the whole
//! measurement to the server's `Ops.RunBench`, which runs the example in the
//! SERVER's checkout and appends the row there. No receipt is written on this
//! machine, so a receipt-versus-warehouse comparison here has nothing local to
//! compare and would be red for the wrong reason. [`run`] reports that case as
//! [`Landing::NoLocalReceipt`] rather than as a defect, and `cargo xtask bench`
//! still proves the row landed — by the warehouse delta it takes around the run,
//! which works in both modes.
use std::collections::BTreeSet;
use std::path::Path;
use serde_json::Value;
/// The receipt, relative to the repo root. Same name the producer writes
/// (`nornir_bench::history::append`), not a second convention.
pub const RECEIPT: &str = "bench_history.jsonl";
/// One run, identified the way both stores can agree on.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]
pub struct RunKey {
/// The timestamp truncated to the second, e.g. `2026-09-20T09:18:53`.
pub second: String,
/// The arm names the run carried, sorted and deduplicated.
pub arms: BTreeSet<String>,
}
impl RunKey {
pub fn label(&self) -> String {
format!(
"{} [{}]",
self.second,
self.arms.iter().cloned().collect::<Vec<_>>().join(", ")
)
}
}
/// The timestamp of one run object, truncated to the second.
///
/// `timestamp` first and `date` second: a run written before the timestamp
/// column existed carries only a date, and it is still a run.
fn second_of(run: &Value) -> Option<String> {
let raw = run
.get("timestamp")
.and_then(Value::as_str)
.or_else(|| run.get("date").and_then(Value::as_str))?;
let cut = raw.find('.').unwrap_or(raw.len());
let s = raw[..cut].trim();
(!s.is_empty()).then(|| s.to_string())
}
fn key_of(run: &Value) -> Option<RunKey> {
let arms = run
.get("results")
.and_then(Value::as_array)
.map(|rs| {
rs.iter()
.filter_map(|r| r.get("name").and_then(Value::as_str).map(str::to_string))
.collect()
})
.unwrap_or_default();
Some(RunKey {
second: second_of(run)?,
arms,
})
}
/// Every run in the receipt, oldest first — the order the producer appends in.
///
/// A line that does not parse is an error and never a silent skip: a truncated
/// or half-written receipt line is exactly the shape of a run that was lost, so
/// dropping it would hide the defect this file exists to find.
pub fn receipt_runs(text: &str) -> Result<Vec<RunKey>, String> {
let mut out = Vec::new();
for (i, line) in text.lines().enumerate() {
if line.trim().is_empty() {
continue;
}
let v: Value =
serde_json::from_str(line).map_err(|e| format!("{RECEIPT} line {}: {e}", i + 1))?;
let key = key_of(&v).ok_or_else(|| format!("{RECEIPT} line {}: no timestamp", i + 1))?;
out.push(key);
}
Ok(out)
}
/// What the warehouse holds for this repo, out of
/// `nornir bench history-show holger --json`.
///
/// An empty table answers `{"ok":false,"data":null,…}` — `data.runs` is then
/// absent, which reads as "no rows", which is exactly what it means.
#[derive(Debug, Clone, Default, PartialEq)]
pub struct Warehouse {
/// Rows this file could key.
pub keys: Vec<RunKey>,
/// Rows it could NOT key, one label each. NEVER silently dropped.
///
/// The mashup incident (law file, "WHY THE BENCH NUMBERS WERE NEVER SAVED
/// OR READABLE", defect 2): the reader stopped on the newest run rather
/// than the newest READABLE one and then reported absence. A run this file
/// cannot key is a run it must NAME, because "I skipped it" and "it is not
/// there" are different answers and only one of them is actionable.
pub unreadable: Vec<String>,
}
pub fn warehouse_runs(doc: &Value) -> Warehouse {
let rows = doc
.get("data")
.and_then(|d| d.get("runs"))
.and_then(Value::as_array)
.cloned()
.unwrap_or_default();
let mut w = Warehouse::default();
for (i, row) in rows.iter().enumerate() {
match key_of(row) {
Some(k) => w.keys.push(k),
None => w.unreadable.push(format!(
"row {} (machine {}, date {}) carries no timestamp and no date",
i,
row.get("machine").and_then(Value::as_str).unwrap_or("-"),
row.get("date").and_then(Value::as_str).unwrap_or("-"),
)),
}
}
w
}
/// What one comparison found.
#[derive(Debug, Clone, PartialEq)]
pub struct Agreement {
/// Runs in the receipt, oldest first.
pub receipts: Vec<RunKey>,
/// Receipts with no warehouse row, oldest first.
pub orphans: Vec<RunKey>,
/// The newest receipt, and whether it landed. `None` ⇒ the receipt is empty,
/// which is not a pass: there is nothing to have landed.
pub newest: Option<(RunKey, bool)>,
/// Warehouse rows that could not be keyed — named, never dropped.
pub unreadable: Vec<String>,
}
impl Agreement {
/// The fraction of receipts that reached the warehouse. `None` for an empty
/// receipt — a ratio over nothing is not 100%.
pub fn ratio(&self) -> Option<f64> {
let n = self.receipts.len();
(n > 0).then(|| (n - self.orphans.len()) as f64 / n as f64)
}
/// RED when the newest measurement is not in both stores.
///
/// An empty receipt is red too. `bench_history.jsonl` absent or empty means
/// no producer has written one, and answering "nothing disagrees" to that is
/// the same empty-is-clean confusion the whole module is about.
pub fn is_red(&self) -> bool {
!matches!(self.newest, Some((_, true)))
}
pub fn summary(&self) -> String {
let ratio = match self.ratio() {
Some(r) => format!(
"{}/{} receipt(s) have a warehouse row ({:.0}%)",
self.receipts.len() - self.orphans.len(),
self.receipts.len(),
r * 100.0
),
None => format!("{RECEIPT} holds no run at all"),
};
let skipped = if self.unreadable.is_empty() {
String::new()
} else {
format!(
" · {} warehouse row(s) UNREADABLE, not counted as absent: {}",
self.unreadable.len(),
self.unreadable.join("; ")
)
};
match &self.newest {
Some((k, true)) => {
format!(
"newest run {} is in BOTH stores · {ratio}{skipped}",
k.label()
)
}
Some((k, false)) => format!(
"NEWEST RUN {} IS IN THE RECEIPT AND NOT IN THE WAREHOUSE · {ratio}{skipped}",
k.label()
),
None => format!("no run to check · {ratio}{skipped}"),
}
}
}
/// The verdict of one `bench-landed`.
#[derive(Debug, Clone, PartialEq)]
pub enum Landing {
/// Both stores were read and compared.
Compared(Agreement),
/// No `bench_history.jsonl` in this checkout, and the warehouse holds rows
/// for holger. That is what CENTRAL mode looks like from here: the server
/// ran the example in its own checkout and wrote the receipt there.
NoLocalReceipt { warehouse_rows: usize },
}
impl Landing {
pub fn is_red(&self) -> bool {
match self {
Landing::Compared(a) => a.is_red(),
// Not a defect and not a pass: nothing local was measured. The
// caller prints it and exits 0, because `cargo xtask bench` already
// proved the row by the warehouse delta.
Landing::NoLocalReceipt { .. } => false,
}
}
pub fn summary(&self) -> String {
match self {
Landing::Compared(a) => a.summary(),
Landing::NoLocalReceipt { warehouse_rows } => format!(
"no {RECEIPT} in this checkout; the warehouse holds {warehouse_rows} run(s) for \
holger. That is CENTRAL mode: the server ran the example in ITS checkout and \
wrote the receipt there. Nothing local to compare."
),
}
}
}
/// Compare the two stores.
pub fn compare(receipts: Vec<RunKey>, warehouse: &Warehouse) -> Agreement {
let have: BTreeSet<&RunKey> = warehouse.keys.iter().collect();
let orphans: Vec<RunKey> = receipts
.iter()
.filter(|k| !have.contains(k))
.cloned()
.collect();
let newest = receipts
.iter()
.max_by(|a, b| a.second.cmp(&b.second))
.map(|k| (k.clone(), have.contains(k)));
Agreement {
receipts,
orphans,
newest,
unreadable: warehouse.unreadable.clone(),
}
}
/// Read both stores off disk and compare them.
pub fn run(root: &Path, config: &Path) -> Result<Landing, String> {
let doc: Value = serde_json::from_str(&crate::bench::history_json(root, config)?)
.map_err(|e| format!("bench history-show did not parse: {e}"))?;
let warehouse = warehouse_runs(&doc);
let receipt = root.join(RECEIPT);
let text = match std::fs::read_to_string(&receipt) {
Ok(t) => t,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => {
return Ok(Landing::NoLocalReceipt {
warehouse_rows: warehouse.keys.len(),
});
}
Err(e) => return Err(format!("read {}: {e}", receipt.display())),
};
Ok(Landing::Compared(compare(receipt_runs(&text)?, &warehouse)))
}
#[cfg(test)]
mod tests {
use super::*;
/// A receipt line / warehouse run, shaped like the real ones.
fn run_json(ts: &str, arms: &[&str]) -> String {
let results: Vec<String> = arms
.iter()
.map(|a| format!("{{\"name\":\"{a}\",\"x\":1.0}}"))
.collect();
format!(
"{{\"timestamp\":\"{ts}\",\"machine\":\"oden\",\"results\":[{}]}}",
results.join(",")
)
}
fn warehouse(runs: &[String]) -> Value {
serde_json::from_str(&format!("{{\"data\":{{\"runs\":[{}]}}}}", runs.join(","))).unwrap()
}
/// ★ THE 2026-08-12 DEFECT, as a test.
///
/// RED before this file existed, because nothing compared the two stores at
/// all.
#[test]
fn a_receipt_with_no_warehouse_row_is_red_and_says_which_run() {
let receipts = receipt_runs(&format!(
"{}\n{}\n",
run_json("2026-09-19T19:18:53.941849574+00:00", &["blake3_256MiB"]),
run_json("2026-09-20T11:58:34.147389315+00:00", &["memcopy_512MiB"]),
))
.unwrap();
// The warehouse has only the older one.
let wh = warehouse(&[run_json(
"2026-09-19T19:18:53.941849+00:00",
&["blake3_256MiB"],
)]);
let a = compare(receipts, &warehouse_runs(&wh));
assert!(a.is_red(), "{}", a.summary());
assert_eq!(a.orphans.len(), 1);
assert_eq!(a.orphans[0].second, "2026-09-20T11:58:34");
assert!(
a.summary().contains("NOT IN THE WAREHOUSE"),
"{}",
a.summary()
);
assert_eq!(a.ratio(), Some(0.5));
}
/// GREEN, and green for the right reason: the two stores disagree on
/// sub-second precision and must still match.
#[test]
fn nanosecond_receipts_match_microsecond_rows() {
let receipts =
receipt_runs(&run_json("2026-09-19T19:18:53.941849574+00:00", &["a"])).unwrap();
let wh = warehouse(&[run_json("2026-09-19T19:18:53.941849+00:00", &["a"])]);
let a = compare(receipts, &warehouse_runs(&wh));
assert!(!a.is_red(), "{}", a.summary());
assert_eq!(a.ratio(), Some(1.0));
assert!(a.summary().contains("BOTH stores"), "{}", a.summary());
}
/// Two runs inside one second are not each other. The arm set is what keeps
/// the truncation above from becoming a false match.
#[test]
fn the_same_second_with_different_arms_is_not_a_match() {
let receipts = receipt_runs(&run_json(
"2026-09-19T19:18:53.100000000+00:00",
&["blake3_256MiB"],
))
.unwrap();
let wh = warehouse(&[run_json(
"2026-09-19T19:18:53.900000+00:00",
&["memcopy_512MiB"],
)]);
assert!(compare(receipts, &warehouse_runs(&wh)).is_red());
}
/// An empty receipt is not a pass. Nothing to check is not the same as
/// nothing wrong.
#[test]
fn an_empty_receipt_is_red_rather_than_vacuously_green() {
let a = compare(receipt_runs("").unwrap(), &Warehouse::default());
assert!(a.is_red(), "{}", a.summary());
assert_eq!(a.ratio(), None);
assert!(a.summary().contains("no run"), "{}", a.summary());
}
/// A malformed receipt line is an error, never a skipped line: a
/// half-written line is what a lost run looks like.
#[test]
fn a_broken_receipt_line_is_reported_rather_than_skipped() {
let err = receipt_runs("{\"timestamp\":\"t\",\"results\":[]}\n{not json\n").unwrap_err();
assert!(err.contains("line 2"), "{err}");
}
/// The empty warehouse answers `data: null`, and that must read as zero
/// rows rather than panicking or looking like a full table.
#[test]
fn an_empty_warehouse_document_reads_as_no_rows() {
let doc: Value = serde_json::from_str(
"{\"command\":\"bench history-show\",\"ok\":false,\"data\":null,\
\"human\":\"no bench runs recorded for `holger`\"}",
)
.unwrap();
assert!(warehouse_runs(&doc).keys.is_empty());
}
/// holger's OWN shape today, before this lane: a receipt that has never
/// been written and a warehouse with no holger rows is RED, not green.
#[test]
fn nothing_anywhere_is_red() {
let a = compare(Vec::new(), &Warehouse::default());
assert!(a.is_red(), "{}", a.summary());
}
}