crates/storage-git-gix/parked-gix-2/bench-gix/walkbudget-probe.rs
//! **What `DEFAULT_BUDGET` actually counts, and what a walk of that size
//! costs.**
//!
//! ```text
//! walkbudget-probe <repo.git> [repo.git ...]
//! ```
//!
//! `DEFAULT_BUDGET` is 10 000 000 and its comment says "the linux kernel is
//! around ten million objects total". The kernel is **13 816 503**, so the
//! ceiling sits 28% under the repository it names. That would be a tuning
//! mistake if the budget were charged against something incidental. It is not:
//! `Graph::tree_closure` charges against `Emit::len()`, the objects the server
//! is about to **send**, so the ceiling is a ceiling on **clone size** and a
//! bitmap-less clone of a repository larger than it ends in `WalkBudget`
//! instead of in a pack.
//!
//! # The boundary is SEARCHED FOR, not assumed, and that is the whole test
//!
//! The obvious probe asserts that `objects_sent - 1` fails. That probe was
//! written first and it **failed on the first repository**: a budget of 82
//! served bifrost's 83 objects and did not stop. "The budget is the pack's
//! object count" is therefore false as stated, and a probe asserting it would
//! have been reporting a defect it had not established.
//!
//! What was true then was narrower, and it was the defect: `tree_closure` tested
//! `out.len() >= budget` against the tree walk's own emission, while the
//! commits `commit_closure` produced were counted by a different check against
//! a different accumulator. The budget therefore bound on a **fraction** of the
//! pack — 0.84 to 0.94 — and which fraction was a property of the repository's
//! shape rather than of the number an operator typed. Since the charge moved to
//! the ids a walk HOLDS (`Graph::charge_walk`) it binds at 1.057, 1.057 and
//! 1.058 on the three largest corpora here: the same ceiling meaning the same
//! thing everywhere, and above 1.0 because a clone holds its peeled tips and
//! its tags as well as its pack.
//!
//! This therefore binary-searches for `min_budget`, the smallest budget at
//! which the clone completes, and reports it as a fraction of `objects_sent`.
//! Both edges are asserted every time:
//!
//! | budget | must |
//! |---|---|
//! | `min_budget` | complete, sending exactly `objects_sent` |
//! | `min_budget - 1` | fail with `WalkBudget`, naming that limit |
//!
//! Both, not either. "It failed at a small budget" is satisfied by any walk
//! that charges anything at all; only the pair establishes where the edge is.
//! And because the edge is measured rather than assumed, the arithmetic against
//! `DEFAULT_BUDGET` is done on the ratio this found and not on an object count.
//!
//! # The memory, measured rather than estimated
//!
//! `Config::budget`'s own doc says that at `DEFAULT_BUDGET` "the id bookkeeping
//! alone … is on the order of a gigabyte". That is an estimate and it is the
//! number the decision to raise the ceiling turns on, so this measures it:
//! `VmHWM` for a process that serves exactly one clone and does nothing else.
//! One repository per process — `VmHWM` is a high-water mark that never falls,
//! so several corpora in one process would report the largest one's peak for
//! all of them. The parent re-execs itself per repository for that reason and
//! for no other.
//!
//! `bytes_per_object` off the largest corpus is the slope a raised ceiling
//! should be costed against. It is **not** a claim about the kernel, and the
//! `--scale` arm below is why that caveat was worth writing down.
//!
//! # `--scale`: the kernel-sized measurement, and what the extrapolation got wrong
//!
//! The ladder's largest corpus holds 352 396 objects and reports 627.6 B per
//! emitted object. Extrapolated 39× that says a 13.8M-object clone would hold
//! ~8.7 GB. **It holds 5.59 GB.** MEASURED oden 2026-08-05 on a purpose-built
//! 14 042 000-object fixture (1.6% past the kernel's 13 816 503), one pack, no
//! `.bitmap`:
//!
//! | | objects | wire | `VmHWM` | B/object | wall |
//! |---|---|---|---|---|---|
//! | ladder top | 352 396 | 64.9 MB | 219 900 kB | 627.6 | — |
//! | `--scale` | **14 042 000** | 599.0 MB | **5 460 816 kB** | **393.8** | 64 s |
//!
//! The slope was still falling, exactly as the caveat said, and the inferred
//! figure was **1.6× too high**. So the honest reading of the ladder is that
//! `bytes_per_object` is an *upper* bound that loosens with distance, and the
//! only way to cost a ceiling at a scale is to build a corpus at that scale.
//!
//! The other half of that run is the fact the ceiling is about: at the shipped
//! `DEFAULT_BUDGET` of 10 000 000 that clone **is refused**, `WalkBudget`,
//! measured rather than inferred. A kernel-sized repository with no `.bitmap`
//! cannot be cloned from stock gunnar, and the machine that cannot serve it had
//! 5.59 GB of headroom to spare.
//!
//! ## Rebuilding that corpus
//!
//! It is not in the fixture tree and it should not be: 947 MB of disk to hold
//! objects whose only property is that there are fourteen million of them. The
//! recipe is four minutes of `git fast-import`, and it is written down here
//! rather than kept in somebody's shell history, because a measurement nobody
//! can repeat is an anecdote.
//!
//! Per commit: `deleteall`, then `FILES` fresh unique blobs under one directory
//! named for the commit — `FILES` blobs, one subtree, one root tree and one
//! commit, with every earlier commit's objects still reachable from the tip.
//! Chosen for object count per byte of disk: the root tree never grows, so the
//! tree bytes are `FILES` entries once per commit rather than the whole corpus
//! rewritten every time. 14 000 commits x 1 000 files is 14 042 000 objects.
//!
//! ```python
//! import sys
//! commits, files = int(sys.argv[1]), int(sys.argv[2])
//! w = sys.stdout.buffer
//! for c in range(commits):
//! base = c * (files + 1)
//! for j in range(files):
//! body = b"o %d\n" % (base + j)
//! w.write(b"blob\nmark :%d\ndata %d\n%s" % (base + j + 1, len(body), body))
//! cmark, at, msg = base + files + 1, 1700000000 + c, b"c%d\n" % c
//! w.write(b"commit refs/heads/main\nmark :%d\n"
//! b"author Gunnar Bench <bench@gunnar.invalid> %d +0000\n"
//! b"committer Gunnar Bench <bench@gunnar.invalid> %d +0000\n"
//! b"data %d\n%s" % (cmark, at, at, len(msg), msg))
//! if c:
//! w.write(b"from :%d\n" % base)
//! w.write(b"deleteall\n")
//! for j in range(files):
//! w.write(b"M 100644 :%d d%d/f%d\n" % (base + j + 1, c, j))
//! w.write(b"\n")
//! w.write(b"done\n")
//! ```
//!
//! ```text
//! git init --bare kernelscale.git
//! python3 mkcorpus.py 14000 1000 | git -C kernelscale.git fast-import --quiet
//! ```
//!
//! `fast-import` writes one pack and **no** `.bitmap`, which is the shape this
//! arm needs and asserts. Delete it when the measurement is taken; it is
//! disposable and this recipe is the part worth keeping.
use anyhow::{bail, ensure, Context, Result};
use gix_hash::ObjectId;
use gunnar_bench::stages;
use gunnar_store::fs::FsStore;
use gunnar_store::ObjectStore;
use gunnar_wire::upload_pack::{Config, UploadPack};
use gunnar_wire::{Error, DEFAULT_BUDGET};
use std::io::Write;
/// **The walk path, on a repository that has a bitmap.**
///
/// Every mirror in this corpus was made by `git clone --mirror`, which packs
/// and writes a `.bitmap`, and a bitmap-answered clone does not walk at all:
/// the first version of this probe served bifrost's whole 83-object clone at a
/// budget of **one**, because `tree_closure` was never entered. That is a true
/// fact about the shipped path and it is reported as `bitmap` below — but it is
/// not the path `DEFAULT_BUDGET` is a ceiling on.
///
/// So the walk arm runs against this. It cuts **one** seam, `bitmap`, and
/// forwards everything else including `packed` — P-020 rule 4: a wrapper that
/// reaches a second behaviour by leaving a trait default in place is how
/// `NoPackCopy` silently disabled bitmaps for every arm it was the BEFORE
/// column for. Every method below is written out for that reason.
struct NoBitmap<'a>(&'a dyn ObjectStore);
impl ObjectStore for NoBitmap<'_> {
fn hash_kind(&self) -> gix_hash::Kind {
self.0.hash_kind()
}
fn contains(&self, id: &gix_hash::oid) -> bool {
self.0.contains(id)
}
fn contains_without_rescan(&self, id: &gix_hash::oid) -> bool {
self.0.contains_without_rescan(id)
}
fn read<'b>(
&self,
id: &gix_hash::oid,
buf: &'b mut Vec<u8>,
) -> gunnar_store::Result<Option<gix_object::Data<'b>>> {
self.0.read(id, buf)
}
fn header(&self, id: &gix_hash::oid) -> gunnar_store::Result<Option<gix_object::Header>> {
self.0.header(id)
}
fn write(&self, kind: gix_object::Kind, bytes: &[u8]) -> gunnar_store::Result<ObjectId> {
self.0.write(kind, bytes)
}
/// Forwarded. The walk still pack-copies; only the reachability shortcut is
/// gone.
fn packed(&self) -> Option<gunnar_pack::PackedObjects> {
self.0.packed()
}
/// P-020: deliberately not forwarded. **This is the one seam this type
/// cuts, and the whole reason it exists.**
///
/// `DEFAULT_BUDGET` is a ceiling on the WALK, and a bitmap-answered clone
/// does not walk: the first version of this probe served bifrost's entire
/// 83-object clone at a budget of one, because `tree_closure` was never
/// entered. Every mirror in this corpus has a `.bitmap`, so without this
/// `None` the probe would search for the edge of a ceiling nothing on the
/// measured path ever reaches — a whole table of budgets taken off a code
/// path that was not running.
///
/// It is confined to this probe on purpose. Nothing in `crates/` may cut
/// this seam; the shipped read side keeps its bitmap, and the reason the
/// walk still matters in production is that a repository pushed into and
/// never repacked has no bitmap to keep.
fn bitmap(&self) -> Option<std::sync::Arc<gunnar_store::bitmap::PackBitmap>> {
None
}
}
/// A sink that keeps the byte count and throws the bytes away: a clone of a
/// 352 396-object repository is not held in memory to be measured, because
/// holding it would be the measurement.
#[derive(Default)]
struct Sink {
bytes: u64,
}
impl Write for Sink {
fn write(&mut self, data: &[u8]) -> std::io::Result<usize> {
self.bytes += data.len() as u64;
Ok(data.len())
}
fn flush(&mut self) -> std::io::Result<()> {
Ok(())
}
}
/// A field of `/proc/self/status`, in kB. `VmHWM` is the peak resident set of
/// this process since it started; `VmRSS` is what it holds now.
fn status_kb(field: &str) -> u64 {
let Ok(text) = std::fs::read_to_string("/proc/self/status") else {
return 0;
};
text.lines()
.find_map(|line| line.strip_prefix(field)?.trim().strip_suffix(" kB"))
.and_then(|v| v.trim().parse().ok())
.unwrap_or(0)
}
/// One clone served at `budget`, returning the summary or the wire error.
fn serve(
repo: &std::path::Path,
want: &ObjectId,
budget: usize,
walk: bool,
) -> Result<Result<(u32, u64)>> {
served_under(repo, want, walk, |config| Config {
budget: Some(budget),
..config
})
}
/// One clone served under whatever `configure` makes of the shipping config.
///
/// `Config::default()` and not a hand-built one: `Limits::walk_bytes` is `Some`
/// by default and `..Default::default()` on a struct literal would make it
/// `None`, which is how a probe comes to measure a server nobody runs.
fn served_under(
repo: &std::path::Path,
want: &ObjectId,
walk: bool,
configure: impl FnOnce(Config) -> Config,
) -> Result<Result<(u32, u64)>> {
let kind = gunnar_store::fs::stored_hash_kind(repo)?;
let store = FsStore::open_or_init(repo, kind)?;
let objects = store.objects();
let refs = store.refs();
let stripped = NoBitmap(&objects);
let read_side: &dyn ObjectStore = if walk { &stripped } else { &objects };
let format = match kind {
gix_hash::Kind::Sha1 => "sha1",
_ => "sha256",
};
let upload = UploadPack::new(read_side, &refs).with_config(configure(Config::default()));
let mut input = std::io::BufReader::new(std::io::Cursor::new(stages::fetch_request(
want,
&[],
format,
)));
let mut sink = Sink::default();
Ok(match upload.serve_v2(&mut input, &mut sink) {
Ok(summary) => Ok((summary.objects_sent, sink.bytes)),
Err(err) => Err(err.into()),
})
}
/// The tip to clone. Taken from the repository rather than assumed to be
/// `HEAD`: several fixtures here are mirrors whose `HEAD` is unborn.
fn tip(repo: &std::path::Path) -> Result<ObjectId> {
let out = std::process::Command::new("git")
.args([
"-C",
&repo.to_string_lossy(),
"rev-list",
"-n",
"1",
"--all",
])
.output()
.context("running git rev-list")?;
ensure!(
out.status.success(),
"git rev-list failed in {repo:?}: {}",
String::from_utf8_lossy(&out.stderr).trim()
);
String::from_utf8_lossy(&out.stdout)
.lines()
.find_map(|l| ObjectId::from_hex(l.trim().as_bytes()).ok())
.with_context(|| format!("{repo:?} holds no commit, so it cannot be cloned"))
}
/// Which side of the ceiling one budget falls on.
///
/// A refusal is **only** `WalkBudget` naming that budget. Anything else — a
/// missing object, a broken pack, an unparsable commit — is an error and not a
/// data point, because a search that read those as "refused" would converge on
/// the boundary of a different fault.
enum Edge {
Refused,
Served(u32),
}
fn walk_budget(repo: &std::path::Path, want: &ObjectId, budget: usize) -> Result<Edge> {
match serve(repo, want, budget, true)? {
Ok((sent, _)) => Ok(Edge::Served(sent)),
Err(err) => match err.downcast_ref::<Error>() {
Some(Error::WalkBudget { limit }) => {
ensure!(
*limit == budget,
"{repo:?}: a budget of {budget} was refused naming a limit of {limit}"
);
Ok(Edge::Refused)
}
_ => bail!(
"{repo:?}: a budget of {budget} failed with {err:#}, which is not WalkBudget — \
this probe would then be searching for the edge of a different fault"
),
},
}
}
/// One repository, in this process, so `VmHWM` belongs to it alone.
fn one(repo: &std::path::Path) -> Result<()> {
let want = tip(repo)?;
let base_hwm = status_kb("VmHWM:");
// ── the arm that must COMPLETE ─────────────────────────────────────────
let (objects, bytes) = serve(repo, &want, DEFAULT_BUDGET, true)?
.with_context(|| format!("{repo:?}: the clone failed at the shipped budget"))?;
ensure!(
objects > 0,
"{repo:?}: the clone sent ZERO objects, and a walk that emits nothing cannot exceed any \
budget — every arm below would pass against nothing"
);
let hwm = status_kb("VmHWM:");
let rss = status_kb("VmRSS:");
// ── where the ceiling actually binds ───────────────────────────────────
//
// Binary search for the smallest budget that still serves the whole clone.
// A budget of 1 must fail on any repository with objects in it — if it does
// not, the walk is not metered at all and every number below is fiction.
ensure!(
matches!(walk_budget(repo, &want, 1)?, Edge::Refused),
"{repo:?}: a budget of ONE object served the entire clone, so this walk is not metered \
and the ceiling this probe is about does not exist on this path"
);
// The upper bound is SEARCHED FOR, not taken from the object count.
//
// It used to be `objects`, on the reasoning that a budget the size of the
// pack must serve the pack. That stopped being true when the charge moved
// from what a walk emits to what it holds: a clone also holds its peeled
// tips and its tags, so the ceiling binds a little ABOVE the object count
// and the search settled on a budget at which the clone does not serve.
// The probe's own re-check caught it — "the search settled on 190, at which
// the clone does not serve" — which is the assertion earning its keep.
let mut hi = objects as usize;
loop {
match walk_budget(repo, &want, hi)? {
Edge::Served(_) => break,
Edge::Refused => {
hi = hi
.checked_mul(2)
.context("no budget below usize::MAX serves this clone")?
}
}
}
let mut lo = 1usize;
while lo + 1 < hi {
let mid = lo + (hi - lo) / 2;
match walk_budget(repo, &want, mid)? {
Edge::Refused => lo = mid,
Edge::Served(sent) => {
ensure!(
sent == objects,
"{repo:?}: a budget of {mid} served {sent} objects where the shipped budget \
served {objects}. A budget that changes the ANSWER rather than refusing is \
a far worse defect than the one being measured."
);
hi = mid;
}
}
}
let min_budget = hi;
// Both edges, every time. `lo` is `min_budget - 1` by construction and the
// search already saw it refuse; this re-asserts that the refusal is
// `WalkBudget` naming that exact limit and not some other failure the
// search would have read as "refused".
match walk_budget(repo, &want, min_budget - 1)? {
Edge::Refused => {}
Edge::Served(_) => bail!(
"{repo:?}: the search settled on {min_budget} but {} served the clone",
min_budget - 1
),
}
let (again, _) = serve(repo, &want, min_budget, true)?.with_context(|| {
format!("{repo:?}: the search settled on {min_budget}, at which the clone does not serve")
})?;
ensure!(
again == objects,
"{repo:?}: the same clone sent {objects} then {again} objects"
);
let tight = min_budget - 1;
let grew = hwm.saturating_sub(base_hwm);
println!(
"{{\"repo\":{:?},\"objects\":{},\"wire_bytes\":{},\"vmhwm_kb\":{},\"vmrss_kb\":{},\
\"walk_growth_kb\":{},\"bytes_per_object\":{:.1},\
\"min_budget\":{},\"budget_fraction\":{:.4},\"default_budget\":{}}}",
repo.file_name().unwrap_or_default().to_string_lossy(),
objects,
bytes,
hwm,
rss,
grew,
(grew as f64 * 1024.0) / f64::from(objects),
min_budget,
min_budget as f64 / f64::from(objects),
DEFAULT_BUDGET,
);
eprintln!(
"{:<30} obj {:>8} wire {:>11} VmHWM {:>7} kB grew {:>7} kB {:>6.1} B/obj \
binds at {:>7} ({:.3} of the pack), refuses at {} ✓",
repo.file_name().unwrap_or_default().to_string_lossy(),
objects,
bytes,
hwm,
grew,
(grew as f64 * 1024.0) / f64::from(objects),
min_budget,
min_budget as f64 / f64::from(objects),
tight,
);
Ok(())
}
/// The kernel's object count, as reported by `git count-objects -v` on a full
/// mirror. Carried as a number so the arithmetic below is visible rather than
/// prose, and labelled everywhere it is used.
const LINUX_OBJECTS: u64 = 13_816_503;
/// **The arm that does not search: one repository at kernel scale.**
///
/// [`one`] binary-searches for the binding edge, which costs `log2(objects)`
/// whole clones. At 14 million objects that is two dozen multi-minute clones to
/// re-establish a fraction the ladder has already measured four times, so this
/// arm asks the two questions the ladder cannot:
///
/// | budget | must |
/// |---|---|
/// | [`DEFAULT_BUDGET`] | fail with `WalkBudget`, naming it |
/// | `objects` | complete, sending every object, and report `VmHWM` |
///
/// Both. The first alone is satisfied by a repository that cannot be served at
/// all; the second alone does not establish that the shipped ceiling was in the
/// way. Together they are "the shipped server refuses this clone, and the
/// machine can serve it".
///
/// **The bitmap is asserted absent rather than shimmed away.** [`NoBitmap`]
/// exists because every mirror in the ladder corpus has a `.bitmap` and a
/// bitmap-answered clone never enters the walk. A fixture written by
/// `git fast-import` has none, so this arm reads the real store and *checks*
/// that — which is the production shape (pushed into, never repacked) rather
/// than a shim standing in for it.
fn scale(repo: &std::path::Path) -> Result<()> {
let want = tip(repo)?;
let kind = gunnar_store::fs::stored_hash_kind(repo)?;
{
let store = FsStore::open_or_init(repo, kind)?;
ensure!(
store.objects().bitmap().is_none(),
"{repo:?} has a .bitmap, and a bitmap-answered clone never enters the walk this \
measures — every number below would be taken off a code path that was not running"
);
}
let counted = counted_objects(repo)?;
let base_hwm = status_kb("VmHWM:");
// ── 1. the ceiling that SHIPPED must refuse ────────────────────────────
//
// `DEFAULT_BUDGET` is no longer the default; this asks for it explicitly,
// so the before-picture stays measured rather than becoming a claim about
// a version nobody can run any more.
match walk_budget(repo, &want, DEFAULT_BUDGET)? {
Edge::Refused => {}
Edge::Served(sent) => bail!(
"{repo:?}: an object budget of {DEFAULT_BUDGET} served {sent} objects, so this corpus \
is not past that ceiling and proves nothing about a repository that is"
),
}
// ── 2. the SHIPPING configuration must serve it ────────────────────────
//
// `Config::default()`, untouched: the bar is that a kernel-sized
// repository with no `.bitmap` clones off a stock server, not that it
// clones off one tuned by this probe.
let started = std::time::Instant::now();
let (objects, bytes) = served_under(repo, &want, true, |config| config)?
.with_context(|| format!("{repo:?}: the clone failed under Config::default()"))?;
let elapsed = started.elapsed();
ensure!(
u64::from(objects) > LINUX_OBJECTS,
"{repo:?}: the clone sent {objects} objects, which is UNDER the kernel's {LINUX_OBJECTS} \
— this arm would then be reporting a kernel-scale result off a corpus that is not one"
);
let hwm = status_kb("VmHWM:");
let grew = hwm.saturating_sub(base_hwm);
// ── 3. and the bytes ceiling still binds ───────────────────────────────
//
// Without this the arm above is satisfied by a server with no ceiling at
// all, which is the easiest way to pass a memory test and the worst
// possible reason to.
let half = (grew as usize * 1024) / 2;
let refused = served_under(repo, &want, true, |config| Config {
limits: config.limits.walk_bytes(half),
..config
})?;
match refused {
Ok((sent, _)) => bail!(
"{repo:?}: a ceiling of {half} bytes — half of the {grew} kB this clone measurably \
grew by — served all {sent} objects. The bytes ceiling is not what decides."
),
Err(err) => match err.downcast_ref::<Error>() {
Some(Error::WalkTooLarge { limit, .. }) => ensure!(
*limit == half,
"{repo:?}: refused naming a ceiling of {limit}, not the {half} it was given"
),
_ => bail!("{repo:?}: the half-ceiling arm failed with {err:#}, not WalkTooLarge"),
},
}
println!(
"{{\"repo\":{:?},\"objects\":{},\"wire_bytes\":{},\"vmhwm_kb\":{},\"walk_growth_kb\":{},\
\"bytes_per_object\":{:.1},\"seconds\":{:.1},\"old_object_budget_refused\":{},\
\"shipping_default_served\":true,\"half_ceiling_refused_at\":{}}}",
repo.file_name().unwrap_or_default().to_string_lossy(),
objects,
bytes,
hwm,
grew,
(grew as f64 * 1024.0) / f64::from(objects),
elapsed.as_secs_f64(),
DEFAULT_BUDGET,
half,
);
eprintln!(
"{:<24} obj {:>9} wire {:>12} VmHWM {:>9} kB ({:.2} GB) {:>6.1} B/obj {:.0}s \
counted {counted}\n{:<24} refused by the old {DEFAULT_BUDGET}-object budget ✓, \
served by Config::default() ✓, refused at a {half}-byte ceiling ✓",
repo.file_name().unwrap_or_default().to_string_lossy(),
objects,
bytes,
hwm,
hwm as f64 * 1024.0 / 1e9,
(grew as f64 * 1024.0) / f64::from(objects),
elapsed.as_secs_f64(),
"",
);
Ok(())
}
/// A fresh directory for one judge to build a repository in.
///
/// Under `GUNNAR_JUDGE_SCRATCH`, or beside the corpus — **never** `/tmp`, which
/// on oden is RAM: indexing a 600 MB pack there is a measurement of the page
/// cache competing with the thing being measured, on a box where six other
/// lanes are already running.
fn scratch_dir(what: &str) -> Result<std::path::PathBuf> {
let root = std::env::var("GUNNAR_JUDGE_SCRATCH")
.unwrap_or_else(|_| "/home/rickard/scratch/cargo/budget/judge".to_owned());
let dir = std::path::Path::new(&root).join(format!(
"{what}-{}-{}",
std::process::id(),
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_nanos())
.unwrap_or_default()
));
// Removed first, so a rerun judges what this run served and not a pack left
// behind by the last one.
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).with_context(|| format!("making {dir:?}"))?;
Ok(dir)
}
/// **Both judges, both hashes: a clone served under the new ceilings is still a
/// pack.**
///
/// A memory ceiling that quietly changed what went on the wire would be a far
/// worse defect than the one it fixes, and no assertion about `VmHWM` can see
/// it. So this serves a whole clone under `Config::default()` — the shipping
/// configuration, ceilings on — and hands the bytes to the two independent
/// authorities:
///
/// * **stock git**: `index-pack --strict --fix-thin`, then `fsck --full
/// --strict` over the repository it produced. Not `index-pack` alone: a pack
/// can index and still name an object nobody has;
/// * **gitoxide**: `gunnar_pack::index_thin_pack`, which is what gunnar's own
/// receive path runs, so a pack gunnar serves is proven to be one gunnar
/// accepts.
///
/// Both, because they disagree about different things, and at whichever hash
/// the fixture is: a sha256 corpus takes the sha256 path through every one of
/// them.
fn judge(repo: &std::path::Path) -> Result<()> {
use gunnar_store::GixObjects;
let want = tip(repo)?;
let kind = gunnar_store::fs::stored_hash_kind(repo)?;
let format = match kind {
gix_hash::Kind::Sha1 => "sha1",
_ => "sha256",
};
// The clone, served by the shipping configuration through the walk. The
// `NoBitmap` seam again: a bitmap-answered clone never enters the walk the
// ceilings are on, so judging one would be judging the wrong code path.
let store = FsStore::open_or_init(repo, kind)?;
let objects = store.objects();
let refs = store.refs();
let stripped = NoBitmap(&objects);
let upload = UploadPack::new(&stripped as &dyn ObjectStore, &refs);
let mut input = std::io::BufReader::new(std::io::Cursor::new(stages::fetch_request(
&want,
&[],
format,
)));
let mut response: Vec<u8> = Vec::new();
let summary = upload
.serve_v2(&mut input, &mut response)
.with_context(|| format!("{repo:?}: the clone to be judged did not serve"))?;
let pack = stages::demux_packfile(&response)?;
ensure!(
summary.objects_sent > 0 && pack.len() > 1024,
"{repo:?}: {} objects in a {}-byte pack. Both judges accept an empty pack, so a clone that sent nothing would pass every arm below.",
summary.objects_sent,
pack.len()
);
// ── judge 1: stock git ─────────────────────────────────────────────────
let dir = scratch_dir("stock")?;
let into = dir.join("judged.git");
let git = |args: &[&str], stdin: Option<&[u8]>| -> Result<()> {
let mut cmd = std::process::Command::new("git");
cmd.args(args)
.stdin(if stdin.is_some() {
std::process::Stdio::piped()
} else {
std::process::Stdio::null()
})
.stdout(std::process::Stdio::piped())
.stderr(std::process::Stdio::piped());
let mut child = cmd
.spawn()
.with_context(|| format!("spawning git {args:?}"))?;
if let Some(bytes) = stdin {
// A write error is DISCARDED and the child waited for anyway. git
// closes stdin the moment it decides a pack is bad, so the honest
// failure — `fatal: pack has bad object at offset …` — arrives as
// EPIPE here, and returning that instead would replace the judge's
// verdict with a plumbing complaint. Seen exactly that way while
// red-probing this arm with a flipped byte.
let _ = child.stdin.take().expect("piped").write_all(bytes);
}
let out = child.wait_with_output()?;
ensure!(
out.status.success(),
"git {args:?} refused it:\n{}\n{}",
String::from_utf8_lossy(&out.stdout),
String::from_utf8_lossy(&out.stderr)
);
Ok(())
};
let into_s = into.to_string_lossy().into_owned();
git(
&[
"init",
"--bare",
&format!("--object-format={format}"),
&into_s,
],
None,
)?;
git(
&[
"-C",
&into_s,
"index-pack",
"--strict",
"--fix-thin",
"--stdin",
],
Some(&pack),
)?;
// A ref, or `fsck --full` has nowhere to start its reachability walk and
// reports a clean bill on a repository it never looked into.
git(
&[
"-C",
&into_s,
"update-ref",
"refs/heads/judged",
&want.to_string(),
],
None,
)?;
git(&["-C", &into_s, "fsck", "--full", "--strict"], None)?;
// ── judge 2: gitoxide, through gunnar's own receive path ───────────────
let dir2 = scratch_dir("gitoxide")?;
let target = FsStore::open_or_init(&dir2, kind)?;
let target_objects = target.objects();
let pack_dir = dir2.join("objects").join("pack");
std::fs::create_dir_all(&pack_dir)?;
let indexed = gunnar_pack::index_thin_pack(
&mut std::io::BufReader::new(std::io::Cursor::new(&pack)),
Some(&pack_dir),
GixObjects(&target_objects as &dyn ObjectStore),
&gunnar_pack::IndexOptions::new(kind),
None,
)
.with_context(|| format!("{repo:?}: gitoxide refused the pack"))?;
if let Some(keep) = &indexed.keep_path {
gunnar_store::drop_keep(keep);
}
ensure!(
indexed.num_objects as u32 == summary.objects_sent,
"{repo:?}: gunnar said it sent {} objects and gitoxide found {} in the pack",
summary.objects_sent,
indexed.num_objects
);
println!(
"{{\"repo\":{:?},\"format\":{format:?},\"objects\":{},\"pack_bytes\":{},\
\"stock_git\":\"index-pack --strict --fix-thin + fsck --full --strict\",\
\"gitoxide\":\"index_thin_pack\"}}",
repo.file_name().unwrap_or_default().to_string_lossy(),
summary.objects_sent,
pack.len(),
);
eprintln!(
"{:<38} {format:<6} obj {:>8} pack {:>11} stock git ✓ gitoxide ✓",
repo.file_name().unwrap_or_default().to_string_lossy(),
summary.objects_sent,
pack.len(),
);
Ok(())
}
/// What `git count-objects -v` says is in the pack, so the scale arm's budget
/// is the repository's own size and not a number chosen to make it pass.
fn counted_objects(repo: &std::path::Path) -> Result<u64> {
let out = std::process::Command::new("git")
.args(["-C", &repo.to_string_lossy(), "count-objects", "-v"])
.output()
.context("running git count-objects")?;
ensure!(out.status.success(), "git count-objects failed in {repo:?}");
let text = String::from_utf8_lossy(&out.stdout);
let field = |name: &str| -> u64 {
text.lines()
.find_map(|l| l.strip_prefix(name)?.trim().parse().ok())
.unwrap_or(0)
};
let total = field("count:") + field("in-pack:");
ensure!(total > 0, "{repo:?} holds no objects");
Ok(total)
}
fn main() -> Result<()> {
let mut args: Vec<String> = std::env::args().skip(1).collect();
let scale_arm = args.first().is_some_and(|a| a == "--scale");
let judge_arm = args.first().is_some_and(|a| a == "--judge");
if scale_arm || judge_arm {
args.remove(0);
}
if args.is_empty() {
bail!("walkbudget-probe [--scale | --judge] <repo.git> [repo.git ...]");
}
// The judge arm holds nothing across repositories, so it needs no child per
// repository the way a `VmHWM` reading does.
if judge_arm {
for path in &args {
judge(std::path::Path::new(path))?;
}
return Ok(());
}
// The child form: one repository, this process, so `VmHWM` is its own.
if args.len() == 1 && std::env::var("WALKBUDGET_CHILD").is_ok() {
return if scale_arm {
scale(std::path::Path::new(&args[0]))
} else {
one(std::path::Path::new(&args[0]))
};
}
let exe = std::env::current_exe().context("finding this binary")?;
let mut biggest: Option<(u64, f64)> = None;
for path in &args {
let out = std::process::Command::new(&exe)
.args(scale_arm.then_some("--scale"))
.arg(path)
.env("WALKBUDGET_CHILD", "1")
.output()
.with_context(|| format!("re-running this probe for {path}"))?;
std::io::stderr().write_all(&out.stderr)?;
std::io::stdout().write_all(&out.stdout)?;
ensure!(
out.status.success(),
"{path}: the probe's own arm failed, so no row for it is a measurement"
);
// Parse the two numbers the summary needs back out of the child's line.
let line = String::from_utf8_lossy(&out.stdout);
let field = |name: &str| -> Option<f64> {
let at = line.find(&format!("\"{name}\":"))? + name.len() + 3;
line[at..].split([',', '}']).next()?.trim().parse().ok()
};
if let (Some(objects), Some(per)) = (field("objects"), field("bytes_per_object")) {
if biggest.is_none_or(|(o, _)| objects as u64 > o) {
biggest = Some((objects as u64, per));
}
}
}
// The extrapolation is the ladder arm's summary and nothing else's: the
// scale arm MEASURES a corpus past the kernel, so printing an inferred
// kernel figure beside a measured one is exactly the confusion this probe's
// header is about.
if let (false, Some((objects, per))) = (scale_arm || judge_arm, biggest) {
eprintln!(
"── slope off the largest corpus in this ladder ({objects} objects): {per:.1} B per \
emitted object.\n Extrapolating it {:.0}x to the kernel's {LINUX_OBJECTS} says \
~{:.2} GB. DO NOT: `--scale` MEASURED 5.46 GB on a 14 042 000-object corpus, so the \
extrapolation is 1.6x high and the slope is still falling at the top of this table.",
LINUX_OBJECTS as f64 / objects as f64,
(LINUX_OBJECTS as f64 * per) / 1e9,
);
}
Ok(())
}