nordisk/edda

gunnar — performance design

Opened 2026-08-03, against master at 704b4e0 ("Build upstream-bound code as donation-shaped crates"). Companion to .nornir/stinky.md, which is owned by another agent: nothing here edits that file. Entries use the same block format with ids P-001…, so they merge into it unchanged. Where a finding overlaps an existing S- entry it says so and does not restate it.

Load target. One million users on a single 64-core server, Codeberg-class: millions of repositories, thousands of concurrent clones, repositories with millions of objects.

What is MEASURED here, and on what

This is the first time any part of gunnar's cost model has been measured. Four numbers below are measured on oden; everything else is read from source or is arithmetic over measured components, and is labelled.

Machine, and its limits as a proxy for the target. oden is an AMD Ryzen Threadripper PRO 3975WX, nproc = 32, sha_ni / avx2 / bmi2 present. The target is a 64-core server. oden is half the target and every scaling figure below stops at 32 threads. The 64-core behaviour is extrapolated and the extrapolation is not trusted, because the measured efficiency curve is already falling at 32 (63%).

Method. A standalone probe crate at /home/rickard/scratch/cargo/gunnar-perf-probe/, depending on the exact crate versions gunnar's Cargo.lock resolves (gix-hash 0.26.0, flate2 1.1.9, zlib-rs 0.6.6, sha1 0.10.7, sha1-checked 0.10.0), --release, opt-level = 3. Corpus: real .rs files from this tree, git-blob-shaped text: mean 11.3 KB (small corpus, 145 files, 1.57 MiB) and mean 19.2 KB (large corpus, 4000 files, 73.3 MiB). This is a probe, not the bench suite. It is throwaway; P-015 proposes the real, historized one.

Noise, stated up front. Within a single process, two runs of the identical miniz_oxide deflate measured 41.0 and 51.5 MiB/s, a 26% spread. The zlib-rs runs of the same operation measured 85.8 and 85.8. Every ratio below that involves miniz is therefore given as a range, and no ratio here is quotable as a bench result until P-015 re-measures it under the house harness with warmups and best-of-N.

Measurement 1 — SHA-1 (MEASURED 2026-08-03, oden)

gix's SHA-1 runs at ~1/5 of the SHA-1 this CPU can do (4.9×), and — the fact worth sitting with — at ~1/5 the speed of its own SHA-256. A sha1 repository hashes roughly 4.7× slower than a sha256 repository on the same box. DEFAULT_HASH_KIND is already Sha256 (crates/gunnar-store/src/object.rs:13), so new repositories get the fast path. Every mirrored or imported repository, which is all of an existing forge's content, gets the slow one.

Root cause, READ: gix-hash-0.26.0/src/hasher.rs:36 builds sha1_checked::Builder::default().safe_hash(false).build(), and sha1-checked-0.10.0/src/lib.rs:186-190 branches: with detection on it calls its own pure-Rust compress::compress; only with detection off does it reach sha1::compress, which is the SHA-NI path (sha1-0.10.7/src/compress/x86.rs:100, cpufeatures::new!(shani_cpuid, "sha", …)). This matches git's own default (sha1dc), so it is not a gitoxide defect. What it is, is an unpriced cost that becomes the entire bottleneck once P-001 lands. See P-008.

Measurement 2 — zlib backend (MEASURED 2026-08-03, oden)

Identical flate2 API, identical corpus, one feature flag apart.

At L6 the compression ratio is byte-for-byte identical (3.28×) and the throughput roughly doubles. zlib-rs 0.6.6 is already linked into gunnar via gix-zlib → zlib-rs (Cargo.lock:2683-2690), so this costs zero new dependencies and, unlike zlib-ng, no C toolchain. See P-007.

Measurement 3 — does per-object deflate scale? (MEASURED 2026-08-03, oden)

4000 objects, 73.3 MiB, zlib-rs L6, static stride partition across std::thread, best of 3.

20.3× on 32 hardware threads, and a hard plateau at the hardware thread count. Oversubscribing to 48 or 64 buys nothing and 64 is measurably worse than 32. Efficiency is already down to 63% at 32, which says the workload is partly memory-bandwidth-bound, not purely compute-bound. Do not extrapolate 40× at 64 cores from this curve.

Combined with Measurement 2: today's serial miniz path runs at ~38 MiB/s (derived: 79.1 ÷ 2.09); a parallel zlib-rs path measured 1602.8. That is the ~42× ceiling on the deflate component, and P-001 argues most of that work should not happen at all.

Measurement 4 — the allocation micro-optimisation buys nothing (NEGATIVE RESULT)

crates/gunnar-pack/src/write.rs:121 allocates a fresh Vec and a fresh ZlibEncoder per object. Replacing it with a reused, cleared scratch buffer measured 85.8 → 86.1 MiB/s (zlib-rs) and 51.5 → 51.5 MiB/s (miniz): no difference beyond noise.

Deflate dominates so completely that the allocation is invisible. S-015 ("Two heap allocations per packed object") is correct as an observation and its memory-pressure argument stands, but it should not be sold as a throughput fix — measured, it is not one. Recorded here so the next agent does not spend a day on it expecting a number.

The CPU budget — where a served byte actually goes

Derived arithmetic over the measurements above, per 1 MiB of raw object content served (deflates to 0.305 MiB on the wire at the measured 3.28× ratio), one core. This is component arithmetic, not an end-to-end clone measurement. It excludes the graph walk, object lookup, syscalls and the network, which is the whole point of the last row.

Three things fall out of this table, and they set the entire build order.

1. Today, deflate is 86% of the compression+hash budget. Everything else is rounding.
2. P-001 does not make deflate faster — it deletes it. The bytes are already deflated, sitting in the pack the client pushed. Copying them is the whole job.
3. The SHA-1 finding is worth 3% today and ~95% after P-001. Before pack-copy, fixing the hash is a rounding error and would be a bad use of a day. After it, the trailer hash is nearly the entire remaining cost. P-008 is sequenced behind P-001 deliberately, and its value is conditional on it.

Two columns of that table are now real (2026-08-03). P-007 and P-008 both landed and were re-measured end-to-end through PackWriter on the same frozen 1.70 MiB corpus: 47.4 MiB/s → 90.8 MiB/s, a 1.92× whole-writer speedup at level 6, against the 2.0× the table's first two columns predict. The table's arithmetic held. What it could not predict is that with deflate removed — level 0, the shape P-001 produces — the writer goes 358 → 1442 MiB/s, which is the "~95% of what remains" claim confirmed rather than derived.

And the honest bound: once the table's last column is real, serving is no longer limited by compression at all. It is limited by the graph walk (P-010) and by object lookup (P-003). Neither has ever been measured. No end-to-end clone speedup is claimed here by anyone.

All four columns are real as of 2026-08-03, and the bound above arrived exactly as written. P-001 landed and was measured on both sides: 9.9× on a corpus with no deltas to reuse (clone_large, byte-identical output, 3197.72 → 322.59 ms) and 18.3× fewer bytes with 13.0× less server CPU on one with history (vs_git_clone against stock git). The table's 31× was arithmetic over the compression+hash budget alone, and a real clone carries a graph walk and an object lookup per object that the table deliberately excluded. That is why the measured figure is a third of the derived one, and why the derived one was not wrong.

What is left is the sentence above coming true: gunnar now emits the same bytes as git upload-pack, for 16× the server CPU. That residue is P-010 and P-018, and it is now the entire serve-path gap.

The walk has now been measured (2026-08-03) and the first thing it said was that the question was mis-posed. P-010 framed the walk as serial where it should be parallel. Profiled, the largest single item in it was not serialism at all: every commit in a clone was read from the store and decoded twice, once for its parents and once again for the root tree the first decode had already parsed and discarded. Removing that, plus the full-object commit parse and the per-object buffer churn, is 1.295× on walk_1m and 1.025× on clone_large — instructions retired, interleaved, both sides. It is also 3.006 G instructions genuinely deleted, where a perfect 32-way fan-out would have deleted none. See P-010.

The parallelism answer — and the trap in it

gunnar has no parallelism anywhere (S-011, grep -c gatling Cargo.lock → 0). The reflex is "fan everything out with gatling". That reflex is wrong for a forge, and the reason is structural.

READ, from znippy-zoomies/gatling/src/gatling_forkjoin.rs (module doc :19-21): "Pure std::thread::scope; no global pool, no rayon." Every entry point opens a fresh scope, spawns n_workers OS threads, and joins them before returning. There is no pool object, no global runtime, and no reentrancy detection: nesting compiles fine and silently spawns W² threads. nornir's own comment says it plainly (nornir/src/knowledge/symbols.rs:317-318): "Call TOP-LEVEL only — it IS the pool; nesting it under another pool oversubscribes."

A forge is thousands of small concurrent requests, not one big batch job. Wrapping per-request pack generation in gatling_map_balanced at 1000 concurrent clones means 1000 × 64 = 64,000 OS threads, plus 1000 thread-scope spawn/join storms per second. Measurement 3 already shows the box gets slower past 32. That is P-004, and it is a blocker precisely because the obvious fix causes it.

The correct split:

The aggregate-throughput point, stated bluntly. At saturation, parallelising one clone across 32 cores does not raise clones/second at all. It is the same total work, rearranged. It buys latency and tail latency, which matter, but they are not throughput. The items that genuinely reduce total work are P-001 (delete the compression), P-007 (halve what is left), P-009 (send fewer bytes) and P-008 (hash faster). Rank accordingly, and note that this inverts the order a naive read of S-011 would suggest.

The gatling call shape, for whoever writes it

// From tokio: gatling BLOCKS. It must never be called on a runtime thread.
let entries = tokio::task::spawn_blocking(move || {
    // `batch`: aim for ~4 units per worker, never 1 on cheap units — the shared
    // AtomicUsize cursor is a measured 2x regression at batch=1
    // (znippy-zoomies/src/stree.rs:219-227).
    gatling_map_balanced(
        &counts,                    // &[T]
        0,                          // n_workers: 0 => one per core
        BATCH,                      // claim size per fetch_add
        |c| c.uncompressed_size,    // size_of: runs SERIALLY on the caller thread — keep O(1)
        |_idx, c| -> Result<Entry> {// panics LEAK the output Vec: return Result, never panic
            encode_one(c)
        },
    )
}).await?;

Four constraints that are not obvious and are all READ from source or house comments:

• Output is in input order, exactly: results are written back to out[orig] (gatling_forkjoin.rs:468-472). A pack needs ordered entries; this satisfies it.
• size_of runs serially on the caller thread over all N items before any work starts (:436). It must be O(1) and side-effect-free.
• A panic in a worker leaks the Vec<MaybeUninit<T>> (no catch_unwind anywhere). Safe, but silent. House idiom: return Vec<Result<…>>.
• Never nest. Flatten to one union work list and run one pool.

Summary

Counts: 6 blocker · 7 scale · 2 correctness · 3 smell, 18 total. 5 DONE (P-001, P-007, P-008, P-012, P-015), 1 PARTLY DONE (P-010 — the redundant-work half landed and measured; the parallel half withdrawn on the same measurement), 1 DECIDED (P-004, the serving tier stays serial on purpose), 1 WONTFIX (P-009, on the measurement it asked for), 10 TODO.

Build order — ranked by expected win ÷ effort

Not a wish list. Each step's value depends on the ones above it, and that is why the order is not the severity order.

P-016, P-017, P-018 are follow-ons, not blockers on this sequence.

blocker

P-001 — Every served object is inflated and re-deflated, discarding the client's own delta compression

• Found: 2026-08-03

• Status: DONE 2026-08-03. Landed at 6e65beb, measured on both sides on 2026-08-03. 18.3× fewer bytes and 13.0× less server CPU on a clone with history; 9.9× on one without. See the closing note, which also corrects this entry's prescribed fix.

• Severity: blocker

• Where: crates/gunnar-pack/src/write.rs:182-209 (write_pack) · write.rs:106-128 (add_object) · crates/gunnar-store/src/object.rs:37 (ObjectStore::read — the trait only ever hands back inflated bytes)

• What: write_pack calls source.try_find(id, &mut buf), which returns fully inflated object content, and hands it to add_object, which deflates it again from scratch. For any object already sitting deflated inside a packfile, gunnar performs a complete inflate → deflate round trip to reproduce bytes it already had. Under §4's design that is every object in the incoming tier, because a push is stored as the packfile the client sent.

Worse, the client's git ran a full delta search before uploading that pack. Those deltas are in the incoming tier. add_object can only emit Blob/Tree/Commit/Tag headers (write.rs:249-257, and S-003), so every delta the client computed is inflated away and re-sent as a whole object.

The fix is already written, in a crate gunnar already depends on. gix_pack::data::output::entry::iter_from_counts has exactly one Mode variant (READ, gix-pack-0.73.0/src/data/output/entry/iter_from_counts.rs:359-367):

pub enum Mode {
    /// Copy base objects and deltas from packs, while non-packed objects will be
    /// treated as base objects (i.e. without trying to delta compress them). ...
    PackCopyAndBaseObjects,
}

The single-variant enum reads like a limitation and is usually reported as one (including in S-003's framing). It is the opposite: it is precisely the optimisation gunnar is missing. from_pack_entry (entry/mod.rs:65-131) copies an existing pack entry's compressed bytes verbatim and re-points its OfsDelta header (:89-112) — no inflate, no deflate, delta preserved. Only RefDelta is downgraded to a base (:113), and only objects with no pack residency are deflated fresh (from_data, entry/mod.rs:138-159).

• Why it matters at load: Per the CPU budget above, this is 86% of the compression+hash cost of serving, and it is pure waste. The derived figure is 28.49 ms → 0.91 ms per MiB of raw content per core, ~31×. It also recovers delta compression on egress without gunnar ever writing a delta encoder (P-009), because the client already paid for it. Bandwidth is the thing a forge at this scale actually buys.

• The dependency this creates: pack-copy only works if the store can hand back an object's raw pack entry bytes, not its inflated content. Today ObjectStore cannot express that (object.rs:37). This needs a third read method, fn read_pack_entry(&self, id) -> Result<Option<PackEntry<'_>>> returning { header, compressed_bytes, base: Option<ObjectId or ofs> }, with a default impl returning None so mem/fs backends are unaffected and the writer falls back to today's path. Design it alongside P-003's batch seam; they touch the same trait.

• It also decides §4. The znippy cold tier as designed cannot serve pack-copy. See P-005. Settle that before wave 9b.

• Fix: In order. (1) Add read_pack_entry to ObjectStore, defaulting to None. (2) Implement it on the incoming-pack tier over gix_pack::data::File. (3) Route write_pack through gix_pack::data::output — count::objects → entry::iter_from_counts(Mode::PackCopyAndBaseObjects) → bytes::FromEntriesIter — rather than hand-rolling, per LAW 5. Note P-002 before doing so, and P-018 for what that pipeline's serial tail costs.

• Verify: Clone a fixture with real, near-duplicate revision history through gunnar, and assert (a) the served pack is materially smaller than the sum of its whole objects and (b) stats.decoded_and_recompressed_objects is zero for objects resident in the incoming pack. Both must be seen RED first: today (a) is equal by construction and (b) is 100%. Beware the identity value (S-003 names this too): a fixture of unique random blobs deltas to nothing, and would make a working encoder look broken. And do not assert on wall time on a small fixture; assert on bytes emitted and objects recompressed, which are applied output.

• Closed: 2026-08-03. Landed at 6e65beb as crates/gunnar-pack/src/copy.rs (write_pack_copy + PackedObjects), reached from gunnar-wire's upload_pack.rs through a new ObjectStore::packed() seam that defaults to None. MEASURED on both sides, 2026-08-03, and the entry's derived 31× was in the right place for the wrong reason: the CPU win is 9.9× and the bytes win is 18.3×.

How BEFORE was obtained, which is the part worth copying

Not from a memory of an earlier ref. GUNNAR_BENCH_NO_PACK_COPY=1 is an ObjectStore wrapper in the bench that returns None from packed() and changes nothing else. That is exactly the server as it was before this landed, on the same binary, the same fixture and the same machine minutes apart. Its credential is that it reproduces the historical figures to three decimals: 18.336× on the sha1 clone, 17.723× on sha256 and 7.22× on the concurrent arm, all identical to the run recorded at e60c95d under P-015. A switch that produced different wrong numbers would have been a differently broken path rather than the old one.

It is also the LAW 2 red for every assertion below, and it is permanent.

The head-to-head, oden 2026-08-03, stock git 2.53.0, one code seam apart

gunnar.vs_git_clone: sha1, 2048 churned commits over 1024 files, 528 MiB of blob content, 36 172 objects. Both columns inside the pinned container, interleaved, median of three.

sha256 moves the same way: 115.06 → 6.49 MB (17.723 → 1.000), CPU 5.025 → 0.377 s, cpu_ratio_x 169.3 → 12.236, wall_ratio_x 6.316 → 0.854. On that arm, in that run, gunnar finished a clone faster end to end than git upload-pack. Concurrent (16 clients): pack_ratio_x 7.22 → 1.000, cpu_ratio_x 26.239 → 1.695, and rss_total_ratio_x 0.169 for one gunnar process against sixteen git ones, unchanged by this work and still 5.9× better.

gunnar.vs_git_fetch_divergent is the one arm where the copy is not total, and its number is the interesting one: 2841 copied against 71 recompressed, 97.6%. The 71 are what gix-pack cannot copy: RefDelta entries, which it downgrades to bases (entry/mod.rs:113), and deltas whose base did not come along. Wire bytes fell 1.79 MB → 0.47 MB and cpu_ratio_x 5.560 → 2.969.

The other half, isolated

gunnar.clone_large serves a 1 GiB fixture of random blobs, which delta to nothing. There is no delta reuse available on it at all, so it measures only "do not inflate and re-deflate what is already deflated":

9.9×, at byte-for-byte identical output. That is the number the CPU-budget table was reaching for, and the split matters: on a corpus with history the two effects compound (18.3× bytes and 13.0× CPU); on a corpus without it only the second survives. Anyone quoting a single multiplier for P-001 should say which corpus it came from.

What it cost, and what it did not

• Nothing was traded. objects transferred is asserted equal to stock git's on both clone arms; peak RSS moved from 17.33 to 22.57 MB against git's 12.11, which is the generator's chunk buffer and is bounded by copy.rs's named CHUNK_SIZE.

• P-004 holds on the new path. gunnar.clone_concurrent_64 reached write_pack_copy for the first time on 2026-08-03 (its fixture was unpacked before) and peaked at 89 threads for 64 concurrent clones, well inside its ceiling. That guard had never been exercised against the writer that can fan out; copy.rs's thread_limit: Some(1) is measured now, not inferred from a grep. Corrected 2026-08-05: "well inside its ceiling" was luck. 89 is not 64 + slack, it is partway to the then-structural 3 × 64 + 3 = 195, and the same arm read 154 in run 7079f265-85fb-4478-9836-9de801da9cb1 and went red on identical code. It reads exactly 64 + 3 now that 5be4644 has landed. See the amendment on P-004.

• The copy path used to be gated on the client's ofs-delta capability (upload_pack.rs), and a client that did not advertise it got the whole of the pre-P-001 server: no entry copying, no delta reuse, a fresh deflate of every object. CLOSED 2026-08-03, and the gap was the full 18×. MEASURED on the churned fixture, in-process, gunnar.clone_no_ofs_delta:

18.336× → 1.095× on bytes, 12.9× → 1.08× on wall clock. The fix is that OBJ_REF_DELTA needs no capability at all — it is the older of the two forms — so a copied delta can name its base by object id instead of by distance backwards. The delta bytes are byte-for-byte the same; only the header changes. WriteOptions::ofs_delta now selects which name is used, and the capability no longer decides whether copying happens.

The residual 9.5% is arithmetic, not slack: one 20-byte base id per delta entry replacing a 1–3 byte varint, 591 093 bytes over ~33 000 deltas. A sha256 repository pays 32 bytes and would sit higher.

No gix change was needed, which is worth recording because it was the expected answer. output::entry::Kind already distinguishes DeltaRef { object_index } from DeltaOid { id }; only Entry::to_entry_header maps the former to OfsDelta unconditionally, and gunnar already knows the id of the entry at that index because it has just written it. The gix gap is a missing option, not a missing capability, and it is 15 lines to route around.

Verified against stock git, not against gunnar's own reader. crates/gunnar-pack/tests/ofs_delta_off.rs builds a churned repository with real git, serves it both ways, and holds the result to git index-pack --strict and git verify-pack -v. gix's own comment calls an OBJ_REF_DELTA whose base lies inside the same pack "a discontinued practice", so acceptance by a third party was the whole safety argument. Three things the test does that the obvious version would not:

• it asserts the fixture has deltas at all, and that guard fired for real — the first version used 400 lines over 12 revisions and git repack -adf produced zero deltas, which would have made both columns pass on nothing;
• it reads the delta form from the type nibble of the first byte of each entry, at git's own offsets, because git verify-pack -v prints the object type and marks a delta only by trailing columns — the first parser matched on a ofs-delta/ref-delta type column that git never prints and reported a pack full of deltas as holding none;
• it asserts the two packs hold the same number of deltas and the same bases, so a fallback that silently expanded them cannot pass.

Seen RED by making write_pack_copy ignore the flag: the form assertion fails. The companion test (both forms decode to the same objects through stock git cat-file) stays green under that edit, which is stated in its own doc rather than left to be discovered — it catches a different failure.

• The concurrent in-process arm could not be measured cleanly. oden ran at load 70–226 on 32 cores throughout (other agents building), and clones_per_sec on clone_concurrent_64 read 48.5 and 268.6 in two runs of the same code. Only the interleaved head-to-head ratios above are quotable; that arm's absolutes are not. Re-measured with repetitions on 2026-08-03 (see the P-015 arm table): seven runs gave a median of 530.3 clones/s with a 3.65× full range, so the fix is not a quieter box — it is publishing the distribution. Instructions retired over the same seven runs spread 5.4%, which says the arm's work was constant and only its rate was not.

The fix was not the entry's fix, and the difference is P-020

The Fix step above prescribes finishing the pipeline with output::bytes::FromEntriesIter. Doing so would silently have reverted P-008 (that type builds its own gix_hash::io::Write, which hardcodes sha1_checked), and reverted it in precisely the shape where P-008 measured 4.02× rather than 1.06×, because pack-copy is what removes the deflate that used to dominate the write. Nothing would have gone red; a pack written either way is byte-identical. crates/gunnar-pack/src/copy.rs therefore takes gix's entries and keeps gunnar's emitter, and xtask/tests/pack_pipeline.rs fails the build if either half drifts. That is P-020, in .nornir/transport-encoding-design.md.

What is NOT done

• read_pack_entry on ObjectStore was not added. The seam is packed() -> Option<PackedObjects>, i.e. gix_pack::Find, because that is the trait iter_from_counts consumes; a hand-rolled entry accessor would have been a second copy of the one genuinely hard part (LAW 5).
• The cold tier still cannot be pack-copied, because there is no cold tier. P-005 remains the decision that keeps it that way or not.
• Only fs implements it. mem returns None and falls through to write_pack, which is correct and which means every in-memory test measures the fallback.

P-002 — gix-pack's parallel feature is never declared — the appliance build silently uses the serial code path

• Found: 2026-08-03

• Status: DONE

• Severity: blocker

• Where: Cargo.toml:62 (gix-pack = { version = "0.73", features = ["streaming-input"] }) · gix-features-0.49.0/src/parallel/serial.rs:128-143 · crates/gunnar-store/Cargo.toml:15 (fs = ["dep:gix"])

• What: gix-pack has zero cfg(feature = "parallel") gates in its own source; its parallel feature purely forwards to gix-features/parallel. Whether gix_features::parallel::in_parallel is the threaded implementation (in_parallel.rs:49) or a plain for loop (serial.rs:128-143) is decided entirely by whether something else in the resolved graph switches that feature on.

MEASURED by feature resolution, 2026-08-03:

• Whole workspace: cargo tree -e features -i gix-features → parallel IS on, but only via gix → max-performance-safe → max-control, i.e. through the client-side porcelain.
• gunnar-pack alone, --no-dev-dependencies: enabled gix-features features are crc32, default, fs-read-dir, prodash, progress. parallel is ABSENT.

So in the appliance build — gunnar-store's fs feature off, gix porcelain not linked, which is exactly the configuration §2's dependency budget exists to produce and which S-007 is about — every gix-pack parallel call site degrades to the serial one. It compiles clean and says nothing at runtime. The pack indexer on the receive path (crates/gunnar-pack/src/read.rs:27-28, 52-56, which passes an IndexOptions::thread_limit and is today the only parallelism in the tree per S-011) is single-threaded in production and multi-threaded in every test that links the porcelain.

• Why it matters at load: It is a silent-green of the exact family the fleet keeps finding (see the silent-green-guard-disease finding): the thing works in dev, is measured in dev, and is quietly serial where it ships. It also means any measurement of the receive path taken with the fs feature on is not a measurement of the appliance.

• Fix: Declare it explicitly, gix-pack = { version = "0.73", features = ["streaming-input", "parallel"] }, so it is owned rather than inherited. Then extend crates/gunnar-cli/tests/feature_split.rs (which S-007 already faults for being too narrow) with an assertion that gix-features/parallel is enabled in the appliance feature set, not merely somewhere in the workspace.

• Verify: The new feature_split.rs assertion must be seen RED against Cargo.toml:62 as it stands today. That is a one-line proof and it will pass on the first try only if written wrong: the guard must resolve the appliance feature set, not the workspace's. Then a receive-path bench (P-015, gunnar.push_large) run in both feature configurations: if the two are identical, the flag did nothing and the guard is blind.

• Closed: 2026-08-03. Declared gix-pack = { version = "0.73", features = ["streaming-input", "parallel"] } in the workspace manifest. Re-MEASURED after the S-007 porcelain removal, and the finding half-changed. The appliance set (cargo tree -p gunnar-cli --no-default-features --features server -e features -i gix-features) now DOES enable parallel, but only through gunnar-store/fs → gix-odb/parallel + gix-ref/parallel, i.e. inherited from a neighbour, exactly the fragility this entry is about. gunnar-pack resolved alone (--no-dev-dependencies) still showed crc32, default, fs-read-dir, prodash, progress and no parallel, so the crate that actually passes IndexOptions::thread_limit was not asking for its own threading. Guarded by crates/gunnar-cli/tests/feature_split.rs::the_pack_indexer_asks_for_its_own_threading, which asserts the feature in both the gunnar-pack-alone and the appliance feature sets, checks the tree is non-empty first (an empty cargo tree is shaped exactly like "feature absent"), and was seen RED against the pre-change declaration. NOT done: no receive-path bench in the two configurations. P-015 does not exist, so the size of the win remains unmeasured. What is fixed is that it is no longer an accident.

P-003 — ObjectStore has no batch seam, so the stree batch lookup §4.1 is built around can never be reached

• Found: 2026-08-03

• Status: TODO

• Severity: blocker

• Where: crates/gunnar-store/src/object.rs:23-53 (the whole trait) · .nornir/TOTAL-PROMPT.md:504-575 (§4.1) · consumers at crates/gunnar-wire/src/graph.rs:74-96, 177, 228, 273 and crates/gunnar-pack/src/write.rs:202, 231

• What: ObjectStore exposes contains, read and header, all strictly one object at a time. Every consumer is a scalar loop.

§4.1 chose stree over the fst trie on one decisive argument, quoted from the spec: "The batch path is what actually decides it. Serving a pack means hundreds to thousands of object lookups at once, and stree has lookup_batch_pipeline::<P> / batch_stream::<P> — software-pipelined and prefetched — for precisely that shape. fst has no batch path at all. Git's hot path is a batch path."

The trait that the cold tier must implement has no way to express a batch. The argument that selected the index cannot be acted on through the seam the index sits behind. Whichever agent writes ZnippyObjects in wave 9b will implement contains(&self, id) and the batch path will be dead on arrival. Being an optimisation that silently never activates, it will look exactly like working code.

READ, on what is being given up: STree64Mmap::lookup_batch_pipeline::<P> (znippy-zoomies/src/stree.rs:408) routes P queries in flight with _mm_prefetch per level, sorts by leaf block, issues one madvise(MADV_WILLNEED) over the touched byte range (:378-385), scans leaves sequentially, then scatters back by original index. Its doc claims 4–8× on cold pages (:359-362). Scalar find_exact gets none of it. Note also that the batch methods are single-threaded: the parallelism they exploit is memory-level, not thread-level, so this composes with P-004 rather than competing with it.

• Why it matters at load: At a million objects the store is memory-latency-bound, not compute-bound (P-012). Batching is how you hide that latency, and it is the single structural lever the storage design already committed to.

• Fix: Add batch methods with scalar default impls, so no backend breaks:

fn contains_batch(&self, ids: &[ObjectId], out: &mut [bool]) { /* default: loop */ }
fn headers_batch(&self, ids: &[ObjectId]) -> Result<Vec<Option<Header>>> { /* default: loop */ }

Then convert the three call sites that are genuinely batch-shaped: missing_objects (write.rs:227-232, the want pre-flight, a natural batch), the have scan in negotiate (negotiate.rs:53-58), and tree_closure's per-level child set (graph.rs:257-265, which already collects blobs: Vec<ObjectId> before checking them, so the batch is already materialised there and then consumed scalar-wise). Do not add a read_batch that returns owned Vec<u8>s; that reintroduces the allocation storm. P-001's read_pack_entry is the right shape for bulk reads. Two stree landmines the spec records and an implementor will hit: find_exact returns a member of a duplicate-key run and its leaf scan walks only ≤ B+1 forward, so expand in both directions; and id_at uses i64::from_le_bytes (znippy-zoomies/src/stree.rs:568-571), so the record's leading 8 bytes must be the derived key stored little-endian. You cannot point stree at a raw big-endian oid prefix at stride 20.

• Verify: A bench (P-015, gunnar.lookup_1m) that resolves 10,000 random oids against a million-object store, scalar vs batch, reporting lookups_per_sec. Plus a correctness guard that the two paths return identical results including on a deliberately constructed 8-byte-prefix collision. The collision test must be seen RED against a verify step that is missing or wrong; §4.1 already requires this and it must not be skipped because collisions are rare.

• Closed:

P-004 — gatling has no shared pool; per-request fan-out at forge concurrency spawns tens of thousands of threads

• Found: 2026-08-03

• Status: DECIDED — the serving tier is deliberately serial

• Severity: blocker

• Where: design-level. znippy-zoomies/gatling/src/gatling_forkjoin.rs:19-21, 87, 343, 455 · the fix S-011 proposes · S-008 (no connection limit) · S-009 (gRPC blocks tokio workers)

• What: S-011 correctly identifies that gunnar has no parallelism and correctly names gatling as the tool (LAW 3). The straightforward reading of that fix is a serious regression, and it should be written down before someone implements it.

READ: gatling has no pool object and no global runtime. Module doc (gatling_forkjoin.rs:19-21): "Pure std::thread::scope; no global pool, no rayon…". Every call spawns n_workers fresh OS threads and joins them before returning (:87, :162, :239, :343, :455, :547, :613). n_workers = 0 means one per core.

At the load target that is a disaster in three independent ways:

1. Thread explosion. 1000 concurrent clones × 64 workers = 64,000 OS threads. With S-008's missing connection limit there is nothing capping the multiplier.
2. Spawn/join churn. ~64 clone() syscalls per fan-out, per request.
3. Measured negative return. Measurement 3: the box plateaus at 32 threads and is slower at 64. Oversubscribing 2000× is far past that cliff.

And a fourth, from S-009: gatling blocks. Calling it from an async fn stalls a tokio worker for the whole fan-out. The house bridge is tokio::task::spawn_blocking(move || gatling_…), as nornir does at nornir/src/server_grpc.rs:149, 175, 951. But spawn_blocking's default pool is 512 threads, which S-008 already identifies as gunnar's accidental concurrency cap.

• Why it matters at load: This is the difference between "gunnar uses all 64 cores" and "gunnar thrashes". It is also why the aggregate-throughput items (P-001, P-007, P-009) rank above the fan-out items in the build order: at saturation, intra-request parallelism rearranges work rather than reducing it.

• Fix: Two tiers, explicitly separated, and written down in design.md before any fan-out lands.

  • Batch tier (repack, gc, index building, LFS chunking): gatling_map_balanced with n_workers = 0. These are offline, one-at-a-time, and are what gatling is for. Serialise them with a semaphore of 1–2 so two repacks never run at once.
  • Serving tier: default to no intra-request fan-out. Concurrency comes from serving many requests at once, each costing about one core. Add a global Semaphore(cores) admission gate (which S-008 needs anyway), and permit intra-request fan-out only when a transfer exceeds a size threshold and the admission gate reports the box is below its core count, with n_workers computed as available_permits, never 0.
  • Never call gatling from an async context without spawn_blocking, and size the blocking pool deliberately rather than inheriting 512.

• Verify: A concurrency bench (P-015, gunnar.clone_concurrent_64) that runs 64 simultaneous clones and reports p50/p99 latency and peak thread count. Seen RED by first implementing the naive per-request gatling_map_balanced(…, 0, …) and observing the thread count and the p99. That red is the deliverable: it is the only thing that will stop the naive fix being written twice. Do not validate this with one clone on an idle box; that configuration makes the wrong design look best.

• Closed: 2026-08-03. The trap was not walked into. S-011 was closed as a measured decision not to parallelise the per-request serving path, and this entry is the reasoning it was closed on. Concretely:

  • Serving tier: no intra-request fan-out, and an admission gate instead. S-008 landed gunnar-ssh's Admission: a server-wide max_concurrent_transfers (256) and a per-principal cap (32), refusing visibly and counting every refusal. That is this entry's prescribed "global Semaphore(cores) admission gate", in the form the SSH front end could actually enforce it.
  • Blocking pool sized deliberately rather than inherited: gunnar_cli::server::serve::GIT_BLOCKING_THREADS = 512, asserted by test to stay ≥ 2× the transfer limit.
  • No gatling call site was added, so the thread-explosion arithmetic in this entry never gets a chance to happen. For the record, availability is not the reason: gatling ships as rotaryengine 0.1.10, pure Rust, no C and no cmake (P-006's toolchain cost is OpenZL's, not gatling's). The reason is that there is no non-per-request batch workload in the serving path. NOT done: the size-threshold escape hatch this entry proposes — fan out only above a transfer-size threshold and only when the admission gate reports the box below its core count, with n_workers = available_permits and never 0 — is unbuilt, and the gunnar.clone_concurrent_64 bench that would justify it needs P-015. Until then a single clone of a huge repository on an idle box is one core, which is the acknowledged cost of this decision.

• Amended 2026-08-05 — the decision held, the arithmetic was off by 3×, and the three became one the same day. thread_limit: Some(1) bounded the workers, not the thread count. gix_pack::…::iter_from_counts ends in gix_features::parallel::reduce::Stepwise::new, which upstream spawns num_threads(thread_limit) consumers plus one input feeder, both unconditionally and neither gated on its 4000-count fan-out threshold. So one served clone was three OS threads: its own, one consumer, one feeder.

MEASURED on oden, 2026-08-05, gunnar.clone_concurrent_64, peak_threads out of /proc/self/status at concurrency = 1, at b9900c2 — the tree run 7079f265-85fb-4478-9836-9de801da9cb1 was taken on — with nothing changed between the rows but the thread_limit literal in gunnar_pack::copy:

+3 for +3 consumers, feeder and 3-thread process baseline unchanged.

This was not the trap this entry names, and the distinction is the point: the cost was a constant 2 per request, independent of core count, and the work stayed serial — the same arm measured telem_proc_cores_max = 1.49 of 32 while serving 64 concurrent clones. "About one core per transfer" held even where "one thread per transfer" did not.

5be4644 then deleted the two threads by vendoring Stepwise::Inline, so at thread_limit == Some(1) the iterator consumes on the calling thread and spawns nothing. RE-MEASURED at f2bfd3b, same binary and fixture:

Exactly N + 3, no scatter. At S-008's ceiling of 256 concurrent transfers the serving tier therefore holds ~259 OS threads, not the ~768 it held that morning — worth having as a number either way, because nothing before this had one.

The bench arm asserted peak_threads <= 2 × concurrency + 8, a ceiling below the then-structural maximum 3N + 3 for every N ≥ 5, so whether it passed was decided by how much the clones happened to overlap — by box load. It read 100–124 at N=64 on an idle box and 154 in run 7079f265-85fb-4478-9836-9de801da9cb1, and went red at N=8 and N=16 on the identical binary. The ceiling is now structural (1×N + 3 + 5) and the slope is emitted as threads_per_clone, so losing Stepwise::Inline would put N=64 at 195 against a ceiling of 72.

P-005 — The znippy cold tier forecloses pack-copy and re-compresses every git object at OpenZL level 19

• Found: 2026-08-03

• Status: TODO

• Severity: blocker

• Where: .nornir/TOTAL-PROMPT.md:462-503 (§4, "objects keyed by oid hex as relative_path") · znippy-common/src/common_config.rs:104 · index.rs:1212-1235

• What: §4 stores each git object in znippy under its oid hex, as inflated content. That decision has a cost the storage reasoning does not price, and it is the same cost P-001 is about.

A git object in a packfile is already deflated, and often a delta. Putting it into znippy means: inflate it, hand it to OpenZL, store the OpenZL frame. Serving it later means: OpenZL-decode, then zlib-deflate for the wire. Every cold-tier object pays a full recompression round trip on every fetch, forever, and can never be pack-copied. git's own cold tier is a .pack, which preserves both the zlib bytes and the delta chain, and that is exactly why git upload-pack can serve most objects with a memcpy.

Compounding it: znippy's write path compresses at level 19 on the real pipeline (common_config.rs:104, let compression_level = 19;; the CLI append default is 3 and the compress verb has no flag at all), and git objects hit the already-compressed detection gap described in P-016, so they are all fed to the codec. MEASURED in znippy's own suite (.nornir/README-full.md:107, oden, 2026-07-22): archive_random_500mb → 103.62 MB/s compress, compressed_chunks = 0, ratio_x = 1. 103 MB/s spent to achieve nothing. Already-deflated git objects are that workload.

• Why it matters at load: It converts §4's cold tier from an optimisation into a permanent per-fetch tax, and it takes P-001 — the largest single win available — off the table for every object that has aged out of the incoming tier, which over time is almost every object.

• Fix: Store the deflated pack entry, not the inflated object. Concretely, the cold tier's value for oid X becomes the bytes of X's pack entry: the type/size varint header, the delta base reference if any, and the zlib stream, exactly as they sat in the pack repack() folded. Then:

  • Pack-copy works from the cold tier as well as the incoming tier (P-001).
  • Delta chains survive repack() instead of being flattened.
  • The stored bytes are already compressed, so znippy's n >= len fallback stores them raw and the level-19 cost becomes pure waste to be removed by P-016.
  • blake3 per-chunk verification (§4's property 2) is unaffected: it verifies whatever bytes are stored. The cost is that a plain object read (ObjectStore::read, for the graph walk) must now inflate, and for a delta entry must resolve its base. The cold tier becomes a real pack rather than a key-value store. That is the trade, and it is the trade git makes. Settle this before wave 9b writes ZnippyObjects.

• Verify: A bench (P-015, gunnar.clone_cold_tier) serving a clone entirely from the cold tier, reporting bytes_out and objects_recompressed. Seen RED against a cold tier holding inflated content, where objects_recompressed is 100% by construction. Assert on those two applied outputs, never on a round-trip through the store: a store that returns the right bytes proves nothing about how they were stored.

• Decided, in part, 2026-08-03 (while building S-006's repack()): (b), the cold tier stores the deflated/packed representation, not inflated content. The argument above is not answerable: an inflated tier makes P-001 unreachable for every object that has aged out of the incoming tier, which over time is almost every object, and P-001 is the largest single win in this file.

But (b) collapses the znippy question rather than settling it, and that part is Rickard's. If the cold tier stores pack entries — the type/size varint, the delta base reference, the zlib stream — then it is a packfile, and §4 explicitly rejects "packs-in-znippy" as strictly worse than either alone. §4 and this entry's fix cannot both be satisfied. Two further entries push the same way: P-006 (znippy's OpenZL codec cmake-builds C/C++ and fetches from GitHub during cargo build, against §2's appliance rule) and P-014 (znippy has no writer lock of any kind, so gunnar would have to supply the one-writer guarantee znippy will not).

So repack() was built over a consolidated packfile, git's own cold tier, which is representation (b) by construction. The choice is measured rather than asserted: 11 of 12 objects pack-copied on the sha256 fixture, the twelfth being a loose object with no pack residency to copy from. RepackOutcome carries objects_copied_from_pack and objects_recompressed for exactly the reason this entry's Verify demands. The §4.4/D14 swap protocol is written against the tier as a seam and does not change when the tier does. See crates/gunnar-store/src/repack.rs and S-006's closing note.

What is left: whether znippy earns a place under this at all, given that (b) makes it a wrapper around the format it was meant to replace, and given P-006's build-time cost. This entry stays TODO until that is answered.

• Closed:

P-006 — znippy requires a C toolchain, cmake and network at build time — §4's "no C toolchain" claim is false

• Found: 2026-08-03

• Status: TODO

• Severity: blocker

• Where: .nornir/TOTAL-PROMPT.md:462 and the §4 note ending "No C toolchain is required either way (no cc, no zstd/lz4 sys crates), so the hard constraint holds." · znippy-common/Cargo.toml:37 · openzl-sys-rs-0.2.0/build.rs:5-6, 52-59

• What: §4 states the hard constraint explicitly and concludes it holds. It does not hold for any build that includes znippy. READ, znippy-common/src/codec.rs:1-2: "Codec layer: OpenZL compression/decompression. OpenZL is znippy's codec — it wraps zstd+lz4 with improved framing." And openzl-sys-rs-0.2.0/build.rs:

  • :5-6: downloads facebook/openzl from GitHub at a pinned commit and cmake-builds it at compile time.
  • :52-56: cargo:rustc-link-lib=static=zstd and …=static=lz4.
  • :59: cargo:rustc-link-lib=stdc++.

So a znippy build needs cmake, a C and C++ toolchain, network access during cargo build, and links libstdc++. The pure-Rust decoders in the workspace (lgz/ljar/lbzip2/ruzstd) are behind the host-decompressors feature for parsing package files and are not the archive codec.

This compounds S-045 ("russh forces two C crates — the 'no C toolchain' rule is not achievable"). S-045 establishes the rule is already broken; this establishes that the storage decision breaks it much harder: a network fetch and a cmake invocation inside the build are a different class of problem from two vendored C crates, especially for a reproducible appliance image.

• Why it matters at load: Not a throughput issue — a shippability issue for the appliance, and a documentation-integrity one. §4 reaches its conclusion partly on this claim, and the claim is wrong. §4's own dependency arithmetic was already corrected once on 2026-08-03 (the arrow/redb paragraph); this is the second bad premise in the same section.

• Fix: Amend §4's constraint sentence to state what is true: znippy brings cmake, a C/C++ toolchain, a build-time network fetch and libstdc++. Then make the decision deliberately rather than by inheritance: either accept it and record it as a cost alongside S-045, or scope znippy to a non-default feature so the minimal appliance can be built without it. Do not let this be discovered by the first person who tries to build the appliance in a sandbox with no network.

• Verify: Attempt a cargo build of the appliance feature set with the znippy backend enabled in a network-isolated container with no cmake installed, and observe it fail. That failure is the evidence. Then assert the minimal appliance set still builds in the same container, seen RED first if znippy is not properly feature-gated.

• Closed:

scale

P-007 — flate2 resolves to miniz_oxide while zlib-rs sits linked and idle — 1.7–2.1× for one line

• Found: 2026-08-03

• Status: DONE 2026-08-03. Landed, re-measured, and the headline speed claim reproduced (1.70–1.94×). The "byte-identical ratio" claim is corrected below: it is identical to four decimals on the 1.6 MiB corpus this file was written from, but on a 72 MiB corpus zlib-rs produces 0.89% more bytes. Still effectively free; not literally identical.

• Severity: scale

• Where: Cargo.toml:114 (flate2 = "1") · crates/gunnar-pack/Cargo.toml · crates/gunnar-pack/src/write.rs:83, 121 · Cargo.lock:1548-1555, 2683-2690, 6397-6400

• What: gunnar's workspace declares flate2 = "1" with default features. flate2 1.1.9's default is rust_backend = miniz_oxide. Meanwhile gix 0.86 moved off flate2 entirely: gix-pack and gix-odb depend on gix-zlib 0.1.0, whose only deps are thiserror and zlib-rs 0.6.6 (Cargo.lock:2683-2690).

So gunnar links two independent zlib implementations, and its own pack writer — the server's hot path — uses the slower one, while the faster one is already compiled in and used by gix for everything gix touches.

MEASURED 2026-08-03 (Measurement 2 above): deflate L6 41.0/51.5 → 85.8 MiB/s (1.7–2.1×) at a byte-identical compression ratio of 3.28×; inflate 308.4 → 554.6 MiB/s (1.8×).

Two stale beliefs to retire while here, both READ:

• The advice to enable zlib-ng for "2.5×" (from gitoxide's crate-status.md) is moot for gix 0.86. gix-features 0.49 has no zlib feature at all: the whole zlib / zlib-ng / zlib-stock family is gone. gix-0.86.0/Cargo.toml.orig:259-260 says so: "Deprecated: gix always uses zlib-rs, so this is equivalent to max-performance." zlib-rs is the fast path; it is the Rust port of zlib-ng's algorithms.

• zlib-ng would also break the no-C-toolchain constraint. zlib-rs does not. Given P-006 and S-045, do not add another C dependency to win something a pure-Rust crate already in the tree wins for free.

• Why it matters at load: deflate is 86% of the compression+hash budget of serving (see the CPU budget). Roughly doubling it, for one line, with zero new crates and no output change, is the best win-per-effort item in this file. It stays worthwhile after P-001 because objects with no pack residency are still deflated fresh (gix-pack .../entry/mod.rs:138-159).

• Fix: flate2 = { version = "1", features = ["zlib-rs"] } in the workspace Cargo.toml. Confirm with cargo tree -p gunnar-pack -i zlib-rs that flate2 now routes through it, and that miniz_oxide has left the graph. Consider also whether the server default compression_level: 6 (write.rs:31) is right: L1 measured 257.5 MiB/s (3.0× faster than L6) for 11% larger output, which is a real trade for a busy forge and is already a knob (write.rs:36). Measure before changing the default; after P-001 most objects are copied, not compressed, and the trade changes completely.

• Verify: gunnar.pack_deflate bench (P-015) reporting deflate_mbs and ratio_x, run on both backends. The ratio must be unchanged, which is the guard that this is a free win and not a quality trade. Seen RED by asserting the ratio moves, which it must not. Also add a feature_split.rs assertion that miniz_oxide is not in the appliance graph, seen RED today.

• Closed: 2026-08-03. Cargo.toml now carries flate2 = { version = "1", default-features = false, features = ["zlib-rs"] }. cargo tree -p gunnar-pack -e features -i flate2 shows any_zlib on and no rust_backend, which is what selects the backend: flate2/src/ffi/mod.rs only compiles its miniz_oxide module when any_zlib is off.

RE-MEASURED 2026-08-03 on oden (Threadripper PRO 3975WX, 32 threads), through gunnar's real PackWriter rather than a raw flate2 loop, best-of-7 after a warmup, one core (taskset), same frozen corpus both sides:

Both land inside the predicted 1.7–2.1×. The ratio_x guard the entry asked for is crates/gunnar-pack/tests/deflate_backend.rs, on a deterministic in-repo fixture where the two backends differ by 26 bytes out of 172 KiB (0.015%).

Two corrections to what this entry claimed.

1. "a byte-identical compression ratio" is too strong. The deflate streams are not byte-identical at all, which is how backend_is_not_miniz_oxide can detect the backend, and the ratio moves by up to ~0.9% depending on corpus. It is a free win in every practical sense and it is not a no-op on the bytes. Anything that assumed pack files keep their old names across this change would be wrong; nothing does, because a pack checksum is over whatever bytes were written.
2. "assert miniz_oxide is not in the appliance graph" cannot be satisfied and the entry should not have promised it: russh 0.61 depends on flate2 with default features, so rust_backend → miniz_oxide is unified back into Cargo.lock no matter what gunnar declares. The crate is compiled and unused; its backend module is #[cfg(not(feature = "any_zlib"))]. The guard therefore asserts the property that actually matters, the bytes are not miniz's, instead of the property that is unreachable.

L1 is confirmed as a real trade, not a free one: 218.4 MiB/s (2.7× over L6) for 10.8% larger output on the small corpus and 11.7% on the large one. compression_level: 6 is left alone, exactly as the entry advised.

Seen RED: reverting the flate2 line makes backend_is_not_miniz_oxide fail with both sides equal.

P-008 — The pack trailer pays SHA-1 collision detection it cannot benefit from — 4.9× on the hash

• Found: 2026-08-03

• Status: DONE 2026-08-03, as a trailer-only change. Collision detection stays on for every object id. See the closing note for the judgement and its limits.

• Severity: scale

• Where: crates/gunnar-pack/src/write.rs:64, 78, 142, 163 (PackWriter::hasher, emit) · gix-hash-0.26.0/src/hasher.rs:36 · sha1-checked-0.10.0/src/lib.rs:186-190

• What: PackWriter hashes every byte it writes (emit, write.rs:163) with gix_hash::hasher(hash_kind), to produce the pack's trailing checksum. That hasher is sha1_checked::Sha1 with UBC collision detection on (hasher.rs:36), which routes to a pure-Rust compression function and never reaches this CPU's SHA-NI unit (sha1-checked/src/lib.rs:186-190; the SHA-NI path is sha1::compress, sha1-0.10.7/src/compress/x86.rs:100).

MEASURED 2026-08-03: 353.5 MiB/s vs 1778.9 MiB/s, 4.9×. And, counterintuitively, gix's SHA-256 runs at 1601–1841 MiB/s, so a sha1 repository hashes ~4.7× slower than a sha256 one on identical hardware.

The important distinction, and the reason this is not simply "turn it off":

• On the receive path, collision detection is exactly the security property it exists for: the input is a packfile from an untrusted client, and sha1dc is what git itself runs there. Keep it. Per-MiB, hashing is ~61% of the receive pipeline (derived: 2.83 ms hash vs 1.80 ms inflate), and that cost is the price of the property.

• The pack trailer is not an object id. It is an integrity checksum over the pack's own bytes, read back by the receiver to detect truncation and corruption. A collision attack against it accomplishes nothing an attacker could not do by simply sending different bytes. Collision detection there is pure cost with no corresponding property.

• Why it matters at load: Today this is ~3% of the serve budget and not worth a day. After P-001 it is ~95% of what remains (0.86 ms of a 0.91 ms budget per MiB), because pack-copy deletes the deflate that currently dwarfs it. That sequencing is why it sits at #5 in the build order and not #2. Do not do this first; do not skip it after P-001.

• Fix: Give PackWriter a hasher chosen for the checksum role rather than the object-id role: sha1::Sha1 for sha1 packs, keeping gix_hash::hasher everywhere an actual object id is computed. Do not reach for sha1_checked::Builder::default().detect_collision(false) as a global switch: the point is to separate two roles that currently share one call, not to weaken both. Document the reasoning at the call site, because the next reader will assume it is a mistake.

• Verify: gunnar.pack_trailer_hash bench reporting hash_mbs, plus a byte-for-byte test that a pack written with the fast hasher has a trailer identical to one written with gix_hash::hasher. The two must agree exactly, since they are the same function absent an attack. Seen RED by perturbing one input byte. That equality test is the whole safety argument and must not be skipped.

• Closed: 2026-08-03. crates/gunnar-pack/src/checksum.rs introduces TrailerHasher: plain sha1::Sha1 for SHA-1 trailers, gix's hasher untouched for everything else. PackWriter holds that instead of gix_hash::Hasher. Nothing else moved.

Where detection is load-bearing and where it is not, decided explicitly.

• Object ids: KEPT, and it was never close. A SHA-1 object id names content the server was handed by an untrusted client. Two colliding blobs sharing one id is precisely the attack sha1dc exists to catch, and a forge ingesting pushes is exactly where it would land. gunnar computes ids through gix_object::compute_hash (crates/gunnar-store/src/object.rs:113), which still routes to gix_hash::hasher. Verified by grep: after this change, gix_hash::hasher has exactly one production call site outside the object-id path, and it is the non-SHA-1 arm of TrailerHasher.
• The pack trailer: pure cost, removed. It is a checksum over bytes the writer itself just produced, appended to those same bytes. It names nothing. An attacker who found a second byte string with the same trailer would have achieved what he can already do for free: send a different pack with its own honest trailer. There is no property to lose.
• Not done, deliberately: sha1_checked::Builder::default().detect_collision(false) as a global switch, which the entry warned against and which would have weakened the one path that earns the cost. Also untouched: the receive path, where gix-pack's bundle writer does its own hashing of an incoming pack.

MEASURED 2026-08-03 on oden, through the real PackWriter, best-of-35 interleaved runs of the before/after binaries under identical load (the box was busy with other agents; best was stable to ~1% while median and worst were not, so best is what is quoted). Corpus: 150 files, 1.70 MiB.

The SHA-256 row is the control that makes the rest believable: the SHA-256 arm still calls gix, and it did not move. Confirmed on the 72.63 MiB corpus too (L6 1.04×, L0 358.3 → 1458.0 = 4.07×).

The entry's sequencing argument is confirmed by measurement, and its number was slightly low. Today's win is +6% on a whole pack write, not 3%, because P-007 landed first and made deflate cheaper, which raised the hash's share. At L0, which is the shape P-001's pack-copy produces, it is 4×. Also worth recording: after this change a SHA-1 repository hashes at the same speed as a SHA-256 one (1442 vs 1389 MiB/s at L0). The 4.7× sha1-versus-sha256 asymmetry Measurement 1 found is gone from the serve path. It remains on the object-id path, where it is the price of the property.

Verified: crates/gunnar-pack/tests/trailer_hash.rs pins the trailer against sha1sum / sha256sum — coreutils, not gunnar's own arithmetic — for a five-object pack, an empty pack, and both hash kinds, and asserts plain SHA-1 and gix_hash::hasher agree exactly on the same body. Seen RED: XOR-ing one bit into the finalized digest turns 4 of the 5 tests red (the SHA-256 one correctly stays green). The 17 real-git interop cases in gunnar-wire and git fsck --strict in gunnar-client/gunnar-store are all still green, which is the end-to-end statement that stock git accepts these trailers.

P-009 — gitoxide has no delta encoder at all; what that costs and what pack-copy recovers

• Found: 2026-08-03

• Status: TODO

• Severity: scale

• Where: research finding. Settles the open question S-003 raises ("whether gix-pack's data::output pipeline can produce the delta encoding, or whether the window search has to be hand-written").

• What: Settled: gitoxide cannot compute a delta. It can only copy one. Evidence, READ and exhaustive:

  • grep -rn -iE 'fn .*(delta_encode|encode_delta|create_delta|diff_delta|make_delta|delta_index)' across every vendored gix-* crate → zero hits.
  • gix-pack-0.73.0/src/data/delta.rs contains only apply and decode_header_size: the decode half, with no encode counterpart.
  • No window or depth logic anywhere in gix-pack. iter_from_counts::Options (:372-396) has no window/depth field, only a TODO: "could this become the window size?" (:385).
  • gix-diff 0.66 is text/tree diff (imara-diff), not a binary delta encoder.
  • output::Entry::from_data (entry/mod.rs:138-159), the only path for a non-packed object, sets kind: Kind::Base(obj.kind) unconditionally.

So S-003's question has a definite answer: the window search would have to be hand-written. A real encoder means size-sorted sliding window (git's default: window 10, depth 50), per-candidate delta attempt, chain-depth accounting. Substantial work, with a large parallel search that is a good gatling_map_balanced candidate.

But do P-001 first, and then re-ask whether this is needed. Under §4's design a push arrives as a packfile the client's git already delta-compressed. Pack-copy serves those deltas straight back. The objects that would still need a fresh delta search are only those with no pack residency, plus RefDelta entries which gix-pack downgrades to bases (entry/mod.rs:113). The client already paid for the delta search; gunnar's job is to stop throwing the result away, which is a far smaller job than writing an encoder.

• Why it matters at load: Bandwidth is what a forge at this scale actually buys, and delta compression is the largest determinant of clone size. No multiplier is claimed here — nothing has been measured, and repeating a general-knowledge figure as if it came from this repository is exactly what LAW 7 forbids.

• Fix: (1) Land P-001. (2) Measure: clone a real repository with heavy revision churn through gunnar and through stock git; record both pack sizes and both wall clocks; additionally record what fraction of served objects were pack-copied vs deflated fresh. That measurement is the deliverable, and it is what decides whether an encoder is worth writing at all. (3) Only then, if the residual fraction justifies it, write the window search, and fan it out with gatling_map_balanced (LAW 3, never rayon).

• Verify: Per S-003: a fixture with near-duplicate revisions, not unique random blobs. The identity-value trap here is that random blobs delta to nothing and would make a working encoder look broken. Assert on served pack bytes, seen RED against today's writer where whole-object and delta sizes are equal by construction.

• Closed: 2026-08-03 as WONTFIX for now, on the measurement this entry asked for. Step (2) is done. P-001 landed and the head-to-head was re-run on the same churned fixture, and the residual fraction the decision turns on is:

And the pack that comes out is 1.000× stock git's on the clone and 1.018× on the fetch. There is no room left for an encoder to work in. A window search would be a substantial piece of work — size-sorted sliding window, per-candidate attempt, chain depth accounting — competing for the 0–2.4% of entries that gix could not copy, on a path that already matches git byte for byte.

The residue is accounted for: RefDelta entries, which gix-pack downgrades to bases (entry/mod.rs:113), and deltas whose base was not in the requested set. Both are properties of the request, not of the compressor.

Where an encoder would still be needed, and it is not the serve path: a repository whose objects have never been in a pack at all: the loose tier immediately after a gunnar init and a series of small writes. repack() folds those, so the question is how fresh the cold tier is (P-011), not whether gunnar can compute a delta. Reopen this entry if a measurement ever shows a meaningful fraction of served entries recompressed; gunnar_recompressed is emitted on every head-to-head clone arm precisely so that fraction is visible rather than assumed.

Not done: crates/gunnar-delta exists (a D21-shaped encoder crate, owned separately) and nothing links it. That remains true and this entry is not an argument for deleting it, only for not putting it on the serve path on the strength of an unmeasured hunch.

AMENDED 2026-08-04 — the last paragraph is now wrong, and the conclusion above is still right. Two things link gunnar-delta today. gunnar-client computes deltas on push, which closed the 0.0%-deltified hole this entry could not see. And repack() now computes them too (DeltaPolicy, crates/gunnar-store/src/pathwalk.rs), because the "reopen if" condition was met from an angle this entry did not consider: not the fraction served fresh, but the quality of what is copied. Pack-copy is 100% faithful and that is exactly the problem — a repository seeded by a client with a poor encoder keeps those deltas for ever, and gc could not improve them because it short-circuited on a single-pack repository (repack.rs:647, fixed). The serve path is unchanged and must stay unchanged: no encoder runs per request, cargo tree still proves gunnar-wire, gunnar-ssh, gunnar-http and gunnar-store resolve without one, and this entry's WONTFIX for the serve path stands.

P-010 — The closure walk is one serial thread and is the bottleneck the moment P-001 lands

• Found: 2026-08-03

• Status: PARTLY DONE 2026-08-03 — the walk was profiled and its redundant work removed (measured, both sides). The parallel half is not landed and is now argued against; see "What was actually measured" below.

• Severity: scale

• Where: crates/gunnar-wire/src/graph.rs:155-202 (commit_closure), :210-269 (tree_closure), :337-461 (select_objects), :470-483 (verify_connectivity)

• What: Every walk is a single-threaded VecDeque/Vec loop with a HashSet seen-set. S-011 names this; this entry is about what to replace it with and why it moves up the list.

Once P-001 removes compression from the serve path, the walk is the clone: the CPU budget's last column is 0.22 ms/MiB, and a million-object graph traversal with a million scalar store lookups is nowhere near that. The walk becomes the bottleneck by construction, and it has never been measured.

The shapes differ and want different primitives:

• commit_closure (:155-202) is a level-synchronous BFS, exactly gatling_bfs(starts, neighbors, max_radius, n_workers) (znippy-zoomies/gatling/src/gatling_bfs.rs:105), which dedups at each level barrier. The stop/boundary/restrict sets become filters inside neighbors.

• tree_closure (:210-269) is a frontier expansion over independent subtrees with a shared seen-set. The seen-set is the contention point; the gix precedent is a sharded map. gix_hashtable::sync::ObjectIdMap is 256 parking_lot::Mutex shards keyed on the oid's first byte (gix-hashtable-0.16.0/src/lib.rs:29-48), which at uniformly random oids gives low contention. Reuse that idea rather than inventing one (LAW 5).

• select_objects step 4 (:391-409), the exclusion closure, is per-commit independent and is the most straightforwardly parallel of the three.

• Why it matters at load: It is the other half of the answer to "gunnar pegs one core". But note P-004: this fan-out belongs to the serving tier, so it must be gated by admission control and must not be a bare gatling_bfs(…, 0) per request.

• Fix: Convert commit_closure to gatling_bfs and tree_closure to a level-batched gatling_map_balanced over the frontier with a sharded seen-set, both behind P-004's admission gate and both reached via spawn_blocking. Weight by header(id).size so the LPT scheduler has something real to sort on. And remember size_of runs serially on the caller thread, so it must come from the batch header lookup (P-003), not from a per-item store read. This composes with P-003: a level-synchronous walk produces exactly the batch of oids a batch lookup wants. Do not touch select_objects' set cloning here; that is S-004, owned separately.

• Verify: gunnar.walk_1m bench reporting objects_per_sec on a million-object fixture, serial vs parallel. Then a guard that both paths return identical object sets, compared sorted, seen RED by deliberately dropping one subtree from the parallel path. Do not assert "it was faster": on a small fixture the threaded path is slower and the guard would be tuned away (the same warning S-011 gives).

What was actually measured, 2026-08-03 — and what the entry got wrong

The instrument first, because it is why these numbers exist at all. oden was not idle for any of this: /proc/loadavg read 9.4 to 110.5 on 32 cores throughout, with 17 concurrent cargo build processes and three other agents' suites running. On that box gunnar.walk_1m's own objects_per_sec moved 2.36× across five repetitions of one unchanged binary (190 996 → 450 019), and clone_large's clone_ms moved 3.79× across eight runs. Wall clock cannot resolve anything here.

Instructions retired can. Under perf stat, the same five repetitions came to 13.194, 13.197, 13.201, 13.193 and 13.192 G instructions — a 0.29% spread against wall clock's 236%. Every before/after ratio below is instructions retired, from binaries run interleaved (B, A, B, A, B, A) so that any drift in the box's load falls on both sides. Process CPU seconds are quoted as a second, independent witness and agree to within 1.4%.

The profile: what the walk actually spends itself on

perf record -F 999 over gunnar.walk_1m (333 334 commits × (blob + tree + commit) ≈ 1.0 M objects, MemObjectStore). Shares are of the whole arm, fixture build included, which is stated because it is the denominator:

The single biggest finding is not in that table: every commit in a clone was being read and decoded twice. commit_closure decoded each commit for its parents and threw away the tree field it had just parsed; select_objects steps 5 and 6, and verify_connectivity step 3, then asked the store for that same commit again purely to get the tree back. On the fs backend the second read is a pack-index probe, a zlib inflate and a delta chain — the odb handles are built refresh_never() with no object cache configured anywhere in gunnar-store, so nothing memoised it.

This is measured, not inferred, by an independent counter — the Counting store's read tally, which knows nothing about the profiler:

The difference is exactly one read per commit, on both backends and both directions of the protocol.

What landed

Three changes, all in crates/gunnar-wire/src/graph.rs, all of which reduce total work rather than rearrange it — which is what §"The parallelism answer" ranks above fan-out, and which is the only kind of change P-004's "the serving tier stays serial" decision permits at all:

1. commit_closure returns the trees it already read (CommitClosure { ids, trees }). Kills the second decode. Costs one commit-sized Vec<ObjectId>: +10.63 MB peak RSS measured on the 333 334-commit fixture against 10.17 MB predicted from 32 bytes × commits, which is the whole of it.
2. commit_links replaces CommitRef::from_bytes on the walk path. CommitRefIter decodes only as far as it is driven and git's grammar puts tree and every parent first, so stopping at the first non-parent token leaves the committer, the encoding, the extra headers and the message unparsed. Graph::commit keeps the full decode for negotiation, which is the only caller that wants committed_at and asks about a handful of commits rather than all of them.
3. Buffers hoisted out of the per-object loops — commit_links takes the caller's buf and parents; tree_closure's per-tree children and blobs vectors are cleared rather than reallocated.

The numbers

Six interleaved repetitions per side, perf stat, oden 2026-08-03, loadavg 54–79:

Both figures are real and the gap between them is the result. 1.295× and 1.025× are the same change measured on two corpora, and the difference is entirely what fraction of the served bytes hang off a commit. walk_1m is a third commits by object count and serves 127 MB; clone_large is 2048 commits inside 6144 objects and serves 1 GiB of blob content, so its walk is a rounding error next to its pack copy and a 1.295× walk buys 2.4% of the arm. Anybody quoting one multiplier for P-010 has to name the corpus, the same way P-001's 9.9×/18.3× split does. A commit-heavy repository — a kernel-shaped history, ~13% commits by object count — sits between them.

What the arm's own guard turned out to be

walk_1m asserted store_touches >= objects. It passed only because of the bug. 333 334 of the 1 000 639 touches were the redundant reads; removing them dropped the arm to 667 305 and made it go RED against its own floor. The floor had quietly encoded the inefficiency as the invariant — the only way to satisfy it was to keep doing the extra work — and blobs, which are checked with contains and not read at all, never contributed to it in the first place. It is replaced by two bounds:

• floor: commit_reads >= commits && tree_reads >= trees, per kind, so a stubbed walk still cannot pass;
• ceiling: commit_reads <= commits + 1 (the + 1 is the tip peel, spelled out rather than absorbed into a tolerance). Seen RED: the pre-fix walker reports 2.000 reads per commit and fails it.

commit_reads_per_commit is now a published metric, so the regression has a number rather than a code review.

What was NOT done, and why the entry's own Fix is now argued against

• gatling_bfs was not landed, and on this evidence should not be. The entry proposes fanning the walk out across cores. But P-004 is DECIDED — serving tier stays serial, and the walk is on the serving tier; the entry's own "Why it matters" already concedes it must be gated. More to the point, §"The aggregate-throughput point" in this file says plainly that parallelising one clone across 32 cores does not raise clones/second at all — it is the same total work rearranged. The three changes above delete 3.006 G instructions; a perfect 32× fan-out would delete none. The remaining serial walk is now the smaller of the two problems, and the honest next step is more deletion, not threads. If the tail latency of a single very large clone ever becomes the binding constraint, this reopens — but it must be reopened on a latency measurement, which nobody has taken.

• The oid hasher was not changed, and it is the next measured item. ObjectId is stored in std::collections::HashSet, which runs SipHash over a value that is already a uniformly random hash: ~6.5% of the walk_1m profile in hash_one and DefaultHasher::write, next to 12.3% in __memcmp_avx2_movbe doing the probes those hashes drive. gix solved this already — gix_hashtable is hashbrown with a passthrough hasher over the oid's own first bytes, and this file already cites its sync::ObjectIdMap for the sharded seen-set. Not landed here because it is 52 sites across four files and a new direct dependency on gix-hashtable, which is a §2 dependency-guard decision, not a perf decision. Sized at ~6.5% of this profile, unmeasured as a delta.

• Frontier::new still calls Graph::commit for each have's tree. Left alone: it is bounded by what the client sent, not by the repository, which is the same line OidSet's doc draws.

• No fs-backed commit-heavy fixture exists. clone_large is 2048 commits inside 1 GiB of blobs and walk_1m is MemObjectStore, whose read is a memcpy rather than an inflate. The corpus where this change should pay most — many commits, real packs — has not been measured, and the 1.295× is therefore a floor for that shape, not a ceiling. Building it is a P-015 follow-on.

• Closed: partly, 2026-08-03. The redundant-work half is done and measured on both sides. The parallel half is not done and is argued against above; it stays open only as a latency question.

P-011 — repack(), gc and index building are the true gatling batch jobs, and none exist

• Found: 2026-08-03

• Status: TODO

• Severity: scale

• Where: S-006 (repack() does not exist) · S-035 (the three znippy git indexes are unbuilt) · .nornir/TOTAL-PROMPT.md:575-610 (§4.2)

• What: S-006 and S-035 establish that these do not exist. This entry records what shape they must have when written, because they are the workloads gatling was actually designed for and getting them right is most of the value of adopting it.

All four are offline, one-at-a-time, whole-repository batch jobs: no request is waiting, the input set is known up front, and the box can be given entirely to them. That is gatling_map_balanced's exact case, and unlike P-010/P-004 there is no admission problem.

• Why it matters at load: A forge repacks constantly. S-006 notes no cold tier ever forms without it; the further point is that repack latency bounds how fresh the cold tier can be, and a serial repack of a million-object repository is minutes of one core. Also relevant: znippy's seal is single-threaded and O(total rows). MEASURED in znippy's own suite, footer_seal at 1 M rows is 1545 ms and random_lookup_cold p50 at 1 M rows is 281 ms (znippy/.nornir/benchmarks.md:86-94, baseline 2026-06-18, znippy 0.9.0 — date it when quoting; it predates the OpenZL codec change). That serial tail sets a floor no amount of gatling removes.

• Fix: Write them with the shapes above. Serialise the whole batch tier behind a semaphore of 1–2 so two repacks never contend, and keep a long-lived ArchiveReader open per repository so the 281 ms cold-lookup cost is paid once, not per request.

• Verify: gunnar.repack_1m bench reporting objects_per_sec and seal_ms separately. Separately, because the seal is the serial floor and lumping it in hides whether the fan-out worked at all. Add the house core_saturation gate (default: peak core-busy ≥ 0.9 × cores, nornir/src/release/gate.rs:124): a parallel bench that peaks below that is a bug, not Amdahl, and this is exactly the bench that should carry it.

• Partly measured, 2026-08-03, and gunnar.repack_1m was measuring a loop that repack() does not run. repack() exists (S-006); the arm named after it did not call it. It hand-rolled for id in ids { store.read(id); writer.add_object(…) }, which is the whole-object writer P-001 exists to get off the serve path, and published 87 054 objects/s under the name of a repack that runs neither that loop nor that writer. Rewritten to drive FsStore::repack on a throwaway copy of the fixture (a repack mutates its repository, and a second one is a documented no-op).

Measured, oden 2026-08-03, 1 000 000 objects over 64 packs, sha256, load ~175:

Three findings, in descending order of how much they change the entry.

1. The batch tier is already on P-001's path and always was. A repack of a packed repository copies every entry: 1 000 000 copied, 0 recompressed. The PackCopyAndBaseObjects generator is what repack() drives, so P-005's "pack-copy works from the cold tier" and this entry's "under P-005 this becomes a pack-entry copy, not a recompress" are both true today. The old arm's 87 054 objects/s was 3.7× pessimistic and of the wrong code.
2. It already fans out, and gatling is not needed for it. This entry's table prescribes gatling_map_balanced over objects for repack(). It is already parallel, through gix-features' own std-thread pool, at 1.30–1.55× on 32 cores across three runs. Replacing that with gatling would be twinning a scheduler (LAW 5) to chase a fan-out that is present; the reason the speedup is 1.3× rather than 20× is P-018's serial tail plus the count/verify/index passes around it, and gatling does not touch any of those. The gatling_map_balanced recommendation for repack() is withdrawn. It still stands for the three index-building jobs, none of which exist.
3. RepackOptions::thread_limit does not bound the repack. It reaches count::objects and iter_from_counts and nothing else; FsStore::write_multi_pack_index runs inside repack() and gix_pack::multi_index::write_from_index_paths takes no thread limit at all (multi_index::write::Options carries only object_hash). Hence 33 peak threads either way. An operator setting thread_limit(1) to keep a repack off the serving cores gets a slower repack, not a narrower one. Recorded on the field's own doc comment; fixing it means a knob upstream or splitting the index write into a step the caller schedules, and neither was worth guessing at here.

NOT measured, and both are gaps this entry asked for:

• seal_ms. FsStore::repack exposes no seam at the seal, because FromEntriesIter hashes as it writes, so the only figures available from outside are the whole repack and the bytes. Deriving a seal from a byte count and another arm's hash rate would be arithmetic wearing a stopwatch's name, so the arm emits nothing rather than that.

• core_saturation. Same reason as everywhere else in this file: the telemetry sampler reads the whole host and oden was at load 175 from other agents.

• gc, dead-blob reclaim, and the three index builders remain unwritten, so the parts of this entry about them are unchanged.

• Closed:

P-012 — Object lookup is O(packs) on a miss, and every thread carries its own unshared cache

• Found: 2026-08-03

• Status: DONE 2026-08-03. Misses 128× faster over 64 packs, hits 21×, and the degradation itself is gone: miss_degradation_x 23.2 → 1.01. The diagnosis this entry opened with was only a third of the cost; see Closed below.

• Severity: scale

• Where: gix-pack-0.73.0/src/index/access.rs:276-297 · gix-odb-0.83.0/src/store_impls/dynamic/find.rs:113-121, 282-284 · gix-odb-0.83.0/src/store_impls/dynamic/mod.rs:31-33 · gix-odb-0.83.0/src/cache.rs:226-229

• What: Three properties of gix's lookup path that matter at a million objects and thousands of concurrent transfers, all READ:

  1. Single-pack lookup is a 256-entry fanout then binary search (access.rs:276-297). At 1 M entries the fanout narrows to ~3906, so ~12 probes, each a 20-byte read at 20-byte stride across a ~20 MB mmap. INFERRED: memory-latency-bound, order 1–2 µs cold and ~100 ns warm; the first few probes reliably miss L3. This is the cost P-003's batch path exists to hide.
  2. A miss probes every pack. dynamic/find.rs:113-121 loops over all indices until a hit; a miss must probe all M before falling through. There is an MRU promotion (find.rs:282-284, snapshot.indices.swap(0, idx)) which helps locality-sorted traversal and does nothing for random-oid workloads. A forge that does not repack accumulates packs, and every have from every client is a lookup that may miss. S-006's missing repack() therefore has a lookup cost, not only a storage cost.
  3. The odb handle is !Sync: snapshot, inflate and the pack cache are all RefCell (dynamic/mod.rs:31-33, cache.rs:226-229). There is no lock contention on reads, but equally no cache sharing: every worker clones the handle and gets its own delta-base cache, inflate state and snapshot. Memory grows linearly with concurrency and hit rate is diluted N-fold. At 64 workers with the stock 64-slot, 96 MB-capped LRU that is up to 6 GB of caches with 1/64 the hit rate of one shared cache.

Note gunnar's server never opens gix::Repository (it bridges through crates/gunnar-store/src/gix_adapter.rs), so none of gix's cache defaults apply. GixObjects::try_find (gix_adapter.rs:69-75) is a cache-free forward to ObjectStore::read. Whatever caching exists must be gunnar's own, and today there is none.

• Why it matters at load: It sets the floor P-010's walk runs against, and it is the reason P-004's "one core per transfer" model is the right default: N threads sharing a repository do not share its cache.

• Fix: (1) P-003's batch seam, so the latency can be pipelined. (2) A shared, bounded object/delta-base cache per repository behind the ObjectStore, sized once rather than per worker; gunnar's to build, since the gix defaults are bypassed. (3) repack() (S-006, P-011), which is what actually bounds M. (4) When the znippy cold tier lands, hold one long-lived ArchiveReader per repository (P-011).

• Verify: gunnar.lookup_1m (shared with P-003) reporting lookups_per_sec for hits and misses separately, and gunnar.lookup_1m_64pack with the same object set spread over 64 packs. The miss number must be seen to degrade with pack count; if it does not, the bench is not producing real misses and is measuring nothing.

• Closed: 2026-08-03.

The diagnosis, which is not what this entry predicted

This entry said a miss is slow because it must be disproved against every pack index, so cost is linear in pack count. That is true and it is the smallest of the three terms. Measured before touching anything, on oden, over the bench suite's own fixtures, by a probe that reproduces each candidate cause in isolation:

A miss ran 11× faster at ONE pack, where there is no O(M) term at all to explain it. gix-odb's default RefreshMode::AfterAllIndicesLoaded (gix-odb-0.83.0/src/store_impls/dynamic/mod.rs:49-58) means that when a lookup falls through every loaded index, Handle::contains (find.rs:320-345) calls load_one_index, which calls consolidate_with_disk_state (load_index.rs:106-112, 201): a full re-scan of objects/ and objects/pack, behind the store's global write lock, on every single miss. Not a pack probe: a directory scan. Measured directly, a bare statx of the pack directory is 640 ns and a bare read_dir is 2.4 µs at one pack and 30 µs at 128 entries, and the full consolidation is ~10 µs at one pack and ~296 µs at 64.

The second term was gunnar's own and this entry did not name it: FsObjects built a fresh handle for every call (fs.rs:300-302, before this change). Store::to_handle_arc() runs collect_snapshot(), which clones one Arc per loaded index into a new Vec, so handle construction is itself O(packs), and unlike the refresh it is paid on hits as well. That is the whole of the hit-path degradation: 1 073 582 hits/s over one pack against 100 323/s over 64, with no miss anywhere in sight.

Only the third term is the O(M) probe this entry is named for, and it is worth 7.2× across 64 packs once the other two are gone.

What was chosen, and why not the alternatives

1. Pool the gix-odb handles (crates/gunnar-store/src/odb.rs), bounded at 64. The handle is Send and not Sync — its snapshot and inflate state are RefCells — so a lease is the shape that fits. This also answers this entry's third original point: memory is bounded by the pool rather than by concurrency.
2. RefreshMode::Never on every pooled handle, which is git's own answer: has_object() passes OBJECT_INFO_QUICK by default and re-scans only when a caller opts in with HAS_OBJECT_RECHECK_PACKED, precisely because misses are the common case on a server. git then makes the writer call reprepare_packed_git(). gunnar cannot: the code that drops a pushed pack into objects/pack is gunnar-cli's PackIngest, and gunnar-wire's only contract is "after ingest returns, the objects must be readable from the ObjectStore". So the re-scan is put back where it is needed by a one-statx watch on objects/pack, trusted only when the mtime it memoised was already in the past when it was taken, because a filesystem timestamp comes from a coarse clock and a change inside the same tick as the scan is invisible. That is git's own "racily clean" rule applied to a directory. Its cost is that for one second after a pack lands a miss pays the full consolidation, which is what it paid always before, so the new path is never slower than the old one. That price is now a published metric, lookups_per_sec_miss_after_write, rather than a claim in a comment.
3. A multi-pack index (crates/gunnar-store/src/midx.rs) for the O(M) term.

Not an stree. §4.1/§4.2 intend one and znippy-plugin-git has one, and LAW 5 says reuse rather than twin. But the measurement that justified stree also recorded that its serial path is a wash against a binary search (0.99× at a million keys) and that only the batch path wins, 1.5–2.6×. A want that misses is a single serial lookup, and P-003's batch seam does not exist.

Not a Bloom or cuckoo filter. After (1) and (2) a miss against one index over a million objects costs 2.3 µs, of which one statx and one loose-object stat are syscalls no filter can remove. A filter could delete the binary search and nothing else, and would need to be built, persisted, invalidated on every push and kept in agreement with the pack set: a second index with its own staleness protocol, which is the twin LAW 5 forbids.

The reuse that pays is git's own, and it is already in the tree: gix-pack writes a multi-pack index (multi_index::write_from_index_paths) and gix-odb reads one (use_multi_pack_index defaults to true; load_index.rs:495-533 collapses every .idx the index names into a single snapshot entry). Zero new formats, zero new lookup code, one file.

The trap, measured before it was believed

A multi-pack index on its own makes the miss path worse. Over 64 packs, misses/s:

With the default refresh mode still on, adding the index costs a further 28%: the per-miss consolidation has one more file to stat and validate. It only pays once the miss has stopped scanning the directory. Shipping the obvious fix alone would have moved the benchmark the wrong way.

BEFORE and AFTER — oden, 2026-08-03, one million distinct objects

Both sides measured in one process, interleaved, best of seven, over two byte-identical copies of the same fixture. BEFORE is not a memory of an earlier run: it is the exact code FsObjects used to run, store.to_handle_arc().exists(id) with the default refresh mode, against a copy that never had a multi-pack index written into it. Best-of rather than mean because oden was at load 45–47 on 32 cores throughout, with other agents building; the best run is the least interfered with, and interleaving means neither side gets the quiet minute. Reproduced twice within 1%.

miss_degradation_x 23.2 → 1.01, and hit_degradation 10.7 → 0.97. The AFTER numbers are pack-count-independent to within 1%, which is the actual claim: the degradation is not reduced, it is gone. Nothing was traded: the hit path improved at both pack counts, which is the regression this could most easily have hidden.

The bench arms, and how the red survived the fix

gunnar.lookup_1m_64pack's red condition is miss_degradation_x > 1, which exists to catch a probe set that produces no real misses. A fix that makes 64 packs cost what one pack costs deletes that guard if the arm only measures the shipped configuration. So the arm now measures the same repository both ways, back to back:

• miss_degradation_x, multi-pack index removed. The raw O(M) term, and what the original red condition is still asserted against, unchanged. Full size: 6.921.
• miss_degradation_x_midx, index in place: 1.005.
• multi_pack_index_gain_x = 6.885, asserted > 1, and miss_degradation_x_midx < miss_degradation_x asserted as well, a relative assertion, because a hard threshold would pass on a machine slow enough.

Full-size row, oden 2026-08-03, single_pack_drift_x 1.052:

⚠ These absolutes are not comparable with the ones published earlier today, and the reason is a second hollow fixture; see the correction under P-015. In short: the corpus these arms ran on contained 16 712 distinct objects, not 1 000 000, so lookup_1m was searching a sixteen-thousand-entry index that fits in L2. The corpus is fixed and the ratios above are all from the corrected one.

What is asserted, and what was watched failing

crates/gunnar-store/tests/multi_pack_lookup.rs holds nine tests, both hash kinds, both directions of wrongness, before and after a repack. Each red below was produced by breaking the implementation, not the test:

1. Turning the refresh off can make a present object report absent. With refresh_if_stale returning false unconditionally — "just set refresh_never and ship it" — a_pack_written_out_of_band_is_found_without_an_explicit_refresh fails on the first contains of the second batch, and a_miss_answered_before_the_pack_landed_is_answered_again_afterwards fails with "the store went on answering false for an object whose pack had landed". Both write the pack the way a real push does: index_pack straight into objects/pack, telling the store nothing.
2. A multi-pack index can make an absent object report present. With repack() made to leave the index alone, a_repack_invalidates_the_multi_pack_index_it_made_stale and a_repacked_repository_reopens_without_a_stale_index fail with "object b724…5a42 is not in the store": contains still answered true out of the stale index and the pack it points into had been unlinked. Stated honestly: disabling only the removal inside the swap does not fail these, because the rewrite at the tail removes the index a moment later. The removal in the swap closes the window for a concurrent reader and no test here earns it.
3. A short index. With indices_in made to drop one .idx: the_multi_pack_index_spans_every_pack_and_every_object fails with "the index does not span all five packs", and objects_resolve_through_the_multi_pack_index_alone fails because the objects of the unnamed pack became unreadable the instant their .idx was gone, which is the direct evidence that lookups go through the multi-pack index rather than around it. Writing the file proves nothing on its own.

A data-loss bug this uncovered in repack(), and the guard that now stands there

real_git_clones_a_repacked_repository went red the moment the pool began scanning at open: "trying to write ref 'refs/heads/main' with nonexistent object". repack() builds its object set from Handle::iter(), which reaches load_all_indices(), documented to load every index the store already knows about, passing RefreshMode::Never; it never looks at the directory. Meanwhile the set of packs it deletes comes from a fresh read_dir. Before this change the store happened to be uninitialised at that moment, so the first iter() consolidated and the two agreed by luck. Initialising the index at open removed the luck: the repack carried one loose object across and then unlinked three packs holding everything else.

Two things now stand there. A refresh after the deletion snapshot and before the iteration, so the object set is by construction at least as new as the deletion set. And verify_folded_into, which reads the index of each pack about to be unlinked and refuses unless every object in it is in the new pack. That is the mirror of the existing verify_pack_holds, which checks the new pack against the request and so cannot see a request that was short. Watched red by removing the refresh: three tests fail with "…idx holds 51b2664f… and the repacked pack does not, so nothing was deleted": a refusal that deletes nothing, in place of silent data loss.

What is NOT done

• A shared object/delta-base cache (this entry's fix (2)). Not built. FsObjects still reaches gix_object::Find on the raw dynamic handle, which decodes with no pack cache at all, so there is nothing yet to share. The pool bounds the memory that concern was about; it does not add caching.
• P-003's batch seam (fix (1)) is untouched, and remains the prerequisite for the stree design §4.1 wants.
• The multi-pack index is written by maintenance, never from a lookup, as in git. FsStore::write_multi_pack_index() is public for gunnar gc; repack() rewrites it; and FsStore::open_or_init writes one when a repository is already at or above MULTI_PACK_INDEX_THRESHOLD (16) packs, which is the only automatic reach available from this crate. A repository that crosses the threshold while a store is open — a long-lived handle taking pushes between garbage collections — does not get one until it is reopened or gc runs. It still gets the pool and the refresh fix, worth 18× on misses there. Wiring a periodic write_multi_pack_index() into the server belongs to gunnar-cli, which this change does not touch.
• Concurrency was tested but not measured. concurrent_lookups_share_the_handle_pool_without_lying runs 16 threads and asserts answers, not throughput; the pool's contention under forge-scale concurrency is unmeasured, and clone_concurrent_64 is not a lookup arm.

P-013 — The iceberg backend cannot be multi-process — skade's RedbCatalog takes an exclusive file lock

• Found: 2026-08-03

• Status: TODO

• Severity: scale

• Where: .nornir/TOTAL-PROMPT.md:418-461 (§4 Backends table, iceberg row) · skade/src/store.rs:12-17, 292, 322 · skade/.nornir/reasoning.md:32-36

• What: §4 lists iceberg as a backend ("OFF by default; drags skade"). The analysis below is the verdict my brief asked for, and it is not a brochure.

What Iceberg genuinely buys a forge at a million users:

• Object storage economics. Blobs on S3 instead of local disk: cheap, durable, effectively unbounded, and the natural home for a cold tier that is written once and read rarely.
• The whole forge queryable in SQL. This is the real prize and it is already half delivered without Iceberg: znippy's lookup sub-index is a valid Arrow IPC stream, so DuckDB/Polars/DataFusion can read it with zero gunnar code (§4). Iceberg adds schema evolution, snapshots and a catalog over that.
• Time travel. Iceberg snapshots over the object index give "what did this repo contain last Tuesday" as a query rather than as a feature to build.
• Horizontal read scaling, in principle. Many readers over the same S3 data.

What breaks, and why the verdict is what it is:

1. The catalog is single-process, by design and by file lock. skade/src/store.rs:12-17 funnels every write through a tokio::sync::Mutex<Database> over redb, and skade/.nornir/reasoning.md:32-36 states the cost outright: "single process. redb takes an exclusive file lock, so two processes can't open the same catalog… if you need multi-writer, you need a service, and this crate is the wrong tool." A second gunnar process cannot open the catalog at all; it fails to start. So the headline benefit, horizontal read scaling, is exactly what this catalog cannot provide. The designed escape is skade-katalog serve (one lock-owning process fronting REST/gRPC), which the design docs mark as NEW/bench-filed: a plan, not something to depend on today.
2. Per-object read latency is wrong by orders of magnitude. A git object read is a sub-millisecond operation on a hot path taken thousands of times per clone. An S3 GET is tens of milliseconds. Iceberg's metadata layers add more. This is survivable only for a cold, bulk-read tier, never for the object store a graph walk hits.
3. IcebergZnippyReader::open is eager and O(all rows): it scans every table in the namespace and builds an in-memory HashMap<String, FileEntry>, with no lazy or trie path. At a million objects per repository across millions of repositories that does not close.
4. extract_file does not verify checksums, explicitly (znippy-iceberg/src/lib.rs:341-343). §4's property 2 ("per-chunk blake3 verification on every read") is lost on the Iceberg path. That is a silent downgrade of an advertised integrity guarantee.
5. Write granularity is one Iceberg commit per sub-index, at seal time, with all sub-index IPC bytes held in RAM until finish(). That is a batch-publish shape, not a serving shape.

Verdict: keep iceberg off by default and do not treat it as a scaling story. Its honest role is a cold-archive and analytics tier: publish sealed, immutable per-repository archives into Iceberg for SQL over the whole forge, time travel, and cheap S3 durability, while the serving path never reads through it. Marketing it as the answer to horizontal read scaling would be wrong on the evidence: the catalog is a single process holding an exclusive file lock, which is the opposite of horizontally scalable. If gunnar ever genuinely needs multi-process, the answer is skade-katalog serve or a different catalog, and that is a project, not a feature flag.

• Fix: Amend §4's iceberg row from a backend to an export tier, and state the single-process constraint and the lost checksum verification explicitly next to it. Nobody should discover the exclusive lock by starting a second replica.

• Verify: Start two gunnar processes against one Iceberg-backed store and observe the second fail to open the catalog. That failure is the evidence, and it should exist as a test so the constraint cannot quietly change.

• Closed:

correctness

P-014 — znippy has no writer lock of any kind; two concurrent appends silently corrupt the archive

• Found: 2026-08-03

• Status: TODO

• Severity: correctness

• Where: znippy-common/src/meta_sink_append.rs:526, 551, 583 · .nornir/TOTAL-PROMPT.md:462-503 (§4, znippy cold tier) · §4.5 (the __gunnar_refs__ log)

• What: READ, by exhaustive grep across the znippy workspace for flock, LOCK_EX, try_lock, O_EXCL: zero hits. create_archive does a bare File::create (meta_sink_append.rs:583); append_files does a bare open_existing (:551).

INFERRED, high confidence, from the append algorithm: append_files computes blob_end as the lowest index_offset over the sealed archive, pwrites new blobs past it, and rewrites the entire metadata tail. Two concurrent appends both compute the same blob_end, both pwrite to overlapping offsets, and both rewrite the tail. The result is mutual corruption with no error from either writer.

gunnar's design makes this reachable. §4.3 takes one archive per repository, and a repo can be pushed to concurrently by several clients; repack() (S-006) is a whole-archive rewrite; and S-010 notes the open-repository handle cache is unbounded, so nothing currently guarantees one writer per archive. znippy will not tell gunnar it got this wrong: there is no error path, only a corrupt archive discovered later.

Two related hazards, same source: a replacing append orphans the old blob bytes with no GC (AppendReport::rows_replaced), and finish() re-sorts and rebuilds the whole metadata tail, i.e. O(total rows), not O(new), which §4.2 already knows and is the stated reason the object archive is never appended per push.

• Why it matters at load: Silent data corruption in the cold tier of a git host is about the worst failure available. It is also the kind that shows up under concurrency and never in a test.

• Fix: Enforce one writer per archive in gunnar, since znippy will not. A per-repository write lock held across the whole repack()/append, taken from the same place that owns the repository handle (S-010's cache, once it is bounded). Prefer an advisory flock on the archive path over an in-process mutex, so a second gunnar process or a stray CLI invocation cannot bypass it. And note S-019's lesson that an O_EXCL lock which does not self-clear turns a crash into a permanent outage, so use flock (released by the kernel on process death), not a lock file. Additionally: file the missing writer lock upstream against znippy, and the missing GC for orphaned blobs.

• Verify: A test that runs two concurrent append_files against one archive and asserts the archive is still readable and contains both sets of entries. Seen RED first: today it must fail or corrupt, and if it passes the test is not actually concurrent. Then the same test with gunnar's lock in place, asserting serialisation. Assert on archive contents, not on the absence of an error: neither writer errors today, which is the entire problem.

• Closed:

P-015 — No bench harness exists; the proposed nornir-bench suite and its red conditions

• Found: 2026-08-03

• Status: DONE

• Severity: correctness

• Where: S-044 (nothing has ever been load-tested or profiled) · S-033 (the CI workflow is not valid YAML and runs nothing) · no benches/ and no xtask/examples/nornir-bench.rs anywhere in the tree

• What: S-044 establishes the absence. This entry is the concrete suite, in the house standard: nornir-bench, not criterion. Nothing else in this file can be honestly claimed without it; that is why it is #2 in the build order and not last.

Layout. gunnar already has an xtask crate, so the harness goes at xtask/examples/nornir-bench.rs (the runner auto-detects this location and adds -p xtask). The file is one line:

fn main() -> anyhow::Result<()> { nornir::bench::api::run_main_json() }

Arms register via nornir::register_bench!(TheStruct { … }); the macro takes one expression and nothing else; id, metrics and workload all come from the Bencher trait (id() + run() -> Result<BenchResult>). Metrics are a free-form JSON map, and the suffix is the type system: _mbs/_per_sec/ops_sec are higher-is-better, _ms/_us/p50/p99 are lower-is-better, _x/_pct are neutral. Name ids gunnar.<scenario>.

⚠ Correction to a working assumption. The brief said results are "historized by git SHA". They are not. READ: bench_runs has columns run_id, repo, ts_micros, date, version, machine, cores. No git_sha, no release_id (nornir/src/warehouse/iceberg_schema.rs:127-140), and build_bench_runs_batch writes exactly those seven. SHA attribution is indirect, via release_lineage + regression_trace. Bench numbers themselves are keyed by (repo, machine, version, timestamp). Do not design gunnar's suite assuming per-SHA keying it will not get. (znippy/.nornir/benching.md:18-21 and nornir/.nornir/bencher.md:23-25 both claim otherwise; the code is the authority.)

The suite.

The gate. House default is a >10% regression = red, configurable per repo as [repo.gunnar.gates] max_regression_pct (some repos use 5.0). Baseline is the newest prior run on the same machine, not best-ever and not a median; first run on a machine bootstraps green. A regressed run is not persisted, so it can never become a future baseline. Direction is inferred from the metric-name suffix. Add core_saturation (peak core-busy ≥ 0.9 × cores) to repack_1m and clone_concurrent_64: a parallel bench that peaks below that is a bug, not Amdahl. Do not wire that rule against telem_cores_busy_max — measured 2026-08-03, it reads the whole host and returns 32/32 for a single-threaded arm, so the gate can never go red. The input has to be the process's own task-clock ÷ wall. See the core_saturation bullet under P-015.

The trap to avoid on day one. inventory collects arms at link time, so a bench crate that dev-deps nornir inherits nornir's entire arm set. This really happened (2026-07-22: nornir bench run skade ran 213 nornir.mimir.* arms and reached zero skade.* arms). Set NORNIR_BENCH_ONLY=gunnar. and make a zero-arm run a hard failure. An empty run is RED, never a silently green zero-arm success, exactly the disease S-026 and the fleet's silent-green-guard-disease finding are about.

• Why it matters at load: Every number in this file is a probe result or arithmetic. None of it is defensible as a regression gate until it runs under this harness on a named machine.

• Fix: Build the harness first (gunnar.capabilities, pack_deflate, pack_trailer_hash; cheap, no fixtures), then the fixture-backed arms. Use NORNIR_BENCH_PREVIEW=1 for a shrunk workload during development; it stamps the machine as <host>-preview so it can never overwrite a published row.

• Verify: The harness itself needs a red. (Historical note: P-007 and P-008 landed on 2026-08-03 without this harness, against a disposable probe plus permanent byte-pinning tests; see the appendix. That was the right trade for two small changes and is not a precedent for the larger items; nothing below P-015 in the build order should be claimed without it.) Land pack_deflate before P-007, so the backend swap is observed as a real improvement against a real recorded baseline rather than asserted. If the first thing the suite does is confirm a change that already landed, it has proved nothing.

• Isolation: container, KVM, or neither. Per workload, and the answer is mostly "neither":

  • Neither for pack_deflate, pack_trailer_hash, walk_1m, lookup_*, repack_1m, negotiate_divergent. These are pure CPU/memory in one process. The house harness runs arms sequentially on the main thread and its entire isolation strategy is same-machine comparison: machine is mandatory and the gate filters history to the same machine. There is no documented requirement for a container or VM anywhere in bencher.md. Adding one would add noise, not remove it.
  • Container for push_large and clone_large only if they need a stock git binary at a pinned version. S-026 shows the interop suites currently self-skip and report green, and a container is the cheapest way to make the toolchain's presence non-optional. dwarves' brokkr container machinery is the existing path. Otherwise neither.
  • Container for P-006's build check, which is precisely a "no network, no cmake" assertion and cannot be made on a developer box.
  • KVM (tunnr) for none of them. tunnr's tunnr-vm / QEMU+OVMF harness exists to boot an appliance off a disk and prove the boot chain; it is the right tool for "does the gunnar appliance come up", and the wrong tool for a CPU benchmark, where the virtualisation layer adds variance and buys no isolation the process does not already have. The one exception worth reserving: if a bench ever needs a known core count and a quiesced scheduler to make core_saturation meaningful, a pinned-vCPU KVM is the honest way to get it. That is not needed for any arm above.

• Closed: 2026-08-03. Built and run. bench/ (detached crate, own [workspace], publish = false), one source: bench/examples/nornir-bench.rs. 11 of the 12 arms above exist; gunnar.clone_cold_tier does not, and why is below.

Two deviations from the plan above, both deliberate:

1. Not xtask/examples/. xtask/Cargo.toml declares "ZERO dependencies, deliberately". It is the §2 dependency guard, and a workspace member, so a nornir dev-dep there would take part in resolver-2 feature unification and be built by cargo test --workspace. bench/examples/nornir-bench.rs is the third accepted location in nornir/.nornir/bencher.md and exists for this case. Verified: cargo metadata at the repo root still resolves 12 packages and does not list gunnar-bench; the root Cargo.toml was not touched.
2. Dev-deps nornir-bench, not nornir. nornir re-exports nornir_bench verbatim (nornir/src/lib.rs:159), so the API is identical. But linking nornir links every register_bench! in nornir/src/bench_registry.rs, which is the day-one trap this entry warns about. Linking only the leaf means the linked arm set is gunnar's arm set. NORNIR_BENCH_ONLY=gunnar. is still set from main(), because a future dependency could reintroduce the hazard silently.

The git_sha correction is CONFIRMED, not merely repeated. Read directly: nornir/src/warehouse/iceberg_schema.rs:127-141. bench_runs() builds a schema of exactly seven fields, ids 1–7: run_id, repo, ts_micros, date, version, machine, cores. No git_sha, no release_id. build_bench_runs_batch (nornir/src/warehouse/iceberg.rs:2440-2458) writes exactly those seven columns in that order and nothing else. znippy/.nornir/benching.md:18-21 and nornir/.nornir/bencher.md:23-25 are wrong; the code is the authority. gunnar's suite is designed for (repo, machine, version, timestamp) keying.

First full-size run — machine oden, 2026-08-03, 32 cores, sha_ni, sha256 repositories, NORNIR_BENCH_PREVIEW unset. All 11 arms passed.

Read these numbers with the host load in hand. oden was running four other build agents; telem_cores_busy_max is emitted per arm and several arms saw 32/32 from work that was not theirs. An earlier full run of the same suite, at load average 57, gave hash_mbs 1616 (vs 1890) and pack_mbs 28.14 on a differently-generated corpus. Treat absolute values as ±30% until a quiesced run exists. The ratios inside one arm (miss_degradation_x, parallel_speedup_x) are bracketed or back-to-back and are far more trustworthy than the absolutes.

What the first run already says, beyond "it runs":

• P-012 is real and large, and when it was actually diagnosed, only a third of it was this. CLOSED 2026-08-03, see P-012: misses 128× faster over 64 packs, hits 21×, miss_degradation_x 23.2 → 1.01. The bullet below stands as what the first run said; the pack count turned out not to be the dominant cause. Spreading one million objects over 64 packs instead of one costs 23.7× on the miss path (82 798 → 3 355 lookups/s) and 4.1× on the hit path. Measured back to back over the same probe set with a drift bracket of 0.959, so it is not host noise.

• P-002's parallel index path DOES engage at this size: thread_limit(1) gives 49.6 MB/s against 121.2 MB/s unrestricted, a 2.45× speedup. It did not engage at preview size (parallel_speedup_x 0.999 on a 332 KB pack); the fan-out has a floor. Note this is a runtime probe, not the two-cargo-feature comparison P-002 asks for; see the gap list.

• negotiate_divergent costs 154 seconds and 18.9 million commit visits for a 20 k/20 k divergence, over 1250 rounds. This is the arm that most deserves a look before anything else in this file is optimised.

Looked at, 2026-08-03, S-047 in stinky.md: the headline was misleading. Neither the 1250 rounds nor the 18.9 M visits was the cost. 1250 is 40 000 ÷ 32, the bench's own fixed-window client; real git doubles its window and abandons negotiation after MAX_IN_VAIN, so it would take ~10 rounds here. And the single most expensive thing in the arm never touched the object store at all: a linear Vec::contains de-duplicating common, which was 81% of the wall clock while changing commits_visited by exactly zero. The real shape is that a v2 client re-sends its whole have list every round, so a round answered from scratch is O(rounds²). MEASURED in a worktree pinned at 87524d6, same corpus, same box: 197 248 → 17 905 ms and 18 901 251 → 6 401 251 commit reads, repeated at 173 757 → 16 140 ms. The remaining O(rounds²) term is the reachability walk being rebuilt per round, which S-047 names rather than claims to have fixed.

• P-001 has not landed: objects_recompressed is 2048 of 2048 blobs. Every object in a clone is inflated and re-deflated. That is the assertion, and it fails the arm if it ever stops being true, which is how P-001 landing will be noticed.

⚠ It did not work, and the arm went on reporting 2048 after P-001 landed. See the fifth hollow fixture below. What noticed P-001 was vs_git_clone, whose guard went red for the right reason; this arm stayed green on the wrong one.

Reds actually watched failing (LAW 2: no green accepted that has not been seen red). All four are reproducible from the environment, not from a code edit, except where noted:

1. Zero-gunnar.*-arm run. NORNIR_BENCH_ONLY=gunnar.nonexistent. → exit 1, RAGNARÖK: this run would execute 0 arm(s) and NOT ONE of them is a 'gunnar.' arm. run_main_json's own empty-run bail is not sufficient on its own; it only fires when something was linked and filtered away, so main() asserts the post-filter set contains a gunnar. arm before handing over.
2. clone_large's objects_recompressed (expectation deliberately shifted by one): arm reports passed: false and emits no result row.
3. negotiate_divergent's commits_visited > 0, via the permanent red switch GUNNAR_BENCH_NEG_NO_SHARED=1: "commits_visited = 0 after 16 rounds — no have was ever common, so the server never walked for a cut point".
4. lookup_1m_64pack's degradation, via GUNNAR_BENCH_PACKS=1: "misses against 1 packs run at 71589/s versus 61592/s against one pack … so no real misses are being produced (P-012)".

Two hollow guards were found in this suite's own first version, by trying to break them:

• The cross-arm degradation comparison was worthless. Asserting lookup_1m against lookup_1m_64pack and then pointing the second arm at a one-pack fixture should have failed instantly. It did not: it reported miss_degradation_x = 30.9 comparing a fixture against itself, because the two arms ran minutes apart on a contended host and single-pack miss rates measured at different moments differed by 3× on their own. The comparison now happens inside one arm as an A/B/A bracket (one pack → M packs → one pack, same probe set), and single_pack_drift_x is emitted so a reader can see how noisy the host was. Only after that fix did GUNNAR_BENCH_PACKS=1 produce a red.
• negotiate_divergent measured nothing and passed. Its first version offered only the client's divergent commits as haves, never the shared base, so common was always empty, negotiate() returned before building a Reachability, and the arm reported commits_visited = 0 with a beautiful sub-millisecond negotiate_ms. It passed, because rounds > 1 was the only assertion and sixteen empty rounds satisfy it. rounds > 1 is necessary and not sufficient; commits_visited > 0 is now asserted too.

⚠ A FOURTH hollow fixture, found on 2026-08-03 while closing P-012, in the same generator and by the same mechanism. packed_fixture asked build_blobs for a million blobs and got 16 712 distinct ones: Lcg::text assembles an object out of whole lines from a 128-line pool, and a 96-byte object is two or three of them, so the corpus could only ever hold a few thousand. oids.bin held 1 000 000 entries and 16 712 unique ids. gunnar.lookup_1m and gunnar.lookup_1m_64pack were resolving against a sixteen-thousand-object index small enough to sit in L2, under a name that says a million, and gunnar.repack_1m shares the fixture. Every absolute in those three rows above is against that corpus and is not comparable with anything measured after the fix. The ratios — miss_degradation_x 23.7 among them — held up: the corrected corpus gives 23.2.

It was caught by an applied output, not by inspection: the new multi-pack index reports how many objects the file that landed actually holds, and the arm asserted that against the objects the fixture claimed. build_blobs now prefixes each blob with its own index line, packed_fixture asserts every id is distinct, and the fixture cache key carries PACKED_CORPUS_VERSION so a stale corpus is regenerated rather than reused.

⚠ A FIFTH, found 2026-08-03 while closing P-001, and it is the one this suite was specifically built to prevent. gunnar.clone_large, described above as "the headline" and as P-001's applied-output assertion, served a repository with zero packs in it. build_history writes loose objects and nothing packed them, so write_pack_copy had nothing to copy from, fell through to a fresh deflate for every object, and the arm reported objects_recompressed = 2048 (the pre-P-001 number) for the whole morning after P-001 had landed and was working. The assertion could not fail, because on a corpus with no packs a working pack-copy and a dead one produce the same output. That is the identity value, and P-001's own Verify step names it.

Two further holes in the same arm, both found by fixing the first:

• FetchRequest::default() leaves ofs_delta false, and the copy path is gated on it (upload_pack.rs:1075). Every real git client sends the capability, so the head-to-head arms got it for free; an arm that hand-builds the request does not. The first run against the newly packed fixture duly reported 0 copied, 6144 recompressed on a repository with one pack holding all of them.
• gunnar.clone_concurrent_64 had the same unpacked fixture, so P-004's peak_threads guard (the whole reason that arm exists) was watching the writer that cannot fan out, while gix's iter_from_counts, which can and which copy.rs holds to thread_limit: Some(1), was never entered. It is entered now: 89 peak threads for 64 concurrent clones.

Both fixtures are repacked, both cache keys carry a -p generation so a stale loose tree cannot be silently reused, and the arm now asserts the copy split from the generator and blob_reads == 0 from the object store: two witnesses produced by different crates. The red is GUNNAR_BENCH_NO_PACK_COPY=1.

A third correctness bug, in the fixture generator rather than a guard: the synthetic corpus deflated at 13.2×, then (after a fix) at 1.35×, the latter being the entropy bound, i.e. deflate's match finder doing literally nothing. ratio_x is P-007's guard, so a corpus that no match-finder change can move is a guard that cannot fire. Cause: an LCG's low bits have period 2^k, so next_u64() % 128 is a permutation cycle and no line ever repeated inside one object. Fixed (high bits, 128-line pool, measured against a zlib reference) to 2.85×, where real source trees sit.

Not built, and what that leaves unmeasured:

• gunnar.clone_cold_tier, impossible today. There is no znippy cold tier in the tree: znippy appears in gunnar only in two comments (Cargo.toml:123, crates/gunnar-cli/Cargo.toml:41, the latter saying the fs backend is "the only ObjectStore backend that survives a restart today"), and P-005 cites .nornir/TOTAL-PROMPT.md §4, a design section, not code. There is no ObjectStore over a znippy archive to point a clone at. Deliberately not stubbed: an arm that self-skips and reports green is S-026 exactly, and it would let P-005 be closed against a benchmark that never ran. When the tier exists this arm is CloneLarge with the store swapped and the same objects_recompressed assertion.

• The stock-git interop container. Not built BUILT 2026-08-03, see the head-to-head section below. The eleven arms above still measure gunnar against itself; the five gunnar.vs_git_* arms measure it against stock git 2.53.0 on the same fixture, inside the container this entry called for.

• P-002 in two cargo feature configurations. Not reachable from one detached bench workspace: this crate links gunnar-store/fs → gix-odb/parallel → gix-features/parallel, and Cargo unifies features per graph, so the serial configuration cannot be built here at all. The runtime thread_limit(1) probe above is the honest substitute; a real two-config comparison needs two bench workspaces.

• core_saturation. MEASURED 2026-08-03, and the metric is worse than contaminated — it cannot produce a red. The claim above was that telem_cores_busy_max is the right input but is polluted by other tenants. It is not the right input. The sampler counts, once a second, how many /proc/stat per-core lines were >50% busy across the whole host — a number gunnar's process does not appear in. Run the strictly single-threaded gunnar.walk_1m and it reports telem_cores_busy_max 32 of 32 and telem_cpu_pct_avg 99.85–99.95% in five consecutive repetitions, while /usr/bin/time says the process itself occupied 0.40–0.72 of one core (mean 0.56). The proposed gate — peak core-busy ≥ 0.9 × cores, or it is a bug — would therefore pass on an arm with no parallelism in it at all, and pass on every arm, forever, on any box with a second tenant. That is silent-green-guard disease with a rule attached, and it is the reason it must not be wired as written.

The host-independent replacement is the process's own task-clock ÷ wall, which is "mean cores occupied by the thing under test" and is what the gate meant. Measured, seven repetitions of gunnar.clone_concurrent_64, oden at loadavg 37 on 2026-08-03:

So the concurrent arm really occupies about 11.7 of 32 cores, a saturation of 0.37, where the published metric says 0.94. perf stat -e task-clock gives the real number today with no code change; the sampler could give it from /proc/self/stat's utime+stime instead of /proc/stat, which is a one-field change in nornir-bench's telemetry and is the actual fix. The pinned-vCPU KVM this entry reserved would make the host reading meaningful but is not needed for the self reading, and the self reading is the one the gate wants.

• S-018's pushes-per-second / fsync ceiling and S-008's socket-level throughput knee. clone_concurrent_64 runs 64 threads in one process against a shared store; it is not an accept loop and does not find the knee.

Run it: cd bench && NORNIR_MACHINE=oden cargo run --release --example nornir-bench (add NORNIR_BENCH_PREVIEW=1 for the shrunk development loop, which stamps the machine oden-preview so it can never overwrite a published row). Fixtures are generated lazily and cached under $GUNNAR_BENCH_FIXTURES, default /home/rickard/scratch/gunnar-bench-fixtures: md1, never /tmp, which is RAM here.

The head-to-head — gunnar versus stock git, added 2026-08-03

Eleven arms measuring gunnar against gunnar catch regressions and answer nothing about whether the project is competitive. S-044 asked for "clone wall clock and pack size against stock git on the same fixture"; five arms now do that.

gunnar.vs_git_clone · gunnar.vs_git_clone_sha256 · gunnar.vs_git_push · gunnar.vs_git_fetch_divergent · gunnar.vs_git_clone_concurrent

How the comparison is made fair

  git clone "ext::<bench binary> %S <repo>"     <- the CLIENT is stock git on BOTH sides
        │
        └─ spawns: <bench binary> git-upload-pack <repo>          ← THE METER
                      │  pumps the pipe both ways, counting bytes
                      │  reaps the server with wait4 → CPU seconds + peak RSS
                      └─ spawns the SERVER:
                           git    :  git upload-pack <repo>
                           gunnar :  <bench binary>, gunnar-wire in-process

The meter is the same process on both sides, so its cost — one extra process, two pumping threads, one memcpy of the stream — is in both columns and cancels out of every ratio. wait4 is what makes CPU honest: Linux returns RUSAGE_BOTH, so git upload-pack's fan-out into git pack-objects is counted. A server that wins wall clock by burning four cores loses cpu_ratio_x, which is the point of measuring it.

The container, and why both sides are in it. P-015 above calls for a container for exactly these arms and for exactly one reason: to make the stock-git toolchain's presence and version non-optional, because a self-skipping interop suite that reports green is S-026. The image is docker.io/library/ubuntu:26.04 + git=1:2.53.0-1ubuntu1, the host's own distribution and the host's own git package, so the pinned git inside is the same build as /usr/bin/git outside and the glibc inside is the host's, which is what lets the bench binary run in there unchanged. Both sides run inside one podman run: containerising only the stock-git arm would have added a namespace-and-cgroup asymmetry far larger than the pinning it bought. The container is entered once per arm, outside the timed region. bench/Containerfile; build with podman build -t gunnar-bench-git:2.53.0 bench/. No KVM, per P-015.

Measured — machine oden, 2026-08-03, 32 cores, git version 2.53.0

⚠ This table is the run at e60c95d, i.e. BEFORE P-001. It is kept because it is the argument that made the case for pack-copy, and because a before is worth nothing once it has been overwritten. The after is the next section, measured the same afternoon on the same fixtures.

Fixture: 2048 commits over 1024 files in 32 directories, 16 KiB blobs, ~2% of the lines of eight files rewritten per commit. Churn, not fresh random blobs. The eleven arms above generate a new random blob per commit, which gives delta compression nothing to find and would have flattered gunnar enormously here, since delta reuse is precisely what P-001 costs. 528 MiB of blob content, 36 172 objects in a full clone, git repack -adf at default window and depth. Three interleaved rounds per side, median reported, one warm-up per side discarded.

All five arms are from one run (NORNIR_BENCH_ONLY=gunnar.vs_git), so the columns are comparable with each other rather than stitched together from separate sittings.

The ref is part of the measurement. These are gunnar at e60c95d. An earlier run of the identical arms at 87524d6 put vs_git_fetch_divergent at cpu_ratio_x 6.647 and wall_ratio_x 2.151; e60c95d's negotiation fix (197 s → 17.9 s on negotiate_divergent) moved them to 5.560 and 1.430. The clone rows did not move at all (pack_ratio_x is 18.336 in both runs to three decimals), because negotiation is not what a clone spends its time on. Quoting a vs_git_* number without its ref is how a stale ratio outlives the change that fixed it.

What it says

1. gunnar loses the serve path, badly, and P-001 is the whole reason. 528 MiB of content leaves gunnar as 108.70 MB (a 4.9× plain deflate) and leaves git as 5.93 MB (89×, because git ships deltas it has already computed and stored). 18.34× is the delta-reuse gap made concrete. This is the single strongest argument in this file for P-001, and it is now a measurement rather than an inference: P-001's §"31× on the compression+hash budget" was derived arithmetic; this is a clone.
2. The hash algorithm is not the cause. sha256 moves the byte ratio from 18.336× to 17.723× and the CPU ratio from 219.963× to 176.413×. Anyone reading P-008 should note that switching hash cannot rescue this row.
3. 219.96× on CPU is worse than 10.8× on wall clock, and that gap is the finding wall clock alone would have hidden. git served the clone in 97.5 ms of server wall for 22 ms of CPU: I/O-bound on a pack it copies, using under a quarter of a core. gunnar spent 4.875 s of CPU in 4.926 s of server wall: one core, saturated, doing deflate work P-001 argues should not happen at all.
4. gunnar wins the receive path outright: 22% less wall clock, 26% less CPU and 38% of the memory of git receive-pack. The wire bytes are equal by construction (the client builds the pack and the client is stock git in both columns), so this row is purely server-side cost, and gunnar's index_thin_pack + connectivity walk beats git's index-pack --fix-thin + check-connected.
5. gunnar wins total memory under concurrency by 7.1× (19.23 MB against 135.95 MB), because it is one process where git is sixteen. rss_ratio_x on that row reads 2.182 against gunnar and is the misleading number: it compares one gunnar process with one of git's sixteen, which is why rss_total_ratio_x is emitted beside it.
6. FsStore::open_or_init takes a cross-process exclusive lock on the repository (gunnar-store/gunnar-store.lock, the Arrow log's writer lock). Run process-per-connection like git upload-pack, fifteen of sixteen gunnar servers die with "the store is locked by another writer". This is not a bug and not a bench limitation: it is why gunnar serve is one process with a thread per connection, and it means gunnar has no process-per-connection deployment shape at all. The concurrent arm therefore runs each side in its own native shape and says so on the row.
7. gunnar negotiates slightly better than git, which was not expected: 2912 objects against 2979 for the same 500/500 divergence over a 2000-commit base, with the same 1008 have lines offered to both.

AFTER P-001 — the same five arms, oden, 2026-08-03

One run, NORNIR_BENCH_ONLY=gunnar.vs_git, at 4d69510, same container, same pinned git 2.53.0, same fixtures. oden was at load average 186 on 32 cores from other agents. So the ratios, which are interleaved gunnar/git inside each arm, are what may be quoted, and the absolute milliseconds may not.

What changed, and what did not.

1. The 18.34× is gone, exactly. gunnar's clone is now byte-for-byte the same size as git's, because it is shipping the entries git repack wrote. Points 1 and 3 of What it says above are answered; point 2 (the hash is not the cause) is confirmed rather than overturned: sha256 tracks sha1 to three decimals on the byte ratio in both runs.
2. CPU is still 16× git's, and that is now the whole story of this arm. 0.378 s against 0.023 s for identical output. git does the same job in a twentieth of the CPU because git upload-pack reuses its own reachability bitmaps and streams a contiguous region of one pack; gunnar walks the graph object by object (P-010) and reassembles entries one at a time through gix's generator (P-018). Those two entries are now the whole of the remaining serve-path gap, which is what the CPU-budget table predicted would happen.
3. vs_git_push did not move, as expected: the receive path never touched write_pack_copy. gunnar still wins it on CPU and memory. The wall-clock column flipped from 0.779× to 1.089× and that is host load, not a regression; the CPU column, which is the load-insensitive one, is 0.737× against 0.736×.
4. The concurrent arm's memory advantage is unchanged at ~5.9×, and its CPU ratio fell from 26.481× to 1.695×.

Every asymmetry that could not be removed

Stated here rather than left to be discovered, because a number with an unstated caveat is worse than no number:

• gunnar's SSH layer is excluded. gunnar serve is an SSH listener with no stdin/stdout upload-pack mode, so the gunnar column is gunnar-wire's UploadPack/ReceivePack — the same code the SSH server calls — over a pipe. The brief allowed equalising the transports or stating the difference; this equalises them, and the consequence is that these numbers are optimistic for gunnar: no key exchange, no publickey auth, no channel framing, no sshsig control-plane work. Nothing here may be quoted as "gunnar over SSH".
• sha1 is the primary fixture and gunnar's default is sha256 (D11). sha1 because that is what every repository anyone would migrate actually is, and because a sha256 fixture would put stock git in the unusual configuration instead. vs_git_clone_sha256 measures the difference rather than asserting it is small.
• The gunnar server process is the bench binary, which carries code git-upload-pack does not. Both clone arms therefore also serve a one-commit repository and report *_rss_floor_mb; the workload's own memory is the difference.
• One extra process and one memcpy, added to both sides by the meter. Symmetric, and it is what buys exact byte counts and an atomic rusage.
• Two different measurement mechanisms on the concurrent arm: wait4 summed over 16 git children, /proc/<pid> for gunnar's one process. A shared process cannot attribute its usage to one of sixteen connections, and inventing a per-connection share would be a number with nothing behind it. Totals are comparable; per-connection RSS on that row is not.
• oden was at 32/32 cores busy from other agents throughout. The interleaved gunnar, git, gunnar, git… ordering and the median are the mitigation, and gunnar_drift_x (first gunnar round over last) is emitted so the reader can price it: 0.996 on the clone arm, i.e. the host did not move under that comparison.
• The fixture is synthetic. Churned text with a fixed line pool, not a real repository. It is deterministic and it deltas the way source deltas; it is not a statement about any particular project.
• git clone --bare, so no working tree is written on either side.
• protocol.ext.allow=always and GIT_PROTOCOL=version=2 are set identically on both sides; git 2.53.0 refuses ext:: otherwise, and without the second both sides fall back to v0.

Reds actually watched failing

Three permanent switches, all reproducible from the environment, plus four genuine failures watched while building the arms (three below, and the cross-process store lock in point 6 above, which killed fifteen of sixteen gunnar servers the first time the concurrent arm ran).

1. Toolchain absent. GUNNAR_BENCH_H2H_IMAGE=localhost/gunnar-bench-git:no-such-tag → every head-to-head arm fails with "the pinned head-to-head image … does not exist … (a head-to-head arm that skips because its toolchain is missing is S-026)". The arms also assert git --version equals git version 2.53.0 and fail otherwise: a pack-size ratio is a statement about a specific git.
2. GUNNAR_BENCH_H2H_SAME_SIDE=1: serve gunnar in both columns while still labelling one git. All five arms fail: "both sides put exactly 120108 bytes on the wire, which means one server was measured twice". This is the exact hollow guard lookup_1m_64pack had in its first version.
3. GUNNAR_BENCH_H2H_SHALLOW_GIT=1, asking the git column for --depth 1. Both sides return valid, fsck-clean repositories with plausible byte counts; the arms fail with "gunnar transferred 190 objects and git transferred 22. The two sides did not answer the same request, so no ratio below is meaningful." Without that assertion, a gunnar that quietly sent a smaller object set would report a flattering pack ratio.

A hollow guard was found in this suite's own first version, by trying to break it. Switches 2 and 3 were set on the host and never reached the driver inside the container: h2h_run forwarded only the variables it knew about, so both red runs reported a cheerful green. A switch whose entire job is to make a guard fire, silently doing nothing. The whole GUNNAR_BENCH_ family is now forwarded across the boundary. Found by running the reds, not by reading the code (LAW 2).

Three more reds that were real bugs rather than switches, all seen failing:

• podman run -w <work> put the driver inside the directory it deletes and recreates. Every path resolution in every child then returned ENOENT, including FsStore::open_or_init on an absolute path that plainly existed, in the container and not on the host. The driver now leaves for / before clearing the tree.
• writeln!(f, "{line}") is not one write. write_fmt issues a separate write per format piece, so sixteen concurrent meters interleaved mid-line and the stats file stopped being JSON ("expected ':' at line 1 column 46"). A single write_all of an already-formatted line is atomic under O_APPEND, and that is what the concurrent arm leans on.
• The equal-objects assertion fired for real on the negotiated fetch (2912 vs 2979) and, at preview size, on the push (16 vs 15). Both are legitimate — where two negotiators stop trading haves is what negotiation decides, and thin-pack base completion differs between index_thin_pack and index-pack --fix-thin — so those two paths assert a 10% spread and the applied output instead: the resulting tip, and a git fsck --connectivity-only run by the receiver's own stock git.

Still not measured

• S-018's pushes-per-second / fsync ceiling. No arm.
• S-008's socket-level knee. vs_git_clone_concurrent uses 16 real client processes against real server processes, but over ext:: pipes, not gunnar's SSH accept loop, which is where the knee actually is.
• gunnar serve over SSH, end to end. Deliberately out of this comparison (see the asymmetries), and therefore the one thing that would make these numbers worse for gunnar rather than better.

smell

P-016 — znippy's incompressible list is extension-only, and git objects have no extension

• Found: 2026-08-03

• Status: TODO

• Severity: smell

• Where: znippy-common/src/index.rs:1212-1235 (is_probably_compressed / should_skip_compression) · znippy-compress/src/slot_packer.rs:244-252, 549 · znippy-common/src/meta_sink_append.rs:645-651

• What: READ, the detection is 100% extension-based: a compile-time matches! over 38 extensions (zip, gz, jpg, parquet, age, iso, pdf, …) with no magic-byte sniffing and no entropy estimation, and the critical line:

} else {
    false          // no extension => compress it
}

Git objects have no extension. §4 keys them by oid hex as relative_path, which matches nothing in the table, so every git object is fed to OpenZL at level 19.

Correctness is safe. There is a content-based fallback, but it is "compress first, then discard": slot_packer.rs:244-252 compresses into scratch and, if n >= len, stores raw. So a deflated pack entry or a zlib loose object is never inflated or damaged. The cost is the problem, not the correctness. MEASURED in znippy's own suite: archive_random_500mb → 103.62 MB/s compress, compressed_chunks = 0, ratio_x = 1 (znippy/.nornir/README-full.md:107, oden, 2026-07-22): full level-19 CPU spent to discover the data was incompressible.

Worse for gunnar specifically: the append path has no extension check at all (meta_sink_append.rs:645-651 always runs the codec first), so append_files / create_archive pay the futile cost on 100% of already-compressed input regardless of name. And there is no per-entry "store raw" hint: ArchiveEntry is { relative_path, data, pkg_type, repo }, with no compressed field.

So, on whether the list needs to become dynamic or content-sniffed for gunnar. Under P-005 the answer is yes, but the sniff is trivial and gunnar already knows the answer. A packfile is deflated; a loose object is zlib-wrapped; the pack-entry bytes P-005 proposes storing are deflated by construction. gunnar does not need to detect anything; it needs a way to tell znippy what it already knows. A general content sniffer (entropy estimate over a prefix, or a zlib-header magic check for 0x78 0x01/0x9c/0xda) is the wrong shape here: it re-derives, heuristically and fallibly, a fact the caller holds with certainty.

• Fix, concretely, in preference order:

  1. Add a per-entry hint upstream: a store_raw: bool on ArchiveEntry and a matching parameter on append_files, honoured on both the packer and append paths. ~20 lines in znippy plus a field. This is the correct fix and it also fixes the append path's unconditional codec call for every other znippy user.
  2. Failing that, add a content sniff at the codec boundary: check for a zlib header and a high-entropy prefix before invoking OpenZL. Cheaper than the current compress-then-discard, still heuristic.
  3. Do not name entries <oid>.gz to trip the extension table. It works, it is a lie to the plugin matches_path dispatch, and it will be copied by whoever reads it next.

• Why it matters at load: At 103 MB/s of wasted compression, a repack of a large repository spends most of its time proving that already-compressed data is already-compressed. It also, per P-011, sits on the one job whose latency bounds cold tier freshness.

• Verify: A bench arm on repack_1m reporting compress_mbs and compressed_chunks. With the hint wired, compressed_chunks must be ~0 and compress_mbs must rise sharply. Seen RED today, where compressed_chunks is already ~0 but compress_mbs is floored. That pair is the whole point: the current state looks correct on one metric and is wrong on the other, so a guard on compressed_chunks alone would be blind.

• Closed:

P-017 — LFS: chunking, and whether gatling's balanced map fits it

• Found: 2026-08-03

• Status: TODO

• Severity: smell

• Where: .nornir/TOTAL-PROMPT.md:1250-1262 (§12) · S-036 (spec'd and unstarted)

• What: S-036 records LFS as unstarted. This is the performance shape it should have when written, so the chunking decision is not made by accident.

Chunk size: take znippy's, do not invent one. znippy already splits files above file_split_block_size = 10 MiB into rows sharing a relative_path, ordered by chunk_seq and placed at fdata_offset (§4 records this). An LFS object is exactly the shape that was built for: large, immutable, content-addressed. Reuse it (LAW 5): an LFS object becomes a znippy entry keyed by its oid, chunked by znippy, with per-chunk blake3 verification for free. No second chunking layer.

Do not reach for content-defined chunking (rolling hash / FastCDC). It exists to deduplicate near-identical large files across versions. LFS objects are already content-addressed and deduplicated by oid at the protocol level; CDC would add real complexity to deduplicate something the protocol has already deduplicated.

gatling_map_balanced fits, cleanly, but for the batch tier, not per request (P-004):

• Units = chunks of one object. Known count, known sizes, so size_of is |c| c.len() as u64 and is O(1), which matters, because it runs serially on the caller thread.
• At 10 MiB per unit, units are expensive and the shared AtomicUsize cursor is not a contention risk; batch = 1 is right here (unlike cheap units, where the measured 2× regression at batch = 1 applies; znippy-zoomies/src/stree.rs:219-227).
• Output order is preserved, which chunk reassembly needs.
• A 1 GiB LFS object is 103 chunks, enough to fill 32 cores at ~3 units each, which is close to the ~4-units-per-core rule of thumb.
• But a 12 MiB object is 2 chunks and will not fill anything; gate the fan-out on object size, exactly as gatling_scanlines gates on min_rows_for_threads.
• Per P-004, an LFS upload is a request, so this fan-out needs the same admission gate as P-010: 50 concurrent 1 GiB uploads each spawning 64 threads is the same trap.

Transport, unchanged from §12: git-lfs 3.0's pure-SSH git-lfs-transfer protocol, same session, same auth, no HTTP server. Verify against the spec before committing; §5's exec dispatch must take a third command as one match arm.

• Verify: gunnar.lfs_chunk_1gb reporting chunk_mbs and chunks, serial vs parallel, plus a guard that reassembly is byte-identical to the input, seen RED by deliberately swapping two chunks, which an order-preserving map makes easy to get wrong in exactly one place.

• Closed:

P-018 — The gix output pipeline's serial tail — sort, reorder, hash — is one thread by construction

• Found: 2026-08-03

• Status: TODO

• Severity: smell

• Where: gix-pack-0.73.0/src/data/output/entry/iter_from_counts.rs:73, 101-137 · gix-pack-0.73.0/src/data/output/bytes.rs:20-40, 74, 111-117 · gix-features-0.49.0/src/parallel/in_order.rs:7-13, 50 · gix-pack-0.73.0/src/data/output/count/objects/reduce.rs:33

• What: Recorded now because P-001 routes gunnar onto this pipeline, and its ceiling should be known before it is adopted rather than discovered after. This is an upstream property, not a gunnar defect, but it bounds what P-001 can deliver.

Serial stages, all READ:

1. Counting's reduce. reduce::Statistics::feed runs on the calling thread and does self.counts.extend(counts) (reduce.rs:33). At 1 M objects that is a single-threaded Vec grow and memcpy of 1 M Count structs.
2. The sort. counts.sort_by on (pack_id, pack_offset) (iter_from_counts.rs:108-117) is fully serial: ~20 M comparisons with a branchy comparator over Option<Location> at 1 M counts, followed by a serial partition_point scan (:119-130).
3. enough_counts_present = counts.len() > 4_000 (:73): below 4001 counts the pack-location resolve is fully serial, silently. Most fetches are below that.
4. The reorder. Workers return chunks out of order tagged with a SequenceId; InOrderIter buffers them in a BTreeMap (in_order.rs:10, 50) until they can be emitted in sequence. One slow chunk head-of-line-blocks the writer and inflates memory by every completed-but-not-emittable chunk.
5. The hash and the back-references. FromEntriesIter writes through gix_hash::io::Write (bytes.rs:24), which hashes every byte on the consuming thread (gix-hash/src/io.rs:99-103), and maintains pack_offsets_and_validity for delta back-references (bytes.rs:74, 111-117). The back-references are why the tail must be serial and in order. Not an oversight.

So the tail — reorder, back-reference, hash, write — is one thread, by construction, and it is the ceiling on packs/second regardless of how many cores the counting and encoding stages get. P-008's hash fix acts directly on this thread, which is the second reason it matters more than its 3%-of-today figure suggests.

• Fix: Nothing to fix in gunnar. Record the ceiling, and when P-001 lands, measure where the pipeline actually saturates (gunnar.clone_large's pack_mbs against core_saturation). If the tail proves to be the binding constraint, the options are upstream: a tree hash is not possible (the trailer is defined as a linear SHA-1 over the stream), so the realistic lever is P-008 plus reducing bytes through the tail, i.e. P-009. Note also P-002: none of the parallel stages above run at all in the appliance build today.

• Verify: gunnar.clone_large with core_saturation attached. If pack serving peaks well below 0.9 × cores after P-001 and P-002, this tail is why, and that measurement, not this entry, is the evidence.

• Partly answered, 2026-08-03, now that P-001 has landed on this pipeline. Two things are measured and one is not.

Measured: the tail is nowhere near the constraint on a serving request, and the reason is that gunnar deliberately does not use the pipeline's parallel half at all. crates/gunnar-pack/src/copy.rs passes thread_limit: Some(1) (P-020, P-004), so the whole generator is one thread and there is no reorder buffer to head-of-line-block. On that shape gunnar.clone_large serves 6144 copied entries at 200.11 MB/s against 20.19 MB/s for the same objects re-deflated. The tail moved from irrelevant to dominant relative to the rest of the request, and is still an order of magnitude faster than the writer it replaced. Stage 3 of the list above (counts.len() > 4_000) therefore has no effect on gunnar's serve path either way, which removes the "silently serial below 4001 counts" concern from that path entirely. It is serial above it too, on purpose.

Measured: this tail is what the remaining head-to-head gap is made of. After P-001, gunnar emits byte-identical output to git upload-pack for 16.2× the server CPU (0.378 s against 0.023 s, vs_git_clone, 2026-08-03). git reads a contiguous region of one pack guided by reachability bitmaps; gunnar reassembles entry by entry through the stages above and hashes every byte on the consuming thread. So the entry's claim that this bounds P-001 is right, and the bound is not on P-001's win; it is what is left after it.

core_saturation: MEASURED 2026-08-03, and the gate input is wrong. telem_cores_busy_max reads the whole host and returns 32 of 32 for the strictly single-threaded walk_1m arm, whose process actually occupied 0.56 of one core. It cannot go red, so it cannot answer this entry's question either. The number that can is the process's own task-clock ÷ wall; for this entry's subject — whether pack serving peaks at one core — that reading is available today from perf stat with no code change. See the core_saturation bullet under P-015 for the measurement and for the one-field fix in nornir-bench.

• Closed:

P-025 — The serve path inflated every object in the repository to find out how long its entry was

• Found: 2026-08-04

• Status: FIXED — landed, byte-identical output asserted on 19 real repositories

• Severity: blocker (it was 51.3% of a clone's CPU profile)

• Where: gix-odb-0.83.0/src/store_impls/dynamic/find.rs:361-441 (location_by_oid) · gix-pack-0.73.0/src/data/output/entry/iter_from_counts.rs:75-98 (the resolve phase) · crates/gunnar-pack/src/copy.rs (PackExtents, the fix)

• What: iter_from_counts resolves every count to a pack location before it produces a single entry. gix-odb's location_by_oid answers that question like this:

buf.resize(size, 0);                                   // zero the inflated size
pack.decompress_entry(&entry, &mut inflate, buf)       // inflate the WHOLE object
    .map(|entry_size_past_header| Location {           // keep only the LENGTH
        pack_id, pack_offset,
        entry_size: entry.header_size() + entry_size_past_header,
    })

The inflated bytes are then thrown away. On P-001's copy path they are never wanted at all — the entry goes onto the wire still compressed — so a clone inflated the entire repository in order to compute byte ranges it was about to memcpy. It is not gix's defect: location_by_oid has to answer for callers that have nothing but the pack data. It is gunnar's, for asking that way.

MEASURED oden 2026-08-04, perf record -F 4999 over a whole-repository clone of znippy.git through stage-probe fetch, taskset -c 0-7:

• Fix, and why it is exact rather than approximate: pack entries are contiguous — entry n ends where entry n+1 begins — so an entry's on-disk length is the distance to the next offset in the pack, which the .idx already holds. PackedObjects (gunnar's own gix_pack::Find) now overrides location_by_oid and answers out of a table built once per request from Find::pack_offsets_and_oid. Nothing is forked and nothing is re-derived: this is what git's pack-revindex.c exists for, and builtin/pack-objects.c's write_reuse_object sizes a reused entry the same way (revidx[1].offset - offset).

Two cases decline and fall back to the inflate, at one object each per pack: the pack's last entry (no successor to measure against; the trait cannot say how long the pack file is), and the first lookup into a pack (the location gix resolves is the only way to learn a pack id through the trait). MEASURED across the corpus: the index answered 99.94% of a clone's locations at the median, 99.66% of a fetch's.

A pack whose offsets are not strictly ascending is refused outright rather than sized wrong.

• The correctness condition that was NOT obvious, and cost a rewrite. An object can live in two packs — a forge between gcs is full of them — and the two copies are different entries: different delta base, possibly a different deflate. Both are valid. gix-odb picks by move-to-front (snapshot.indices.swap(0, idx) at the end of every location_by_oid), so a table that answered out of whichever pack it happened to hold picks the other one. MEASURED 2026-08-04 on a two-pack fixture built so that A-only, B-only and A∩B are all non-empty: 287 638 bytes served where gix-odb serves 287 691, and in three runs of four the length matched and the contents did not, because HashMap iteration order carries a per-process seed.

So PackedObjects mirrors gix's front index and answers only for that pack. gix-odb searches index 0 first, so if the object is in the front pack gix returns the front pack, whatever its MRU history; if it is not, this declines and gix does its own search, and its answer becomes the new front here exactly as it becomes index 0 there. Byte-identity is then by construction rather than by fixture. tests/pack_extents.rs::overlapping_packs_do_not_let_a_hash_seed_choose_the_bytes is that fixture, and removing the front rule fails it 4 times out of 4.

Nothing in the corpus is multi-pack, so this cost nothing there: the corpus table above is the same to within noise before and after the rewrite (clone median 2.079× → 1.965× across two runs an hour apart on a box whose load ran 9–24).

• A second correctness condition, found 2026-08-05, and this one served a corrupt pack. "The distance to the next offset in the pack" is the end of an entry only if the table holds every offset that pack has, and once a multi-pack-index exists it does not. The multi-pack format holds one deduplicated oid table across every pack it spans, so an object living in two packs is recorded against exactly one of them, and gix-odb's IndexLookup::iter answers pack_offsets_and_oid for the other pack by filtering that table by pack — a list short by precisely the objects the index attributed elsewhere. The entry before each hole was then sized across it and copied with its neighbour's bytes as a tail, and both receivers refused the result: fatal: pack is corrupted (SHA1 mismatch) from stock git 2.53.0, a trailer mismatch from gitoxide, in both hashes.

The list is now held to the pack's own 12-byte header, the one object count no index can filter. MEASURED oden 2026-08-05: 48 surplus bytes on a 287 691-byte two-pack fixture; 46 080 on knut.git at 16 packs (8 066 803 refused → 8 020 723 accepted, 0.57%). The refusal is per pack, so P-025 survives where the index left a list whole: 82.0% of that clone's locations still came out of an index, against 89.1% before, a loss of 230 of 2915 lookups (7.9%). objects_copied_from_pack/objects_recompressed were 1559/1714 on both arms — what changed is how a length is learnt, never whether an entry is copied.

Pack count was never the trigger; an object in two packs is. bench/year clones clean at 1139 packs because its packs share nothing, and the floor here is two — the fewest a multi-pack index can span. MULTI_PACK_INDEX_THRESHOLD = 16 only decides when a store starts having the index that exposes it, which is why the shipped 12-pack corpus never showed it although 19 of 19 of its repositories hold at least one duplicated object (1 funnel-core … 380 nornir) — tunnr, the sha256 member, among them at 13. The 99.94% figure above therefore stands only for a store with no multi-pack index.

• What it cost, MEASURED oden 2026-08-04, taskset -c 0-7, load 12–22, whole-process CPU (getrusage(RUSAGE_SELF), so iter_from_counts's worker thread is counted), best of 5 per arm, bench/examples/packwrite-probe.rs, all 19 workspace_nordisk members, 18 sha1 + tunnr sha256:

Noise band: the worst per-row CPU spread across the five repetitions was a median of 6.9% (clone) and 7.1% (fetch). Two rows are much worse and are named rather than averaged away: znippy-zoomies fetch, 64.8% on the disabled arm at 3.96 ms, and korp fetch, 21.8% on the enabled arm. Every clone row is outside its own band. The weakest row in the table is korp's fetch at 1.111×, and korp is the corpus's large-binary-blob member: its 42 ms of CPU is mostly not this.

Peak RSS, /usr/bin/time -f %M, znippy.git clone, three runs per arm: 69 288–70 204 kB → 54 580–54 744 kB, −21.4%.

Persisted, nornir bench run gunnar → bench_runs, gunnar.clone_large (a synthetic 2048-commit churn of one large file, 6144 objects, 67.69 MB out — not the corpus, and the ratio is much larger because the objects are much larger): 197.71 / 205.05 / 208.92 MB/s → 914.01 / 1099.56 MB/s, i.e. 4.4–5.6×, clone_ms 309–327 → 58.7–70.6. bytes_out, objects_copied_from_pack (6144) and objects_recompressed (0) are identical on both arms, which is a fourth independent witness that nothing moved. Run ids: off 4cfb4708 adc6b38d e9bf0adc, on 9c3243dc 34f0c033. Re-run after P-027 made nornir-bench's own Counting<S> forward bitmap(), in case that arm had been on the walk path: 199.85 / 209.76 → 1037.40 / 1045.93 MB/s, unchanged, because that fixture's linear history carries no .bitmap for either arm to use (82f4d9b6 63476c27 off, 2a95c674 bd0514cf on).

One run of this arm — 4b65a303, 187.19 MB/s — is in bench_runs against another lane's worktree: nornir bench run gunnar resolved the repository by name and built /home/rickard/scratch/wt/gunnar, not this branch. Recorded because the row is in the warehouse and is not a measurement of anything named here. The fix is --config <dir>/nornir.toml with [repo.gunnar] path naming the worktree, since the repo root is resolved from the config's grandparent joined with path.

Stage decomposition (stage-probe fetch, nornir.git, 12 186 objects): the whole saving lands on the calling thread, because the resolve phase is already inline at thread_limit: Some(1) — pack_write exclusive CPU 61.84 → 16.73 ms (3.70×), and its share of the attributed request fell from 86.28% to 63.22%, leaving select.bitmap as the second real cost at 36.78%.

• What it does NOT fix, said plainly. Whole-process CPU for that clone went 129.71 → 84.91 ms, and 68 ms of the remainder is still on iter_from_counts's worker thread. That is entry_by_location's data.to_owned() — one heap allocation per object served, which P-023 independently found as the largest futex call site. It is unreachable from gunnar without abandoning iter_from_counts entirely, because gix_pack::find::Entry owns its Vec<u8> and the trait has no borrowing form. After this entry it is the largest remaining item on the read path, worth an estimated further ~1.3–1.4× (memmove + page-init + allocator churn is ~28% of what is left).

• A lever that was named and is not there: "pre-size the output buffer". A stage lane measured pack_write at 123 allocations totalling 19.36 MiB for a 4.52 MiB pack and read it as Vec doubling worth removing. Reproduced here at 115 allocations / 38.19 MB for an 8.91 MB pack, and then traced: it is bench/src/stages.rs's own let mut out: Vec<u8> = Vec::new(), the sink the probe serves into. Doubling to 8.9 MB sums to ~33.5 MB of realloc, and the count and the size both fit. The server has no such buffer: the real sink is BufWriter<Metered<&mut dyn Write>> (gunnar-cli/src/server/transport.rs:340), and every buffer write_pack_copy_thin owns — counts, offsets, written_ids, the deflate scratch — is already built with with_capacity. Pre-sizing that Vec would make the benchmark faster and the server exactly as fast, which is the third kind of instrument artifact found in this tree today.

• Bearing on P-023: that entry measured zlib_rs::inflate at 32.4% of the server's CPU at 32 concurrent clients. Most of that is this. P-023's futex finding is untouched — the per-object hand-off is the entry phase, not the resolve phase.

• Bearing on P-018: the 16.2× head-to-head server-CPU gap against git upload-pack was measured 2026-08-03 with this inflate in the path. It has not been re-measured here; the arithmetic says the gap should be roughly halved on a clone, and arithmetic is not a measurement.

• Verify: crates/gunnar-pack/tests/pack_extents.rs, three cases, and bench/examples/packwrite-probe.rs over the corpus. Both assert two things, because the change is defined to emit identical bytes and a digest comparison alone is therefore blind by construction: the bytes are identical to the GUNNAR_PACK_EXTENTS=off arm and gunnar_pack::extent_lookups() says the index is what answered. git index-pack --strict is the third opinion, because both gunnar arms share everything downstream of the location and could be wrong together.

Seen RED five ways, 2026-08-04:

• entry length + 1: byte-equality fails and git index-pack --strict says pack has bad object at offset 275: inflate returned 1;

• extent_location returning None always: "the enabled arm still inflated 72 entries" and "no location came out of the index";

• the strictly-ascending guard removed: the refusal case fails on a pack whose offsets repeat;

• the front-index rule replaced by "any table we hold": the overlapping-packs case fails 4 of 4, three times on content at equal length and once on length;

• and the overlapping-packs case itself was first written blind — it served only the shared region, which leaves one pack learned and nothing to be ambiguous about, and it passed with the rule deleted. It was rewritten to serve the whole graph before it was believed.

• Closed: 2026-08-04.

P-026 — P-001's headline is a synthetic fixture's; on real repositories it is 3.4× the bytes and 18× the CPU, not 18× and 220×

• Found: 2026-08-04

• Status: FIXED — the number is re-taken and the doc comment everyone reads it from is corrected

• Severity: blocker (it is the figure the entry-copy argument is quoted from)

• Where: crates/gunnar-store/src/object.rs (ObjectStore::packed's doc) · .nornir/performance-design.md §vs_git_clone at e60c95d · bench/examples/packwrite-probe.rs (the re-take)

• What: "108.70 MB against 5.93 MB, and 4.875 s of server CPU against 0.022 s" — 18.336× and 219.963× — is quoted in six places in this tree and was quoted to Rickard repeatedly on 2026-08-04. It is gunnar.vs_git_clone at e60c95d, 2026-08-03, and the table it comes from says so. What none of the six quotations say is what it was measured on: a fixture built to maximise delta reuse (2048 commits rewriting ~2% of the lines of eight 16 KiB files, 528 MiB of blob content, 36 172 objects). A synthetic churn deltas far better than a source repository.

Two things it is NOT. It is not contaminated by the bitmap() wrapper defect: ObjectStore::bitmap landed at ffb09f3, 2026-08-04 16:05, a day after that measurement, so neither column had a bitmap. And it is not a NoPackCopy A/B at all — it is gunnar-before-P-001 against stock git across two commits. Both of those were worth checking before correcting anything, and the correction is narrower for it.

What is contaminated is every BEFORE column taken through gunnar_store::NoPackCopy between 16:05 and b3bf145 at 20:47 the same day — GUNNAR_BENCH_NO_PACK_COPY=1, crates/gunnar-wire/tests/pack_copy.rs — because that wrapper did not forward bitmap() and the trait default is None. See P-027.

• The re-take. MEASURED oden 2026-08-04, all 19 workspace_nordisk members (18 sha1 + tunnr sha256), each mirrored and git repacked without -f so upstream delta chains survive — the post-gc single-pack state a forge maintains, which is also the state in which there is most to copy. NoPackCopy is the BEFORE column and changes nothing but packed(); the bitmap is on in both columns and every serve asserts it out of the store rather than assuming it. Whole-process CPU (getrusage(RUSAGE_SELF), so iter_from_counts's worker thread is counted), taskset -c 0-7, best of five, box load 9–20:

With P-025 on top the CPU column becomes 33.52× (clone, median) and 18.65× (fetch, median) against the same BEFORE column; the byte column is unchanged because P-025 is byte-identical by construction.

So the shape of the claim survives and its size does not. Pack-copy is worth 3.4× the bytes and 18× the CPU on a median real repository. That is a large win and it is not the one that was being quoted.

• P-001 is engaging, and that was the other open question. The same run reports objects_copied_from_pack / objects_recompressed per repository: 64 080 of 64 528 clone objects copied (99.31%; median per repo 99.57%, worst 97.52%) and 5 672 of 5 827 fetch objects (97.34%; median 98.60%, worst 91.76%). A hypothesis that the 54%-of-cycles-in-zlib finding meant "a large fraction of objects are not being copied" is therefore wrong: the zlib was location_by_oid inflating every object — copied or not — purely to learn its entry length, which is P-025.

• Verify: bench/examples/packwrite-probe.rs over the corpus. It refuses to report rather than reporting a mixed comparison: it fails if the two columns disagree about the bitmap, if NoPackCopy copied anything at all, or if copied/recompressed move between the P-025 arms.

• Closed: 2026-08-04.

P-027 — Five ObjectStore wrappers turned an optimisation OFF by not mentioning it, and no wire byte differed

• Found: 2026-08-04 (StageCounting, by the decomp lane), extended the same day

• Status: FIXED — all five, plus a rule that makes a sixth impossible

• Severity: blocker (two of the five were the BEFORE column of a published comparison)

• Where: crates/gunnar-store/src/object.rs (the two defaults) · xtask/src/pack_pipeline.rs (Rule::SilentOptOut, the fix by construction)

• What: ObjectStore::packed and ObjectStore::bitmap both default to None, and both defaults are arguable for a backend — a mem store has no pack, an un-repacked repository has no .bitmap. For a wrapper they are lethal: None is not "I did not answer", it is "turn it off". select_objects consults store().bitmap() first and returns before the walk when it answers, and upload_pack consults packed() and falls back to write_pack when it does not. So a wrapper that forwards read/header/contains/write and omits one of these silently reroutes everything measured through it.

Nothing could catch it. The bitmap path is defined to select the same object set as the walk, so wire bytes, object counts and every fidelity digest across all 19 corpus repositories agree either way. That is the definition of a change no test can see.

Five instances, all live on 2026-08-04:

The last two were found by the rule below, not by reading. A sixth, gunnar_store::conformance::BrokenObjects, forwarded neither and would have made the conformance red probes run against a store that was also missing two optimisations.

• Fix — the shape, not a fifth patch. xtask's pack-pipeline guard grows a third rule: every impl ObjectStore in the tree must write out packed() and bitmap(), even to answer None. Six leaf backends and test doubles now do, at six lines each, and a wrapper cannot omit a forward without failing the build. The trait defaults stay for out-of-tree implementors.

The rule reads a wider file set than the other two — crates/*/tests and bench/ included — because three of the five instances were instruments rather than servers, and an instrument is exactly where a silently-disabled optimisation becomes a published number.

• Verify: xtask/tests/pack_pipeline.rs, four cases. The rule is asserted per-impl rather than per-file, because "does this file contain fn bitmap(" is green on bench/examples/nornir-bench.rs, which held one complete impl and one incomplete one. Seen RED by deleting the real forward from bench/src/readside.rs and watching the tree scan fail on it by name.

• Closed: 2026-08-04.

P-028 — A have that misses costs a statx, because one contains serves two callers with opposite correct answers

• Found: 2026-08-04, out of gunnar.have_lookup's finding that the miss path is 13.8× the cheapest index candidate and only 1.13× between the two candidates — so the index is not the term

• Status: FIXED, and honestly sized: unmeasurable end to end, and that is the result

• Severity: smell (a real defect with a known cause on a path where misses are normal; not a blocker, and the entry says so rather than dressing it up)

• Where: crates/gunnar-store/src/object.rs (ObjectStore::contains_without_rescan) · crates/gunnar-store/src/odb.rs (OnMiss, OdbPool::lookup_without_rescan) · crates/gunnar-wire/src/negotiate.rs, graph.rs, upload_pack.rs

• What: P-012 put RefreshMode::Never on the pooled handles and put the re-scan back as a one-statx watch on objects/pack, taken on every miss. That watch is right for the caller P-012 was looking at and wrong for the one it was not, and the two share a method:

Negotiation is the second row and it is the common case: a have the server does not hold is the expected answer, so the syscall is spent to re-derive a false that was already correct. git draws the same line — has_object() passes OBJECT_INFO_QUICK and re-scans only under HAS_OBJECT_RECHECK_PACKED.

• Fix: a second trait method, contains_without_rescan, defaulting to contains. The negotiation's four membership sites call it; want validation, the walk's Missing::Skip and everything else keep contains. Skipping the re-scan can only produce a false negative — a snapshot is a subset of the disk, never a superset — and every consumer of a have degrades in that one direction: it drops out of common, so less is excluded; it drops out of Selection::client_has, so no thin delta names it as a base; ready may be withheld a round longer. A false positive would be fatal and no snapshot can make one.

• MEASURED, oden 2026-08-04, gunnar.have_refresh, one binary, both columns interleaved, taskset to 8 cores, three raw runs at loads 54/26/21 plus one persisted run (bench_runs fb8e5ab4). Minima of 31 reps; the median is published beside every figure and agrees to within 2%:

Across all four runs the saving per miss is 440–790 ns — one statx — and the allmiss128 stage ratio is 1.43–1.44× every time.

• What it is NOT, which is the part worth keeping. gunnar.have_lookup attributed the whole 13.8× gap between the shipping miss path and a 16-bit fanout to RefreshMode. It is 30% of it. Shipping is 2 479 ns per have on the all-miss shape and this takes it to 1 724, against 302 ns for git's own .idx search and 181 ns for the fanout. So 13.8× → 9.5×, and the ~1 500 ns still standing is neither the refresh nor the index: it is the pool lease, the gix-odb handle and Exists::exists itself. That is the open question this entry leaves behind, and it is larger than the one it closed.

• End to end it is inside the noise on every shape, which was predicted before it was measured and is the honest result. Where the gap exists the negotiate stage is 0.52% of the request; where the stage is a real 3.7% the haves are hits and there is nothing to win. 16 calls × 755 ns is 12 µs against a request of 3 200. The arm computes the verdict rather than narrating it: *_e2e_outside_noise is 0 on all three shapes in all four runs.

• A second regime, found and NOT quoted as the headline. The watch is only usable as a memo once its recorded mtime is older than RACY_WINDOW. Inside that second, refresh_if_stale cannot trust it, consolidates, reports that it did, and the retry loop goes round again: LOOKUP_ATTEMPTS = 4 full directory consolidations for one miss, measured at 128 probes and 128 consolidations for 32 misses. Pre-existing, contains's business rather than this split's, and the bench arm refuses to run unless the consolidation count is zero — quoting the worst case as the case would have inflated the headline by orders of magnitude.

• Verify — three tests, each seen RED, and one of them is the only thing in the tree that can see the change at all. Every ordinary check is green either way: the two methods agree on a current snapshot by construction, so answers, wire bytes, object counts and contains totals are identical by design.

  • gunnar-store/tests/multi_pack_lookup.rs: contains_still_finds_a_pack_that_landed_after_the_snapshot, the_cheap_path_answers_a_miss_where_the_refreshing_path_finds_the_pack, the_cheap_path_never_invents_an_object, only_the_refreshing_path_asks_the_filesystem. Seen RED three ways: the new method left on the trait default (2 fail), contains itself routed onto the cheap path (5 fail, including P-012's own two), and the cheap path answering true unconditionally (3 fail).
  • gunnar-wire/tests/have_miss_is_the_answer.rs: a want in a pack that landed after the snapshot is still served (RED by routing wants_exist onto the cheap path — the fetch is refused); and a stale have widens the pack, measured at 90 objects against 60, a strict superset, both accepted by git index-pack --strict in a repository holding what the client offered (RED with the split unwired — the two packs come out identical).
  • bench/examples/have-parity.rs: all 19 workspace_nordisk mirrors, both hash kinds (18 sha1 + tunnr sha256), clone AND negotiated fetch, both columns interleaved out of one store — byte-identical on all 38, 60 887 clone objects and 302 fetch objects served. On a current snapshot the two methods are defined to agree, so this is a regression check on WHICH call sites were rerouted, not a semantic one. Seen RED twice: with the wrapper's forward deleted the byte check stays green and only the as_have counter fires (0 of 16), which is the whole reason that counter exists; with one column answering a have wrongly the packs come out 6 810 398 against 80 265 bytes.
  • The observable is FsStore::rescan_probes, a count of the syscall, because the syscall is the only thing that moved.

• A property found on the way, which narrows the exposure further: check_wants peels every advertised ref before it validates a want, and peeling a ref into a just-landed pack is itself a read down the refreshing path. So on any request whose refs cover the new pack, the want path cures the staleness before negotiation ever runs. The wire test has to delete the branch to observe a stale have at all.

• Not done, and deliberately: contains_without_rescan is not added to P-027's OPTIONAL_BUT_LOAD_BEARING guard. That rule exists for defaults whose absence changes what the server does; this one's default is contains, so a wrapper that forgets it is slower and never wrong — the opposite polarity. Making every leaf write out a body identical to the default would be noise. The four in-tree wrappers forward it explicitly regardless, and the bench decorator asserts it fired (IndexedContains::as_have), because forgetting it in an instrument would publish the BEFORE path under the AFTER label.

• Closed: 2026-08-04.

S-003 — where a server-side delta encoder belongs

Moved here from the front page 2026-08-05.

The client computes deltas; repack() preserves them and the serve path computes none (Mode::PackCopyAndBaseObjects, crates/gunnar-store/src/repack.rs). MEASURED 2026-08-04, 9 corpora pushed by stock git into a bare repo, --thin against --thin --window=0: a serve-time encoder is worth 1.14×–4.90× freshly pushed and exactly 1.0000× after gc, 9 of 9. So the encoder belongs in the batch tier and not the serve path, and the pack is the store. Pack-copy already recovers 100% of a clone and of a fetch (gunnar_pack::write_pack_copy_thin).

Amended 2026-08-05 — the thin voucher meant visited, and one receiver noticed

Selection::client_has vouched for what the client holds plus the objects being sent, on the argument that naming an in-transfer base is an OBJ_REF_DELTA against an object in the same pack and stock git accepts that. Stock git does. gitoxide does not: gix_pack::data::input::LookupRefDeltaObjectsIter resolves every OBJ_REF_DELTA of a thin pack against the receiver's object database and the bases it has already spliced in, and never looks inside the pack it is reading. gitoxide is what gunnar-client and receive-pack both end in, so gunnar was serving packs its own client could not read.

It needs a multi-pack store to happen at all: the base is stored twice — whole in the pack whose push introduced its dependant (spliced there by index-pack --fix-thin), and as a delta in the pack whose push introduced it — so gix_pack's copier cannot re-point the second delta and names its base by object id. One pack, one location, no such entry. That is why every arm in crates/gunnar-wire/tests/pack_copy.rs was green: its fixture is git repack -adf, one pack. A repository that has been pushed to and not gc'd is the shape, and it is also exactly the repository with no .bitmap — so this was the walk path and never the bitmap path.

Found by gunnar.year, which is the only arm that pushes concurrently and then fetches: it died at week 3–7 on The object <tree> could not be decoded or wasn't found.

The voucher is now the exclusion closure alone, on both paths. MEASURED on oden 2026-08-05, one negotiated fetch per repository per cut over the 19 real workspace_nordisk repositories (bifrost included — the corpus mirror of it is intact, whatever its working checkout is), rebuilt in the production shape (12 pushes each, 12 packs, no .bitmap, sha1 and tunnr's sha256), 46 cases, both judges on the same bytes:

The 13 failures are 11 of the 19 repositories, at both hash kinds.

What it costs. On the same 46 cases, fetchbytes-probe: thin bases refused 4 → 74 (Edges) and 0 → 73 (Full), objects re-deflated 897 → 967 (+7.8%), and packfile wire bytes +0.532% (Edges) / +0.542% (Full). None of the new refusals is wrong in the sense the probe judges: the base is an object of the transfer, which the receiving repository genuinely does not hold before the fetch.

And it costs nothing where there is nothing to refuse. On the shipped real-corpus (one repack -adq pack and a .bitmap each) all five arms are byte-identical before and after, 46 of 46 cases, pack_wire unchanged to the byte — bitmap arms included, and answered_from_bitmap is 46/46 there. The walk's own extra work is one Vec<ObjectId> push and one hash removal per object sent, which is O(objects in the transfer) against a walk that already reads and deflates every one of them. Isolated on scale-deep-1500c-256k-20d with GUNNAR_BITMAP=off, 9 interleaved runs pinned to 4 cores: same 76 objects, same 71 store reads, same 4 987 665 wire bytes, serve_ms median 6.99 → 7.03 (+0.57%) inside an 8.5% / 5.5% spread. Below the instrument floor and not claimable as a cost.

Amended 2026-08-05 — the unselected base was decided by offset, and a multi-pack store has two

WriteOptions::base_closure names a delta's unselected base and adds it to the pack rather than exploding the delta. It decided "unselected" by pack offset, which is the question gix_pack's Entry::from_pack_entry asks — it searches potential_bases for a matching (pack_id, pack_offset) — and it is the wrong question to ask of a pack. A pack may hold an object once, and that is a question about an object id.

On a single-pack store the two are the same question. On a multi-pack store they come apart, and it is the same duplication the amendment above turns on: an object stored in two packs is one id at two locations, location_by_oid answers with one of them, and a delta in the other pack names an offset the selection does not hold whose object it does. The closure added it, and git index-pack refuses such a pack outright — fatal: The same object <id> appears twice in the pack.

Found by gunnar.year again, at week 12, on the quarterly stock-git clone. It is a CLONE: no common, thin == false, client_has never consulted, so it is not the voucher and the two defects are one after the other.

MEASURED oden 2026-08-05, one real git clone --single-branch per repository with transfer.fsckObjects=true followed by fsck --full --strict, over the same production-shaped corpus (12 pushes, 12 packs, no .bitmap, sha1 and tunnr's sha256), 18 repositories (bifrost excluded: its SOURCE pack fails git fsck):

The 13 are all sha1. tunnr, the corpus's one sha256 repository, is green in both arms before and after — its production-shaped store does not contain the condition — so sha256 here is a non-regression and not a reproduction. The key that changed is an ObjectId and nothing in the fix is hash-width dependent, but that is an argument and it is recorded as one.

The closure now declines a base the pack already carries, counted as WriteOutcome::closure_bases_declined, and declines the whole chain with it: the entry that collided is the one the delta above it names, so nothing above it could be copied either.

It costs none of the item. On the shipped real-corpus the served packs are byte-identical before and after in both arms, 18 of 18, and the unselbase-probe table reproduces to the object: 64 445 → 64 533 objects (+0.14%), 94 575 097 → 91 903 865 wire bytes (−2.82%), 448 → 320 re-deflated, server CPU −52.78% at 8f79cd7 and −52.33% here (5 repetitions each, best-of, 4 pinned cores on a box at load 6.3). The negotiated-fetch path is untouched by construction and measured so: 55 cases across 19 repositories × 3 cut depths, both judges, identical output.

Why no fixture in the tree could fail on it. git clone --bare repacks. Every fixture built that way holds each object once, one object has one location, and the two membership questions cannot come apart — the same trap the amendment above turned on, one defect later. crates/gunnar-wire/tests/pack_copy.rs now carries the arm, and its DUP_FILES doc carries the pack-count sweep that says the condition needs five packs before it appears at all.

A separate defect the sweep uncovered, and it is NOT this one. On this fixture, write_pack_copy emits a pack both receivers refuse the moment the store has a multi-pack index — stock git with fatal: pack is corrupted (SHA1 mismatch), gitoxide with a failed index — where NoPackCopy is clean on the same bytes and GUNNAR_BASE_CLOSURE=off makes no difference. closure_bases_declined also falls to zero there, so the closure stops seeing the store at all.

The multi-pack index is the cause and not the co-ordinate, and that is measured rather than inferred: FsStore::open_or_init writes one at gunnar_store::midx::MULTI_PACK_INDEX_THRESHOLD packs, and moving that constant from 16 to 8 moves the cliff from 16 packs to 8, exactly.

  threshold 16   packs 15 ok · 16 FAIL · 17 FAIL
  threshold  8   packs  7 ok ·  8 FAIL ·  9 FAIL

What is not claimed: that every multi-pack-index store is affected. gunnar.year runs to 1 139 packs on this commit and clones clean, so the shape needs more than a midx. MEASURED at sha1 only — this fixture builder has no sha256 arm. The item is open.

Appendix — reproducing the measurements

The probe is at /home/rickard/scratch/cargo/gunnar-perf-probe/ (scratch on md1, not in this repo, and deliberately disposable; P-015 is the real harness). Two binaries:

cargo build --release                 # flate2 default => miniz_oxide
cargo build --release --features zrs  # flate2/zlib-rs
./target/release/gunnar-perf-probe    # SHA-1, deflate, inflate, allocation shape
./target/release/scale                # deflate thread scaling, 1..64 threads

Everything it measures is a component. No end-to-end gunnar operation has been benchmarked by anyone, and this file does not claim one has. That is S-044, and P-015 is the answer to it.

The P-007 / P-008 closing probe (2026-08-03)

The numbers in those two closed entries come from a second probe, at /home/rickard/scratch/cargo/gunnar-p007-p008-probe/, also disposable. It differs from the one above in the way that matters: it path-depends on gunnar-pack and drives the real PackWriter, so what it reports is a whole pack write — entry headers, deflate, trailer hash — not a raw flate2/gix-hash loop.

cargo build --release
./gunnar-p007-p008-probe [corpus-dir] [byte-cap]   # default: gunnar's own crates/

Method, and its limits:

• best-of-7 after a warmup, printed with median and worst beside it so noise is visible. oden was shared with other agents for part of this; where load made median and worst collapse, best stayed stable to ~1% and is what is quoted. Every such figure says so.
• Frozen corpora. The first attempt compared runs against gunnar/crates, which another agent was editing; the corpus grew by 3 files mid-comparison. The small corpus is now a snapshot at /home/rickard/scratch/cargo/corpus_small; the large one is ~/.cargo/registry/src, which is immutable.
• Before/after built from the same source tree, one change apart, and for P-008 run interleaved so both binaries saw the same machine load.
• The SHA-256 rows are controls, not results: nothing in either change touches them, so a SHA-256 number that moved would mean the measurement was wrong.

The permanent guards these numbers justify are crates/gunnar-pack/tests/deflate_backend.rs and crates/gunnar-pack/tests/trailer_hash.rs. They are correctness guards, not benches: they pin bytes, not milliseconds. P-015 is still the missing piece for timings.

The P-001 before/after (2026-08-03)

No probe crate at all this time, and that is the point: P-015's harness is now the instrument, and the BEFORE column is produced by the same binary as the AFTER.

cd bench
podman build -t gunnar-bench-git:2.53.0 .          # once
NORNIR_MACHINE=oden NORNIR_BENCH_ONLY=gunnar.vs_git \
  cargo run --release --example nornir-bench        # AFTER

NORNIR_MACHINE=oden-preview GUNNAR_BENCH_NO_PACK_COPY=1 \
NORNIR_BENCH_ONLY=gunnar.vs_git \
  cargo run --release --example nornir-bench        # BEFORE — arms go RED

Method, and its limits:

• GUNNAR_BENCH_NO_PACK_COPY=1 is an ObjectStore wrapper returning None from packed() and changing nothing else, which is exactly the pre-P-001 server. Its credential is that it reproduces the e60c95d figures to three decimals (18.336×, 17.723×, 7.22×). It reaches the spawned server and the long-lived daemon the concurrent arm uses. It did not at first, and that arm stayed green through a run where the other two went red.
• NORNIR_MACHINE=oden-preview on the BEFORE run. A deliberate red must never be able to become a future baseline; the -preview machine name is the harness's own mechanism for that.
• A failing arm emits no result row, so the BEFORE numbers are carried in the failure message. There is nowhere else for them to come from, and an arm whose red says only "assertion failed" makes its own before-column unobtainable.
• oden was at load average 70–226 on 32 cores from other agents for every run. Ratios are interleaved gunnar/git inside one arm and are quotable; absolute milliseconds are not, and clones_per_sec on clone_concurrent_64 read 48.5 and 268.6 in two runs of identical code, which is what that looks like when it bites. The instrument that survives this is instructions retired (perf stat -e instructions): on 2026-08-03 walk_1m's wall clock moved 2.36× across five repetitions of one unchanged binary while its instruction count moved 0.29%. Every P-010 ratio in this file is instructions retired, taken interleaved. Use it for any before/after taken on a box that is not provably quiet — which, on oden, is all of them.
• Two corpora, deliberately. clone_large is random blobs: no deltas exist, so it isolates "do not re-deflate" at 9.9× with byte-identical output. The head-to-head fixture is churned text; deltas exist, so it measures both effects at once, 18.3× on bytes. Quoting one multiplier for P-001 without naming the corpus is wrong in either direction.

The permanent guards these numbers justify are gunnar.clone_large's copy-split and blob_reads == 0 assertions and h2h_assert_delta_reuse on both head-to-head clone arms, all in bench/examples/nornir-bench.rs, plus xtask/tests/pack_pipeline.rs for P-020's two source markers.

Re-measured 2026-08-03 17:09 UTC, after the pack_copy repair

Both columns re-run from the same binary at 569682b, NORNIR_BENCH_ONLY=gunnar.vs_git, against stock git 2.53.0 in localhost/gunnar-bench-git:2.53.0.

gunnar_copied / gunnar_recompressed on the AFTER clone arms: 36 172 / 0 and 52 112 / 0. That split, and not the byte count, is what says these are copies.

This box was NOT idle, and the section above is the reason the distinction is written down rather than assumed. Load average was 17.7 rising to 28.1 on 32 cores across the two runs (determinism-8ea at 713% CPU, errcode_raise_s at 384%, osmium at 125%), and the harness's own telemetry agrees: telem_cores_busy_avg 15.2 on the AFTER clone arm, 32.0 on the concurrent one. So:

• Quotable. Every byte column, pack_ratio_x, and the copy/recompress split. They are counts of bytes and entries, deterministic, and a busy neighbour cannot move them. The 18.336× / 17.723× / 7.220× BEFORE figures reproduce e60c95d to three decimals, which is the same credential the red switch has always had.
• Not quotable. Every millisecond, and the CPU-second ratios derived from them, which is why they are given to one decimal above and not three. BEFORE and AFTER are separate processes at different loads; CPU time is more robust to contention than wall clock but is not immune to it.
• Not the same claim as parity with git. pack_ratio_x is 1.000 — gunnar puts the same bytes on the wire that git does. Server CPU is not at parity: 0.404 s against git's 0.021 s on the sha1 clone, 18.8×, on this run at this load. P-001 closed the wire gap; it did not close that one, and P-015 still owns it.

The red switch was re-proven in the same sitting, which mattered this time: NoPackCopy moved out of the bench and into gunnar-store at 569682b so that gunnar-wire/tests/pack_copy.rs could cut the seam at the same place (LAW 5), and a switch that had stopped reaching the server would have shown up as a BEFORE column that quietly matched the AFTER one. It did not: all three arms went red with the figures above, and vs_git_clone_concurrent — the arm that once stayed green through a red run because the switch missed its long-lived daemon — went red with the rest.

P-029 — the ~1 500 ns P-028 could not attribute is two syscalls in gix-odb, and neither is what it named

Closes the open question P-028 left behind. Its words were: "the ~1 500 ns still standing is neither the refresh nor the index: it is the pool lease, the gix-odb handle and Exists::exists itself." That is refuted. Those three together are at most a fifth of it. Arm: a standalone contains_without_rescan probe against a real mirror, 2026-08-04, oden.

• Counted, not inferred. 20 000 contains_without_rescan calls for object ids that are not present, against the nornir mirror (12 328 objects, one pack, zero loose directories), under strace -c -f:

The same binary asked 12 328 ids that are present issues neither. Both are miss-only, and a syscall count is exact whatever the box is doing, which is why it is the primary evidence here and the nanoseconds are the secondary.

• Where they come from.

  • statx — gix-odb/src/store_impls/loose/find.rs: contains is hash_path(id, self.path.clone()).is_file(). One PathBuf heap clone and one statx per loose database. gunnar's repositories have no loose objects at all, and it is asked anyway, because a loose object has no index to consult.
  • sched_yield — gix-odb/src/store_impls/dynamic/load_index.rs, the Err(_nothing_more_to_load) arm of load_next_index: an unconditional std::thread::yield_now() before the loop that waits on num_indices_currently_being_loaded. It fires under RefreshMode::Never, with nothing loading and no contention. gitoxide's own comment on that loop is TODO: potentially hot loop - could this be a condition variable?.

• Attributed by moving it, three arms, one difference each. gix-odb vendored into scratch (never into gunnar's tree) so each term could be removed on its own. Minimum of nine runs of 200 000 misses, release, interleaved:

Arm C is wrong as a shipped change and exists only to size the term; the syscall counts confirm each arm removed exactly what it claims (B: 0 sched_yield, 20 046 statx; C: 44 statx, all of them process start-up).

Replicated at a different load, because a three-way split on a box at load 200 is a claim that has to survive being taken twice. Second run at loadavg 131: 1081.5 / 797.8 / 199.2 ns, giving 283.7 ns (26 %) / 598.6 ns (55 %) / 199.2 ns (18 %). Totals agree within 3.5 % and every share agrees within one percentage point.

• So the shape of the answer is the opposite of the guess. 81 % of a miss is two syscalls that gix-odb makes on the way to the index. The residue — which is the pool lease, the handle and the lookup itself — is 197.9 ns, against the 302 ns P-028 measured for git's own .idx search on the same box. gunnar's own data-structure work is already at git's speed. The 6× is the journey, not the destination, and the fanout16 candidate cannot touch it.

• These are warm-loop numbers and therefore a LOWER bound. The in-request cost is what to quote: P-028's 1 724 ns against this probe's 1 045 ns is 1.65×, so in a real negotiation the terms are roughly 970 ns of statx, 430 ns of sched_yield and 330 ns of everything else. Do not quote the warm-loop column — the same trap oid-index-reasoning.md records at 22.86 ns vs 302 ns.

• Not worth fixing, and the arithmetic rather than the opinion. Removing both syscalls would take a miss from ~1 724 ns to ~330 ns, saving ~1.4 µs. HaveMemo answers each distinct have once per connection, INITIAL_FLUSH is 16 and doubles, and MAX_IN_VAIN's 256 was never reached at any divergence up to 200 commits. So a typical negotiation is 16 misses = 22 µs against a request of 2 349 µs — 0.95 %; the worst shape observed, 128 misses, is 178 µs, and that request is correspondingly larger. Against that, one term is correct behaviour that cannot be removed (the loose statx) and the other is a synchronisation whose author left a comment explaining why it is there. Report it, do not patch it. .nornir/upstream-gix-odb-miss-yield.md.

• What could not be determined. Whether dropping the unconditional first yield_now() is safe upstream. The comment above it describes a real slot-map race and this lane did not model it; arm B is a measurement, not a proposal, and the upstream report asks the question rather than answering it.

P-030 — every push paid 50 ms of sleeping, and no gunnar-side lever existed

FIXED, carried as the one patched dependency in the tree. vendor/gix-features + [patch.crates-io]; the diff is +30/−2 lines in one function. Red-first proof: crates/gunnar-pack/tests/index_floor.rs. Upstream report: .nornir/upstream-gix-slice-watcher-sleep.md. 2026-08-04, oden.

• The defect, and why it is a floor and not a slowdown. gix_features::parallel::in_parallel_with_slice spawns an interrupt-watching thread that sleeps out its whole interval, and sets stop_everything only after every worker has been joined. The watcher lives inside the same std::thread::scope, so the sleep is appended to the call rather than overlapped with it. gix-pack's Tree::traverse asks for 50 ms and is unconditional in write_data_iter_to_stream, so every index_pack — the whole of the receive path — cost 50 ms of nothing.

• A five-file commit is 8 objects: five blobs, two trees, one commit, 490 bytes packed. That is the ordinary push, and it paid the floor in full.

• MEASURED, release, 101 calls on that 8-object pack, on a box at loadavg 317:

The flat before-column is the evidence. Those samples were taken while 32 cores carried a load average of 317, and the spread is 50.22–55.01 ms, because the sleep dominates everything the machine is doing. The after-column's spread IS that load, now visible. CPU is unchanged: a wait was removed, not work.

• Load-independent confirmation, because a clock on a box at load 317 is not evidence on its own. strace -f -c -e trace=clock_nanosleep over one call: exactly one clock_nanosleep(CLOCK_MONOTONIC, {tv_nsec=50000000}) before the patch and zero after. That is the whole finding in two syscall counts.

• End to end, the ordinary push through a real server. Stock git 2.53.0 pushing the 8-object commit over real SSH into a fresh repository, 31 pushes per round, six rounds, min per round:

Best conditions either arm saw: 66.84 → 18.90 ms, 3.54×, Δ −47.9 ms — the floor, and essentially nothing but the floor. Every round favours the patch; the spread between rounds is the fleet, not the change. The stored pack is byte-identical in all 12 runs and all 372 pushes: sha-256 f7ceed18….

• Against stock git, interleaved so all three arms saw the same box. The same 8-object pack, minimum of 25 each, loadavg 223:

And across the size range, interleaved the same way, minimum of nine each, at a quieter loadavg of 64:

The floor helps at every size and hurts at none, and CPU is flat throughout — 12 021 objects: 1088.8/1095.1 ms of process CPU stock against 1081.2/1074.3 patched, inside 2%. gunnar was already ahead of git at 12 000 objects before the patch; what the patch changes is the small end, which is where the ordinary push lives.

The stated ratio needs its caveat and it is not a small one: git's column is a whole subprocess, fork and exec included, and gunnar's is an in-process call. The comparison is fair as a server comparison — stock git's receive-pack forks index-pack and gunnar does not — but it is not a like-for-like of the two indexing implementations, and anyone quoting "gunnar is twice git's speed at indexing" from this row is quoting it wrong. What the row does establish without any caveat is the unpatched column: 20.6–24.2× is not a process-startup effect at that magnitude.

• There was no gunnar-side lever, and this was checked rather than accepted. The 50 ms is a literal in gix-pack. The one knob that reaches the watcher — should_interrupt, via periodic() returning None — aborts the traversal. thread_limit: Some(1) pays it too, because the watcher is spawned regardless of the worker count.

• Patched at gix-features, not at gix-pack's literal. The interval is not the defect; an uninterruptible wait is. Fixing the layer that owns the wait leaves gix-pack stock — one vendored crate rather than two — fixes every other caller (LAW 5), and keeps 50 ms as what it was meant to be: the latency of noticing a Ctrl-C. Dropping the literal to 1 ms would have removed the floor too, at the price of waking a thread a thousand times a second to poll one atomic.

• What upstream main already did, found by checking the claim rather than asserting it. "Tree::traverse is the only caller" is true of the published crate and false of main: main has rewritten the delta traversal to resolve::all, which does not call in_parallel_with_slice at all. main's gix-pack is still numbered 0.73.0, so that rewrite is unreleased, and the crates.io sparse index confirms 0.73.0 is the newest published version. So gunnar cannot upgrade out of the floor — there is nothing to upgrade to — and the patch is the only route. It also means the exit may come from the gix-pack side rather than the gix-features side: when a new gix-pack ships, try it with vendor/ removed, because index_floor.rs may already be green. The upstream PR was corrected to say all of this before a maintainer had to.

• It costs the dependency graph nothing. Same crate, same version, same dependencies; §2's guard counts third-party by name. MEASURED before and after across all twelve resolved graphs, byte-identical: appliance 280 crates / 267 third-party, 0 violations, 7 tracked deviations.

• The residue, named rather than left for the next lane. With the floor gone, the 8-object push's remaining server cost is thread creation: strace -c -e trace=clone3 counts 33 clone3 per index_pack — 32 workers and a watcher — because IndexOptions::thread_limit is None and oden has 32 cores. For a pack of 8 objects that is 33 threads to resolve 8 entries, and it is the whole of the ~3 ms that is left. Sizing thread_limit to the object count is the obvious next move and is not made here: it changes a knob on the receive path and deserves its own red-first proof.