nordisk/edda

Every gix entry point on the serve path, counted, costed, and ranked — reasoning

Class: reasoning. A census, not a plan. Every gix-* API gunnar reaches while answering ls-remote, clone, fetch or push, with how often it is called per request, what share of the request's cycles it owns, and a verdict of keep / replace / already done / unknown. Measured on oden, 2026-08-04.

Read §0.1 first. This lane's #1 row landed on origin/master while the lane was measuring it. Everything below is re-measured at 1a4df9a with the fix in, and the superseded figures are kept beside the current ones because the before/after is the census method validating itself.

0. What was measured, and on what

• Trees. The census was taken at origin/master 4d4d8af, re-taken at 1a4df9a after the rebase. Both are named on every figure.

• Box: oden, Threadripper PRO 3975WX, 32 threads. Load average 8.6–28.9 across the session; every timed arm pinned with taskset -c 24-27.

• Noise band, measured rather than assumed. Five repetitions of the same 20-request fetch loop, same binary, same corpus, at 1a4df9a: 26.04 / 26.62 / 26.62 / 26.72 / 31.72 ms per request. Median 26.62, max/min 1.22×. (At 4d4d8af, on the pre-fix path: 44.17 / 44.37 / 44.43 / 45.99 / 52.19, band 1.18×.) Nothing inside those bands is claimed as a difference.

• Corpus: the 19 mirrored workspace_nordisk repositories at /home/rickard/scratch/gunnar-bench-fixtures/real-corpus, 65 737 objects. Built by pushing over file:// then git repack -adq — not -adf, so upstream delta chains survive; never git clone --bare, which silently repacks.

• Instruments.

  • crates/gunnar-pack/src/oidprobe.rs — extended this lane from 9 counters to 22, still behind the oidprobe cargo feature that only bench/ turns on. With the feature off bump is an empty inline function and the statics do not exist.
  • bench/examples/gix-census.rs — new. counts (the four-command sweep), loopfetch / looppush (long, steady loops to hang perf record on), entrysize (the seam A/B for the row that landed).
  • bench/src/stages.rs — reused; ls_remote and serve_on added (§7).
  • perf record -F 4999, release with debug = 1, symbols resolved.

• Guard baseline, re-measured, not quoted: cargo xtask guard → 280 crates / 267 third-party / 7 tracked / 0 violations, PASS. Matches the stated baseline, so the tree measured is the tree the baseline describes.

• Suite, both sides, cargo test --workspace --no-fail-fast: origin/master 1a4df9a 1107 passed / 0 failed, this branch 1107 / 0. The brief's figure of 1094 is stale — the merge that brought fc2e0c3 also brought crates/gunnar-pack/tests/pack_extents.rs. fold_and_clone.rs's four known failures do not appear under --workspace: they need gunnar-store/repack-delta, which workspace feature unification turns on and a per-crate cargo test -p gunnar-coldtier does not. Same cause, from the other side, for gunnar-wire/tests/bitmap_cost.rs, which fails 2 of 3 under -p gunnar-wire on a pristine origin/master and passes under --workspace. Two red conditions that depend on how the suite is invoked, recorded rather than reported as a regression.

0.1 The correction that has to come first

This lane was sent with a hypothesis to confirm: that a perf profile blaming zlib for 54% of a fetch had been taken with reachability bitmaps off, since the pack-copy path is not supposed to inflate anything.

The hypothesis was wrong, and so was the direction of the error. With the bitmap path asserted by the stage table, zlib was 74.6% of a negotiated fetch at 4d4d8af. The cause was gix_odb's location_by_oid, which answers "where does this entry live and how long is it" by fully inflating the object into a zeroed buffer and keeping nothing but the byte count — called by gunnar's pack-copy path exactly 1.000 times per object served, measured (60 887 calls over 60 887 objects on clone; 27 489 over 27 489 on fetch).

That row was ranked #1 and sized at 80–140 lines with a predicted end-to-end win of 1.60–1.96×. It then turned out to have been landed six hours earlier by the pack-write lane, at fc2e0c3 — "An entry's length is in the index, so stop inflating the whole repository to find it" — reaching origin/master at 1a4df9a while this census was being written. Their reported medians: clone 1.965×, fetch 1.663×.

Re-measured here as a same-binary A/B via their own GUNNAR_PACK_EXTENTS switch, znippy negotiated fetch, 60 requests, byte-identical wire on both arms (395 175 600 bytes):

The independent prediction landed inside the band it predicted. That is the one thing this document can offer as evidence that its other estimates are worth anything, so it is reported rather than quietly deleted — as is the fact that the lane spent its afternoon measuring something another lane had already fixed.

1. The census at 1a4df9a — the current tree

Ranked by measured share of request cycles. n/req is per request; /obj is per object served, summed over the corpus rather than averaged per repository.

fetch — a negotiated fetch

perf, znippy, 60 requests, bitmap path asserted, harness digest kept out of the loop:

Every remaining share is larger because the denominator shrank by 1.62×; in absolute cycles only inflate moved.

Stage frame (stage-probe fetch, znippy, 1a4df9a): pack_write 89.98% wall / 90.59% CPU, select.bitmap 9.21 / 9.41, everything else under 0.7%. Table accounts for 93%+ of process CPU.

stage-probe's absolute milliseconds are not the shipping cost, and the census does not quote them. It installs CountingAlloc as the process's #[global_allocator] and opens the store cold on every call. Measured directly: through stage-probe the extents A/B is 80.77 → 63.86 ms of wall and 51.15 → 52.03 ms of CPU — flat, where the same A/B through loopfetch (no counting allocator, store held open, production shape) is 48.64 → 29.81 ms of CPU. The counting allocator's own per-allocation cost swamps a change whose whole effect is to stop allocating a big zeroed buffer. Stage fractions come from stage-probe; every absolute in this document comes from loopfetch.

clone and fetch, per-command counters (19 requests each, 1a4df9a)

The two re-deflation causes are disjoint by command, which changes what each fix is worth:

• on a clone every re-deflate is gix returning None from from_pack_entry — an object with no usable pack residency, or a RefDelta;
• on a fetch every re-deflate is gunnar's own thin veto (§2).

push (19 requests, 60 887 objects indexed, 1a4df9a)

index_thin_pack, broken down — which nobody had done. It is one gix call. perf, 8 pushes of 5 000 objects, 407.4 ms each, 1a4df9a:

The stage table accounts for only 40.0% of process CPU on a push: Bundle::write_to_directory runs with thread_limit: None and fans out to every core, and no stage boundary exists on those threads. 380.4 ms wall against 452.9 ms of process CPU is 1.19–1.37 cores. Every stage percentage above is a share of the attributed 40%, and is labelled as such rather than presented as a share of the request.

ls-remote (19 requests, 1a4df9a)

0.212 ms of process CPU per request (4.025 ms over 19), against 28.6 ms for a fetch and 246.6 ms for a push on the same sweep. Every pack entry point is 0/request, measured — location_by_oid, entry_by_location, iter_from_counts, PackBitmap and Bundle all read zero. ArchiveRefStore (another lane) already serves this without the filesystem. Nothing here is worth a second look, and this census touched none of it.

2. The row that is now #1: the thin veto refuses 75.8% of what gix offers

crates/gunnar-pack/src/copy.rs:451. gix emits Kind::DeltaOid for a delta whose base sits outside the transfer; gunnar checks selection.client_has and, if it will not vouch, writes the object out whole through add_object — an inflate of the entire delta chain and a fresh deflate.

MEASURED, 19 repositories, negotiated fetch, 1a4df9a: 1 465 DeltaOid entries offered, 1 110 vetoed — 75.8%. On znippy: 247 offered, 182 vetoed out of 1 796 objects served (10.1%), and those 182 objects own essentially the whole of the fetch's 44.5% deflate and most of its residual 13.7% inflate. Together that is ~55% of a negotiated fetch's cycles bought by 10% of its objects.

This lane's first reading of the same data was backwards, and the correction is the point. GIX_PACK_TRY_FIND_CACHED reads 0 on a fetch beside GIX_PACK_RECOMPRESSED 182, and the obvious conclusion — "the veto never fired" — is wrong: the veto's fallback read goes through PackedObjects::handle directly rather than through the counted gix_pack::Find impl, so the pre-existing counter cannot see it. GIX_PACK_THIN_VETOED was added for exactly that and reports 182, matching RECOMPRESSED to the unit on every repository.

Verdict replace — and the thing to replace is client_has, not gix. copy.rs's own docs record the same effect measured from the other side on the h2h-divergent-2000b-500d fixture: 71 expanded non-thin → 8 with Exclusion::Edges → 0 with Exclusion::Full. On this corpus GUNNAR_EXCLUSION=full changes nothing — 182 either way, byte-identical wire — because the bitmap answered the selection and that knob only reaches the walk. So the vouching set the bitmap path hands to client_has is narrower than the one the walk path builds, and that is the defect.

What has to be written. Widen Selection::client_has on the bitmap path in crates/gunnar-wire/src/bitmap.rs to the full common set the negotiation established, as graph.rs's Exclusion::Full already does for the walk. Roughly 40–80 lines.

The guard, and it must not be a digest. The thin arm and the whole arm emit different bytes — that is the point — so stage-probe wirehash cannot check this the way it checks the extents change. The instrument is the counter pair: GIX_PACK_THIN_VETOED must fall, wire bytes must fall, and a stock git client must still be able to complete the fetch, which is what crates/gunnar-wire/tests/pack_copy.rs and the real_git_interop tests are for. A thin delta whose base the receiver does not hold is not a smaller pack, it is a failed fetch, so the correctness bar here is higher than the last row's, not lower.

Expected win, stated to be falsified: if the vouching set were complete the fetch's deflate goes to zero and its inflate to near it — up to ~55% of cycles, ~2.2× — against a noise band of 1.22×. The honest floor is lower: some of those 182 bases genuinely are absent from the client, and only a measurement after the change can say how many.

3. 64.8% of a push is collision-detecting SHA-1 — and that is probably correct

gix_hash::hasher(Kind::Sha1) is sha1_checked with UBC detection on. performance-design.md Measurement 1 has it at 353.5 MiB/s against RustCrypto's SHA-NI at 1 778.9 MiB/s — 4.9× — and 1 948.1 MiB/s with detection off.

gunnar already fixed this where it owns the code: P-008's gunnar_pack::checksum::TrailerHasher uses RustCrypto for gunnar's own pack trailer, which is why a fetch's profile shows sha1::compress::x86::digest_blocks (hardware) at 13.72% and no sha1_checked at all. copy.rs's module docs record rejecting FromEntriesIter for precisely this reason.

The push path cannot reach it. Bundle::write_to_directory builds its hasher internally and takes no parameter for one.

Verdict keep. Not because it is cheap — it is the single largest term in the most expensive command — but because this is the receive path, the input is untrusted, and detecting a SHAttered-class collision in a pushed pack is what the cost buys. Stock git makes the same call (DC_SHA1 is its default), and gunnar's push is already 2.45× faster than gitea at 3.63× less CPU, so this is not where gunnar is losing. It is recorded as a row rather than dropped because it answers "why does a push cost 8.6× a fetch per object", and because if it is ever revisited the lever is a gix API that admits a hasher — not anything in gunnar.

4. The rows that earn their keep

Stated with numbers, because a census that recommends replacing everything is worthless and so is one that keeps everything by default.

• gix_pack::index::File::lookup — 3.59 probes per object served on a fetch and 0.85% of cycles here, against the fanout lane's independently measured 0.93% for all oid-keyed probing. Two lanes, two instruments, same order. A 16-bit fanout moves this seam 4.37× and the request 1.0016–1.0076× — inside its own noise. Keep; the fanout lane's "built, correct, off by default" stands.
• gix_object::FindHeader::try_header — 0.445 calls per object. This was read() + buf.len(), a full inflate of the object and every base under it, and is now decode_header, which walks entry headers only; it took a znippy fetch from 2.2–2.6× to 1.17–1.26×. Already done — and it is the template the two landed wins share: a gix method whose cost is wrong for how gunnar uses it, replaced by the metadata that already answers the question.
• gix_object::CommitRefIter / TreeRefIter — the whole graph walk lives inside select.bitmap's 9.41%, and that stage issues 628 store reads totalling 0.22 MiB. The Token::Parent early-exit is already the right shape. Keep.
• gix_ref — 3 calls per request, ≪0.5% on every command including the one that is nothing but refs. ArchiveRefStore already serves ls-remote from __gunnar_refs__ with no filesystem. Already done; untouched here.
• gix_object::Exists::exists on push — 1 412 calls per push inside connectivity, 6.66% of attributed CPU. Real, an order below pack_ingest, and doing necessary work. Keep.
• gix_odb::Handle::location_by_oid — now 1.000 per request rather than per object, and 0.70% of cycles. Already done (fc2e0c3).

5. What could not be measured, and what it would take

Named rather than estimated. LAW 7.

1. pack_object_ids — gix_pack::index::File::at + ::iter, once per push. gunnar-cli/src/server/transport.rs:886,889 re-opens the .idx that Bundle::write_to_directory just wrote and iterates every entry, to subtract the thin bases and hand the policy gate the introduced set. bench/src/stages.rs::push uses its own Ingest and does not run it, so the census's push column reports GIX_PACK_INDEX_OPEN 0 — a measured zero for the harness and a real 1/push in production. It is O(objects in the pack) and runs immediately after a call that already had every id in hand. To measure it: make stages::push drive gunnar-cli's PushIngest rather than its own, which means making that type reachable from bench/. Half a day, and the row is worth it only if the answer is large.
2. gix_pack::Find::pack_offsets_and_oid — called by iter_from_counts when it must name a thin base's oid; O(objects in the pack) per call, cached per pack per chunk. gunnar's impl at copy.rs:452 delegates without a probe. One bump away, and it is upstream of §2: if the vouching set widens, this is called more, not less.
3. The cache.put asymmetry. gix-pack inserts one entry per decode (decode/entry.rs:442) where git inserts every intermediate on the way back up (packfile.c:1715-1719) — O(1) against O(depth). It can only matter where delta chains are decoded, which on a fetch is the 182 vetoed objects and on a push is inside Bundle::write_to_directory. Unknown, and it is downstream of §2: fix the veto and most of those decodes stop happening. Measure after, not before.
4. iter_from_counts's third thread and entry_by_location's to_vec. Another lane owns replacing both; not duplicated here. Included with its numbers: the thread that thread_limit: Some(1) does not prevent is why 40.6% of server CPU is the kernel futex hash-bucket spinlock, and to_vec allocates once per object served — 1.000/object, confirmed by this lane's GIX_PACK_ENTRY_BY_LOCATION on both clone and fetch. The 8.99% of memmove/memset in the current fetch profile is that copy, and it is now the third-largest row.
5. Multi-pack. Every corpus repository has exactly one pack. The extent table that landed at fc2e0c3 handles multi-pack by consulting only the pack gix-odb would have searched first, and its commit message records the two-copies-of-one-object hazard that forced it. This census did not re-verify that; packed-1000000o-64p-v2 exists as a fixture and would.
6. Cold tier (GUNNAR_COLD_TIER=znippy). gunnar-coldtier's ZnippyFind has its own copy of the defect fc2e0c3 just fixed — crates/gunnar-coldtier/src/objects.rs, decompress_entry called from location_by_oid to learn an entry's on-disk length. Not on the default serve path and not measured here, but it is the same fix and LAW 5 says one writer.

6. Red probes: every guard here has been seen to fail

LAW 2. Three, and all three fired.

1. loopfetch refuses to profile the walk. It runs one instrumented fetch before the loop and calls StageRun::expect_path(SelectPath::Bitmap). Proven red with GUNNAR_BITMAP=off. Without it the §0.1 finding would be the same unfalsifiable measurement it was sent to check. (This lane first wrote its own rule — select.bitmap present, select.tree_closure absent — and then found stages.rs had grown a stricter one that also reads whether a .bitmap was on disk, so a repository that has one and did not use it is named as such. The local copy was deleted in favour of it; LAW 5.)
2. entrysize's equivalence check. GIXCENSUS_RED=lastentry drops the trailing-hash sentinel and it fails on the last entry of every repository and nothing else: "22b463a3… at offset 2157969: the index says 64 bytes, the inflate says 44" — 20 bytes, the SHA-1 width. Green, it proves the index-derived entry length identical to the inflated one for 65 737 entries across all 19 repositories, zero mismatches, which is the independent confirmation that fc2e0c3's rule is sound. Seam ratio 41.7–233.4×; the pack-order table costs 14.8–18.3 ns per object to build, once per store open.
3. A guard this lane needed and shipped without. The three entry-kind counters read 0 for two full runs because the bump call site never landed in copy.rs. GIX_PACK_KIND_BASE 0 is shaped exactly like "this fetch emitted no base entries", which is a plausible sentence. check_entry_partitions now asserts that base + delta_ref + delta_oid and copied + recompressed each equal the objects served — two independent partitions of the same entries. A missing bump breaks the first; a miscounted veto breaks the second.

7. What this lane changed in the tree

Instrument only. No serve-path behaviour changed, no verdict acted on.

• crates/gunnar-pack/src/oidprobe.rs — 13 new counters (9 → 22) and bump_by, all behind the existing oidprobe feature.
• crates/gunnar-pack/src/copy.rs — 6 bump sites: entry kinds, the copied/recompressed split, entry bytes, iter_from_counts, the thin veto.
• crates/gunnar-pack/src/read.rs, crates/gunnar-store/src/fs.rs, crates/gunnar-store/src/gix_adapter.rs, crates/gunnar-cli/src/server/transport.rs — one bump each.
• bench/src/stages.rs — ls_remote (the fourth command had no driver at all) and serve_on, which is wire_digest_on minus the FNV pass. That pass is byte-at-a-time over a 6.59 MB response and was ~8% of the first profile, all of it harness, sitting in the denominator of every share reported. wire_digest_on is now defined in terms of serve_on, so the digest the fidelity check compares is a digest of exactly the bytes the profiled loop produced (LAW 5).
• bench/examples/gix-census.rs + its [[example]] stanza — new.

Reproduce:

cd bench
cargo build --release --example gix-census \
  --target-dir /home/rickard/scratch/cargo/gunnar-gixcensus-bench
B=/home/rickard/scratch/cargo/gunnar-gixcensus-bench/release/examples/gix-census
F=/home/rickard/scratch/gunnar-bench-fixtures/real-corpus
R=$F/znippy.git
TIP=$(git -C $R for-each-ref --format='%(objectname)' refs/heads/ | head -1)
OLD=$(git -C $R rev-parse $TIP~200)

CORPUS=$F taskset -c 24-27 $B counts 200                     # §1, all four commands

taskset -c 24-27 perf record -F 4999 -o on.data  -- $B loopfetch $R 60 $TIP $OLD
GUNNAR_PACK_EXTENTS=off \
taskset -c 24-27 perf record -F 4999 -o off.data -- $B loopfetch $R 60 $TIP $OLD
perf report --stdio --no-children -g none --percent-limit 0.5 -i on.data --sort symbol

taskset -c 24-27 perf record -F 4999 -o push.data -- $B looppush $R /home/rickard/scratch/pw 8 $TIP
perf report --stdio --no-children -g none --percent-limit 0.6 -i push.data --sort symbol

for d in $F/*.git; do taskset -c 24-27 $B entrysize $d 100000; done   # §6.2
GIXCENSUS_RED=lastentry $B entrysize $F/skade.git                     # its red
GUNNAR_BITMAP=off       $B loopfetch $R 6 $TIP $OLD                   # §6.1's red

8. The ranked list, which is the deliverable

At 1a4df9a, current tree. Twenty-two counted entry points; two are worth writing code for, and one of those is already somebody else's.

Everything below rank 5 is under 1% of any command and the census says keep or already done. The one open row worth writing is #2, and its own numbers say it is worth up to ~2.2× on a negotiated fetch against a 1.22× noise band — which is exactly the shape of claim this document exists to make falsifiable rather than to assert.