nordisk/edda

What git actually does, what the forges measured, and whether Arrow buys gunnar anything

Research lane, 2026-08-04. No production code was written. Every claim below is tagged MEASURED BY (someone ran it and published the number), ASSERTED BY (someone stated it without a measurement), or INFERRED (mine, from reading source).

Primary source for all file:line citations: git.git at 5b247172, v2.55.GIT, 2026-08-03, cloned to /home/rickard/scratch/research/git.git. Line numbers move; the commit does not.

0. The one thing to read if you read nothing else

gunnar's only measured loss is fetch wire bytes, 1.2458× against gitea (.nornir/forge-bakeoff-benchmarks.md, oden 2026-08-04T13:29:40Z). That doc explains it as "gunnar has no server-side delta compression, so a fetch that a Go forge answers with a thin pack is answered by gunnar with whole objects."

I checked that explanation and it is wrong, and the table disproves it.

• S-003 (thin-pack emission, crates/gunnar-wire/src/upload_pack.rs:1269-1304) landed in fab91d4, 2026-08-04 13:18:03Z. The bake-off ran at 13:29:40Z and git merge-base --is-ancestor fab91d4 14e49e6 says yes. The 1.2458× is a post-S-003 number.
• The fetch row's own columns say gunnar sent a thin pack: wire 369 602 B, on-disk after index-pack --fix-thin 418 630 B. --fix-thin only adds bytes when bases were omitted. gunnar omitted 49 028 B of bases; gitea omitted 87 084 B (296 717 → 383 801).

So the loss is not thin-pack support. gunnar's thin pack elides 11.7% of its objects' bytes as external bases; gitea's elides 22.7%. The residual gap is delta search quality — which objects get deltified against what — and nothing else. That is a much more specific problem than the doc states, and it points at exactly one place.

The ceiling on that place is not 1.0×, it is 3–29×, and it lives in the pack you copy from. Because gunnar serves by copying pack entries verbatim, the delta quality of every answer is the delta quality of the on-disk pack. Stolee measured git repack -adf --path-walk against plain repack -adf: microsoft/fluentui 438 MB → 148 MB (2.96×), a private monorepo 130 049 MB → 4 432 MB (29.3×). MEASURED BY Derrick Stolee, path-walk cover letter, 2024-10. https://www.spinics.net/lists/git/msg488026.html

gunnar's architecture makes the repacker the only place serve performance can come from, and the repacker is where git has moved furthest in the last three years.

(One housekeeping item for whoever owns that file: the prose at .nornir/forge-bakeoff-benchmarks.md:54-57 and :126-131 is stale — it says "gunnar sends no thin pack" while the table two rows up shows it did. Not my file to edit.)

1. How git works, from the source

1.1 The packfile is the delta store, and gc is the encoder

gunnar measured this and got it right: a serve-time delta encoder is worth 4.795× on a freshly-pushed bare repo and exactly 1.0000× after gc — 1 057 553 bytes both ways, byte-identical, nine of nine corpora (.nornir/cold-tier-decision.md:84-94).

git's name for that is --reuse-delta, on by default in pack-objects. Once repack has found a delta, every later pack copies it rather than recomputing it. The corollary git states explicitly and gunnar has not written down:

"This is an important optimization for serving fetches, because it means the server can avoid inflating most objects at all and just send the bytes directly from disk. This optimization can't work when an object is stored as a delta against a base which the receiver does not have (and which we are not already sending). In that case the server 'breaks' the delta and has to find a new one, which has a high CPU cost." — Documentation/git-pack-objects.adoc:417-426

So the 1.0000× finding is not "delta encoding is worthless." It is "delta encoding is a maintenance-time cost, not a serve-time one." gunnar has never run a maintenance pass in a benchmark. That is the gap, not the encoder.

1.2 git's verbatim reuse is P-001 plus four things gunnar does not do

builtin/pack-objects.c:1098-1310, entry points write_reused_pack_verbatim and write_reused_pack.

The word-granular trick is load-bearing and cheap: the front of a bitmapped pack is, by construction, the objects reachable from the preferred tip, so a clone's result bitmap is a solid run of ones and git memcpys the whole prefix without looking at a single object header.

Multi-pack reuse is worth 6.50× ± 0.03 on a repo deliberately fragmented into ~100 packs, git pack-objects --revs --stdout --use-bitmap-index --delta-base-offset, hyperfine 10 runs: 6.234 s ± 0.026 single-pack vs 959.5 ms ± 2.4 multi-pack. MEASURED BY Taylor Blau, cover letter v2, 2023-12. https://patchwork.kernel.org/project/git/cover/cover.1702592603.git.me@ttaylorr.com/

INFERRED: gunnar has crates/gunnar-store/src/midx.rs and crates/gunnar-bitmap/src/pack_order.rs, so the two prerequisites exist. The 6.50× only appears on a repo with many packs, which is exactly what a forge that never runs gc becomes.

1.3 The delta base cache — the answer to the 12.4× chain question

A lane measured that a 50-link delta chain costs 12.4× a single delta in znippy, and blamed the absence of a delta base cache. That diagnosis is right. Here is git's actual design:

• Where: packfile.c:1167-1325, a hashmap keyed on (struct packed_git *p, off_t base_offset) plus an intrusive LRU list (delta_base_cache_lru).
• Size: DEFAULT_DELTA_BASE_CACHE_LIMIT = 96 * 1024 * 1024 — pack-objects.h:12. 96 MiB per thread, tunable as core.deltaBaseCacheLimit (Documentation/config/core.adoc:447).
• Eviction: LRU, evicted from the head until delta_base_cached <= limit, after the new entry's size has already been added — packfile.c:1303-1311. Note that this admits an entry larger than the whole limit and then evicts everything else; there is no admission control.
• What it caches: not just the bottom of the chain. unpack_entry (packfile.c:1523-1728) is three phases — drill down pushing each link on a stack, inflate the base, then apply deltas back up — and phase 3 inserts every intermediate result into the cache as it goes (packfile.c:1715-1719). Reading object N of a chain therefore leaves links 1..N-1 cached.
• Phase 1 short-circuits on a hit (packfile.c:1544-1553) and detaches the entry (removes it from the cache, transfers ownership), so a hit is not a copy. The read path that does not consume it, cache_or_unpack_entry (packfile.c:1252-1267), copies with xmemdupz instead.

This is why git's chains are cheap and znippy's are not. git's default depth is 50 (builtin/pack-objects.c:230) with a window of 10 (:228) — the same 50 the znippy measurement used. Walking a pack in pack order, every object's base was almost certainly just materialised by the previous object, so the amortised cost of a 50-link chain is ~1 inflate + 1 patch, not 50. A reconstructor without the cache pays O(depth) per entry, i.e. O(depth²) for a range — and 12.4× is roughly what you get for depth 50 with locality.

git has a perf test for exactly this: t/perf/p0003-delta-base-cache.sh, whose header notes the effect is emphasised by long chains from gc --aggressive "though cache is still quite noticeable even with the default depth of 50."

1.4 Delta islands — gunnar has nothing, and probably should not yet

delta-islands.c (556 lines). Each ref is assigned to an island by regex (pack.island, capture groups form sub-islands); island_bitmap marks propagate down the object graph; pack-objects then refuses a delta whose base is not present in all of the delta's islands. builtin/pack-objects.c:2672-2676 folds island_delta_cmp into the delta-search sort so island-mates cluster.

The problem it solves is narrow and specific: one object store shared by many forks, or many refs/pull/* / refs/changes/* outside refs/heads. Absent either, git says it "tends to work automatically" (Documentation/git-pack-objects.adoc:428-431).

ASSERTED BY git docs, no published number: islands produce "slightly larger packs (because we miss some delta opportunities), but guarantee that a fetch of one fork won't require large amounts of data from another fork's objects."

GitLab's Gitaly tracked this as issue #1418. https://gitlab.com/gitlab-org/gitaly/-/issues/1418

My call: not worth it for gunnar now. gunnar is one repo per store, no fork network, no refs/pull. Islands are a cost with no upside until a fork network exists. Revisit if gunnar ever shares an object store across repos.

1.5 The name-hash, and --path-walk — the highest-value finding in this report

git's delta search does not compare all pairs. It sorts candidates by (type, name-hash, preferred_base, [island], size desc) — type_size_sort, builtin/pack-objects.c:2651-2690 — and then slides a window of 10 over that order. The "name hash" is a hash of the path, and version 1 depends primarily on the final 16 bytes of the path (Documentation/git-pack-objects.adoc:378-386). In a monorepo with many index.ts/BUILD/package.json at different depths, that collides catastrophically and the window fills with unrelated files.

Two fixes shipped:

• name-hash v2 (git 2.49): considers each path component separately with a shift, salting the low bits with parent directory names (Documentation/git-pack-objects.adoc:388-396). Not allowed when writing bitmaps — it silently downgrades to v1.
• --path-walk (git 2.51): abandons revision order entirely for the first pass and emits all objects at a given path together, then a second pass across paths (Documentation/git-pack-objects.adoc:398-406).

MEASURED BY Stolee, git repack -adf vs git repack -adf --path-walk:

Same source; only fluentui is reproducible. He also reports a private monorepo where git push sends 60–100 MB and path-walk packs the same objects into a thin pack under 1 MB. That last one is an assertion in prose, not a table.

INFERRED, and this is the actionable part: gunnar's fetch loss is measured on a small fixture where the delta search barely matters. On a real monorepo the difference between a good and a bad repacker is a factor of 3 to 29, and gunnar's serve path inherits it entirely. Whatever encoder gunnar ends up with, the ordering is worth more than the encoder.

1.6 Reachability bitmaps: git's selection policy

pack-bitmap-write.c:1014-1101.

Commits are sorted newest-first, then next_commit_index(i) decides the stride:

idx <= 100          → stride 0   (bitmap EVERY commit)
idx <= 20 000       → stride min(idx-100, 100)
idx  > 20 000       → stride clamp(idx-20000, 100, 5000)

— pack-bitmap-write.c:1014-1036. Within a stride, a commit already flagged NEEDS_BITMAP wins; failing that, the last merge commit in the window wins (:1078-1090). Fewer than 100 commits total → bitmap all of them (:1052-1058).

Three storage tricks on top:

• XOR chaining: each stored bitmap may be written as a XOR against one of the previous 10 (MAX_XOR_OFFSET_SEARCH = 10, pack-bitmap-write.c:347), picking the offset that minimises the result.
• Name-hash cache (BITMAP_OPT_HASH_CACHE, 4 bytes/object at the end of the .bitmap): exists specifically to feed the delta heuristic "when serving a fetch between an older, bitmapped pack and objects that have been pushed since the last gc" — Documentation/config/pack.adoc:181-188. That is verbatim gunnar's fetch scenario.
• Lookup table (BITMAP_OPT_LOOKUP_TABLE): <commit_pos, offset, xor_row> triplets so a reader can seek to one commit's bitmap instead of parsing all of them — Documentation/technical/bitmap-format.adoc:76-84.

Pseudo-merge bitmaps (git 2.47, experimental) cover the many-refs case: one bitmap pair per group of refs, (commits bitmap, reachability-closure bitmap), grouped by regex with a decay function f(n) = C·n^-k, default maxMerges = 64, threshold = 1.week.ago, stableThreshold = 1.month.ago (Documentation/config/bitmap-pseudo-merge.adoc, background in Documentation/gitpacking.adoc:27-98). ASSERTED BY git docs; no public number.

What bitmaps were worth, measured. GitHub, 2015-09: object counting "that used to take several minutes of CPU time … now takes less than 3 ms"; a Linux-kernel clone "can take up to 8 minutes of CPU time"; the bitmap index "takes between 5 and 10% of the original size of the repository"; and after deployment "average CPU time spent on Git network operations was reduced by more than 90%." MEASURED BY Vicent Martí, GitHub. https://github.blog/open-source/git/counting-objects/

gunnar's own 0.278× and the 1598→18 inversion sit comfortably inside that. The 90% figure is the one to compare against, and gunnar has not measured serve-time CPU on a repo big enough for it to show.

1.7 MIDX, .rev, .mtimes — what they are actually for

• .rev (Documentation/gitformat-pack.adoc:321-339): a table of index positions sorted by pack offset. It is the inverse of the .idx, and it is what makes "given a byte offset in the pack, which object is this, and where does it end" O(1). Without it, verbatim entry copying cannot find an entry's length. This is crates/gunnar-bitmap/src/pack_order.rs in gunnar — already present.
• MIDX (Documentation/technical/multi-pack-index.adoc): O(log N) object lookup across many packs, "for some large repositories, repacking into a single packfile is not feasible." Keeps one record per oid, resolving duplicates in favour of the preferred pack, else most-recently-modified. Incremental MIDX chains exist but do not yet support reachability bitmaps (same doc, "Design state").
• .mtimes: per-object mtimes, used by cruft packs so unreachable objects can age out without becoming loose.

MEASURED BY GitHub, on their monorepo, for cruft-free MIDXs (git 2.51): 38% smaller MIDX, 35% faster writes, 5% faster reads. https://github.blog/open-source/git/highlights-from-git-2-51/

1.8 Generation number v2 — why topological levels were not enough

Documentation/technical/commit-graph.adoc:37-125.

v1 = topological level: root = 1, else 1 + max(parents). v2 = corrected committer date: root = its committer date, else max(own date, 1 + max(parents' corrected dates)); a root at timestamp 0 gets 1 so it is distinguishable from "uncomputed."

Both satisfy the cutoff property (gen(A) <= gen(B) ⟹ A cannot reach B), so why change? Because v1 throws away the date. git's pre-commit-graph heuristic was "if commit-time(A) < commit-time(B) then A probably cannot reach B," which is a good heuristic and is used for git log's default order. A topological level cannot recover it, so a v1 graph made some walks worse than no graph at all. v2 is a generation number that is also a monotone repair of the commit date — it keeps the cutoff guarantee and stays usable as a date-ordered priority key. Storage: 31-bit offset from the commit date in the GDAT chunk, with a 64-bit overflow chunk (Documentation/gitformat-commit-graph.adoc:101-125).

Mixed chains are the hazard: old git writing a split graph without corrected dates on top of a new one, so the format forbids naively taking the newest layer's number (same doc, :301-307).

gunnar has no commit-graph and no generation numbers (grep -ril commit_graph crates → empty). Its reachability walk is crates/gunnar-wire/src/graph.rs plus bitmaps.

1.9 reftable — the direct competitor to gunnar's Arrow ref log

This is the comparison Rickard should care about most, and it is not Iceberg.

Documentation/technical/reftable.adoc states the problem in gunnar's own terms: "Atomic pushes modifying multiple references require copying the entire packed-refs file… Support atomic push with O(size_of_update) operations."

That is exactly gunnar's design goal, and gunnar's answer (one Arrow RecordBatch per push, append-only, periodic compaction into snapshot + bounded reflog window, old batches archived —crates/gunnar-store/src/arrow_log.rs:1-100) is structurally the same answer reftable gives: append a new table, compact periodically, keep the log as the reflog.

Where they differ, with reftable's own published numbers (866k refs, android):

Documentation/technical/reftable.adoc:50-84. MEASURED BY Shawn Pearce (originally JGit, ~2017); the table is carried in git.git unchanged.

The mechanism behind the 33.9 µs is sorted, prefix-compressed, variable-size blocks with a block index — binary search into a cold file, no full read.

gunnar's log is not sorted and not indexed; the current state is obtained by replaying every batch into an in-memory map (arrow_log.rs:21, "replay, last-wins per name. Refs are few, so this is instant"). Its own measurement: 680 bytes of framed IPC per single-ref push, and 180 000 pushes ≈ 122 MB replayed at open (arrow_log.rs:28-35). Compaction bounds it — MEASURED oden 2026-08-04, 10 000 pushes: 6 800 448 B uncompacted → 541 744 B at the default 1024/256 policy → 3 864 B fully folded (arrow_log.rs:40-46).

INFERRED: the model is sound and the compaction numbers are good. The exposure is the constant: reftable amortises to 33 bytes per ref; gunnar's framed IPC message is 680 bytes per single-ref push, ~20×. That is invisible at 50 refs and fatal at android's 866 000. If gunnar ever needs a large-ref-count repo, the answer is not a bigger Arrow batch — it is a sorted, block-indexed on-disk layout, i.e. reftable, or an Arrow file with a sorted key column and an offset index.

1.10 Arrow IPC alignment: gunnar's own best Arrow datum

crates/gunnar-store/src/arrow_log.rs:540-551 drops IpcWriteOptions alignment from Arrow's recommended 64 bytes to 8. MEASURED oden, 10 000 pushes: 15 360 448 B at 64-byte alignment → 6 800 448 B at 8-byte — a 2.26× size tax that the alignment recommendation was charging, for a file nothing mmaps. Batch counts unchanged, which is what proves the change moved bytes and not policy.

This belongs in the Arrow section below because it is the clearest measured statement anyone in this repo has about what Arrow costs: on many small batches, Arrow's zero-copy alignment guarantee is a 2.26× size multiplier you are paying for a property you are not using.

1.11 Thin packs, partial clone, promisors

• Thin pack is not an optimisation, it is the fetch protocol's normal mode. gitea/Forgejo exec git pack-objects --revs --thin --stdout --delta-base-offset --include-tag (observed by podman top during the bakeoff, .nornir/forge-bakeoff-benchmarks.md). git's own fetch perf test (t/perf/p5311-pack-bitmaps-fetch.sh) always passes --thin.
• Partial clone (Documentation/technical/partial-clone.adoc): blob:none, blob:limit=, tree:0 filters, promisor remotes, on-demand fetch. gunnar has crates/gunnar-wire/src/filter.rs, which the module doc says must never accept a filter and then pack everything anyway — good instinct, that is the classic silent green.
• Large Object Promisors (Documentation/technical/large-object-promisors.adoc) is git's in-flight alternative to LFS: a promisor remote holding only large blobs, server-side. gunnar has crates/gunnar-lfs. LOP is where upstream is going; worth watching, not worth chasing.

2. What the big forges published

Everything here is someone else's measurement unless marked otherwise. Where a famous claim turns out to have no measurement behind it, that is said.

2.1 The two largest verified wins in the whole corpus are both selection, not infrastructure

Bitmaps, and they came from JGit, not GitHub. Colby Ranger implemented reachability bitmaps in JGit at a 2013 hackathon with Shawn Pearce; Vicent Martí and Jeff King ported them to C git for 2.0. The original measurement is Ranger's Gerrit User Summit 2012 deck, linux kernel, 2.7M objects / 335k commits: https://docs.google.com/presentation/d/1t2qzSO4z5MbrN21Cz4RrLWz9IZ5ylJNwQassajKyPk8/htmlpresent

Also: full bitmaps 1 MiB total / 26.8 KiB avg; XOR-compressed 2.4 MiB / 9.8 KiB avg (~60% saving); hybrid index overhead ~1%; bitmaps every 100 commits recent, 10 000 for old history — which is within a factor of two of what next_commit_index still does in 2026 (§1.6).

The fetch-1-commit regression is the row that matters to gunnar. Ranger measured bitmaps making the nearly-current client slower. gunnar measured the opposite: 10 commits behind cost 1598 store touches on the exact walk and 18 with a bitmap. Both can be true — Ranger's classic path had a cheap early-exit that gunnar's exact walk apparently does not — but it means gunnar's 0.278× headline is not the shape the literature expects, and it deserves a second look rather than a victory lap.

The C-git port's numbers, in the commit message of 6b8fda2 (Git 2.0), full torvalds/linux, 3 237 103 objects: no bitmaps 44.111 s; bitmaps, no pack reuse 5.413 s; bitmaps + verbatim pack reuse 1.636 s — ~27× wall clock. https://github.com/gitster/git/commit/6b8fda2

Note the middle row. Verbatim pack reuse is worth 3.3× on top of bitmaps in git's own founding benchmark. That is P-001, measured by git in 2013.

Path-walk / name-hash v2, §1.5. GitHub's own Git 2.49 highlights gives the timing the cover letter omitted: microsoft/fluentui repack with --name-hash-version=2 went ~96 s → ~34 s and the pack 439 MiB → 160 MiB. MEASURED BY Taylor Blau, 2025-03. https://github.blog/open-source/git/highlights-from-git-2-49/ The path-walk series gets fluentui to 148 MB by a different route, so the two land in the same place — INFERRED: they are attacking the same defect (name-hash v1 collisions), not stacking.

2.2 GitHub

MEASURED BY Vicent Martí, 2015-09, production: counting objects "several minutes of CPU time → less than 3 ms"; bitmap index 5–10% of repo size; >90% reduction in average CPU time on git network operations. https://github.blog/open-source/git/counting-objects/

MEASURED BY Taylor Blau, 2021-04, production (https://github.blog/engineering/scaling-monorepo-maintenance/):

• linux kernel full object traversal >33 s → 1.57 s with bitmaps
• .rev reverse indexes, fleet-wide: peak-24h CPU ~10.8 s → ~7 s, ~35%
• Homebrew/homebrew-core fetch CPU ~80% reduction at p50
• multi-pack bitmaps in production: 5.67 CPU-days saved per hour
• average repack ~1 min → 15 s after geometric repacking

Careful with units: "5.67 CPU-days per hour" and GitLab's "5.2 → 4.2 hours-of- repacking per hour" are accumulated CPU per wall hour, not latency.

MEASURED, Git 2.51, GitHub's main monorepo, cruft-free MIDX: 38% smaller MIDX, 35% faster to write, ~5% faster reads. Repo unnamed, no absolutes. https://github.blog/open-source/git/highlights-from-git-2-51/

Spokes/DGit is thin. Reynolds 2016 gives 3 replicas, writes need 2/3, and a CPU load "~3× lower at peaks" read off a graph; Haggerty 2017 gives cross-continental RTT 60–80 ms and a ref update still costing ~1/3 second after optimisation, 4 round-trips per 3-phase-commit. Everything else is assertion. https://github.blog/engineering/architecture-optimization/introducing-dgit/ · https://github.blog/2017-10-13-stretching-spokes/

Stolee's "Git's database internals" (5 parts, 2022) contains essentially no measurements, and part V none whatsoever. Cite it for mechanism, never for figures.

The best public partial-clone data set is Abbaspoursani 2020-12, https://github.blog/open-source/git/git-clone-a-data-driven-study-on-cloning-behaviors/. torvalds/linux clone avg round-trip: full 5 min (60 s git CPU); --depth=1 1.2 min; --filter=tree:0 2.4 min; --filter=blob:none 3 min. And the counter-intuitive result worth remembering: shallow fetches are 4–25× slower than full fetches (linux: 6 000 ms vs 350 ms) because computing the shallow boundary is expensive.

2.3 GitLab — the best-measured forge write-up that exists

Jacob Vosmaer, 2022-01, "How we reduced Git fetch CPU 9x". https://about.gitlab.com/blog/git-fetch-performance/

Method, stated: 20 simultaneous shallow clones of the same commit, 87 MB pack (1 740 MB egress), GCE c2-standard-8, GitLab FOSS 13.7.9 vs 14.6.1. Results: wall clock 27 s → 7.5 s (3.6×), server CPU 100% → 40%, and a 30 s / 99 Hz CPU profile showing busy samples 21 720 → 2 328 (9.3×), of which git itself 18 552 → 923. The wins came from the pack-objects cache (GitLab 13.11), RPC transport work (14.5), and git 2.31 / 2.34 / 2.35 upstream.

This is the template gunnar's bake-off should copy: a concurrency-N workload, CPU profile broken out per process, and named versions on both sides.

Patrick Steinhardt, 2023-11, geometric repacking + cruft packs in production (https://about.gitlab.com/blog/2023/11/02/rearchitecting-git-object-database-mainentance-for-scale): fleet-wide repacking time ~5.2 h/h → ~4.2 h/h (−20%); the gitlab-org monorepos ~0.7–1.0 h/h → 0.12–0.15 h/h (−80%); and an artificial benchmark packing loose objects on linux.git ~13 s → 174 ms (>90×). Rollout 26–28 April 2023, two-week before/after windows.

GitLab adopted delta islands and never published a before/after. Gitaly #1418 and the #1575 rollout issue contain a checklist and the assertion "over time the performance of deduplicated forks will degrade" — no storage or CPU outcome. https://gitlab.com/gitlab-org/gitaly/-/issues/1418 · https://gitlab.com/gitlab-org/gitaly/-/issues/1575

Gitaly's cgroups work has motivating incident data but no benchmark: one snapshot showed 218 git + 19 ruby = 251 processes in a single communal cgroup with a 90 G memory limit, and the pre-cgroups mitigation was to rate-limit SSHUploadPack from 120 → 15 concurrent gRPC calls. https://gitlab.com/gitlab-org/gitaly/-/issues/3049

Gitaly Cluster / Praefect: no published benchmarks at all — only requirements (single-digit ms network latency, health checks within 1 s).

2.4 Microsoft

Brian Harry, 2017-05, the canonical Windows-repo figures: ~3.5M files, ~300 GB,

250 000 commits in 4 months, ~3 500 of ~4 000 engineers, 8 421 pushes/day. With GVFS, 80th-percentile status 10–15 s, checkout ~10 s. Clone: Redmond with proxy 127 s; North Carolina ~25 min → 70 s with a proxy. https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-planet/

The famous pre-GVFS baseline — git clone >12 hours, git status ~10 minutes — is from the 2017 GVFS announcement and is reported without methodology.

There is no public, methodologically clean "Scalar vs plain git clone of a 300 GB repo" number. The Scalar story post gives one hard figure — sparse-checkout pattern matching took 40 minutes on the monorepo before cone mode, and cone mode gives 3–4 s git status — and otherwise asserts that partial clone + fsmonitor "reached target performance." https://github.blog/2022-10-13-the-story-of-scalar/

fsmonitor is measured properly, Hostetler 2022-06, Git 2.37: git status on Chromium (393k files) 17.6 s → 0.827 s; synthetic 2M files 85.1 s → 0.750 s; refresh_index alone at 2M files 73.2 s → 0.096 s. https://github.blog/engineering/infrastructure/improve-git-monorepo-performance-with-a-file-system-monitor/

2.5 JGit / Gerrit — and where their numbers stop

Beyond Ranger's 2012 bitmap deck (§2.1), Google published almost nothing measurable. Shawn Pearce's 2017 "DFS reftable at Google" post gives exactly one number — 866k refs in a Gerrit repo, the reason their ref store stopped coping — which is the same 866k in reftable's own table (§1.9). https://www.eclipse.org/lists/jgit-dev/msg03389.html

org.eclipse.jgit.benchmarks (JMH) exists and publishes no results. There is also a jgit-dev thread titled "Writing packs with bitmaps sometimes slower than without" that is worth reading before assuming bitmaps are free: https://www.eclipse.org/lists/jgit-dev/msg03811.html

2.6 gitoxide — the closest peer, and the reason gunnar's push arm wins

All self-reported by Sebastian Thiel, hardware stated, hyperfine or /usr/bin/time.

Clone of linux.git (2.65 GB pack, ~9M objects), 2022-11: https://github.com/GitoxideLabs/gitoxide/discussions/579

INFERRED, and I think it is the right reading of gunnar's push arm: gunnar's push win — 2.45× wall, 3.63× less CPU, and RSS 97 MB against gitea's 391 MB — is almost entirely gitoxide's index-pack being faster and leaner than C git's, not anything gunnar's own code does. The memory ratio in particular is the same shape as Thiel's 0.54×. That is not a criticism; it is a statement about where the win is banked and therefore where it can be lost.

And the honest counter-case, which gunnar should copy the discipline of: on an M1, gitoxide's index-pack loses to git — 43.29 s vs 31.14 s, 207 s user vs 90 s, 2.06 T instructions vs 496 G — because git uses ARM's SHA-1 crypto extensions. https://github.com/GitoxideLabs/gitoxide/discussions/46 gunnar's bake-off is x86-only. On ARM the push arm may invert.

Thiel also reports gix blame up to 30% slower than git where commit-graph caches are used https://github.com/GitoxideLabs/gitoxide/discussions/1855 — a concrete case where §1.8's commit-graph earns its keep. gitoxide's ODB design (LRU object cache → fixed-size delta-base cache → multi-pack) is documented with no numbers: https://github.com/GitoxideLabs/gitoxide/discussions/302

2.7 Codeberg / Forgejo / Gitea

No published performance-engineering write-up with numbers exists from any of them. That is the finding. What exists is a Codeberg incident thread with user-reported timings (a 1.24 KB push taking 1 m 41 s; a concurrency cap of 16 simultaneous pushes that queue-jammed) and a team diagnosis that is asserted, not instrumented. https://codeberg.org/Codeberg/Community/issues/887

Forgejo v12.0 (2025-07) claims Git-op improvements from git merge-tree --write-tree; no before/after numbers.

The structural point for gunnar's bake-off: both forges shell out to stock git pack-objects and do no bitmap-, geometric- or MIDX-aware scheduling of their own. Every optimisation in §§2.1–2.4 reaches them only via whichever git version they happen to invoke. The bake-off is therefore not gunnar-vs-gitea; it is gunnar-vs-git pack-objects-at-whatever-version-and-maintenance-state.

Third-party Gitea-vs-GitLab benchmark posts circulate with numbers like "120 ms vs 200 ms at 50 concurrent users." I could not verify hardware or methodology for any of them. Do not cite them.

3. Arrow and Iceberg, seriously

3.1 What Arrow IPC actually guarantees, and what it does not

From the spec, Columnar.html: https://arrow.apache.org/docs/format/Columnar.html

• Framing. Continuation marker 0xFFFFFFFF, int32 LE metadata length, flatbuffer Message, pad to 8, body. End of stream is 8 bytes: 0xFFFFFFFF 0x00000000. gunnar's "the IPC frame boundary is the transaction boundary" claim is sound.
• Alignment is a recommendation; only 8 is binding. Verbatim: "Implementations are recommended to allocate memory on aligned addresses (multiple of 8- or 64-bytes)". The one normative sentence is "The complete serialized message must be a multiple of 8 bytes so that messages can be relocated between streams." 64 exists for AVX-512. gunnar's 64→8 change (§1.10) is therefore inside the spec, not a shortcut, and it was worth 2.26× on a 10 000-push log.
• Zero-copy evaporates three ways: any body compression, endian byte-swap, or misalignment. Arrow C++ says IPC is "predominantly zero-copy" but "may have to allocate memory in some cases (for example if compression is enabled)" (https://arrow.apache.org/docs/cpp/ipc.html). In arrow-rs the default is require_alignment = false, which means a misaligned array is silently reallocated and copied rather than rejected. A live dev@arrow thread (Raphael Taylor-Davies, 2025-03-27, https://www.mail-archive.com/dev@arrow.apache.org/msg33371.html) says arrow-cpp's Flight client "often produce[s] unaligned buffers" and calls it "a bug not a feature." Cross-implementation zero-copy is something you assert, not something you assume. gunnar's code already knows this — arrow_log.rs:540-544 notes it reads with StreamReader over a Cursor, which copies regardless.
• Arrow IPC has no checksum of any kind. Block is {offset:long, metaDataLength:int, bodyLength:long} and nothing else (https://raw.githubusercontent.com/apache/arrow/main/format/File.fbs). No dev@ thread proposing one exists. Compare git's .idx v2, which carries a CRC32 per packed object, "computed over the entire packed object, including the header … and the entire compressed object." A torn body write with an intact metadata length is undetectable in Arrow IPC and detectable in git. gunnar's torn-write recovery is a message-boundary scan (arrow_log.rs:20); that catches a truncated message and not a corrupted complete one.
• 2 GiB ceiling per Binary/Utf8 array per batch (i32 offsets). Not theoretical: apache/arrow #37069, open since 2023-08-08 — Feather files of 2.02–2.18 GB fail with "Invalid IPC message: negative bodyLength." https://github.com/apache/arrow/issues/37069 LargeBinary is i64.
• The spec warns against exactly what a git archive would do: "Applications that already use compression in their storage or transport protocols should avoid using buffer compression. Double compression typically worsens performance and does not substantially improve compression ratios." git objects arrive zlib-deflated.

Random access is per-RecordBatch and there is no index. The file footer is Footer{version, schema, dictionaries:[Block], recordBatches:[Block], custom_metadata}. That is the entire random-access apparatus: batch-granular byte offsets. No row index, no key index, no statistics, no page index, no Bloom filter — Parquet has all three of the last (tustvold & alamb, 2022-12-26, https://arrow.apache.org/blog/2022/12/26/querying-parquet-with-millisecond-latency/). The "O(1) random access" Arrow advertises is within an already-materialised array (tustvold & alamb, 2022-10-05). Getting from a key to (batch, row) is your problem, and it is precisely what git's .idx fanout plus binary search solves.

Arrow's own FAQ is blunt about the intended lifetime: the on-disk format "does not prioritize the requirements of long-term archival storage," and "Parquet files are often much smaller than Arrow IPC files." https://arrow.apache.org/faq/ DuckDB/Voltron Data/Columnar, 2025-05-23, agree: Arrow IPC is the "better choice for query result transfer and ephemeral caching," Parquet "for archival storage."

Batch size is the whole ballgame, and one-batch-per-push is the worst case. DataFusion defaults to 8192 rows per batch, and the measurement behind it (Dandandan, ARROW-12136, 2021-03, TPC-H SF=10 Q1) is monotone the other way: 32000 rows 996 ms, 16000 828 ms, 8000 715 ms, 4000 685 ms. A minimal schema message is ~56 bytes; an unbuffered writeBatch() costs ~30 syscalls per batch (Khare, 2025-08-12, https://kharekartik.dev/writing/what-arrow-actually-demands-from-you/). gunnar writes one batch per push, which is batch size 1 — hence 680 B for a single-ref push against reftable's amortised 33 B/ref (§1.9). That ~20× is Arrow being used at the bottom of its efficiency curve, not Arrow being used wrong. Compaction is the right mitigation and gunnar already has it.

(One thing that does not apply to gunnar: the growing-footer problem. The Arrow file format accrues a 24-byte Block per batch that must be parsed whole on open; gunnar uses the stream format, which has no footer. It pays with a full replay instead, which compaction bounds. No published measurement of Arrow footer parse cost at high batch counts exists — nobody uses it that way.)

3.2 Is Iceberg a real analogue for git? No, and the spec says why

Read from format/spec.md on main (https://raw.githubusercontent.com/apache/iceberg/main/format/spec.md); the rendered site is client-side and yields only navigation to a fetcher.

Four structural failures, each spec-backed:

1. Merge is unrepresentable. A snapshot has parent-snapshot-id — a single long. There is no second-parent field. The publish primitive is ManageSnapshots.fastForwardBranch, which fast-forwards "if from is an ancestor of to" and throws otherwise; no merge method exists on that interface. The Spark procedures are fast_forward, cherrypick_snapshot, publish_changes, rollback_to_snapshot — nothing named merge. snapshot-log is "a list of timestamp and snapshot ID pairs": a log, not a DAG.
2. No content addressing anywhere. data_file and manifest_file carry no content-hash field. Identity is file_path — a mutable URI. The only hash in the spec is Murmur3 for the bucket partition transform, and write.object-storage.enabled hashes the path for S3 prefix spreading, not the bytes. Two byte-identical files are two paths and two manifest entries. Zero cross-snapshot dedup.
3. Full metadata-root rewrite per commit. Every commit rewrites the whole table metadata JSON — all snapshots, all schemas, all partition specs re-serialised — plus a new manifest list and at least one manifest, then swaps a pointer. git writes new objects plus a ref.
4. Single-pointer optimistic concurrency, with commit.retry.num-retries 4 and total-timeout-ms 1 800 000.

The commit rate is the killer. MEASURED BY Adobe, 2020-12-22: "Initial tests with Iceberg showed a limit of 15 commits per minute per dataset" against a need for "at least 30 times more." Their stated cause is structural, not a bug: "Iceberg is scanning metadata files of the previous snapshot and composing a new snapshot after the data is written… and increases with time as more and more metadata are added." Their fix was to stop committing per write. https://medium.com/adobetech/high-throughput-ingestion-with-iceberg-ccf7877a413f Apache's own Spark Structured Streaming guidance says "use a trigger interval of 1 minute at the minimum," and each batch produces a new snapshot; Flink's worked example assumes 5 minutes. There is no Apache first-party maximum-commits-per-second figure — do not quote one. A forge's unit is one push = one commit.

The tell is that Project Nessie exists specifically to add what Iceberg lacks — "the first technology to provide an open facility for cross-table transactions," and a merge operation. Dremio's own comparison concedes Iceberg's branching is "limited to individual tables." Both git-semantics products bolt on merge, cherry-pick and multi-object atomicity: the three things the spec does not have.

The honest one-liner: Iceberg is a git repo with one file, HEAD-only linear history plus fast-forward-only branches, whose index is rewritten in full on every commit.

What is worth stealing is vocabulary, not format: sequence numbers for ordering concurrent writers, and the position-delete vs equality-delete distinction (a tombstone naming a location vs one naming a predicate). gunnar's ref log already has the ordering.

3.3 Who has actually stored VCS or content-addressed data in columnar formats

Nobody stores git objects in Arrow, Parquet, Iceberg or ORC. Two lanes searched independently, five-plus phrasings each, including the git mailing list. This is a genuine white space, not a search failure — and the nearest neighbours all explain why.

Software Heritage is the only VCS graph in a columnar format at scale, and the columnar copy is always additional to the object store, never a replacement. Blobs live in a separate content-addressed S3 bucket. MSR 2020 (arXiv:2011.07824): 1.13 B revisions, 4.42 B directories, 5.08 B contents, ~1 TB per download format; Athena queries 3 min 40 s scanning 150.68 GB. Current exports, 2026-06-04: 58.7 B nodes / 1.1 T edges, 33 TiB ORC against 16 TiB for the compressed graph. The traversal side is WebGraph/BVGraph at 4.9 bits per arc versus ~8 bytes naive (~13×), 80 ns/arc lookup, ~2 M nodes/s single-threaded. https://docs.softwareheritage.org/devel/swh-export/graph/dataset.html · SANER 2020, Boldi/Pietri/Vigna/Zacchiroli. The purpose-built representation is 2.06× smaller than the columnar one.

Columnar layout actively defeats content-addressed dedup, and the fix had to be bolted on. HF measured (2024-10-05) a 2 GB Parquet file with 1000-row row groups: append 10 000 rows dedupes 99.1% (+20 MB); modify one row dedupes 89% and costs +230 MB; deleting one middle row makes ~half the file new blocks under compression. Cause: "absolute file offsets are part of the Parquet column headers," so any modification rewrites all column metadata even when the data did not change. https://huggingface.co/blog/improve_parquet_dedupe

The fix — content-defined row groups, Krisztián Szűcs, dev@arrow 2025-01, shipped in Arrow/PyArrow ≥21 on 2025-07-25 — is measured over 170 GB of revisions: vanilla Snappy 182.6 GiB → 148.0 GiB stored; CDC Snappy 178.3 → 75.6 GiB; CDC ZSTD 109.6 → 55.9 GiB. Per operation on a ~96 MB file: identical re-upload 0.00 B; insert 1 000 rows 89.8 MB → 6.00 MB; drop a column 37.7 kB. https://huggingface.co/blog/parquet-cdc

Read that as the general law: to get dedup out of a columnar container you must bolt content-defined chunking onto the writer, and even then you land near 50% on shifted data where content-addressed objects get 100% on anything unchanged.

INFERRED, and it validates a choice gunnar already made: this is exactly why the znippy archive must be an Arrow IPC index plus a separate raw blob region, not blobs inside a LargeBinary column. And arrow-rs will not help you — its view-array IPC writer says so in a code comment: "Current implementation just serialize the raw arrays as given and not try to optimize anything," because pruning unreferenced data buffers "is much more nuanced." BinaryView's N-views-one-blob property does not survive serialisation as a dedup mechanism.

Dolt rejected git-as-storage from the other side: "Git storage is not built for fast seek… You really need something close to a B-tree with special version control properties." Their answer is a prolly tree, ~4 KB content-defined chunks. Their compression study is the best piece of self-correcting measurement in this survey: across 170 public DoltHub DBs over 100 MB, zstd dictionaries gave 24.73% mean reduction by first-1000-chunks, 35.44% random-sample, 38.87% grouped by history graph; 79 of 170 DBs had zero groups; DBs with under 10 commits compressed negatively — and they explicitly retracted an earlier "50% on most repos" claim as selection bias. https://www.dolthub.com/blog/2025-04-11-archive-update/ Their post titled "Comparing Dolt and Git" contains zero numbers; do not cite it.

Hugging Face Xet is the best-documented CDC rollout: GearHash, target 64 KB chunks (range 8–128 KB), BLAKE3 chunk IDs, xorbs ≤64 MB. MEASURED: CORD-19 over 50 incremental updates, storage 8.9 → 3.52 GB (−60%), download 51 → 19 min; gemma-2-9b-it-GGUF 191 GB → ~97 GB (−49%); observed Hub dedup 30–85%; a 5 GB SQLite DB with a 1 MB delta went 13 min → ~0.1 s. https://huggingface.co/blog/from-files-to-chunks · https://huggingface.co/blog/xet-on-the-hub

lakeFS is less git-like than the marketing suggests: its ranges are SSTables named by content hash and commit IDs are hashes of record IDs, but refs and uncommitted state live in PostgreSQL/DynamoDB, and the data objects are never hashed, chunked or deduplicated by lakeFS — only the metadata index is content-addressed.

Two adjacent results worth knowing. Tigris objgit (2026-06-23) put a go-git server on pure object storage and issued 8 500 GetObject calls to clone a 200 KB packfile; a 100 k-object push implied ~200 k calls, 30+ minutes. Relevant to any cold tier that talks to S3. https://www.tigrisdata.com/blog/objgit/ And World of Code (arXiv:2010.16196), a git-object warehouse on custom KV stores with no columnar anything: >18 B objects, blobs 487 GB compressed / 102 TB uncompressed, >170 k lookups/s/thread — and the number that matters, a full sequential sweep takes 38 hours while 1 M random lookups take 56 hours.

3.4 The counter-question: git already built a column store, twice, and measured it

The columnar thesis is not wrong. It is already implemented in git, exactly where git's access pattern became a scan over one attribute — and nowhere else.

commit-graph is a chunked, fixed-width, one-column-per-attribute sidecar: OID chunk, 256-entry fanout, fixed-width records, octopus overflow chunk. MEASURED BY Derrick Stolee, 2018-06-25:

Plus merge-base −60–88%, branch --contains −83–99%. Default since Git 2.24. https://devblogs.microsoft.com/devops/supercharging-the-git-commit-graph/

Changed-path Bloom filters (Git 2.27) are a zone map on that sidecar: 10 bits per element, 7 hashes, ~1% false positive rate, under 100 MB for linux.git, ~6× typical and up to 20×.

Every measured columnar win in git's history comes from metadata scans, not object storage. That is the pattern, and it is the same pattern as Software Heritage's ORC export. It is also, precisely, what gunnar's Arrow ref log is.

And git tried columnar object storage and abandoned it. Pack v4 (2013, Nicolas Pitre and Nguyễn Thái Ngọc Duy) added a pack-level sorted SHA-1 table, a frequency-sorted ident table and a tree path dictionary — dictionary encoding of a column. 90 commits on np/pack-v4. Marked [Stalled] in What's cooking by late 2013, never merged; git still writes v2 only. I found no post-mortem stating why. Do not invent one. https://www.spinics.net/lists/git/msg216294.html

What the packfile has that Arrow does not:

• .idx v2: 256-entry fanout → O(1) narrowing plus O(log n) binary search, and a CRC32 per object.
• .rev: before it, printing an object's size was "more than 62 times slower" than printing its contents; after, "roughly the same amount of time." Production: homebrew-core fetch CPU −80%, fleet daily peak 10.8 s → 7 s.
• Bitmaps: counting "up to 8 minutes of CPU" → "less than 3 ms"; index 5–10% of repo size.
• Cross-object binary delta against an arbitrary base. git.git's v2.37.0 root tree: 19 388 B raw → ~14 KB zlib → 50 bytes as an offset delta. Arrow has no delta encoding at all — dictionary and RLE are the only cross-row encodings, and IPC compression is whole-buffer LZ4/ZSTD.
• A tuned depth/size knob. Jeff King, gc: default aggressive depth to 50: going from depth 250 to 50 on linux.git costs +1.2% size and buys −21.6% rev-list CPU.

For refs specifically — which is what gunnar stores in Arrow — git already shipped the purpose-built format and it becomes the default in Git 3.0. reftable's numbers are in §1.9. git's own BreakingChanges doc, verbatim: "Writing many references at once is slow with the 'files' backend… The 'reftable' backend significantly outperforms the 'files' backend by multiple orders of magnitude."

So, honestly, what does Arrow add?

Three things, all real: cross-language queryability (DuckDB reads the ref log with no gunnar code), transactionality from the framing so there is no transaction manager to prove correct, and typed columns for metadata so "who moved main last year" is a SQL question. Those are complexity and capability wins.

What it costs: ~3% size against the pack; 2.26× on small batches at the recommended alignment; no content addressing, no dedup, no delta encoding, no key index, no checksum, no tombstones; a 2 GiB ceiling per Binary array per batch; zero-copy that evaporates on compression or misalignment; and a ~20× per-record constant at batch-size-one.

I found no measurement, anywhere, showing a columnar format beating a packfile at any git-server operation. Not speed, not size. The evidence runs the other way twice: Software Heritage's columnar export is 2.06× its purpose-built representation, and columnar layout defeats content-addressed dedup until you bolt CDC onto the writer.

The ~3% is not an Arrow bug. It is the format working as specified — offsets, validity bits, 8-byte padding, per-batch flatbuffers — over payloads that are already compressed and opaque, so none of Arrow's actual mechanisms apply to them. There is no speed win to find either, because Arrow's speed win is decode avoidance, and there is nothing to decode in an opaque blob.

Verdict: Arrow in gunnar is justified by queryability and by the absence of a transaction manager, and by nothing else. Say that in the README rather than leaving the speed question open — leaving it open invites someone to go hunting for a win the evidence says is not there.

4. What a serious git-server benchmark measures that ours does not

git ships its own harness, t/perf/, and it is the reference for what the git project thinks is worth measuring. t/perf/README documents the shape: best-of-N (GIT_PERF_REPEAT_COUNT, default 3), two corpora (GIT_PERF_REPO "at least git.git size", GIT_PERF_LARGE_REPO "about linux.git size"), output as Elapsed(User + System) so CPU is always beside wall-clock, and ./run comparing two git revisions rather than two products.

The tests that matter here:

Four things git's harness does that the gunnar bake-off does not:

1. Sweeps the client's distance from the tip. One divergence point is one sample of a curve that gunnar's own bitmap work already showed is non-monotonic — 10 commits behind cost 1598 store touches on the exact walk, 35× further behind cost fewer. A single diverge fixture cannot see that.
2. Measures the client too. index-pack --fix-thin is where a thin pack's savings are repaid. A server that sends fewer bytes by pushing work onto the client has moved cost, not removed it, and p5311 catches that in one line.
3. Runs the many-packs and stale-bitmap states deliberately. Both p5303 and lib-bitmap.sh's "partial bitmap state" construct the exact condition gunnar's fetch loss lives in — an old bitmapped pack plus fresh un-repacked history — and gunnar's fixtures reach it only by accident.
4. Reports CPU next to wall-clock always. gunnar does report net CPU, which is good, and the bake-off's own conditions row (loadavg 62.16 on a 32-core box) is honest about why the ms columns are noise.

The bake-off does three things t/perf does not, and they are better: a byte- counting relay in front of every engine so one instrument produced all figures, read-back cgroup quotas rather than requested ones, and interleaved rounds. Those should stay.

No published bake-off number has ever been taken off a maintained repository — and another lane got there first, hours ago. A maintained-state fetch arm landed in 598b563 (2026-08-04 15:29:31Z) and f7813d1 (15:36:59Z), and it is careful in the right way: it reads pack count and pack bytes on both sides of the gc and treats a gc that changed nothing as a finding rather than an abort (bench/examples/forge/mod.rs:613-618, 2988-3061). But bench_history.jsonl's last row is 2026-08-04T13:29:40Z and contains zero maintain* keys. The arm exists; it has not produced a row. Everything currently published is from a state neither server would be in after a week of production.

5. Ranked: what gunnar should do

Ranked by expected value, with the evidence and my cost estimate. Estimates are mine and are INFERRED, not measured.

Worth doing

R1 — Delta ordering in the repacker. Not a delta encoder. Evidence: §0 shows the residual fetch loss is delta-search quality, not thin-pack support. §1.5 and §2.1 show what ordering is worth, measured, by two independent routes: --path-walk takes fluentui 438 MB → 148 MB and a private monorepo 130 049 MB → 4 432 MB (29.3×); --name-hash-version=2 takes the same fluentui 439 MiB → 160 MiB and 96 s → 34 s, i.e. it is faster and smaller. Because gunnar copies entries verbatim, every byte of that lands on the wire for free at serve time. Cost: real — a path-grouped enumeration pass in crates/gunnar-store/src/repack.rs, plus the sort key. But it is the only lever in this report with a published order of magnitude behind it, and it needs no serve-time encoder, which P-009 already declined for good reasons.

Start with the name-hash, not path-walk: it is a sort key, roughly 30 lines, and it got fluentui to within 8% of what full path-walk achieved.

R2 — Run the maintained-state arm that already exists, and sweep divergence depth. Evidence: §1.1 — after gc a delta encoder is worth exactly 1.0000×, nine of nine corpora, so every published bake-off number is from a state neither server would be in after a week. The arm is already written (598b563, f7813d1) and has produced zero rows (§4). Sweeping depth is the missing half: p5311 measures a client 1/2/4/8/16/32/64/128 days behind and times the client's index-pack --fix-thin too; gunnar samples one divergence and only the server. Cost: a run, plus fixture work for the sweep. Risk, and it is the point: this may widen the gap, because gitea gets stock repack and gunnar gets repack.rs. Better to know before R1.

R3 — Name-hash cache in the bitmap file. Evidence: §1.6 — pack.writeBitmapHashCache exists for exactly one purpose, "serving a fetch between an older, bitmapped pack and objects that have been pushed since the last gc" (Documentation/config/pack.adoc:181-188), which is gunnar's losing arm word for word. 4 bytes/object. Cost: small; gunnar already writes .bitmap. Only pays off after R1 — it is a cache for a delta heuristic, and gunnar has no heuristic to feed yet. Sequence it second, not first.

R4 — Two things the delta cache that just landed does not do. Evidence: §1.3. A delta cache landed in the archive backend in 9ce1cca (2026-08-04 16:31Z) using gix-pack's MemoryCappedHashmap. That closes the big hole. Two gaps survive it, and I checked both in the source rather than assuming.

Part one: gix-pack caches only the final result of a decode, not the chain. decode_entry walks down probing cache.get at every cursor, but on the way back up it issues exactly one cache.put, keyed on first_entry.data_offset — the entry the caller asked for (gix-pack-0.73.0/src/data/file/decode/entry.rs:239 and :442). git inserts every intermediate result in the same pass (packfile.c:1715-1719). In a sequential pack-order walk the difference washes out, because each base is itself requested and cached. Under random access over a deep chain it does not: git warms the whole chain in one read, gix warms one entry per read. That is O(1) versus O(depth) cache warm-up, and it is an upstream-fixable difference, not a gunnar bug.

Part two is the bigger half and it is why gitoxide loses to git at verify-pack today: git streams the pack and applies deltas inversely — decompress a base once, then recurse into every dependent that names it — while a random-access mmap design re-walks each chain from the bottom independently. A cache narrows that gap; inverting the traversal closes it, and it is the right shape for a full-archive verify or repack where every entry is visited anyway. https://github.com/GitoxideLabs/gitoxide/issues/1

R5 — Multi-pack verbatim reuse. Evidence: §1.2 — 6.50× ± 0.03 MEASURED on ~100 packs; §2.1 — verbatim reuse was worth 3.3× on top of bitmaps in git's founding benchmark. gunnar has both prerequisites (crates/gunnar-store/src/midx.rs, crates/gunnar-bitmap/src/pack_order.rs). Cost: moderate. Ranked below R2 on purpose: the 6.50× exists only in the many-packs state and gunnar has not measured whether it lands there. Let R2 decide.

R6 — Steal p5311's and p5303's shape, and GitLab's profile method. Evidence: §4 and §2.3. The specific borrowings: sweep client distance; time the client; construct the many-packs and stale-bitmap states deliberately; and run a concurrency-N workload with a per-process CPU profile rather than one client. That last one is what turned GitLab's "3.6× wall clock" into "9.3× CPU, of which git itself was 18 552 → 923 samples" — a number you can act on. Cost: harness only.

Not worth it, and why

Delta islands. §1.4. One repo per store, no fork network, no refs/pull. Pure cost until that changes.

commit-graph / generation numbers. §1.8. gunnar already has bitmaps at 0.278× the exact walk, and generation numbers accelerate the same walks bitmaps short-circuit. Two mechanisms, one problem. The counter-evidence is real but off-target: Thiel measures gix blame up to 30% slower than git precisely where commit-graph caches exist (§2.6) — but blame is a client operation and gunnar is a server. Build it only if gunnar grows merge-base or --topo-order endpoints.

Pseudo-merge bitmaps. §1.6. Experimental upstream, no published number, and the problem they solve is "so many refs that bitmapping each tip is infeasible." gunnar has no such repo. Watch, do not build.

Replacing the Arrow ref log with reftable. §1.9. The 680 B/push constant is 20× reftable's 33 B/ref, but at gunnar's ref counts the absolute numbers are trivial and compaction already flattens the curve. This becomes urgent at ~10⁵ refs and not before. The cheap hedge is to know the number: measure open latency at 100 000 refs once, and if it is fine, stop thinking about it.

Anything Iceberg. §3.2. A snapshot's parent-snapshot-id is a single long, so a merge is not representable and ManageSnapshots has no merge method — Project Nessie exists to add it. Identity is file_path, a mutable URI, so there is no content addressing and no cross-snapshot dedup. Adobe measured 15 commits/min per dataset against a need for ~450, and the cause is structural. A forge's unit is one push = one commit. The resemblance to gunnar's Arrow log is real but it is a resemblance to reftable's design, arrived at independently — read §1.9 instead.

Zstd on the wire. Jeff King measured zstd=1 for git at −15.8% repack time and −18% log -S for +5.6% pack size, and said "there's basically no downside at all to using zstd" — and it was never adopted, because zlib is in the wire format and a transition destroys the ability to serve bytes straight off disk. https://www.spinics.net/lists/git/msg289028.html gunnar's P-001 depends on the same property. Zstd inside znippy's own container is a different question and stays open; zstd on anything a client parses as a packfile is not.

Putting object bytes inside an Arrow columnar buffer. §3.3. Xet measured a one-row modification in a Parquet file costing 230 MB of new storage at 89% dedup, because absolute file offsets live in the column headers. gunnar's archive already keeps a raw blob region separate from the Arrow IPC index; that is the right answer and there is now published evidence for it. Do not "simplify" it into a LargeBinary column.

6. What I could not find out

Where the numbers are missing is itself a finding. Six of the most-cited references in git-scaling folklore publish no measurements at all.

• Delta islands: no before/after has ever been published, by anyone. GitLab adopted them (Gitaly #1418, rollout #1575) and shipped only a checklist. git's own docs say "slightly larger packs." That is the entire public record for a feature every large forge runs. It is why R-not-worth-it §5 rests on structure, not numbers.
• Pseudo-merge bitmaps: no public number. Experimental upstream since 2.47.
• Stolee's "Git's database internals" (all five parts, 2022) contains essentially no measurements; part V none whatsoever. Widely cited as if it were evidence.
• Taylor Blau's Git Merge 2025 deck, "New techniques in repacking monorepos", has no production numbers. I extracted the full text with ghostscript (gs -sDEVICE=txtwrite); every figure on the incremental-MIDX slides (13 320 / 2 894 / 1 431 / 680 / 214) is an illustrative object count in a diagram. https://ttaylorr.com/presentations/git-merge-2025.pdf
• There is no clean "Scalar vs plain git clone of a 300 GB repo" number in public. The famous 12-hour-clone / 10-minute-status figures are 2017 GVFS-era and reported without methodology; nobody published the Scalar-era equivalent.
• JGit's org.eclipse.jgit.benchmarks (JMH) publishes no results. Google's reftable-at-scale post gives exactly one number: 866k refs.
• Gitaly Cluster / Praefect: no published benchmarks, only requirements.
• Gitaly's pack-objects cache doc gives no hit rate and no CPU percentage — only a page-cache observation (cache reads >100 MB/s hourly while disk reads stay under 1 MB/s) and a capacity estimate (worst case ~30 GB, 1.5% of free space).
• Codeberg / Forgejo / Gitea have published no performance engineering with numbers. What exists is an incident thread with user-reported timings. Their git serving is stock git pack-objects, confirmed by the bake-off's own podman top probe rather than by any document of theirs.
• The path-walk table's Repos B/C/D are private. Only microsoft/fluentui is reproducible. The 29.3× is real but unverifiable by us.
• No shipped system stores git objects in Arrow, Parquet, Iceberg or ORC. Two lanes searched independently, including the git mailing list, where nobody has ever proposed it. That is a white space, not a search failure — but it also means there is no prior art to copy and no cautionary tale to learn from.
• I found no measurement anywhere of a columnar format beating a packfile at any git-server operation (§3.4). Absence of evidence, but I looked in the obvious places: Software Heritage, Dolt, XetHub/HF, lakeFS, World of Code, Irmin.
• Nobody has published packfile vs tar+zstd on the same object set, either.
• Pack v4, git's own columnar object-storage attempt, stalled in 2013 with no public post-mortem. 90 commits, a tree path dictionary, marked [Stalled] in What's cooking and never merged. The reason is not on record.
• No published Arrow figure for memory amplification when building batches, and none for Arrow IPC size overhead as a percentage over raw. I looked for both; the only evidence is anecdotal issue reports. gunnar's own 1.013–1.077× is consistent with the arithmetic (offsets + validity + ≤7 B padding + per-batch flatbuffers) and needs no other explanation.
• No Apache first-party maximum-commits-per-second figure for Iceberg exists. The 15/min is Adobe's, from 2020. Do not quote a newer one; the circulating ones are blog-grade.
• Software Heritage has never publicly explained why ORC rather than Parquet.
• I could not verify the methodology of any third-party Gitea-vs-GitLab benchmark. Several circulate with confident numbers. Do not cite them.
• I did not verify gunnar's own baseline numbers. Everything cited from .nornir/* is another lane's measurement read as given — except §0, where I did check the commit ordering with git merge-base --is-ancestor and the fetch row's wire-vs-on-disk columns, and found the doc's explanation wrong.