nordisk/edda

Making the transfer ordering the default, and the two defects found under it

MEASURED on oden, 2026-08-04, continuing .nornir/transfer-shape-reasoning.md. Subjects: /home/rickard/scratch/cargo/gunnar-xferdef/release/gunnar (26 386 808 B, 22:47:25Z, the merged lane) and /home/rickard/scratch/cargo/gunnar-xferdef2/release/gunnar (26 393 976 B, 23:06Z, that plus the changes below). Client is stock git 2.53.0. Every byte is counted by one relay (cargo xtask wireproxy) in front of one server, server→client, and every relay row was ok.

Corpus: the same 19 real workspace_nordisk repositories, pushed into a gunnar store by stock git, never clone --bared. Rebuilt for this lane with a deeper depth ladder — 50, 100, 200, 400, 800 — which is the only structural difference from the previous lane's corpus: six pushes per repository instead of three, so the never-repacked store carries more push residue (70 packs against that lane's ~57) and its clone row is correspondingly fatter. The fetch rows are unaffected, since a fetch sends a suffix.

Bitmap state, on every row. The never-gc'd store carries 0 .bitmap files; every gc'd arm carries 19, one per repository, written by the repack. That is production-accurate on both sides — gunnar's push path writes no bitmap and its repack does — but it means the skip row and the gc rows do not differ only in the pack. The ordering comparison lives entirely among the gc'd arms, which all carry 19 bitmaps, so it is unaffected.

The instrument's noise floor is 0.071%, and it was measured rather than assumed. Two arms, f_gc and f_gctv2, served byte-identical stores — their pack-after.tsv files are identical and both logged recompute=false on all 19 — so the difference between their wire totals is pure measurement noise: clone 0.071%, fetch@50 0.000%, fetch@100 0.020%, @200 0.022%, @400 0.012%, @800 0.058%. Every ratio below that differs by more than ~0.1% is a result.

Two source repositories moved under the run. $SRC is a live checkout other agents push to, and reading rev-parse HEAD at verify time made workspace_nordisk read FAIL on a perfectly good clone. transfer-arm.sh and transfer-ceiling.sh now read a frozen heads.tsv — repository, pushed head, expected object digest — taken once from the store's own refs/heads/main. Every clone and fetch now also runs under transfer.fsckobjects=true with transfer.unpackLimit=1, so stock git's index-pack --strict judges every pack at receive time, before the fsck --full --strict that follows.

1. The headline: the ordering default is INERT on the path that ships

gunnar gc — no flags, the shipping default — logged recompute=false on 19 of 19 repositories. DeltaPolicy::Auto's threshold is 0.20 and the corpus's delta fractions run 0.4096 to 0.6934, mean 0.5693. The policy declines, no search runs, and RepackOptions::delta_order is never read.

Proven at the byte level rather than argued: gunnar gc and gunnar gc --delta-order transfer-v2 produced the same 19 packs, byte for byte.

So "make transfer-v2 the default" is a change to what --aggressive (and any DeltaPolicy::Recompute) does. It does not, on its own, fix the gc that doubles a shallow fetch. That is said first because every number below could otherwise be read as fixing something it does not touch.

2. gunnar gc was rolling dice, and the wire was the die

Three gunnar gc runs, default flags, over three byte-identical copies of the same store — the policy declining on all 19, so every run was a pure copy:

Pinned to one CPU, the same runs are byte-identical — and nornir comes out 15 854 870, 7.6% smaller than the best of the three parallel runs and 14.2% smaller than the worst. So the parallel path was not merely random, it was systematically worse.

The cause is duplicate objects. A push sends a thin pack and index-pack --fix-thin appends the bases it was thin against, so every object that was a base for the next push is physically in two packs. output::count::objects records whichever pack the lookup that ran first happened to find; iter_from_counts then sorts by (pack_id, pack_offset) and can copy a delta only when its base precedes it. Which end of each delta pair survives as a delta was decided by thread scheduling — and gunnar's serve path copies pack entries verbatim (P-001), so that decision was the wire, for every clone and every fetch, until the next gc.

Fixed by counting single-threaded (repack.rs, write_pack_file). The count is a lookup per object over an mmap'd index; iter_from_counts keeps the operator's full thread limit, which is where the work is. Three 16-CPU runs after the fix: byte-identical, and equal to the single-CPU control.

What it is worth, on the wire, gunnar gc before and after:

A second nondeterminism exists and is NOT fixed. A store's pack load order is its packs' mtime order; several pushes into a fixture land inside one timestamp tick, and a duplicated object then resolves to a different pack. That lives in gix's odb slot assignment, not in this module. It was found by the new test failing after the fix, which is the only reason it is known at all; two_copy_repacks_of_one_store_produce_the_same_pack therefore copies the store with cp -a so that mtimes are held constant and the thread race is the only variable left. Two stores that differ only in pack mtimes may still repack differently.

3. What transfer-v2 is worth, clone and fetch as separate rows

gc --aggressive, everything else identical, on the fixed binary. git thin is stock git's pack-objects --thin over the identical object set — the ceiling.

All 16 repositories improve at depth 50, 0.303× (korp) to 0.850× (ordning) — reproducing the previous lane's range exactly.

The arithmetic, per repository rather than asserted

Per operation over the corpus: one clone costs +299 462 B (5 689 772 B over 19 clones) and one 50-commit fetch saves 342 385 B (5 478 156 B over 16), so the ordering pays from 0.87 fetches per clone onwards. At depth 100 that is 0.74, at 200 0.90, at 400 1.42, and at 800 it never pays.

That average hides a 19× spread, so here is every repository — bytes one clone costs against bytes one 50-commit fetch saves, and the ratio of the two:

Fourteen of sixteen pay for themselves inside 1.12 fetches per clone. The two that do not are the two largest, and they are the two the chain limit is failing (§5): at --depth 250 nornir's break-even falls 2.68 → 0.20 and workspace_nordisk's 5.28 → 2.07, with the corpus per-operation figure falling 0.87 → 0.29. korp's clone gets smaller at depth 250, so its break-even is negative — it costs nothing at all.

A real repository sees fetches per clone in the hundreds or thousands. Nothing in this corpus can supply that ratio, and it is not being claimed; what is claimed is the two numbers it multiplies.

4. The crossover, measured out to 800 and to the whole history

transfer-v2 / name-hash-v2, both at depth 50:

The second column holds membership constant, so the curve is not confounded by short repositories dropping out; it crosses in the same place. On the all-repo aggregate the crossover interpolates to ~520 commits; on constant membership ~655.

Absolute depth is the wrong axis, and the per-repository table says so. As a fraction of each repository's own history the crossover sits at:

15 of 16 repositories do not cross until a client is 60–90% of the repository's history behind. A client that far behind is doing a clone in all but name. That is why this is a default and not a flag. The one outlier, workspace_nordisk, crosses at ~15%.

5. Why nornir got worse — it is the chain limit, not the window

The previous lane reported nornir at 1.480× against the never-gc'd store and left it unexplained. Reproduced here at 1.475×, and then isolated.

First, what it is not. git verify-pack over the fetch set, nornir 50 commits behind, 482 objects sent:

transfer-v2 restores nornir's client-side bases from 79 to 141, against the never-gc'd store's 143. The direction is fixed. And --window 50 changes the base distribution by nothing at all — the collision hypothesis (that nornir, being the largest repository, has enough name-hash collisions that the ten-object window cannot reach the previous revision) was tested and is refuted.

The stored bytes for those same 482 objects say where the loss is:

96 whole objects costing 426 122 B against the never-gc'd store's 112 costing 257 012 — fewer whole objects, 1.66× the bytes. Twelve objects carry 235 811 B of it, and one carries 191 971 B: src/bin/nornir.rs, a 703 104-byte blob that the never-gc'd store ships as a 54-byte delta and transfer-v2 ships whole, at 192 025 B.

The mechanism. Under an epoch order a file's revisions are one chronological chain. git's pack.depth = 50 was chosen for the short bushy chains a size-descending order produces; under transfer-v2 a file with more than 50 revisions must break its chain, and at every break the object travels whole. If a break lands inside the last fifty commits, the fetch pays for it — and on nornir the file it landed on is 703 KB.

The prediction, and the test. Raise the chain limit and the regression must disappear. gc --aggressive --delta-order transfer-v2 --depth 250, same 482 objects:

Gone, and depth 1000 is identical to 250, so 250 is not a cliff edge. facett, the other repository with a chain at the cap, moves the same way (467 977 → 396 436). korp, whose chains never reach the cap, does not move (321 871 → 321 820).

So nornir is not a different kind of repository. It is the repository with a large, long-lived file whose revision count exceeds the chain limit.

6. The chain limit is a real trade, and it is not this change's to make

Depth 250 is better on every wire row and on the archive:

At depth 250 gunnar beats stock git's own pack-objects --thin at every depth, and the crossover against name-hash-v2 moves out past 800.

And it is paid for, exactly, by the reader. Total delta applications a full read of the corpus costs, from verify-pack -v, 64 903 objects:

The default chain limit stays at git's 50. Going to 250 buys 6.7% off a 50-commit fetch and 3.7% off a clone for 2.82× the delta applications every reader performs — a bill that lands on clients' index-pack and cat-file, not on the forge, which never applies a delta (P-001). The delta-application counts above are exact; the wall-clock reader cost is not reported, because every timing taken tonight was on a box running at load 50–110 and the numbers came out self-contradictory. Choosing a new default on a cost that could not be timed is the same mistake as Auto's 20%, and it is not being made here.

What ships instead is the knob: --window and --depth on gunnar gc, and RepackOptions::delta_search. They were a hardcoded 10/50 reaching through a ..Default::default() that no caller could touch. An operator whose repository shows the nornir shape — a large file with more revisions than the chain limit — now has the remedy, and it is in the log line, window=10 depth=50, on every re-encoding repack.

7. Is DeltaPolicy::Auto's 20% right?

The number is defensible. The observable is not.

The number: Auto was built for a client that shipped 0.0%-deltified packs. Against that, 0.20 has a factor of two of margin below (this corpus's minimum is 0.4096) and separates the two populations without splitting a hair. Nothing in this measurement argues for moving it.

The observable is the problem. Auto asks how many deltas a pack has. The defect asks which way they point, and the two are uncorrelated: korp's delta fraction moved 67.3% → 66.0% while every one of its client-side bases was destroyed. Worse — before §2's fix, three gc runs over one store produced packs 6.1% apart with the same delta fraction on every one. A number that does not move when the thing moves cannot be the trigger for it.

So the honest verdict is that 20% is not the determining variable it was taken for; on this corpus it never fires at all, and no value of it would have caught either defect in this document. A trigger that could is reversed_fraction — the share of a pack's deltas whose base is newer than its target, which the same walk that computes the epochs already has the data for. That is designed here and not implemented: it needs its own threshold, chosen with its own measurement, and choosing one tonight would be repeating the mistake.

8. --aggressive that cannot help, and why it still searches

Not fixed, and the reason is worth writing down. A sound cheap decline is a memo, not a heuristic: record (pack ids, order, window, depth, outcome) after a repack and, if the same key comes back unchanged, skip the search — the outcome is a pure function of the inputs, which §2's fix has just made true. A heuristic ("this pack looks too good to improve") would be a silent green: it would report that --aggressive ran when it had not. The memo needs a state file in the repository's gunnar-store/, an invalidation key and its own red test, and it was left rather than half-built.

9. What was seen red

• two_copy_repacks_of_one_store_produce_the_same_pack — restore thread_limit: options.thread_limit on the count and it fails on run 1 (368 323 against 370 936 on the fixture; 6.1% on the real corpus). It also went red with the fix in its first form, which is how the mtime-ordering nondeterminism in §2 was found at all.
• the_delta_window_reaches_the_search_and_changes_the_pack — revert the window:/depth: lines to ..Default::default() and both window repacks produce the same pack.
• the_delta_depth_is_its_own_wire — feed the depth from delta_window and only this test fails while the window test stays green, so the two guards are not one guard twice.
• gc_default_matches_the_command_line — derive Default on GcArgs instead of writing it and it fails on window, 10 against 0. That is not hypothetical: every GcArgs { .., ..Default::default() } in the tests would have run a delta search with a window of zero and still passed.
• The corpus's own verification went red for real: workspace_nordisk read objects_match=FAIL on a perfectly good clone because the source repository's HEAD had moved mid-run. That is what the frozen heads.tsv is for.

10. The refusal, which the default made reachable — and which was wrong

Making TransferV2 the default turned three green tests red at once: gc_repacks_every_repository_it_lists, a_repack_invalidates_the_multi_pack_index_it_made_stale and a_repacked_repository_reopens_without_a_stale_index. All three build the same thing — a repository holding objects written straight into the store and no reference at all — and all three now got Error::backend("the transfer ordering dated no object …") out of a plain gunnar gc.

That is not a shape to dismiss: a 0%-deltified repository is precisely where DeltaPolicy::Auto does fire, so the default ordering reaches the search there and nowhere else on this corpus.

Reading the guard against the walk it guards shows it was aimed at a state the walk cannot produce. pathwalk::group stamps epoch = rank + 1, never zero, so aged() == 0 implies no root was descended at all — no reference reaching a commit or a tree. In that state there are no paths either, so NameHashV2 is exactly as degenerate as TransferV2, nothing is masquerading as anything, and this crate's own documented contract for it is "0 means no information, and that is the honest default". The guard could only ever fire on the legitimate case and could never fire on the fault it names.

Grouping::roots — the number of root trees descended — separates the two, and the condition becomes roots > 0 && aged() == 0: there was history and the walk did not date it. That is a sharpening and it is asserted in both directions:

• roots_separates_an_empty_repository_from_a_failed_walk goes red if the out.roots = assignment is deleted, i.e. if the two cases become indistinguishable again;
• stubbing epoch to 0 in group while the roots are still descended makes the repack refuse, with the new message, proving the guard is reachable and still doing its job.

It is also asserted that the two counts are not interchangeable: aged() and grouped differ by construction, because the root tree's own path is the empty string and so it is dated but not named (2 against 1 on the fixture). An equality assertion there would have been a guard that fails on healthy input, and it was written, run, and corrected.

11. What this does not answer

• Wall-clock reader cost. The delta-application counts are exact; the timings are not reportable (see §6).
• The skip column is not a baseline, it is a ceiling. The corpus pushes a cut at exactly each measured depth, so the never-gc'd store holds a pack that is stock git's own thin pack for that fetch — nornir@50 is 413 870 against git's 408 229, 1.014×. "Eight of sixteen are worse than never gc-ing" therefore means "eight of sixteen fall short of git's optimum", and a real never-gc'd server has its packs at whatever cuts its pushes happened to fall on. The comparison that decides the default is against name-hash-v2, where all 16 improve.
• workspace_nordisk crossing at ~15% of its history was not isolated the way nornir was. It is the one repository where transfer-v2 loses inside a range a real client could occupy, and depth 250 does not fully fix it (fetch@200 0.919×, fetch@800 1.078×).
• workspace_nordisk's shape. It is the only repository where transfer-v2 loses inside a range a real client could occupy, and it is the only one the chain limit does not explain.