Your browser sucks, JavaScript disabled! Gunnar's console needs JavaScript and WebAssembly. Git over SSH on port 2222 does not.

nordisk/edda

crates/storage-git-gix/parked-gix/.nornir/stinky.md

gunnar — stinky

Opened 2026-08-03, against master at 0191154 ("Add gunnar-cli: serve both listeners, and clone over real SSH"), 27182 lines of Rust across crates/*/src. crates/gunnar-cli/ was being written while this pass ran; findings that cite it are marked and may have moved.

Everything below is judged against one load target: gunnar carries the git traffic of somewhere between one organisation and a very large forge — millions of repositories, thousands of concurrent clones, repositories with millions of objects.

This file and the front page must not drift. Several findings here were stated publicly in README.md, which turns them from observations into commitments. When an item is marked DONE here, amend its public claim in the same commit. When a public claim changes, add or update the entry here. A page that still advertises a limitation that was fixed is as misleading as one that hides a limitation that was not.

2026-08-05: the README no longer carries a "Not wired" list. It was a front page listing what is absent, and its own title had gone false — the first bullet read "znippy cold tier — wired, off by default". Every claim moved, none was deleted: the cold tier to cold-tier-decision.md, the three index sections to write-read-cycle.md §3, S-003 to performance-design.md, and multi_ack_detailed / replication / partial clone to wire-design.md. Those four files are what this rule now points at.

Nothing in gunnar has ever been load-tested, profiled or benchmarked. There is no benches/ directory anywhere in the tree and no CI. Every performance claim in the source comments is therefore unverified, and every number below that is not marked MEASURED is derived from reading the code or from arithmetic over the serde/struct shapes. Say which when you quote it (LAW 7).

How to use this file

Each finding is one ### S-NNN block. A fixing agent takes one id, fixes it, and edits only that block.

Summary

id title severity status
S-001 The identity table is one JSON blob, parsed 4× per SSH connection blocker DONE
S-002 The catalogue is one JSON blob; every repo lookup parses all of it blocker DONE
S-004 A fetch materialises the client's whole object closure, then clones it blocker DONE
S-005 verify_connectivity claims O(new history) and is O(whole repository) blocker DONE
S-006 repack() does not exist; no cold tier ever forms blocker DONE
S-007 The appliance build resolves gix-worktree, gix-index, gix-filter, gix-command blocker DONE
S-008 No connection limit on either listener; 512 blocking threads are the accidental cap blocker DONE
S-009 The gRPC control plane blocks tokio worker threads and starves the SSH listener blocker DONE
S-010 The open-repository handle cache is unbounded and never evicts blocker DONE
S-011 There is no parallelism anywhere in the serving path blocker DONE
S-012 The arrow ref log never compacts and replays the whole stream at open blocker DONE
S-039 The exit status is lost to a russh close race — a succeeded push reports failure blocker DONE
S-040 gunnar cannot clone from gunnar, though stock git can blocker DONE
S-044 Nothing has ever been load-tested or profiled; there is no benchmark harness blocker IN-PROGRESS
S-003 No delta encoding; every served pack is whole objects scale TODO
S-013 reachable_from_refs() walks the whole repository per non-tip want, uncached scale TODO
S-014 Unbounded client-driven accumulation: haves, wants, commands, push-options scale TODO
S-015 Two heap allocations per packed object, and each object buffered compressed scale DONE
S-016 Every connection materialises and clones the whole ref list scale TODO
S-017 O(n²) Vec::contains on two client-sized lists scale TODO
S-018 A ref transaction fsyncs under a std Mutex on a blocking thread scale IN-PROGRESS
S-019 The O_EXCL store lock does not self-clear; a killed appliance will not restart scale DONE
S-020 multi_ack is not advertised; negotiation costs extra round trips scale TODO
S-021 Refs.ReadObject reads the whole object before applying max_bytes scale DONE
S-022 58 log lines in 27182 lines of code, no spans, no ids, no counters scale DONE
S-047 Negotiation re-derives every round from scratch, so a divergent fetch is O(rounds²) scale IN-PROGRESS
S-048 Every authenticated HTTP request does a single-ref CAS on the identity table scale TODO
S-023 MustNotExist is not a create guard on the fs backend, and the conformance suite cannot see it correctness DONE
S-024 PushAttestation has no producer; three published event fields are constants correctness DONE
S-025 touch_key is never called, so last_used is never set correctness IN-PROGRESS
S-026 The real-git interop suites self-skip and report green correctness DONE
S-027 write_stream allocates a client-declared size with no cap correctness DONE
S-028 Unchecked subtraction in the loose-object decoder correctness INVALID
S-029 expect() on client key bytes in the auth hot path, and on lock poisoning per request correctness DONE
S-030 Five swallowed errors, none logged or counted correctness DONE
S-031 deepen-since / deepen-not / deepen-relative were never run against stock git correctness TODO
S-041 PushGate is handed no new-object set and no pack byte count correctness DONE
S-042 Nested-namespace repos are clonable over SSH and invisible to gRPC correctness DONE
S-043 Policy has no store: Policy.* is Unimplemented and gates read a JSON file off disk correctness TODO
S-032 design.md, storage-reasoning.md and both READMEs describe a design that was dropped smell DONE
S-033 There is no CI system; cargo xtask ci is the gate and a human types it smell WONTFIX
S-034 A 250 ms sleep per completed transfer, in the request-completion path smell TODO
S-035 The three znippy git indexes are unbuilt, and the stree key diverges from the spec smell TODO
S-036 Spec'd and unstarted: mirror replication, LFS smell TODO
S-037 unreachable! on a clap verb match smell TODO
S-038 Three upstream nornir-git items forcing duplication here smell DONE
S-045 russh forces two C crates — the "no C toolchain" rule is not achievable smell WONTFIX
S-046 Two tests are load-flaky: green alone, red under a saturated workspace run correctness IN-PROGRESS
S-049 sync_dir has a third and fourth copy in gunnar-coldtier, unlogged smell TODO
S-050 Every file:line in the cycle diagram was unguarded, and 8 of 24 had rotted correctness DONE
S-051 The build stamp's rerun-if-changed names nothing in a git worktree correctness DONE

Counts: 14 blocker · 13 scale · 15 correctness · 9 smell — 51 total. 29 DONE · 5 IN-PROGRESS · 14 TODO · 1 INVALID · 2 WONTFIX, counted from the table above on 2026-08-05 with awk -F'|' '/^\| \[S-0/ {gsub(/ /,"",$5); c[$5]++} END {for (k in c) print k, c[k]}' rather than by hand.

This line has now gone stale three times in one day, twice in the same direction, and that is the argument for running the awk rather than editing the number. The version immediately before this one read "24 DONE · 14 TODO"; the same awk over the same rows, unchanged, gave 25 DONE · 13 TODO, so it understated by one before this pass even began. The version before that read "46 total" and "11 DONE · 1 INVALID · 6 IN-PROGRESS · 28 TODO", which is 46 against 47 rows: S-046 is filed in the smell group and carries correctness severity, and the severity tally had dropped it. Both stale versions are preserved here because the pattern is the finding. Four status fields have been found disagreeing with their own entries — S-033, S-038 and S-045 all understated, which is the safe direction, and S-007 overstated, which is not and is written up in its own entry. 2026-08-04: both awks were re-run before this edit and the previous line was correct — 26/7/12/1/1 over 47 rows, and the severity tally matched too. The first time it has not gone stale. It moved here only because S-021 closed and S-048/S-049 were filed; the severity tally moved with it, which is the half that was dropped last time.

The one INVALID is S-028. The subtraction it names cannot underflow, measured against gix-object 0.63.0 over 503539 inputs with a detector proved able to report a violation; adding the prescribed checked_sub would have been a guard with an unreachable error arm. The five IN-PROGRESS, as of 2026-08-05, each with the reason it is not DONE rather than a status word on its own (S-033 left this list on 2026-08-05, closed WONTFIX: the CI question is decided, not unfinished — cargo xtask ci, typed by a human, and the worktree hole in it is fixed): S-018 (the cheap half landed, group commit did not) · S-044 (no benchmark harness) · S-025 (still no producer and no sidecar, re-checked a fourth time on 2026-08-04; the deliberately-uncalled comment has landed, the feature has not — and its twin touch_token turns out to be wired already, filed as S-048) · S-046 (control_plane_starvation fixed and proved under load; repack.rs's real_git_clones_a_repacked_repository not reproduced in 399 runs at load average up to 154. The failure is now instrumented, not diagnosed: as of 2026-08-04 the next occurrence records its exit status, both streams and the hash kind, where the one observed failure recorded nothing. It is neither fixed nor explained) · S-047 (three per-round redundancies removed and measured; two remainders named).

S-021 closed on 2026-08-04, the last of its three passes: the object ceiling is 64 MiB — 512 MiB / 8 concurrent inflates, the arithmetic this entry carried for two sessions — settable to the deployment through gunnar serve --max-object-bytes.

Ten entries now carry a Ready to apply block: S-021 (scale) plus the nine correctness ones S-025, S-026, S-027, S-029, S-030, S-031, S-041, S-042 and S-043, worked in one pass on 2026-08-03 while five other agents held gunnar-wire, gunnar-store, gunnar-pack, gunnar-ssh, gunnar-cli, gunnar-core, gunnar-policy and gunnar-grpc, every crate in the tree except gunnar-proto, gunnar-pktline and gunnar-client. Each block names the exact edit and, more importantly, the RED that has to be seen first. Several of them name a naive fix whose guard could not fail, which is why those entries were not landed quickly rather than a sign they are hard. Only S-042 landed, because gunnar-proto was the one crate free.

Corrections to the working assumptions this pass started from

Recorded because they were wrong and the next agent should not re-derive them:

blocker

S-001 — The identity table is one JSON blob, parsed 4× per SSH connection

S-002 — The catalogue is one JSON blob; every repo lookup parses all of it

S-004 — A fetch materialises the client's whole object closure, then clones it

let mut seen = excluded_objects.clone();

a full duplicate of that set. Both live for the whole request.

What changed. select_objects now builds the client's closure once:

  1. OidSet — a newtype over HashSet<ObjectId> that does not implement Clone. The regression this entry is about is now a compile error rather than a line somebody has to notice in review (LAW 5: by construction, not by guard).
  2. Steps 4 and 5 are swapped, so the excluded-commit set is moved into the excluded-object set instead of being copied. The two steps commute, one reading the commit graph and the other the object graph, and that is what makes the move legal.
  3. The exclusion walk's seen set is excluded_objects. One set, one insert per id.
  4. Emit — the tree walk either keeps the ids it finds (the fetch path, which is about to pack them) or tallies them (the exclusion walk, which already has them in seen). That deletes the repository-sized Vec. The traversal budget is charged against either sink identically.
  5. Step 6 takes the exclusion set as its seen set rather than cloning it. Sound because tree_closure skips an id that is in exclude or already in seen, so a seen pre-loaded with everything exclude held decides every id the same way; the exclude parameter stays for verify_connectivity, which must keep the two apart.

The guard, and the RED (LAW 2). graph.rs's a_fetch_holds_the_clients_object_closure_once counts the object ids the walkers hold at once — a cfg(test) meter inside OidSet, so a normal build pays nothing — and asserts a ceiling of one closure plus a bounded scrap and a floor of one whole closure. The floor is not decoration: a select_objects that never built the exclusion set would hold almost nothing, satisfy any ceiling, and send the client the entire repository on every incremental fetch. Seen RED by restoring the pre-fix shape (the clone, the second seen, the Vec):

the walk held 2139 object ids at once for a client closure of 915 — 2.34× one copy, ceiling 943.

A second test, the_budget_bounds_the_exclusion_walk_not_only_the_commit_walk, exists because its first version could not fail: a budget of 10 against 200 commits is tripped by the commit walk, so the assertion was green whatever Emit::Tally did. Both are now proved red by breaking Emit::Tally's len() to return 0: with the budget chosen to sit between the commit count and the object count, that reports the exclusion walk ran to completion under a budget of 400. The first version reported nothing, which is why it is not the version that shipped.

MEASURED on oden, 2026-08-03. Peak resident set of one incremental fetch, three runs each, VmHWM from /proc/self/status (the kernel's own high-water mark, so no sampler can miss the peak). Fixture: a bare sha256 repository of 200000 commits / 600000 objects built by git fast-import, one have (the tip's parent), one want (the tip), done. The fetch itself sends 3 objects / 367 bytes in both builds, so every byte below is the negotiation, not the pack. The measuring process does nothing else; its RSS before the fetch is ~7.8 MB.

peak RSS per excluded object
before (7312df2) 182.9 MB (182.7 / 183.1 / 182.9) 305 B
after 140.8 MB (140.7 / 141.0 / 140.8) 235 B
saved 42.1 MB, 23% 70 B

Scaled to the ten million objects this entry is written against — derived, not measured, and linear because the sets are — that is about 700 MB less per in-flight fetch, which is the same order as the ~900 MB the entry estimated. Sixteen concurrent clones of such a repository is ~11 GB of duplication that no longer happens. What is not claimed: the remaining 140.8 MB is still O(repository), because one copy of the closure is still one copy. Only __gunnar_reach__ changes that.

S-005 — verify_connectivity claims O(new history) and is O(whole repository)

The frontier is the set of reference values that existed before the push, so the walk costs O(newly pushed history) rather than O(repository): a client that pushes one commit onto a million-commit branch is checked against the one commit.

The code calls select_objects(graph, tips, frontier, …). Inside select_objects, the frontier is the excludes argument, and steps 3 and 4 walk the full commit closure of the excludes and then the full tree-and-blob closure of every one of those commits (graph.rs:380-409). The frontier is every pre-existing ref, so that closure is the entire repository. Only after paying that does step 5 compute the small set the doc describes. The stated complexity and the actual complexity differ by the size of the repository. This is a claim that was never tested against a repository large enough to contradict it.

S-006 — repack() does not exist; no cold tier ever forms

The cold tier is a consolidated PACKFILE, not a znippy archive, and that is a deliberate answer to P-005 rather than a shortcut. P-005 establishes that §4's znippy tier stores inflated object content, which forecloses pack-copy permanently and makes every cold-tier fetch pay a full recompression, the exact cost P-001 exists to delete. P-005's own fix (store the deflated pack entry) turns the tier into a packfile with an archive around it, which §4 explicitly rejects ("packs-in-znippy is rejected"). Those two cannot both hold, and the choice between them is Rickard's; P-006 (znippy needs cmake, a C++ toolchain and a build-time network fetch) and P-014 (znippy has no writer lock at all) both argue for not hard-wiring znippy into the appliance until it is made. So repack() was built over the tier git itself uses. The swap protocol is what §4.4 specifies and is unchanged when the tier is replaced: snapshot → write new → verify open → only then remove.

What it does: snapshots the packs and loose objects to fold (so a push arriving mid-repack is safe by construction, not by a lock); writes one new pack through gix-pack's generator in PackCopyAndBaseObjects mode; hands it to gunnar-pack::index_pack, which persists pack-<id>.pack then pack-<id>.idx, and since gix-odb discovers packs by their index, the new tier becomes visible in one atomic rename; reopens the index that landed and looks up every object; fsyncs the directory; and only then unlinks the folded packs (.idx first) and loose objects. A second run with nothing to fold is a no-op, asserted by pack-file mtime, not a rewrite.

P-001 is unblocked, and measured, not assumed. RepackOutcome reports objects_copied_from_pack against objects_recompressed, because a byte count and a wall clock cannot tell a pack-copy from a re-deflate. On the sha256 fixture: 11 copied, 1 recompressed of 12 objects, the one being the loose object, which has no pack residency to copy from. Delta chains survive a repack.

RepackOptions::prune_unreachable (default off) is what finally reclaims a superseded control-document blob. It is off by default because an object written but not yet referenced is unreachable by definition; the hazard is documented at the option, and there is no CLI flag for it yet, so gunnar gc never prunes.

Zero new crates: the appliance graph is 313 packages, the same figure S-007 closed on. gix-pack, gix-features and gunnar-pack were all already in it.

Seen RED twice, deliberately, before being trusted (LAW 2):

  1. dropping one object from the folded set → 4 tests failed, naming the object count, the loose fold and the pruned count;
  2. moving the deletion of the old packs to before the new pack exists, the exact thing D14 forbids → the interrupted-repack and the SIGKILL test both failed on "a pack the repack had not finished replacing was already deleted".

Still open, and not claimed: the znippy tier itself (P-005/P-006 must be settled first), the __gunnar_oid__ stree index (S-035), arrow-log compaction at repack (S-012), and a --prune flag on gunnar gc.

S-007 — The appliance build resolves gix-worktree, gix-index, gix-filter, gix-command

cargo tree --manifest-path crates/gunnar-cli/Cargo.toml -p gunnar-cli \
    --no-default-features --features server -e normal

The appliance build resolves 350 unique packages, 47 of them gix-*, and among them every crate §0 lists as "the server needs none of" and §2 lists as "Forbidden in the S graph specifically":

gix-worktree 0.55.0 · gix-index 0.54.0 · gix-filter 0.33.0 · gix-attributes 0.34.0 · gix-ignore 0.22.0 · gix-glob 0.27.0 · gix-config 0.59.0 · gix-discover 0.54.0 · gix-pathspec 0.19.0

plus a complete client-side git transport stack that a server has no use for — gix-protocol 0.64.0, gix-transport 0.58.0, gix-url 0.37.0 — and gix-command 0.9.1, the crate whose job is spawning git subprocesses, in a build whose specification opens with "Zero shell, both halves. No git subprocess." crates/gunnar-cli/src/server/stores.rs:10-18 admits the fs-backend part of this in prose, re-confirmed when gunnar-cli landed: fs is the only ObjectStore in the tree that survives a restart, so the appliance build has no alternative until S-006's cold tier exists. The rest of the list is not admitted anywhere. feature_split.rs is a good guard aimed at the wrong half. It checks for gunnar-client, nornir-git, rayon, libgit2, git2, openssl, skidbladnir and zbus, and passes. It checks for none of the nine crates above. It is green while the constraint it appears to enforce is broken. The LAW 2 shape, in a guard whose own header is about not trusting declarations.

Re-MEASURED 2026-08-03, the command in the What section, run by hand and not read off a cached tree: the appliance resolves 314 unique packages, 24 of them gix-* (313/24 when this entry closed; the extra one is other work landing, not a regression here), and all thirteen forbidden names — the nine gix-* filesystem crates plus gix-protocol, gix-transport, gix-url and gix-command — are absent.

The guard now exists, in crates/gunnar-cli/tests/feature_split.rs: the_appliance_links_no_worktree_no_client_transport_and_no_subprocess, over the resolved cargo tree -e normal graph, appliance feature set only — the default set is deliberately excluded, because gunnar-client is a git client and legitimately links the transport stack.

Seen RED, by proving the detector rather than by faking the disease. The historical red is no longer reproducible without putting the porcelain facade back, so what was watched failing is the matching itself: adding gix-odb v — a crate that genuinely is in the appliance graph, and is what the fs backend moved onto in the fix — turns the new test red with

the APPLIANCE build links gix-odb v, which §0 and §2 forbid it (S-007).

5 passed; 1 failed. A companion test, the_forbidden_list_would_actually_catch_a_crate_that_is_there, pins that permanently: it asserts gix-odb v is present and that gix-od v is not, so neither an empty cargo tree nor a mis-shaped name can leave the list matching nothing. Without it the thirteen assertions above are thirteen ways of comparing a string to itself.

S-008 — No connection limit on either listener; 512 blocking threads are the accidental cap

load served refused
1 principal, 64 clients 32 32, all per-principal
32 principals, 320 clients 256 the rest

So a client holding one key could never reach the 256 written on the box; that took eight identities. gunnar is an appliance — one team, a handful of keys — so every real deployment silently got one eighth of its capacity, and the 224 idle slots were invisible because the refusal named a limit that was doing exactly what it said. Fixed as max-min fairness in crates/gunnar-ssh/src/admission.rs (Admission::ceiling_for): the per-principal number is now a floor and a share, not a wall.

contended  = occupied (candidate included) ≥ CONTENTION_PERCENT (75) % of the server-wide limit
ceiling(p) = if !contended { the server-wide limit }
             else          { max(configured, server_wide / active principals) }
admit      = the server-wide gate has room AND p holds < ceiling(p)

75 % of 256 is 192, leaving 64 free slots — twice the configured floor — as headroom in which an arriving identity is never asked about fairness at all. Below that threshold free slots exist by definition and refusing anything refuses work the box could do, which is the mistake being corrected. 192 in flight is also 6× the core count the defaults were sized for (performance-design.md Measurement 3), so the machine is deep into queueing before fairness engages. active principals is the length of the per-principal map read under the same lock as the counts, so it is exact rather than sampled — and a stale entry now shrinks everybody's share, which is why the map is still scrubbed of zero counts. The refusal reports the ceiling in force, not the configured one, plus how many identities it is divided between; an operator who read 1 out of a config file and 2 out of a log would otherwise conclude the gate was broken. The old behaviour is still available and is now reachable: SshConfig:: per_principal_limit_is_hard / gunnar serve --hard-per-principal-limit, for a host where an identity is a tenant and the cap is what is being sold. --max-concurrent-transfers and --max-transfers-per-principal are settable too, which this entry's own Fix line asked for in the first place and which nothing had delivered: both numbers were unreachable from the command line, so the two defaults were the only values any deployment could have. Seen RED, both directions, three levels. Two probes inside ceiling_for, each failing a disjoint set — which is what proves the tests measure different things rather than agreeing with each other:

S-009 — The gRPC control plane blocks tokio worker threads and starves the SSH listener

Both listeners run on one new_multi_thread runtime built in main.rs:41-44, and the worker pool defaults to core count.

SSH identification latency median
shared runtime (pre-fix, HEAD) 1.59 s – 3.02 s ~2.4 s
control plane on its own runtime 86 µs – 2.8 ms ~148 µs

~1000× on the worst case. The probe reads the SSH-2.0-gunnar_… line over std::net from an OS thread outside both runtimes; measuring from a task on the starved runtime would have been right for the wrong reason. The test asserts the fixture is big enough to block (one call ≥ 10 ms) and that the control plane really was loaded, so a green cannot be vacuous. Note what the pre-fix server did not do: fail. It answered seconds late, which is exactly the "git is slow" an operator cannot find in a log. NOT done: walk still has no depth cap, entry cap or deadline. A caller can still make the control plane itself unresponsive — that is now the only thing they can make unresponsive, which is the property this fix bought.

S-010 — The open-repository handle cache is unbounded and never evicts

S-011 — There is no parallelism anywhere in the serving path

S-012 — The arrow ref log never compacts and replays the whole stream at open

pushes before (never compacts) after (default policy) after (folded, window 4)
10 15 808 B · 10 batches 15 808 B · 10 batches 8 128 B · 5 batches
1 000 1 536 448 B · 1 000 batches 1 536 448 B · 1 000 batches 8 128 B · 5 batches
10 000 15 360 448 B · 10 000 batches 1 223 104 B · 796 batches 8 128 B · 5 batches

One single-ref push is 1 536 bytes on the wire, so this entry's 180 000-push five-year repository was a 276 MB / 180 000-batch read on every open and is now bounded at ≤1.57 MB / ≤1024 batches, flat. What compaction discards: the reflog, which §4.5 says is this log, so the rows that fall out of the window are moved, not dropped: one complete self-contained Arrow IPC stream per compaction in refs-archive/, named by timestamp, never read at open, still a DuckDB glob. Archive is written and fsynced before the swap, so a crash duplicates a span of history rather than losing it. Secrets are the deliberate exception — secrets.arrow folds with no archive and no window, because a rotated host key must actually stop existing. A log written before this existed is folded at open as well, because the write-path trigger cannot help a repository that is only ever read from; that open pays the replay once and every later one is bounded. Eleven new tests, each seen RED against a deliberate break (2026-08-03): a no-op compaction turns 7 red; compacting in place instead of staging + renaming turns the crash test red and nothing else; dropping the archive turns 2 red; a fold built from a one-transaction-stale state turns the concurrent-writer test red on a lost update (w0 left at oid 0x17 where 0x18 was acknowledged); disabling the open-time fold turns exactly its own test red; dead write-cost counters turn the S-018 guard red. The open-cost measurement's own red-capability is permanent: the_open_cost_guard_catches_a_log_that_never_compacts runs the same helper against CompactionPolicy::never() — the behaviour this module used to have — and asserts it still grows. cargo test -p gunnar-store --features fs: 59 passed, 0 failed, conformance and its four break-wrappers included.

S-039 — The exit status is lost to a russh close race — a succeeded push reports failure

S-040 — gunnar cannot clone from gunnar, though stock git can

Neither half is individually wrong: refusing an unimplemented capability is right (S-020), and the v2 negotiation both ends implement is fine. The defect is the combination, and it was invisible until the two crates were joined. Nothing tested gunnar against gunnar. Stated in README.md, so it is a commitment.

S-044 — Nothing has ever been load-tested or profiled; there is no benchmark harness

arm headline
gunnar.pack_deflate 130.7 MB/s at ratio_x 2.85 (level 6)
gunnar.pack_trailer_hash 1890 MB/s sha256
gunnar.clone_large 1 GiB fixture, clone_ms 3246, 19.9 MB/s, objects_recompressed 2048/2048
gunnar.clone_concurrent_64 p50 988 ms, p99 1044 ms, 60.4 clones/s, peak_threads 67
gunnar.push_large push_ms 255, index 121.2 MB/s vs 49.6 MB/s serial (parallel_speedup_x 2.45)
gunnar.walk_1m 710 341 objects/s over 1 000 002 objects
gunnar.lookup_1m 2 189 547 hits/s, 82 798 misses/s
gunnar.lookup_1m_64pack 529 963 hits/s, 3 355 misses/s, miss_degradation_x 23.7
gunnar.repack_1m 87 054 objects/s, seal_ms 0.002
gunnar.negotiate_divergent 154 490 ms, 1250 rounds, 18 901 251 commits visited

The head-to-head, and gunnar loses it — oden, 2026-08-03

This is the number this entry existed to get, and it is bad. Five vs_git_* arms, same repository, same stock git 2.53.0 client, same ext:: pipe on both sides, both sides inside one pinned ubuntu:26.04 + git=1:2.53.0-1ubuntu1 container. Fixture: 2048 commits over 1024 files in 32 directories, 16 KiB blobs, ~2% of eight files rewritten per commit — churn, not fresh random blobs, because delta reuse is exactly what P-001 costs and a fresh-blob corpus would have flattered gunnar. 528 MiB of blob content; 36 172 objects in a full clone; git repack -adf at default window and depth, i.e. what a server that has run git gc looks like.

arm metric gunnar git 2.53.0 ratio
vs_git_clone (sha1) bytes on the wire 108.70 MB 5.93 MB 18.34× worse
wall clock 4971 ms 462 ms 10.76× worse
server CPU 4.875 s 0.022 s 219.96× worse
peak RSS (floors 6.07 / 4.91) 17.33 MB 12.06 MB 1.44× worse
vs_git_clone_sha256 bytes 115.06 MB 6.49 MB 17.72× worse
wall clock 5107 ms 528 ms 9.68× worse
server CPU 4.994 s 0.028 s 176.41× worse
vs_git_fetch_divergent bytes (1008 haves each side) 1.79 MB 0.33 MB 5.4× worse
wall clock 177 ms 124 ms 1.43× worse
server CPU 0.143 s 0.026 s 5.56× worse
objects sent 2912 2979 gunnar cut better
vs_git_clone_concurrent ×16 bytes 193.17 MB 26.77 MB 7.22× worse
wall clock (batch) 1127 ms 444 ms 2.54× worse
server CPU 7.03 s 0.265 s 26.48× worse
p50 / p99 998 / 1048 ms 388 / 406 ms ~2.6× worse
total RSS for the batch 19.23 MB (1 process) 135.95 MB (16 processes) 7.1× better
vs_git_push wall clock 384 ms 493 ms 0.78×, gunnar wins
server CPU 0.332 s 0.451 s 0.74×, gunnar wins
peak RSS 20.62 MB 54.73 MB 0.38×, gunnar wins
bytes in 1.28 MB 1.28 MB 1.00, the client built both

All five arms are from one run, so the columns are comparable with each other rather than stitched together. Taken at gunnar e60c95d, and the ref matters: an earlier run at 87524d6 put the fetch arm's CPU ratio at 6.65× and its wall ratio at 2.15×, and e60c95d's negotiation fix is what moved them to 5.56× and 1.43×. The clone rows did not move, because negotiation is not what a clone spends its time on.

Read it plainly: on everything a client waits for, gunnar is between one and two orders of magnitude behind stock git, and the cause is known. P-001 has not landed and S-003 (no delta encoding) is open, so every object is inflated and re-deflated whole. 528 MiB of content becomes 108.70 MB out of gunnar (a 4.9× plain deflate) and 5.93 MB out of git (89×, because git ships the deltas it already has). The 18.34× is the delta-reuse gap, made concrete, and it is the strongest argument in the tree for doing P-001 next. The hash algorithm is not the cause: sha256 moves the byte ratio from 18.34× to 17.72× and the CPU ratio from 219.96× to 176.41×.

gunnar wins the receive path outright — 22% less wall clock, 26% less CPU and 38% of the memory of git receive-pack — and wins total memory under concurrency by 7.1×, because it is one process where git is sixteen. Both of those are real, and neither cancels the clone number.

A finding that is not a benchmark result: FsStore::open_or_init takes a cross-process exclusive lock on the repository (gunnar-store/gunnar-store.lock). Run process-per-connection like git upload-pack, fifteen of sixteen gunnar servers die with "the store is locked by another writer". gunnar can only ever be one process per repository, which is what gunnar serve is, and why the concurrent arm runs each side in its own native shape and says so.

Caveats, none of them removable, all of them stated on the row: the gunnar side is gunnar-wire over a pipe, so gunnar's SSH layer is excluded and the numbers are optimistic for gunnar on transport; the gunnar server process is the bench binary, so an RSS floor is measured and reported for both sides; the meter that counts bytes and reaps rusage adds one process and one memcpy to both columns; oden was at 32/32 cores busy from other agents throughout, though gunnar_drift_x on the clone arm was 0.996, so the host did not move under that comparison; the fixture is synthetic.

Two of this entry's four asks are still not done and are not claimed: there is no pushes-per-second / fsync arm (S-018), and no socket-level load test that finds S-008's knee. vs_git_clone_concurrent uses 16 real client processes and real server processes, but over ext:: pipes rather than gunnar's SSH accept loop.

The Verify clause is satisfied in kind rather than to the letter: instead of one deliberate regression, every arm carries an applied-output assertion and four were watched failing: the zero-gunnar.*-arm run (exit 1), clone_large's objects_recompressed, negotiate_divergent's commits_visited = 0, and lookup_1m_64pack's degradation. The last of those found a hollow guard in this suite's own first version: comparing two arms measured minutes apart on a busy host reported a 30× "degradation" between a fixture and itself, so the comparison was moved inside one arm as an A/B/A bracket. See P-015 for the full record.

scale

S-003 — No delta encoding; every served pack is whole objects

S-013 — reachable_from_refs() walks the whole repository per non-tip want, uncached

S-014 — Unbounded client-driven accumulation: haves, wants, commands, push-options

A pkt-line payload is up to 65516 bytes, so each accepted line costs up to 64 KB. read_packet also allocates a fresh vec![0u8; len] per line (wire_io.rs:44) — a per-packet heap allocation on the framing hot path. For balance: gunnar-pktline's decoder itself is genuinely well defended: every length parse is bounded, every slice is guarded, MAX_PAYLOAD is checked before the payload is waited on, and no panic on attacker-controlled length or content was found in decode.rs. The hole is above it, in what the callers are willing to accumulate.

S-015 — Two heap allocations per packed object, and each object buffered compressed

let mut header = Vec::with_capacity(16);            // write.rs:115
let mut deflate = ZlibEncoder::new(Vec::new(), …);  // write.rs:121

The second is the more serious: the encoder's sink is a fresh Vec, so each object's entire compressed form is materialised in memory before a byte reaches the socket. The pack as a whole streams — write_pack correctly reuses one 8 KB read buffer at write.rs:200 and PackWriter is generic over W: Write — but a single multi-gigabyte blob is buffered whole, which contradicts the "streamed, nothing buffered" claim at write.rs:60-61 at exactly the granularity that matters. In the walker, Graph::commit (graph.rs:126) and Graph::peel (graph.rs:106) each allocate a fresh Vec per call, and both are called once per commit in three separate passes of select_objects; CommitFacts.parents is a Vec per commit (graph.rs:302). Packet::Data(buf[4..len].to_vec()) copies every decoded pkt-line.

S-016 — Every connection materialises and clones the whole ref list

S-017 — O(n²) Vec::contains on two client-sized lists

if peeled.kind == gix_object::Kind::Commit && !out.contains(&peeled.id) {   // :948

out is a Vec<ObjectId> and contains is a linear scan, run once per want. The wants list is whatever the client sent and is itself uncapped (S-014). Same pattern at line 1029 over the tag list.

1000 copies of one want cost 3035 store touches and a single want costs 38

Three touches per repeated line, because a repeated want was peeled again and peel is an object read. Fixing commit_wants alone moved it to 2036, not 38: the other two were wants_exist's existence probe and the selection walk peeling every tip. So the fix is one writer, not four patched consumers (LAW 5) — dedupe_oids runs once in each protocol version's want assembly, before anybody reads the list, and order is preserved because pack order is observable. Now 38 == 38. What is fixed but NOT guarded, stated rather than papered over: the Vec::contains → HashSet half, quadratic in distinct wants, has no deterministic instrument. Both spellings return identical output and touch the store identically, so only wall clock separates them, and this crate does not assert on wall clock (S-046 already carries two load-flaky tests). The gap is named in commit_wants' doc comment instead of being covered by a timing test that would go red on a busy box.

S-018 — A ref transaction fsyncs under a std Mutex on a blocking thread

S-019 — The O_EXCL store lock does not self-clear; a killed appliance will not restart

Recovery is at startup, exactly as the Fix clause asked, and not a new operator verb. The lock file now records <pid> <start-time>, the start time being field 22 of /proc/<pid>/stat. The pid alone is not enough, and that is the whole difficulty: a recycled pid reads as "still running" to a /proc/<pid> existence check, so the naive fix never recovers and nothing on a quiet box would say so. No unsafe, no libc, no new dependency; unsafe_code = "forbid" still holds.

The rule is: reclaim only what is PROVABLY gone. Alive refuses. A recycled pid is reclaimed. A zombie is reclaimed, because it holds no descriptors, can write nothing, and its pid cannot be recycled while it sits there, which matters because that is the appliance's real shape for as long as its reaper is slow. Everything undecidable refuses: no /proc on this platform, an owner line that will not parse, or a lock written by a gunnar older than this change that recorded no start time. force_unlock stays for exactly that remainder, and its doc now says so. The reclaim is one attempt, not a loop: if a second starter reclaimed the same stale lock a microsecond earlier it now holds a live one, and this one must refuse. Retrying would be a loop that ends in two writers.

SEEN RED, and the shape of the red is the point. With acquire put back to the pre-fix behaviour (refuse on AlreadyExists, never reclaim) cargo test -p gunnar-store --lib reports 44 passed; 2 failed:

a_lock_left_by_a_dead_process_is_reclaimed_and_one_held_by_a_live_one_is_not panicked: a lock left by a process that is GONE was not reclaimed: StoreLocked { path: "/tmp/.tmpuGUznB/gunnar-store.lock" }

a_recycled_pid_does_not_keep_a_dead_owners_lock_alive panicked: a lock whose recorded start time is not the live pid's is stale: StoreLocked { path: "/tmp/.tmpmRWA9C/gunnar-store.lock" }

The two refusal tests stayed green through that probe, which is the half the Verify clause calls "the whole test": a fix that reclaimed unconditionally would pass the two failures above and be worse than the bug. Five tests in all, and each names the thing it would otherwise pass vacuously on. The live-owner refusal and the reclaim are in one test against one real sleep process, so no fixture drift can separate them; the reclaimed store is asserted to answer a query rather than merely to exist; the zombie test asserts the owner really reached state Z before concluding anything; and the_lock_file_records_this_process_as_its_owner exists because an acquire that wrote no owner line would make every later recovery undecidable, which reads exactly like the bug.

The applied output, in the scenario the entry describes. repack.rs::a_repack_killed_mid_flight_leaves_the_old_state_serving spawns a real child, SIGKILLs it mid-repack, and reopens. That reopen used to require a force_unlock call, and that call is now deleted: the test passes without it, against a real stale lock file left by a real killed process. cargo test -p gunnar-store --features fs: 75 passed, 0 failed.

Docs corrected in the same change rather than left to drift. .nornir/appliance-design.md, .nornir/storage-reasoning.md and .nornir/TOTAL-PROMPT.md item 7 all stated force_unlock as the recovery path and named "a PID-plus-liveness check or a higher MSRV" as an open decision. That decision is now made and recorded: the PID check, not the MSRV bump. Error::StoreLocked's message changed too, because reaching it now means the owner is alive or could not be shown to be gone, which is not what it used to mean.

Not claimed: platforms with no /proc are refused rather than handled, and there is no test for them, because this box has /proc. On the appliance the question does not arise for the memory backing, which has no lock file at all.

2026-08-03 — the recovery is now proved for gunnar serve, not only for the store. The proof above is at the store level: an in-process ArrowLogStore reopened after a child was SIGKILLed. The scenario the entry is about is an appliance that comes back after a power cut, and that had no end-to-end proof, because the only two arms that run the server as a process put it in a container or a KVM guest and every container run starts from a fresh volume — there was never a stale lock there to reclaim.

installtest/src/bare.rs closes that. It runs gunnar serve as an ordinary process, SIGKILLs it, asserts the lock file survived naming that exact pid and start time, starts a second server on the same data root, and requires it to log the reclaim and then answer a real ls-remote with the pushed oid. Measured on oden 2026-08-03 against the binary at 7b6a9ee: reclaim to serving in 100 ms.

Two red switches keep it from being vacuous, and both are run by cargo xtask bare red --arm bare:

The first version of lock-holder-alive was itself blind and is recorded here because the shape recurs: it reused the port, the second server died of a bind failure, and the harness reported READY anyway — wait_for_ssh returns on the first SSH banner and the banner came from the server that was already there. That is the same hazard wait_for_ssh's own header describes for podman's port forwarder, arriving by a different road.

S-020 — multi_ack is not advertised; negotiation costs extra round trips

S-021 — Refs.ReadObject reads the whole object before applying max_bytes

let mut buf = Vec::new();
let Some(data) = objects.read(&id, &mut buf).map_err(store_error)? else { … };
let size = data.data.len() as u64;
let (bytes, truncated) = match max_bytes {
    0 => (data.data.to_vec(), false),
    limit if size > limit => (data.data[..limit as usize].to_vec(), true),
    _ => (data.data.to_vec(), false),
};

The whole object is inflated into buf before max_bytes is consulted, and then copied again with to_vec(). A caller asking for one byte of a one-gigabyte blob costs the server the full gigabyte twice over.

let (bytes, truncated) = match max_bytes {
    0 => (data.data.to_vec(), false),
    limit if size > limit => (data.data[..limit as usize].to_vec(), true),
    _ => (data.data.to_vec(), false),
};

with a truncated computed from size and max_bytes, then buf.truncate(limit) and data: buf, so the buffer moves into the response instead of being cloned out of it. Note data.data borrows buf, so size and kind must be read off data and the borrow dropped before buf is moved. RED: assert Vec::as_ptr of the response's data equals the pointer buf was allocated at, or count allocations around the call. Asserting the bytes are right passes today and proves nothing about the copy.

reading an 8388608-byte object allocated 16825617 bytes, over the 12582912-byte ceiling that sits between one copy and two

16825617 is 2x the object plus ~48 KB of incidental allocation, which is the second copy exactly. The test also asserts a floor (allocated >= BLOB_BYTES), so a stubbed read that allocated nothing cannot pass, and asserts the returned bytes equal the object, so a "no copies" fix returning an empty buffer cannot either. The first inflation is untouched, so a caller asking for one byte of a huge object still costs the server the whole object. That needs the configured ceiling and explicit ResourceExhausted described above, which turns a working request into a refusal and wants a deployment number decided rather than picked by an agent. The test's own module docs say so, so the next reader does not mistake this for the whole fix.

The number that needed no deciding, and has landed. tonic 0.13.1's DEFAULT_MAX_RECV_MESSAGE_SIZE is 4 MiB and the send default beside it is usize::MAX (read out of the vendored source, tonic-0.13.1/src/codec/mod.rs:100-101, not from memory). gunnar overrides neither. So a Refs.ReadObject response over 4 MiB cannot be received by a client using the generated stub's defaults: it fails on the client with a length error after the server has already inflated and serialised it. That is not a deployment number, it is a property of the transport, and refusing such a request costs nobody anything.

read_object now takes the size off ObjectStore::header — free, gix-odb answers kind and size out of the index without inflating — and answers ResourceExhausted naming the object's size when the response would exceed it. On the response, not on the object, which is the whole point: asking for a 1 KiB prefix of a 1 GiB blob is answered exactly as before, so this closes an expensive failure without turning a working request into a refusal. ControlError grew a ResourceExhausted variant mapping to Status::resource_exhausted, because "retry narrower" and "you sent something malformed" are different facts to a generated client.

SEEN RED, and on the allocation rather than on the error, because the error cannot tell the two implementations apart. With the header probe swapped for a full read() — i.e. inflate first, then refuse, returning the identical ResourceExhausted with the identical message:

refusing a 12582912-byte object allocated 12631547 bytes, which is the object itself: it was inflated and only then refused, so the memory amplifier this refusal exists to close is still open (S-021)

The guard is crates/gunnar-cli/tests/read_object_copies.rs's an_undeliverable_response_is_refused_without_inflating_the_object, over the counting #[global_allocator] that file already had. It also asserts the error names the size (or a caller cannot pick a max_bytes that would work) and — the other side, without which a blanket refusal would pass — that a 1024-byte prefix of the same object is still answered, truncated, with the right bytes. reading_an_object_inflates_it_once_and_does_not_copy_it_again's fixture moved from 8 MiB to 3 MiB so it measures the copy instead of hitting the new refusal; its subject is one materialisation against two, which does not depend on the size.

The number that still needs deciding, stated so it is one sentence for Rickard rather than a design question. The first inflation is still unbounded for a caller that passes a small max_bytes: the server materialises the whole object to produce the prefix, because ObjectStore has no bounded read and for a delta-encoded object cannot honestly grow one (the investigation above, against gix-odb 0.83.0). Bounding that needs a ceiling on the object, which genuinely does refuse requests that work today. The ceiling is

  ceiling  =  server memory budget for this RPC  ÷  concurrent inflates

and the denominator is already known: read_object runs synchronously on a control-plane worker (S-009 gave the control plane its own runtime and did not add spawn_blocking), and there are control_plane_workers() of them — cores ÷ 4 clamped to 2..=8, so 8 on oden's 32 cores. So a 64 MiB ceiling is a 512 MiB worst case, a 256 MiB ceiling is 2 GiB. The only missing input is the budget. It is written at MAX_RESPONSE_BYTES in control_plane.rs so the next reader does not re-derive it.

Worth recording alongside: S-008's own "NOT done" note says the gRPC router still has no ConcurrencyLimitLayer. That does not make the denominator unbounded here, because the synchronous call occupies a worker thread and further callers queue rather than allocate — but it does mean the bound moves the day anyone wraps these methods in spawn_blocking, which gunnar-grpc/src/seam.rs:28-34 still asks for.

The number: 64 MiB, from the arithmetic this entry already wrote — 512 MiB budget / 8 concurrent inflates. Both factors are now constants in control_plane.rs carrying their own reasoning. The denominator uses the clamp's upper bound rather than the live control_plane_workers(), deliberately: a default derived from available_parallelism() would refuse different requests on a laptop and on the forge host, which is the class of difference nobody finds until production. The budget is 512 MiB because the control plane is an administrative surface sharing a box with the git serving path, and half a gigabyte is far more than object inspection needs — the number that would want defending is a larger one, not this.

It is settable end to end, because this is the one ceiling that turns a working request into a refusal and so must not be compiled in: --max-object-bytes on gunnar serve → ServeConfig::max_object_bytes → CoreControlPlane::with_max_object_bytes, with 0 for no ceiling spelled explicitly rather than being what you get by configuring nothing. Verified reachable by gunnar serve --help, not by reading the clap derive. The refusal names both the object's size and the ceiling, so an operator can tell whether to raise the limit or the caller to stop asking.

SEEN RED three ways, because the error alone distinguishes none of them.

The guard also asserts the other side three times over, or a blanket refusal would pass everything: the same object through a default-ceiling plane is still read, a 4 KiB object through the same low-ceiling plane is still served, and an uncapped plane returns the whole thing.

A fourth red, which nobody was looking for, and it is this file's own disease. tests/read_object_copies.rs measures with a counting #[global_allocator], which is per process, while cargo test runs a binary's tests on parallel threads. Every measurement was therefore fed by the other tests' allocations and reset by their ALLOCATED.store(0). It was a coin flip and it lied in both directions:

reading an 3145728-byte object allocated 6484018 bytes, over the 4718592-byte ceiling ... (another test's inflation, inside this window)

reading an 3145728-byte object allocated only 48772 bytes, so this is not measuring the inflation at all (another test reset the tally mid-window)

The second shape is the dangerous one: without the floor assertion, "allocated almost nothing" reads as an excellent result. This was latent from the moment the second measuring test was added — the two raced and happened not to lose — so every earlier SEEN RED recorded in this entry was taken under a --test filter that ran one test at a time. A third test made it reliable rather than causing it. Fixed by construction: the exclusion lock is taken by fixture(), not by each test, so a fourth test added later gets it without a usage rule to remember. Three consecutive runs had failed; five consecutive full runs are green.

What is still true and is not claimed fixed: a read that IS allowed still inflates the whole object to produce a prefix. ObjectStore has no bounded read and for a delta-encoded object cannot honestly grow one. The cost of that is now bounded, not eliminated, and the module docs of both the source and the test say so.

S-022 — 58 log lines in 27182 lines of code, no spans, no ids, no counters

crate calls
gunnar-cli 15
gunnar-ssh 14
gunnar-core 7
gunnar-grpc 7
gunnar-client 4
gunnar-policy 3
gunnar-store 3
gunnar-wire 3
gunnar-pack 2
gunnar-pktline 0
gunnar-proto 0
total 58

There is no tracing::span! and no #[instrument] anywhere, no connection id, no request id, no request timing, and no metrics of any kind: no counters, no histograms, no prometheus, no metrics crate. Connection identity in the SSH logs is an ad-hoc ?self.peer (Option<SocketAddr>) repeated at each call site rather than a span field. The distribution is worse than the total suggests:

One span per connection, and one vocabulary. crates/gunnar-core/src/observe.rs is the single definition of both. Six sibling targets, TARGETS enumerating them so a filter can be written against a list rather than a guess:

target what is on it
gunnar::audit control-plane mutations: who called what, and the decision (D9)
gunnar::ssh connections: accepted, authenticated, dispatched, every refusal
gunnar::git transfers: upload-pack, receive-pack, objects, packs, bytes
gunnar::policy the push gate: which gate refused which reference
gunnar::store ref transactions, CAS contention, repack
gunnar::metrics the counter heartbeat

The hierarchy, all of it in one process and none of it restating a field:

  ssh.connection { conn, peer, principal, repo }     gunnar::ssh
    └── git.transfer { service, access }             gunnar::git
          ├── git.upload_pack | git.receive_pack     gunnar::git
          │     └── store.ref_txn { document }       gunnar::store
          └── policy.gate { ref_name }               gunnar::policy

principal and repo are declared Empty at new_client and recorded when auth_publickey and exec_request learn them, so a line at any depth carries all four. Measured in the test capture: gunnar_wire::receive_pack and gunnar_pack::read, crates this change did not touch, come out carrying conn=…, principal=tester, repo=team/levels without a line of their own being edited. Two hand-offs would have dropped the span silently and both are explicit: tokio::spawn (the watchdog and the transfer task take .instrument) and spawn_blocking (WireTransport::serve captures Span::current() and enters it on the blocking thread, which is where the whole git conversation actually runs).

The request id is not a UUID. RequestId is a u64, process seed in the high 32 bits and a relaxed fetch_add in the low 32, rendered {:016x}. No dependency, no allocation, no lock, fixed width so it is greppable.

Timing, and applied outputs rather than "done". Every operation records a duration: git.transfer.finished (admission to last byte), git.upload_pack.served, git.receive_pack.served, policy.gate.*, store.ref_txn, store.repack.*. The lines carry what was applied: objects sent, bytes measured at the wire (a Metered<T> around the stream, not derived from an object count), negotiation rounds, common, protocol as spoken, refs ok / refs rejected, pack bytes, packs and loose objects folded, and copied vs recompressed (P-001's measure).

Counters, 18 of them, one block of AtomicU64 in ServerMetrics. gunnar serve mints one and hands the same Arc to the SSH listener and the transport, so connections_refused and pushes_rejected are readings of one server. No lock is taken to record one; transfers_in_flight is released by a Drop guard, not a call. A repacks counter was written and then removed: gunnar gc runs in a separate process with no ServerMetrics, so it could never have moved, and a counter that cannot move reads as "this never happens".

The heartbeat. One gunnar::metrics line every 60 s (ServeConfig::metrics_interval), skipped when nothing changed, plus one at shutdown. It answers "what is this server doing right now" from a journal an operator already has open, and cost four lines rather than a second listener and a second protocol.

Errors that used to be silent now say so. repo_store failing after the grant check said yes (the client saw a dead channel and the server said nothing); upload-pack/receive-pack failing mid-transfer; the ref transaction's final Err; a CAS that keeps losing (warn! from the second retry, because one lost race is normal and a document that keeps losing them is S-018); a policy engine that refuses without naming a gate.

Three emissions were broken and the suite watched go red, then restored:

broken red
span.record("principal", …) deleted from auth_publickey "the connection span does not carry the principal", 2 tests, 4 still green
metrics.add_gate_refusals(1) deleted "gate_refusals did not move: 0 → 0", 1 test
Metered::write stops counting "bytes_out is 0, which is not a packfile", 2 tests

What each test pins: the connection span's conn/peer/principal/repo as inherited, the span stack ssh.connection → git.transfer → git.upload_pack, and that every line under the connection carries the same id; a refused key producing a warn naming reason=unknown-key and the fingerprint, with no principal on it and auth_faults deliberately not moved; a gate refusal naming gate=branch-protection and ref_name=refs/heads/main, agreeing with the ref-level line; counters read before and after with per-field deltas, including the ones that must not move; that the essential events are at info or above so the shipped default is legible; and that every gunnar:: target is declared, with the remaining module-path targets pinned to a known list so the gap cannot grow silently.

S-047 — Negotiation re-derives every round from scratch, so a divergent fetch is O(rounds²)

# variant negotiate_ms commits_visited rounds
V0 baseline (87524d6) 197 248 18 901 251 1250
V1 + set-based have dedup 37 356 18 901 251 1250
V2 + one read per commit in the reach walk 34 781 12 641 251 1250
V3 + per-connection have memo 17 905 6 401 251 1250

11.0× on wall clock, 2.95× on object reads. The two read reductions land on their predicted values to the object: V1→V2 removes 6 260 000 reads, which is exactly the redundant second read of every commit the walk touched; V2→V3 removes 6 240 000, exactly the re-derived cutoff minus the 20 000 first-time have lookups the memo still pays. Arithmetic that lands on the nose is the evidence that the model of the cost was right, not just the direction.

Repeated back to back in the same sweep, to bracket the noise rather than assert past it: V0 again 173 757 ms, V3 again 16 140 ms, so 10.8×, against the 11.0× above. commits_visited was bit-identical on every repeat.

The wall clock on this box is noisy and is reported as such. V0 measured 158 394 ms on a quiet box, 197 248 ms at load average ~20, against the 154 490 ms the arm first recorded. commits_visited is deterministic and identical across every run, which is why the read counts above carry the argument and the milliseconds only corroborate it.

Each asserts cost(round 5) == cost(round 20) exactly, since a generous bound passes on anything, and asserts a floor of >= WINDOW separately, so a negotiation that stopped touching the store cannot pass by doing nothing.

SEEN RED, twice, in a worktree pinned at 87524d6:

probe result
the pre-fix code the two cost tests fail with per-round costs in exact arithmetic progression, [21, 41, 61, … 401] and [24, 44, … 404]. That progression is the finding, printed.
the fix, with HaveMemo deliberately answering every miss Absent all three fail, including the warm-versus-cold one.

The second probe is why the floor is >= WINDOW and not > 0. Against that broken memo the per-round costs were [1, 1, 1, …] and [2, 2, 2, …]: constant, non-zero, and entirely from fetch probing the want. A > 0 floor passed it. The guard was blind and was caught being blind, which is the only reason it is not blind now.

correctness

S-023 — MustNotExist is not a create guard on the fs backend, and the conformance suite cannot see it

S-024 — PushAttestation has no producer; three published event fields are constants

It is a PARAMETER of authorise, not a builder like with_git_protocol, and that is the whole design. An attach-after-the-fact field can be forgotten, and a forgotten one is silently None, which is the exact shape of the bug being closed. The compiler asks for it instead: adding the parameter turned every call site red until it was supplied, including seven in tests/sandbox.rs and three in tests/anonymous.rs. git_protocol stays a builder for the opposite reason: it comes from the client and must never reach the grant check. None remains a real state, not a missing value: the anonymous path of D23 authenticates with no key at all, and cannot push, so it cannot reach an attestation.

Nothing is recomputed. auth_publickey already had the fingerprint; it is the value the identity table was probed with one line above, so the session keeps it rather than deriving it twice (S-029 had just made it a first-class Option<Fingerprint> at that boundary, which is why this was cheap).

What the guess could never have got right. The replaced code read the identity table back and answered only when the principal held exactly one key, because with two there is nothing in the table that says which one made the connection. So every machine account with a rotation key, and every human with a laptop key and a desktop key, published an empty fingerprint on every RefUpdated, every RefDeleted and every attestation. The audit trail was blank for precisely the accounts it exists for, and no amount of care in gunnar-cli could have fixed it, because the information was not there.

The fixture is two keys, and a one-key fixture is why this survived. With one key the guess is right, and right for the wrong reason, so every existing test agreed with it. the_event_names_the_key_that_authenticated_when_the_principal_holds_two registers a second key for the same principal, pushes with the second over real ssh with IdentitiesOnly=yes, and asserts the published fingerprint is the second key's, taken from ssh-keygen -lf rather than recomputed by gunnar. It asserts its own premise first, that the principal really holds two keys and that the two fingerprint differently, so a degenerate fixture reports itself instead of passing.

SEEN RED TWICE, because one red only proves one of the two things this test claims.

probe result
the pre-fix guess (keys.len() == 1, else empty) left: "" · right: "SHA256:A8uKc/…"
"just use the principal's first key" left: "SHA256:H0mO59…" · right: "SHA256:Rnr9MM…"

The second probe is the one that was not optional. The test's message talks about the empty string, so the empty-string red was always going to appear, and a guard that only catches absence would happily accept a confident, wrong answer, which this entry states plainly is worse: "naming the wrong key in a provenance record is worse than naming none". Watching it reject the wrong key too is what makes that sentence enforced rather than written down.

PushAttestation's producer is proved by applied output, not by grep. The test asserts a PushAttestation arrives on a live Events.Watch subscriber from a real push, carrying the same fingerprint. A surface whose producer is asserted only by the producer's existence is how a neighbouring project's warehouse stayed empty across four sessions. cargo test -p gunnar-ssh -p gunnar-cli --features gunnar-store/fs: all green, push_events 7/7.

S-025 — touch_key is never called, so last_used is never set

The verification, so the next agent need not repeat it: grep -rn 'touch_key\|last_used' crates/*/src returns the definition (auth.rs:519), its // S-025 block (auth.rs:495-518), the field (model.rs:148), three None initialisers and the gRPC read at control_plane.rs:236. grep -rni sidecar crates/*/src finds nothing relevant — every hit is repack.rs's pack sidecars. No sidecar type, no sidecar file, no writer. Both routes this entry names are still unbuilt, so it stays open and half-wiring it would repeat exactly the failure S-024 was about.

What is new: touch_token — the same function for tokens, three hundred lines below touch_key in the same file — IS wired, on the HTTP request path. gunnar-http/src/service.rs:131 calls it for every authenticated request. It is the same DocRef::update — clone the document, modify one shard, flush the blobs, then a compare-and-swap on the single refs/gunnar/auth reference with MAX_ATTEMPTS = 8. That is the very cost this entry refuses to pay for keys, already being paid for tokens.

It is not the same defect, and the difference matters: the call is let _ =, so a lost CAS does not refuse a request that has already authenticated. This entry's stated fear — "a key that authenticated correctly would be refused because a bookkeeping write lost a race" — is avoided there. What is not avoided is the throughput cost, which is filed separately as S-048 rather than being folded in here, because it is a live serialisation point on a working path and not a dead function.

S-026 — The real-git interop suites self-skip and report green

#[test]
fn git_reads_a_sha1_repository_gunnar_wrote() {
    if !git_available() {
        eprintln!("SKIPPED: `git` is not on PATH, so §15 interop was NOT verified");
        return;
    }
    check(gix_hash::Kind::Sha1);
}

The message is honest, and it goes to stderr where no test harness reports it. To cargo test this is a pass with zero assertions executed. The same pattern is in three stock_clients.rs tests for ssh. The contrast is instructive: crates/gunnar-cli/tests/ssh_end_to_end.rs uses assert!(tools_present(), "this test requires gitandssh on PATH"), a hard failure. That is the right shape, in the same repository.

The RED. The four test binaries were built once, then executed directly with PATH pointed at an empty directory, so git and ssh are genuinely absent while the toolchain that built them is not involved. Before this change every one of these reported a pass; each now fails, naming the tool:

binary with git/ssh absent the message
gunnar-wire tests/real_git_interop (harness = false) exit 101, was exit 0 this test requires git on PATH; §15 interop cannot be proven without it
gunnar-store tests/real_git_reads_what_we_wrote 2 failed, 0 passed this test requires git on PATH (:146, :152)
gunnar-store tests/repack 5 failed this test requires git on PATH (:356, :362, :370, :415, :759)
gunnar-ssh tests/stock_clients 3 failed, 0 passed this test requires ssh on PATH, and git and ssh for the ls-remote case

1 + 2 + 5 + 3 = 11 of 11, the exact count this entry re-counted to. Every one is the hard assert! shape crates/gunnar-cli/tests/ssh_end_to_end.rs uses; no #[ignore] was added, so absence stays visible in harness output rather than in stderr. grep -rn SKIP crates/*/tests/*.rs now returns only stock_clients.rs's module doc explaining what the pattern used to be.

The green side is intact on a machine that has the tools: repack reports 7 passed; 0 failed; 1 ignored and real_git_interop reports 17 passed, 0 failed, out of 17, so the assertions did not simply become permanently red.

S-027 — write_stream allocates a client-declared size with no cap

fn write_stream(&self, kind: Kind, size: u64, from: &mut dyn std::io::Read) -> Result<…> {
    let mut buf = Vec::with_capacity(size as usize);
    Read::take(&mut *from, size).read_to_end(&mut buf)?;

size is a declared length, not an observed one, and there is no upper bound check, unlike gunnar_pack::IndexOptions::alloc_limit_bytes (read.rs:29-31), which exists precisely for this class of input. A declared u64::MAX either aborts the process on a capacity-overflow panic or attempts an absurd allocation, before a single byte is read. The buf.len() as u64 != size check at gix_adapter.rs:114 catches the mismatch only afterwards. On a 32-bit target size as usize truncates silently first.

write_stream now refuses a declared size above gunnar_pack::DEFAULT_ALLOC_LIMIT_BYTES (1 GiB) with an InvalidInput error naming the size and the limit, before it allocates, and clamps the pre-allocation as well so the 32-bit size as usize truncation cannot slip a rejected size past the check. A refusal rather than a clamp, for the reason the plan gave: a clamp reads a prefix and stores it under the hash of bytes that are not the object.

The ceiling is the sibling's number, not a second one (LAW 5). It turned out to be reachable without threading anything: gunnar-store already depends on gunnar-pack unconditionally (Cargo.toml; ObjectStore::packed() is why). DEFAULT_ALLOC_LIMIT_BYTES was pub in gunnar-pack's private read module but never re-exported, so the only change needed there was adding it to the existing pub use read::{…} line. No constant was declared in gunnar-store.

The RED, watched, under ulimit -v 25165824 as the fixture rule requires. With the check removed and only Vec::with_capacity(size as usize) left:

panicked at library/alloc/src/raw_vec/mod.rs:28:5: capacity overflow test result: FAILED. 1 passed; 1 failed

The case is u64::MAX deliberately, and the test says why: a merely large declared size (2 GiB) reaches the allocator, and an allocation failure aborts the whole test process instead of failing one test. u64::MAX exceeds isize::MAX, so Vec rejects it arithmetically and never calls malloc. cap + 1 is asserted alongside it, and both entry points, write_stream and write_stream_with_known_id, are driven separately, because "it delegates today" is a fact about the body, not the contract.

The second test is what stops the first being hollow: a_declared_size_within_the_ceiling_still_writes_the_object asserts a normal blob still round-trips and lands under the same id write_buf gives it. A write_stream that refused everything would satisfy every assertion in the first test perfectly. cargo test -p gunnar-store --lib → 42 passed, 0 failed.

S-028 — Unchecked subtraction in the loose-object decoder

The proof is in the callee. gix_object::decode::loose_header ends (gix-object-0.63.0/src/lib.rs:354-370, the version this workspace resolves):

let size_end = input.find_byte(0x0).ok_or(InvalidHeader { … })?;
…
Ok((kind, size, size_end + 1))

size_end is an index of a byte inside input, so size_end <= input.len() - 1 and the returned offset is <= input.len(). There is no path that returns a larger one.

MEASURED, and the measurement was proved able to fail (LAW 2). A standalone probe linking the same gix-object 0.63.0 walked 503539 inputs — every truncation of thirteen hand-forged headers (blob 18446744073709551615\0, blob -1\0, blob\0, \0, …), every legal <kind> <size>\0 shape across four kinds and seven sizes with payload lengths 0-7 and every prefix of each, and 500000 deterministic-fuzz strings over an alphabet weighted to the bytes the parser keys on. Of those, 1027 decoded successfully, min(len - offset) was 0, and offsets past the end: 0. The 1027 is asserted, not reported: an earlier corpus decoded only 19 inputs and the probe refused to draw a conclusion from it. The detector was then run against a deliberately broken decoder returning size_end + 2 and reported 239 violations and min(len - offset) = -1, so it is not blind.

A checked_sub was therefore NOT added, and that is the point of recording this rather than "fixing" it: with the callee's contract as it is, the error arm would be dead code that no test in this repository could ever reach, a guard that cannot go red, which is the class this file exists to catch. The residual risk is a future gix-object changing that contract; the honest defence against that is the length check decode_loose already performs on the next line (header claims {size} bytes but {n} follow), which is a real, reachable corrupt-object error. Nothing in crates/gunnar-store/ was edited. It was held by another agent this session, and there is now nothing to edit.

S-029 — expect() on client key bytes in the auth hot path, and on lock poisoning per request

let blob = key.to_bytes()
    .expect("a public key russh authenticated with always re-encodes");

This runs in auth_publickey, on every SSH connection, over a key the remote client supplied. The reasoning in the message is probably sound for a key russh has already parsed and signature-verified, but it is an expect() on client-influenced input in the authentication path, and D20 already records that this codebase has two incompatible ssh-key copies in the graph, which is exactly the situation where a future point release changes an edge-case encoding. Separately, stores.rs guards its handle-cache mutex with .expect("the repository handle cache lock") in three places. Any panic under that lock poisons it and turns every subsequent request into a panic: a single-fault server-wide outage. For balance: the SSH command parser (command.rs) returns typed CommandError variants for every malformed input with no unwraps, and gunnar-pktline's decoder is clean (S-014). This finding is narrow, not systemic.

thread 'access::tests::a_key_that_does_not_re_encode_is_refused_rather_than_panicked_on' panicked at crates/gunnar-ssh/src/access.rs:101:43: a public key russh authenticated with always re-encodes test result: FAILED. 16 passed; 1 failed

That panic is the production failure this entry describes, reproduced in a test rather than argued about. Two tests, because either alone is hollow. a_key_that_does_not_re_encode_is_refused_rather_than_panicked_on drives the error arm and asserts a real key still fingerprints to the ssh-keygen -lf known answer, so a fingerprint_of_encoding that answered None to everything, which would refuse every login on the server, cannot pass. server::tests::a_key_that_does_not_re_encode_is_rejected_and_counted asserts the applied output at the caller: the value handed back to russh is Auth::Reject, connections_refused moved by 1, and connections_accepted did not. An Auth::Accept would put a key nobody can fingerprint onto the server, which is the one outcome worse than the panic. auth_faults is deliberately left alone: it means "the identity table could not be read", and this key never reached it. The warn carries reason = "key-does-not-re-encode", which is what tells the two apart. cargo test -p gunnar-ssh: 78 passed, 0 failed. Not claimed: the refusal is asserted at the seam and at the helper, not through a live russh handshake. russh will not hand over a key that failed to parse, which is the reason the seam exists.

S-030 — Five swallowed errors, none logged or counted

Retitled 2026-08-03, from "Four swallowed errors, one of which leaks disk forever". The count is now five (a fifth appeared with S-006's repack), and the "leaks disk forever" claim is false since repack() exists. Both are corrected in the re-check below. The original title is kept here because the finding is cited by it elsewhere.

1. mem.rs: a poisoned lock no longer reads as an empty store. len() was .map(|o| o.len()).unwrap_or(0) and contains() was .map(|o| o.contains_key(id)).unwrap_or(false). Both are now recovered guards (unwrap_or_else(|poison| poison.into_inner())), matching what arrow_log.rs already does in this crate. Sound for the reason stated at the function: RwLock poisoning is raised by a panicking writer, and what the lock protects is a map of complete objects, so there is no half-inserted object to observe. contains was not in the audit's list of five and is the worse of the two: false tells connectivity that a reachable object is missing and tells a fetch that the client does not have something it does. Found while fixing the one beside it.

2. repack.rs, the .keep swallow, as ONE writer for both call sites. pub fn drop_keep(path) in gunnar-store, exported from the crate root, emitting error! on gunnar::store with the marker's path, and silent on NotFound, because already-gone is the outcome it wants. It does not return a Result, and the reason is written at the function: a surviving .keep is a stale marker, not lost data and not a lost repack, since PackFiles::all does not skip a pack that carries one. Failing a finished repack over it would turn a cosmetic leftover into an operator-visible error.

SEEN RED, and the first red probe was itself blind, which is worth recording.

assertion failed: a poisoned lock reported an EMPTY store; the object is still there — left: 0, right: 1

The assertion is on the wrong answer, not on an error type: 0 and false are shaped exactly like a correct answer about an empty store, which is the whole finding. the_fallible_paths_still_report_the_fault holds the other side: read must keep returning Error::backend on a poisoned lock, or a recover-everywhere fix would look identical from the outside.

a .keep that could not be removed produced 0 lines, not one. All events: []

The failure is forced by a real EACCES, a marker inside a directory made read-only, rather than by a stub, so it is the error the production call sites actually discarded. removing_a_keep_that_is_there_is_silent_and_actually_removes_it is what stops a drop_keep that logs unconditionally from passing, which would put an error line in an operator's journal on every successful push.

tracing-subscriber is a new dev-dependency of gunnar-store, invisible to cargo tree -e normal and so unable to reach the appliance graph S-007 guards. cargo test -p gunnar-store --features fs: 80 passed, 0 failed.

3. transport.rs: the .keep swallow now calls the shared writer. gunnar_store::drop_keep(keep), one line, so both places that retire a .keep go through one function instead of two copies kept in step by hand (LAW 5). Verified as a property rather than by reading the diff: grep remove_file … | grep -i keep over gunnar-cli/src and gunnar-store/src now returns no second writer, only the two drop_keep call sites and its definition. The stale comment the re-check named, "when gunnar gc learns to repack, this is where it stops", is replaced with what is true now: gc does repack, and dropping the marker here is what makes the pack eligible.

4. fs.rs. The dropped decode failure, fixed by giving the error a third state. Error::RefCas's actual was Option<Target>, filled with from_gix_target(actual).ok(), so "there was something I could not read" arrived as None, the same value as "nothing was there", on the concurrent-push rejection path, which is exactly where the difference matters. It is now Observed { Nothing, Value(Target), Unreadable(String) }, with a Display that says which, and the reason carried rather than dropped.

The finding's own count was off by one and the correction is in the code: it lists map_prepare_error's three .ok() sites, and there are two. The third line it counted is the MustExist arm, which passes Nothing honestly because gix reports no observed value there. That is now written at the arm so nobody "fixes" it again.

SEEN RED, on a reachable value rather than a forged one. MERGE_HEAD is a legal gix FullName and is legal nowhere in gunnar_store::refs, which admits HEAD and refs/… only, so a symbolic target pointing at it decodes in gix and not here. With observed_from_gix put back to from_gix_target(target).ok().into():

a reference that could not be decoded was reported as nothing

The test asserts the premise first (from_gix_target(&undecodable).is_err()), so if a future gix or a future validate() closes that gap the test says so instead of quietly exercising the decodable path. It also asserts the ordinary Object(id) case still comes back as Value, without which an observed_from_gix that answered Unreadable to everything would report every lost push as a corrupt repository.

5. server.rs. Teardown stays best-effort and stops being silent. note_teardown(step, result) emits debug! on gunnar::ssh with the step and the error. debug! and not warn! on purpose: one failed close is the ordinary case of a peer that hung up first, a thousand is descriptor exhaustion, and what tells them apart is a count in a journal rather than a severity on a line that fires constantly.

Genuine, fixed.

  1. gunnar-lfs/src/fs_store.rs, FsPendingWrite::drop's let _ = std::fs::remove_file(temp). This is the one with no backstop, and it is worse than the .keep swallows this entry is named for. The .keep cases cost a stale marker, because PackFiles::all collects a pack that carries one; the repack scratch cases cost nothing, because sweep_stale_scratch retries. Nothing in the tree ever looks at lfs/incomplete/ — grep finds only open_or_init creating it and one test asserting it is empty. So a rejected or severed upload whose temp file cannot be unlinked stays on disk at full size, under a name no oid resolves to, silently and monotonically. Now drop_incomplete, error! on gunnar.lfs, silent on NotFound, still infallible because it runs in Drop. Not gunnar_store::drop_keep, and the reason is written at the function: reaching it would make gunnar-lfs depend on gunnar-store, against that crate's stated dependency budget of gunnar-pktline + gix-hash + thiserror + tracing. Six lines is a poor price for a storage crate. SEEN RED with the swallow restored outright (not moved, which is the mistake the drop_keep probe made in this entry's own second pass):

an abandoned LFS upload that could not be removed produced 0 lines, not one. All events: []

Forced by a real EACCES — the marker inside a directory made read-only — and the test asserts the premise first (temp_files == 1, the file really did survive), so on a machine where the removal succeeds anyway it fails saying so instead of passing having proved nothing. A companion asserts the ordinary cancelled upload is silent, so a drop_incomplete that logged unconditionally cannot pass.

  1. sync_dir existed TWICE, in repack.rs and midx.rs, and the two had already diverged. repack's returned Err on an unexpected open failure; midx's dropped it. Same directory (objects/pack), same durability primitive, opposite answers to the same fault, which is the twin LAW 5 forbids and the divergence was real rather than hypothetical. Fixed by construction: one pub(crate) sync_dir in repack, midx calls it, the second copy is gone rather than watched. The swallowed sync_all inside it keeps its behaviour — a filesystem that refuses to fsync a directory handle is a durability weakening, not a correctness one, and is not worth failing a repack over — but now emits debug!, because a systemic one means every rename in the tree is unordered against a power cut and nothing else would ever mention it.

Named by the audit and NOT of this shape, with the reason, so the next agent does not "fix" them:

What is left of the class, and not claimed: nothing found in this sweep. The six sites from the two passes above plus these two are all of them; bus.rs:96 remains the documented non-finding it always was. RED, with the body replaced by the swallow:

assertion failed: expected exactly one line — the failure — got []

Both directions are asserted in one test, because a note_teardown that logged unconditionally would put a line in an operator's journal on every clean disconnect and destroy the signal it exists to give. The first red probe for this one was worthless and was replaced: it made the helper unwrap() a None, so the test failed on a panic inside the probe rather than on the assertion: green-looking evidence that proved only that the function was called.

Two of the entry's server.rs line numbers are not swallows and are deliberately left alone. let _ = handle_tx.send(running.handle()) is a oneshot to a receiver three lines below that does handle the failure, with a message about the listener ending before it accepted. Reporting it twice would be noise.

tracing-subscriber is now a dev-dependency of gunnar-ssh as well as gunnar-store, for the same reason and with the same property: invisible to cargo tree -e normal, so it cannot reach the appliance graph S-007 guards.

Six sites, not five. contains() in mem.rs was found while fixing the one beside it, and MustExist turned out not to be one at all. The count in the title is left as it was, because the entry is cited by it.

The five sites named as outstanding are all handled at master, verified at the code rather than from the entry: gunnar-lfs/src/fs_store.rs now calls drop_incomplete; sync_dir is one pub(crate) function at repack.rs:993 with midx.rs:191 calling it, so the twin is gone by construction rather than watched; and midx.rs:156 is the pre-clean before File::create, which truncates anyway and reports on the next line — not a silent failure, as the entry says.

Four remaining candidates were read and are NOT of this shape, named so the next sweep does not re-derive them:

Two genuine sites WERE found, both outside every crate this entry ever looked at, and both are gunnar-coldtier, which another lane held: unfold.rs:107 and lib.rs:496 are a third and fourth copy of repack::sync_dir's body — the same directory fsync, the same swallow, and not the debug! that copy was given when this entry deduplicated the first two. Filed as S-049 rather than fixed here.

S-031 — deepen-since / deepen-not / deepen-relative were never run against stock git

S-041 — PushGate is handed no new-object set and no pack byte count

The seam. PushGate::check(&RefCommand, &dyn ObjectStore) is now check(&RefCommand, &PushFacts<'_>), and PushFacts carries the store, the genuinely-new object set and the byte count. AllowAll and the two test gates (protocol.rs's RefuseEverything, real_git_interop.rs's RefuseThese) moved with it; four implementations in the tree, which is why a struct rather than a wider tuple — the next fact does not re-break them.

1. The object set is now exact, not over-approximated, and the subtraction is done where the answer exists. The .idx read-back is unchanged and still over-counts on its own; what changed is that the thin-pack base lookup is wrapped. index_thin_pack is handed RecordingBases, which delegates to GixObjects and records every id it is asked for and finds — by definition an object the repository already held, since gix consults it only for a delta base that is not inside the pack. Then introduced = everything in the .idx − the bases we handed back. That needed no second enumeration of anything (LAW 5) and no change in gunnar-pack, which another lane held.

2. pack_bytes is counted off the wire, by ReceivePack::serve itself. A Counting BufRead wraps the stream for the duration of ingest, so the number is true by construction: it is not IndexedPack's business and not the backend's, and the only code that sees every byte go past is the code that hands the stream over. Deliberately not the size of the file that landed — completing a thin pack writes bases the client never sent, and a store with no filesystem has no file to measure. A deletions-only push reports Some(0), a measured zero, so a configured byte limit refuses an oversized pack rather than a git push --delete.

3. The Arc<Mutex<Vec<ObjectId>>> + Arc<Mutex<Option<String>>> side channel is gone. PackIngest::ingest returns a PackReceipt — the indexed pack and Result<Vec<ObjectId>, String> — so the facts travel on the return path instead of through two locks the gate had to know to read. unknown_objects' refuse-on-undecidable behaviour survives, moved onto PushFacts::new_objects, whose Err arm is the reason it is a Result and not an Option: handing an empty slice for "I do not know" is exactly the shape that lets a configured limit be silently satisfied. The blanket closure impl of PackIngest (which most of gunnar-wire's own tests use) returns PackReceipt::without_object_set, i.e. the refusing default, not an empty set.

SEEN RED, three probes, each watched failing for its stated reason (crates/gunnar-cli/tests/push_facts.rs, real git push over real SSH):

  1. Subtraction removed, the pre-fix over-approximation restored →

the gate was shown a different object set than the push introduced. Extra (bases the repository already held, attributed to this push): ["705ea49b57af308ab5ef36727f16fb094ed9719f33911b7e1eb48e2e6db7ee5c"]. Missing: []

Exactly one extra id, and it is the first commit's blob — the delta base git did not send. Ground truth is git rev-list --objects HEAD --not HEAD^ on the client, not anything gunnar computes.

  1. with_pack_bytes removed → both byte tests fail, with the pre-fix message the entry predicted:

! [remote rejected] main -> main ([limits] a 512-byte packfile limit is configured but the pack size was not measured, so the limit cannot be enforced)

That is the entry's "a size-limit test cannot even be written today", made visible. The over-limit test asserts !told.contains("was not measured") precisely so it cannot pass on that refusal, and a companion test asserts a push under the limit still lands and the reference actually moved, so a gate that refuses whenever a limit is configured fails too.

  1. pack_bytes multiplied by ten → the cross-check fails:

the gate was told the pack was 77770 bytes, but the pack gunnar wrote for the same push is 7777 bytes. Those are not measurements of the same pack.

Written because "packfile is N bytes" with a constant, a zero or the uncompressed file size would satisfy an assertion that only looked for the phrase.

The vacuity trap this file was most exposed to, and what closes it. If git sends a self-contained pack, no base is pulled, the subtraction removes nothing, and probe 1 cannot tell the fix from its absence. So the fixture writes a ~117 KB compressible file and edits one line of it, and the test fails — not skips (S-026) — unless the pack gunnar stored holds strictly more object ids than the push introduced. That inequality is the evidence a base was pulled in. Measured: 4 ids stored, 3 introduced.

cargo test --workspace --all-features: green apart from gunnar-wire's pack_copy::a_client_without_ofs_delta_gets_a_pack_it_can_read and ::a_clone_of_a_packed_repository_copies_entries_instead_of_recompressing_them, which fail identically at origin/master (e4d8a78) in a clean worktree — the regression another lane holds, not this change, and checked rather than assumed.

S-042 — Nested-namespace repos are clonable over SSH and invisible to gRPC

Seen RED first, on assertions and not on a compile error. The constant and the error variant were added inert, so the new suite ran against the old validator: 3 of 4 tests failed, the load-bearing one with

the sandbox serves "team/sub/repo.git" as team/sub/repo, but the control plane refuses to name it: repository path must be exactly <namespace>/<repo>.

The fix is one line of structure and one of policy. parse splits at the last / (rsplit_once) instead of demanding exactly two segments; validate walks the namespace's levels plus the repository name, applies the same per-segment rules to each, refuses a / in the repository half (one component by definition), refuses an empty level, and bounds the total at MAX_COMPONENTS. Content is checked before the count, matching gunnar_core::sandbox, so a traversal is still reported as a traversal.

Nothing else had to change, and that is the measured part. The gRPC choke point is gunnar_grpc::status::repo_path → RepoPath::validate, and the one control-plane entry point is CoreControlPlane::repo_ref, which is path.validate() followed by RepoRef::new, the real sandbox. So the only thing that had ever rejected a nested path was this shape check, and Repos.List already emitted them (control_plane.rs:172, RepoPath::new(&repo.namespace, &repo.name) off a RepoRef that preserves nesting). Green after: 17/17 gunnar-proto, 52/52 gunnar-grpc.

On LAW 5: the rule is still stated twice, deliberately, and the sharing is a test. The entry asks for one validator called from both sides. It is not reachable: gunnar-proto is a leaf that dwarves and korp link for the control-plane types alone, and gunnar_core::sandbox arrives behind gunnar-store and gunnar-pack, and neither crate can take the other as a normal dependency without putting a storage stack in a client or a tonic stack in the appliance's model layer. So the second-best construction: tests/nested_namespace.rs takes gunnar-core as a dev-dependency (invisible to cargo tree -e normal, so feature_split.rs's appliance guard is unaffected) and reads gunnar_core::sandbox::red::LEGITIMATE, the sandbox's own corpus, asserting every path it accepts is nameable over gRPC. It re-derives each row through resolve_repo_path rather than trusting the table, refuses to run unless the corpus holds at least two nested rows (or it would pass against the validator that rejects all of them), and pins MAX_COMPONENTS against the real gunnar_core::sandbox::MAX_COMPONENTS so the one restated constant cannot drift silently. That guard was itself proved able to fail, because it was green from the moment it was written and a green never seen red is worth nothing: setting gunnar-proto's MAX_COMPONENTS to 5 turns the_component_bound_is_the_sandboxs_and_not_a_second_number and accepting_nesting_did_not_open_the_structural_escapes red: two independent detectors of the same drift, the second one because a/b/c/d/e becomes an accept.

Also asserted, because widening a validator is how an escape gets in: six rows from the sandbox's ESCAPES table that this change could plausibly have turned into accepts (a/b/c/d/e, ns//repo, ns/../../etc/passwd, ns/repo/../../../root, ns/repo/../other, ns/..\..\etc) are still refused, each for the right reason, and the test first asserts each row really is in ESCAPES, which caught two rows invented for this test that were not.

Deliberately NOT changed: RepoPath stays laxer than the sandbox. It still accepts non-ASCII, over-long components and shell metacharacters, because it is the shape check at the edge and RepoRef::new is what refuses those one layer down. Making it stricter would recreate this bug from the other side. A leading / is still EmptyNamespace: that is the ssh:// URL form, the control plane never sends one, and the property test skips those rows explicitly rather than quietly.

Ready to apply, in a crate held by another agent: crates/gunnar-cli/src/server/control_plane.rs:28-32 still carries model-mismatch note 1, "RepoPath cannot express a nested namespace … That is a gunnar-proto change, not one to make here." It is now stale and should be deleted (and notes 2 and 3 renumbered). The end-to-end half of this entry's Verify, create team/sub/repo and assert Repos.List returns it, belongs in crates/gunnar-cli/tests/ and is not written; the property test above covers the rule, not the wiring.

S-043 — Policy has no store: Policy.* is Unimplemented and gates read a JSON file off disk

smell

S-032 — design.md, storage-reasoning.md and both READMEs describe a design that was dropped

S-033 — There is no CI system; cargo xtask ci is the gate and a human types it

# green, normal PATH
cargo xtask preflight
#   git          git version 2.53.0
#   ssh          OpenSSH_10.2p1 …
#   preflight: PASS — every external binary the §15 tests need is present     (exit 0)

# red, a PATH containing ssh/ssh-keygen/cargo/rustc but NOT git
PATH=<stripped> cargo xtask preflight
#   preflight: FAIL — the interop tests of §15 cannot be proven:
#     git: No such file or directory (os error 2)
#   Do NOT run the suite without them: the interop tests print `SKIPPED:` and pass,
#   which is a green nobody has seen red.                                     (exit 1)

task_ci returns ExitCode::FAILURE immediately on that (xtask/src/main.rs:260-262), so the whole pipeline goes red before it compiles anything. That is the exact scenario the entry names, "push one that removes git from the test image and see the interop job go red", proven against the gate that exists rather than against the workflow that has never run. What is not claimed: no GitHub Actions run has ever happened, so the workflow's own jobs remain unverified, and the other half of the Verify clause (add a rayon dependency, see the pipeline go red) was not exercised because it means editing a Cargo.toml while eight agents are in the tree.

Neither this entry nor .nornir/ci-options.md §4 noticed. This entry sat open, quoting a python invocation that loads .github/workflows/ci.yml and reporting its keys; §4 settled a whole decision about a file that was gone. Two documents agreeing with each other, both wrong, which is the failure mode the whole stinky.md mechanism exists to catch and did not — the record went stale about the record.

Nothing has been restored. A workflow that has never run is not evidence, and putting one back on release day would re-create exactly the "looks like it has CI" hazard this entry was filed for.

S-034 — A 250 ms sleep per completed transfer, in the request-completion path

S-035 — The three znippy git indexes are unbuilt, and the stree key diverges from the spec

S-036 — Spec'd and unstarted: mirror replication, LFS

S-037 — unreachable! on a clap verb match

error[E0004]: non-exhaustive patterns: &ServiceCommand::Reload(_) not covered --> crates/gunnar-cli/src/service.rs:109:11 error[E0004]: non-exhaustive patterns: ServiceCommand::Reload(_) not covered --> crates/gunnar-cli/src/service.rs:131:11

Two errors, both at compile time, which is the whole of the clause. The probe variant was then removed. No unreachable! remains in any src/; the six hits in the workspace are all in tests/ and examples/.

S-038 — Three upstream nornir-git items forcing duplication here

The verification this entry named is the deletion, and the deletion has happened. crates/gunnar-client/src/ssh.rs is 280 lines of delegation, not duplication: :170 calls nornir_git::ssh::connect_exec_with, :194 calls connect_exec_auth, :179 calls nornir_git::ssh::receive_pack_command, and :34/:41/:48 re-export SshAuth / HostKeyPolicy / SshChannel rather than restating them. Its own module doc at :21-22 names connect_exec and SshAuth as the reason the copy is gone. crates/gunnar-client/src/porcelain.rs:291 calls nornir_git::gitio::porcelain_status_in and its doc at :279 says in as many words that the walk used to be re-implemented here. Not claimed: the push interop suite was not re-run for this closure, because the deletion was landed by the agent holding gunnar-client and that crate's tests belong to that agent's own verification. What is asserted here is the narrow thing this entry tracked, the three upstream items and the twinned code, and all four are gone.

S-045 — russh forces two C crates — the "no C toolchain" rule is not achievable

cargo tree -p gunnar-ssh -e normal       --prefix none | sed 's/ (\*)$//' | sort -u | wc -l   # 244
cargo tree -p gunnar-ssh -e normal,build --prefix none | sed 's/ (\*)$//' | sort -u | wc -l   # 257

S-046 — Two tests are load-flaky: green alone, red under a saturated workspace run

The cause was NOT the one everybody assumed, and that is the finding. The obvious suspect was assert!(worst <= SSH_IDENT_DEADLINE), an absolute 500 ms wall clock, which does measure the machine. It was replaced. But reproducing the flake showed the failure was somewhere else entirely: at load average 105 the test died on its vacuity check, hammered >= 2 * HAMMERS, with

the control plane answered only 18 calls in 4.3s of probing, so it was never actually loaded and the SSH measurement below is a measurement of an idle server

A true statement about the box, reported as a regression. The probing loop ran for a fixed second and the assertion then demanded a fixed amount of work inside it, so a saturated box failed the test by being saturated. Had only the wall-clock deadline been widened — the fix this entry explicitly forbids — the flake would have survived and looked cured.

Both are now relative or waited-for, not clocked.

MEASURED under deliberate load, oden 2026-08-03: 64 busy-loop processes on 32 cores, load average 155–176, the condition this entry describes. 8 runs, 8 green. Probe medians 249–602 µs against baselines of the same order — ratios of 1.1×–2.4× against the 200× bound. The old assertion's quantity, worst, ranged over 4 ms – 94 ms across those runs while the ratio barely moved: that is the load sensitivity that made it the wrong thing to assert on.

SEEN RED with the final loop shape, by restoring S-009's pre-fix shared runtime (tokio::spawn in place of control_runtime.spawn):

SSH identification had a median of 3.048758582s (worst 4.529513194s) … against 243.086µs on the SAME server seconds earlier … That is 12542× slower under load, past the 200× bound (48.6172ms). The control plane is starving the git listener

So the relative assertion still catches the bug it exists for, with two orders of margin.

The first version of this fix introduced a NEW flake, and it took a load run to see it. MAX_PROBES broke the waiting loop, so on a fast box the probe quota filled while the loaded control plane had served 31 of the 32 calls the measurement needs, and the test failed one call short. MAX_PROBES now caps the sample and not the wait. Recorded because a de-flaking change that adds a flake is the same disease.

Following this entry's own instruction — find what fails before touching a number — the test was run under the condition it describes, on oden (32 cores):

condition runs failures
64 busy-loop processes, load average 31 → 51 30 0
96 busy-loop processes plus two concurrent full cargo test --workspace runs, load average 58 → 154 369 0
whole gunnar-store suite under the first condition 3 0

399 isolated runs and 5 whole-suite runs, zero reproductions. The two workspace suites that ran alongside failed only on gunnar-wire's pack_copy pair, which fails identically at origin/master in a clean worktree and belongs to another lane.

What that rules out, which is the useful half of a negative result:

What is therefore still open, stated rather than guessed: the single observed failure has no captured output, so what assertion fired is unknown. The condition that was NOT reproduced here is the one the original run had and this one did not — other agents' cargo invocations competing for the same target directories and the same git binary, rather than synthetic CPU load. The next agent should capture the failure text when it happens rather than run more loops; this run cost about ten minutes and bought only eliminations. This entry stays IN-PROGRESS.

What the report used to drop, each of which narrows what broke:

Also: the loop runs twice, over Sha1 and Sha256, and said which in none of its assertions. "sha1 only" and "both" are different bugs. Every assertion now names it.

The formatting is split out of git() as git_report so it is reachable without catch_unwind — a panic-formatting path that can only be exercised by panicking is a path that cannot be seen red, and this one now is. SEEN RED against the old stderr-only report, driven by a real failing git rather than a synthesized Output:

the report must name the exit status — 128, 137 and 141 are different diagnoses: git ["rev-parse", "--verify", "HEAD"] failed: fatal: not a git repository (or any parent up to mount point /)

The test asserts its own premise first — that git rev-parse outside a repository really does fail — so on a future git that resolved it, it says so rather than passing having exercised the success path.

This entry stays IN-PROGRESS, and deliberately so. The flake is instrumented, not diagnosed. Nothing here explains the original failure and nothing should be read as claiming it does; what changed is that the next occurrence is readable.

S-048 — Every authenticated HTTP request does a single-ref CAS on the identity table

if let Some(hash) = &token {
    let _ = self.auth().touch_token(hash, gunnar_core::now_ms());
}

touch_token goes through DocRef::update, which clones the whole identity document, modifies one shard, flushes the shard blobs, and then does a compare-and-swap on the single refs/gunnar/auth reference, retrying up to MAX_ATTEMPTS = 8 times. So every authenticated HTTP request performs a document clone, a blob write and a CAS on one server-wide reference, synchronously, before the request is served.

S-051 — the build stamp's rerun-if-changed names nothing in a git worktree

cargo:rerun-if-changed=../../.git/HEAD
cargo:rerun-if-changed=../../.git/refs

In a git worktree ../../.git is a file holding gitdir: …, not a directory, so neither path exists. Cargo treats a rerun-if-changed path that does not exist as always dirty, so the trigger did not fire on the thing it watches — it fired on every build instead.

S-050 — every file:line in the cycle diagram was unguarded, and 8 of 24 had rotted

S-049 — sync_dir has a third and fourth copy in gunnar-coldtier, unlogged

if let Ok(f) = std::fs::File::open(&pack_dir) {
    let _ = f.sync_all();
}

Same directory-durability primitive, same swallow, and no debug! — so the reasoning that earned repack's copy a log line has simply not reached these.