nordisk/edda

Gunnar PLAN

The one live plan. What is done, what is next, what is blocked.

Updated: 2026-08-06 · Tests: cargo xtask ci's test step 1678 passed, 0 failed over 210 suites, oden 2026-08-06, with CHAOS_MONKEY_III's three new guards in it. It is not comparable to the 1343 of 2026-08-05: that was a different count, not a smaller run of the same one. fmt and clippy are RED, and were red on ba2c691 before any of this — four files nobody here touched, verified against a clean origin/master rather than assumed; see the CHAOS_MONKEY_III section. The older "517 workspace" figure counted a narrower set; the two are not comparable and no baseline was re-measured to bridge them. · Branch: master

Detail lives elsewhere and is not repeated here. .nornir/TOTAL-PROMPT.md is the specification and decision log, D1–D21, including the corrections made where a decision turned out to be wrong. .nornir/stinky.md is the audit, 45 entries, S-NNN. The scale analysis is .nornir/performance-design.md: 18 entries, P-NNN, measured on this machine.

Status lives in those files, not here. When an item closes, its entry changes there. This file is the ordering and the reasoning for it.

audit    45 entries    7 done · 3 in progress · 35 open      (9 open blockers)
scale    18 entries    all open                              (6 blockers)

Done

In hand

Seven agents. repack() · catalogue and auth cost · backpressure and parallelism · benches and the znippy git format · zlib and hash cost · ref-log follow-ups.

Next, in dependency order

Ordering is by what unblocks what, not by appeal.

1. repack() · S-006

No cold tier ever forms; pushed packs accumulate without bound. Everything about znippy storage is downstream, including delta reuse: there is no packed tier to reuse deltas from until this exists.

2. Settle the znippy storage questions · P-005, P-006, P-013

Before anything is written against ZnippyObjects, or it is rework. As designed the cold tier stores inflated content, which forecloses pack-copy permanently. znippy also pulls a C toolchain, cmake and a build-time network fetch, against §2.

3. Delta reuse · P-001

Gunnar inflates and re-deflates objects that arrived deflated, discarding compression the client's git already paid for. Derived at 28.49 ms → 0.91 ms per MiB per core. gix's single-variant Mode::PackCopyAndBaseObjects reads like a limitation and is the fix, already written. Needs (1).

4. Scale correctness · S-001, S-002, S-004, S-010

The catalogue is O(table) on reads as well as writes, and may() linear-scans every grant — one SSH connection costs four auth-document parses before any git. A fetch clones a repository-sized HashSet per request, ~900 MB at 10M objects. The handle cache never evicts. Fine at ten repositories, fatal at a million.

5. Backpressure · S-008, S-009

No concurrency cap on either listener; tokio's default 512 blocking threads is the accidental ceiling and saturation is invisible. The control plane blocks worker threads and starves the SSH listener.

6. Observability beyond the control plane · S-022 · DONE 2026-08-03

Both halves are closed. gRPC first (D9): every call authenticated, authorised and audited on gunnar::audit. Then the rest of it, as a tracing layer rather than more info! calls:

• one span per SSH connection, carrying a request id and the peer, plus the principal and the repository once those are known, with git.transfer, git.upload_pack / git.receive_pack, policy.gate and store.ref_txn beneath it. Lines from gunnar-wire and gunnar-pack, crates the change never touched, come out carrying the connection id and principal because they are emitted inside it;
• six sibling targets, gunnar::{audit,ssh,git,policy,store,metrics}, defined once in gunnar_core::observe and enumerated in TARGETS, so a filter is written against a list rather than a guess;
• duration on every operation, and applied outputs on every line worth reading: objects sent, bytes measured at the wire, rounds negotiated, refs moved and refused, pack bytes, packs folded, copied vs recompressed;
• 18 counters in one shared block of atomics, so no lock to record one and no allocation on the hot path, plus a 60 s heartbeat line that stays quiet while nothing changes.

info is a usable production default and a test asserts it: the essential events would fail the suite if any of them sank to debug.

Still dark, and named rather than implied: gunnar-wire/gunnar-pack/gunnar-store still log on module-path targets (attributable, but not selectable by RUST_LOG=gunnar::git); there is no metrics surface on gRPC, since Admin.Stats does not report ServerMetrics and that needs a gunnar-proto change; there are no histograms, so "the slow clone" is findable and "clones got slower this week" is not.

7. Benchmarks · S-044, P-015

12 arms proposed, nornir bencher not criterion. Nothing has ever been load-tested, so every performance claim in the specification is an assertion until this exists.

8. Parallelism · S-011, and read P-004 first

There is none in the serving path. But P-004 inverts the naive fix: gatling has no shared pool, per-request fan-out at 1000 concurrent clones means ~64000 OS threads, and the box is already slower past 32. Intra-request parallelism buys latency, not throughput. A measured decision not to parallelise the request path is a legitimate outcome; repack and bulk verification are where fan-out belongs.

Wanted, not scheduled

• Delta computation. gitoxide has no encoder, only reuse. Build as a donation-shaped crate (D21): no Gunnar dependencies, gix's conventions, MIT OR Apache-2.0. Keep it gunnar-* in-tree; publishing under gix-* would squat their namespace.
• Offer the three donation crates upstream. gunnar-delta (no delta encoder in gitoxide), gunnar-bitmap (no EWAH, no .bitmap) and now gunnar-sendpack (no push at all, in either direction). All three pass tests/donation_shape.rs, name no gunnar dependency and are MIT OR Apache-2.0. Nothing has been offered yet, and "gix lacks it" is not "gix wants it" — ask before assuming. gunnar-sendpack is the one to lead with: it is the biggest hole, and it is the only one gunnar itself consumes from both ends, which is the evidence that the shape is right.
• Reachability bitmaps wired in. Built in znippy, unused here.
• The crown: a bootable tunnr appliance ISO. The end of the road for §0. An installer image that boots Gunnar as PID 1 with no operating system underneath it, on real hardware. Containers are the rehearsal; this is the thing itself. tunnr already boots a Rust binary as PID 1 under UEFI in KVM and on metal, so the machinery exists. What is missing is Gunnar packaged into it. Order: container → KVM boot → real hardware, and the container leg is done: 2026-08-03, a scratch image whose entire filesystem is one 15 MB static-PIE binary took a stock git push and returned an identical 9-object graph to a stock git clone (cargo xtask images, and "MEASURED: the container leg" in .nornir/appliance-design.md). Nothing has been booted yet, in KVM or on metal, so §0's no operating system underneath is still an argument where it counts.
• LFS over pure SSH. Same session, same auth, no HTTP. Note that this is now the only gap smart HTTP left behind it: a repository whose blobs are LFS pointers clones its git objects over http:// perfectly and then fails in git lfs smudge, because /info/lfs/objects/batch is not one of the four URLs gunnar-http serves and is refused with the rest of the non-git surface. Measured on tunnr 2026-08-05 — the pack transfer was clean and git fsck passed; only the checkout failed. Whether the batch API belongs on the HTTP listener or LFS stays SSH-only is the open decision.
• TLS terminated by gunnar itself (D25). Today gunnar terminates none: it refuses to read a credential off a cleartext connection unless a proxy in front says the client spoke TLS (X-Forwarded-Proto: https), or unless an operator passes --http-allow-cleartext-credentials for loopback or a link they vouch for. That is the right default and it is not the whole answer for an appliance with nothing in front of it.
• git:// is never built. Not a gap and not scheduled: no authentication, no encryption, disabled by GitHub in 2022. It is the one transport this list rules out permanently rather than deferring.
• Replication to a second Gunnar (D15): warm standby, and a second copy on hardware you control.
• Shard by repository, the multi-node shape. A REST catalog is recorded as a rejected answer: skade's embedded catalog measures 2.0M ops/s at 0.31 µs p50, 492× Nessie and 677× Polaris, so swapping it for REST would surrender ~500× throughput to solve a problem a forge does not have. Repositories are independent: each node owns a set and its own warehouse, and the exclusive file lock never binds because two nodes never want the same one. What is needed is a routing table, not consensus.
• SeaweedFS. Investigate how it avoids filesystem lookup cost, and whether the technique applies to the cold tier. Not researched; do not repeat the claim until it is.
• Group commit for the ref log (S-018). One fsync per transaction today, and the redesign needs the benchmark from (7) to justify it.

Measured today, and it changes things

• A znippy archive of 60000 small git objects came out 3.4× LARGER than the input. 12.24 MB of source sealed into 42.1 MB, about 522 bytes of overhead per object. If the cold tier stores loose objects that way it inflates rather than compresses. This is a second, independent reason the cold tier must store packs rather than objects, and it was not predicted by anything in the spec.
• negotiate_divergent takes 154 seconds: 1250 rounds, 18.9 M commits visited. The most alarming number in the bench suite and not yet an audit entry.
• P-012 is real and large: 23.7× lookup degradation on misses across 64 packs.
• P-002's parallel index engages at scale but not at small size, 2.45× on a large push against 0.999× on a small one.
• znippy's LAW 3 rayon guard has never run. tests/rayon_free_law.rs include!s a file that does not exist, so the guard cannot compile and therefore cannot go red. It is znippy-zoomies' to fix. LAW 3 is unenforced there; the law holds by accident, not by guard.

Gunnar mot BUILD-MATRISEN — vad konstellationen kräver (2026-08-05 kväll)

Källa: workspace_nordisk/.nornir/build-matrix-2026-08-02.md (de fem formerna, 33 rader) och workspace_nordisk/PLAN.md § "Build Thing — nuläge, målbild och kvällens arbete (2026-08-05)" (kolumnerna). Gunnar står i ingendera. Målbilden säger uttryckligen "Alla fem repon (inkl. gunnar) ska ha suite-kolumnen med minst ORIGINAL", så det här är en lucka i gunnar, inte en frihet.

De fem formerna, konstellationens definition

ATOM          ROLE     LINK   VAD
<x>-server *  server   fat    tjansten
<x>           client   FAT    ALLT inkompilerat, INGEN gRPC, funkar utan server
<x>-cli       client   thin   pratar gRPC med en server, ingen lokal store
<x>-ui        client   thin   skrivbords-UI
<x>-solo      solo     fat    UI:t med sin server i, en binar

De fem kolumnerna, samma för alla repon

exe │ rpm │ container │ iso-installer │ iso-live      + PIZZA-axel (gpu)

exe är default och skrivs aldrig ut. container har valbar bas-image (minimal = busybox, rdp-weston). iso-installer och iso-live har alltid bas tunnr.

Gunnar TO BE

Ingen gpu-axel på gunnar. Det är en git-server; ingenting ritas. Den kolumnen är tom av ett skäl, inte av glömska.

Fyra avvikelser, mätta mot dagens deklaration

1. container och iso är ATOMER hos oss och ska vara KOLUMNER. gunnar-image och gunnar-appliance-iso är egna [[build.variant]]-rader (kind = "container", kind = "iso"). I målbilden är de leveransformer av en atom. Så gunnar-server ska kunna byggas som exe, rpm, container, iso-installer och iso-live — fem artefakter ur en rad — i stället för att container och iso är två separata rader utan roll.

Det här är den som ändrar mest, och det är också svaret på "tre filer till en release": tre var fel antal. Leveransen är kolumnerna, per atom, och den ska komma ur atomerna och inte ur handskrivna podman save-rader — build-matrix är uttrycklig: "Varje xtask som namnger ett paket, en feature-sträng eller ett bin-namn är en andra deklaration och måste bort. Likaså varje dok som beskriver ett byggrecept för hand."

2. gunnar är thin, matrisen säger fat. Formen definieras som "allt inkompilerat, ingen gRPC, funkar utan att en server kör" — vilket en git-klient uppfyller per definition. Men linkage-etiketten säger fat i matrisen och thin i vår deklaration, och vårt thin är bevisat med LinkageAudit (0 tonic, 0 prost, 0 gunnar_grpc; 17 936 680 B mot solo 27 725 776 B). Antingen ändras matrisen för gunnars fall eller så ändras etiketten. Detta är ett beslut, inte en bugg.

3. gunnar-ui finns inte. Matrisen kräver raden. Vad ett UI för en git-server ska göra är en produktfråga som ingen har ställt.

4. gunnar-service står utanför formerna. Den är kind = "service" och matchar ingen av de fem. Antingen är den en sjätte form konstellationen saknar, eller så är den en kolumn (som container och iso) förklädd till en rad — samma fel som punkt 1.

Vad som blockerar oss, och som inte är vårt

Målbildens egen lista över vad som fattas i nornir-build-thing:

1  rpm som target                      finns inte alls: ingen kind, cell eller byggare
2  ArtifactKind::Installer + assemble  InstallerBuild finns, armen fattas
3  discovery for wasm-modulen          ABI:n klar, ingen vet var modulen ligger
4  build-thing -> feature-graph        pizza-axeln skriven, testad, obeoende

Punkt 1 och 2 blockerar två av gunnars fem kolumner. rpm kan inte deklareras av någon, och iso-installer når man bara genom den separata dörren nornir build --installer <namn> eftersom assemble()-armen inte finns. De ligger i edda och tillhör den andra flottan; deras arbetsordning i kväll är 4 först, sedan 2.

Ordningen för gunnar

1  gunnar in i matrisen           5 atomer x 5 kolumner, ORIGINAL
2  container/iso -> kolumner      gunnar-image + gunnar-appliance-iso omformas
3  fat/thin-etiketten             beslut, inte fix
4  gunnar-ui                      produktfraga: vad gor ett UI for en git-server?
5  rpm + iso-installer            blockerade av edda, inte av oss

En sak som gäller oavsett ordning: ISO:n är inte byte-reproducerbar — identisk payload gav två olika sha256, och något i mastringen bär en klocka. Så länge det står kan en checksumma i en release-not inte verifieras av någon som bygger om.

CLOSED: det korrupta packet ur ett 16-pack-repo (2026-08-06)

Ett körfält som mätte genomströmning noterade i förbifarten att bygget fdf404c (2026-08-04) serverar ett pack som stock git vägrar — fatal: pack is corrupted (SHA1 mismatch) — ur ett 16-pack-repo, deterministiskt, medan git verify-pack godkänner alla 16 packen på disk och ett 13-pack-korpus går igenom på både fdf404c och master. Det jagade det inte. Jagat nu.

Defekten själv är redan fixad på master, av 0d10e17 (2026-08-05 11:47). Det är inte ett "master är fint" — orsaken är utredd och grinden är sedd röd. PackExtents (P-025) mäter en posts längd som avståndet till nästa offset i samma pack; ett multi-pack-index för bok över ett objekt som ligger i två pack mot exakt ett av dem, och gix-odb bygger det andra packets offsetlista genom att filtrera den avduplicerade tabellen per pack. Listan blir kort med precis de objekt indexet tillskrev någon annanstans, och posten före varje hål mäts tvärs över det.

fdf404cs learn_pack är masters med fullständighetskontrollen borta, rad för rad. Att ta bort den kontrollen reproducerar alltså det bygget exakt — och det gör den.

Fönstret: fc2e0c3 (2026-08-04 20:54, P-025) → 0d10e17 (2026-08-05 11:47). 6c31fa1 (2026-08-03) skrev multi-pack-indexet och är den andra halvan av utlösaren; ingendera commit ensam kan producera felet. fdf404c ligger 2 h 49 min in i fönstret och är inte speciell på något sätt.

Tröskeln är indexet, inte packantalet. MÄTT oden 2026-08-06

Pushat korpus, sexton filer, write_pack_copy utan fullständighetskontrollen, stock git 2.53.0 som mottagare:

threshold 16   packs  2 ok ·  4 ok ·  8 ok · 13 ok · 15 ok · 16 FAIL · 17 FAIL · 20 FAIL
threshold  8   packs  2 ok ·  4 ok ·  8 FAIL · 12 FAIL · 16 FAIL
threshold  2   packs  2 FAIL ·  4 FAIL

Klippan flyttar med MULTI_PACK_INDEX_THRESHOLD, på packet när. Golvet är två pack — det minsta ett multi-pack-index kan spänna över. 13-pack-korpuset var inte rent för att tretton är litet: det höll redan 130 av 240 objekt i mer än ett pack, och 16 av 240 redan vid två pack. Det var rent för att tretton ligger under tröskeln, så det fanns inget index. Med kontrollen på plats accepteras varje antal 2–20 vid varje prövad tröskel.

Varför ingenting såg det

Två flerpack-korpus finns i trädet och inget av dem kan uttrycka defekten. nornir-benchs packed_fixture skriver ids.chunks(per) — disjunkta pack, en residens per objekt — och sonderas bara för uppslag, serveras aldrig. Allt annat är en git clone --bare-utskärning, som packar om till ett pack. Det krävs ett objekt i två pack och en mottagare som tittar på de serverade byten, och ingen arm hade båda.

Två armar, vid det packantal där store:n skriver indexet obedd

• crates/gunnar-wire/tests/pack_copy.rs::a_clone_of_a_store_with_a_multi_pack_index_is_not_corrupt — hel klon ur MULTI_PACK_INDEX_THRESHOLD pushade pack, hållen mot stock git och gitoxide. Konstanten läses, inte stavas, så armen följer policyn om den flyttar. Alla tre villkor sedda röda: mottagaren (201 501 byte vägrade mot 194 782 accepterade, ≈6,7 kB överskott), fixturen vid 4 pack ("no multi-pack-index"), fixturen ompackad till ett pack ("no object … lives in more than one pack").
• bench/examples/nornir-bench.rs::gunnar.multi_pack_serve — samma form på sha256, och den namnger defekten i byte i stället för genom någon annans felsträng: samma store serverad två gånger ur ett öppnat handtag, en gång på GUNNAR_PACK_EXTENTS=off (gix-odbs inflate, som aldrig frågar ett index om en längd) och en gång på den levererade vägen. Lika, annars är skillnaden defekten. Utan kontrollen: 217 314 mot 201 177, +16 137 överskott, 8,0 % av packet — och det är ett A/B på samma fixtur, inte två byggen av en fixtur som bär riktiga tidsstämplar. 132 av 240 objekt i två pack, båda mottagarna accepterar, surplus_bytes 0. Sedd röd på överskottet, på stock git och på gitoxide var för sig.

DUP_FILES står kvar på tolv pack och dess doc säger nu varför det är ett val och inte ett tak: ovanför tröskeln avböjer de kortlistade packen sin tabell och closure_bases_declined faller till 0, vilket skulle göra den armens kanariefågel onåbar. Det var det öppna "femton är taket, av ett skäl som inte är denna defekt" i 7ab9d33 — det skälet var denna defekt, och det är stängt.

Vad som inte är mätt

• Defekten över en socket. Varje siffra ovan är serveringsvägen (UploadPack::fetch, stages::serve_on) med en riktig mottagare som läser byten; ingen gunnar-serverprocess kördes, eftersom korruptionen ligger i packet och transporten inte är i kedjan.
• Kostnaden i produktion. Ovanför tröskeln faller closure_bases_declined till 0 på detta korpus (mot 3–7 under den), så basstängningen ser inte den store:n alls; objects_copied_from_pack håller sig (214 vid 16 pack mot 213 vid 12). På knut.git mätte 0d10e17 avböjandet till 230 av 2915 indexbesvarade uppslag, 7,9 %. Ingen ny mätning gjordes av vad det kostar på ett riktigt korpus.
• nornir-bench sätter "passed": false och avslutar ändå med 0 — STÄNGT 2026-08-06, se nästa avsnitt.

FIXAT 2026-08-06: en bencharm kunde inte fälla någonting, och fixturen kunde inte se formen

De två orsakerna till att packbuggen ovan slank igenom är stängda. Bägge är samma sjukdom: en kontroll som inte kan rapportera rött.

nornir-bench skrev "passed": false och avslutade med 0

JSON:en hade rätt hela tiden; statusen ljög. Varje anropare som dömer en delprocess som ett skal gör det — &&, set -e, ett CI-steg, Command::status() — läste en körning med röda armar som en ren grön. Det stod som stående fara i PRATAs eget huvud och det bet gunnar.multi_pack_serve samma dygn som armen skrevs: armen kunde rapportera korruptionen, grinden kunde inte se den.

Kuvertet skrivs och flushas först, och först därefter avgör verdict exitstatusen, så en röd körning är nu både läsbar och skild från noll — ingenting som parsar stdout förlorar sitt underlag. Felet namnger varje fälld arm, bär dess eget meddelande och anger kvoten.

Fyra anropare hittade, tre var oärliga, alla tre lagade:

Två anropare var redan ärliga och rördes inte: nornir/src/release/pipeline.rs (failed_bench) och Ops.RunBench i server_grpc.rs (ok ur tests). runner-ändringen är vad som håller dem ärliga — hade den behållit sitt bail! hade grinden rapporterat failed_bench_exec utan ett enda armnamn och lagret fått noll rader.

De 20 examples/nornir-bench.rs i trädet är alla fn main() -> Result<()> som returnerar run_main_json() rakt av, så de ärver fixen utan en rads ändring. Kontrollerat, inte antaget.

gunnars åtta skalskript i .nornir/bench/ har alla åtta redan pipefail, med 23 rör till tail/grep/head/wc mellan sig. Ingen | tail-läcka att laga där.

Sedd röd. nornir-bench-redprobe är en riktig binär vars armar faller på beställning, och tests/failed_arm_exits_nonzero.rs läser dess ExitStatus — för verdict som returnerar Err och en process som avslutar skilt från noll är två olika fakta, och defekten bodde i glappet. Med det gamla slutet återställt (println!(…); Ok(())) faller de två rödarms-fallen med exit 0 while redprobe.red carried "passed": false medan det gröna står kvar. Bevisat också på riktigt gunnar-korpus, se nedan.

packed_fixture kunde bara skriva en form, och sa inte vilken

ids.chunks(per) — disjunkta pack, en residens per objekt, inga refs, ingen commit, så ingenting byggt på den kunde serveras till någon. Allt annat i filen är git clone --bare, som packar om till ett pack. Hela sviten höll alltså två former, ett pack och N disjunkta pack, och ingendera är den form en pushad server har.

Formen är nu ett värde (PackShape), den ligger i fixturens cachenyckel, och den mäts tillbaka från disk ur .idx-filerna i stället för att antas: en disjunkt fixtur som håller en dublett, eller en överlappande som inte gör det, faller nu i stället för att tyst byta vad dess arm handlar om. Den sonderade formen behåller sitt gamla namn byte för byte, så inget miljonobjektskorpus byggs om och P-012s serie baseras inte om.

gunnar.multi_pack_extent_cost är konsumenten, och den ställer en fråga ingen annan ställer. MultiPackServe och crates/gunnar-pack/tests/pack_extents.rs svarar bägge på är den serverade klonen korrekt — en vid sexton riktiga pushar, en vid golvet två pack i bägge hasharna. Ingen av dem mäter priset: learn_pack avböjer extent-tabellen för precis de pack indexet lämnade korta, och de objekten går tillbaka genom inflate-vägen som P-025 finns för att slippa. Det är en kvot över ett korpus, alltså en benchfråga. Den går på två sekunder i stället för minuter, eftersom formen byggs direkt i stället för genom git.

MÄTT oden 2026-08-06, 4000 blobbar / 16 pack / överlapp 8:

+12 708 byte överskott, 2,2 % av packet, och stock git vägrar det. De 6,4 % är avsiktligt det pessimistiska hörnet, inte en prognos: 120 dubletter spridda åtta per pack lämnar 15 av 16 pack korta. knut.git i den riktiga pushade formen avböjer 7,9 %. De två talen ramar in kostnaden.

Den första röda proben avslöjade den här commitens egen vakt som blind. GUNNAR_BENCH_MIDX_COST_OVERLAP=0 körde hela armen till en ren grön och rapporterade objects_in_two_packs=0 och extent_hit_frac=0.99 — fixturens kontroll är overlap == 0 ⇒ duplicated == 0, och att be om inget överlapp uppfyller den. Det deklarerade röda villkoret var en mening i en doc-kommentar och ingen assertion. Nu är det en assertion, och samma anrop faller och namnger formen. Fjärde gången på tre dygn att en vakt skriven under jakt på blinda vakter var blind.

Det som blev synligt först nu

gunnar.lookup_1m_64pack föll en gång av tre i preview-form på en box med last 47: misses run at 11441/s with a multi-pack index over 64 packs and 254765/s without one. De två följande körningarna gick igenom. Armens P-012-grind är alltså lastkänslig vid preview-skala, vilket betyder att den kan gå röd av fel skäl. Före i dag hade den röda körningen avslutat med 0 och ingen hade sett den. Inte utrett vidare; noterat.

Vad som inte är mätt

• Överskottet över en socket. Allt är serveringsvägen med en riktig mottagare som läser byten; ingen gunnar-serverprocess kördes.
• De 19 andra repona. Fixen i nornir-bench gör att varje bencharm i konstellationen nu kan fälla ett CI-steg. Ingen av de 19 svitarna kördes för att se om någon av dem redan är röd — den första körningen efter det här kan mycket väl bli det, och det är poängen.
• extent_hit_frac i produktionsform. 6,4 % är den här fixturens överlappsfördelning, inte en pushad servers.

Build thing: receptet, kvittot, och hur receptet distribueras (öppen, inte akut)

Rickards precisering 2026-08-06, och den korrigerar mitt första svar i PRATA. Två saker blandades ihop:

receptet   build thing = ett knippe byggskript, EN per produkt, versionerat
kvittot    vilken recept-version + vilka installningar byggde DENNA artefakt

.nordisk.buildthing-sektionen ska bära kvittot, inte receptet. En pekare — build-thing@<version> plus de valda inställningarna — inte en wasm-modul i varje binär. Mitt PRATA-svar (5ceba91) argumenterade för att modulen åker med artefakten; det är fel för att receptet då dupliceras i varje produkt det byggt. Pekaren är några hundra byte, och den upplöser samtidigt ISO-luckan jag flaggade i samma post: två lager spelar ingen roll när det som ska hittas är litet nog att stå var som helst.

Och då behövs distributionen på riktigt. Rickards förslag, och det starkare valet: ett znippy-arkiv serverat av holger. Skälet är inte smak — holger serverar redan sexton znippy-drivna repository-backends (server/repository/znippy-*). Ett build-thing-arkiv är ytterligare en pakettyp där, inte en ny mekanism. Content-addressed, versionerat, och det fungerar air-gapped, vilket ett crates-register inte gör.

Kratan förblir källan till receptet; znippyn är den distribuerade byggda formen. Samma delning som .rs mot .crate.

En sak att bestämma innan discovery byggs: pekaren måste vara content-addressed, inte bara en version. build-thing@3 säger vilket recept som var tänkt; en hash säger vilket som faktiskt kördes. Skillnaden syns först den dag någon frågar varför två artefakter med samma versionsnummer inte är identiska.

Bootstrap-svaret från PRATA står kvar och påverkas inte: .nornir/build-variants.toml är bootstrappen. Den svarar redan på manifest och variants — de två obligatoriska av elva verb — så en produkt utan modul degraderar till det den kan i dag i stället för att bli osynlig. Checka inte in .wasm.

Inget av detta är akut. Prio från t14s är exe + container + iso-installer för gunnar; rpm är parkerad.

Walk-börsen svälter en fan-out när den är snäv (öppen, mätt)

Byte-taket landade och kerneln klonar: 14 042 000 objekt, bitmapslöst, VmHWM 5,59 GB, 599,0 MB wire, 64 s. Trettiotvå samtidiga bitmapslösa kloner går från 4,66 GB till 1,09 GB med en börs på ~900 MB, och varje vägran är WalkPoolFull på tråden med besked om att försöka igen.

Det som inte är löst: en börs som är snäv mot en vandring svälter en fan-out. Mätt: fyra kernel-kloner mot en börs för två serverade noll. Ingen av dem fick plats, ingen gav upp sin plats, och alla fyra vägrades.

Det är bin-packing utan storlekar — servern vet inte hur stor en vandring blir förrän den har gjort den. Tre alternativa policyer är genomgångna på papper, ingen mätt.

Tills någon mäter dem gäller ordningen: --max-concurrent-transfers är primärgrinden, börsen är backstoppen. En börs satt så snävt att den blir primärgrind ger den här svälten. Default är halva MemTotal, vilket är långt från det läget — men den som sänker den manuellt ska veta vad som händer.

En siffra till, för den som läser gamla briefer: min 9,2 GB-extrapolering var 1,6× för hög. Verkligt tal är 5,59 GB, mätt. Och bytes per objekt spänner 393,8 till 1 572,9 vid samma storleksordning objekt — satt av innehåll, inte av antal, vilket är hela skälet att ett objekt-tak aldrig kunde bounda minne.

Known wrong, and said plainly

Open after the 2026-08-05 sweeps

Everything below was measured, not suspected. Three need Rickard's decision and are marked; the rest are work nobody has started.

Needs a decision, not a patch

• DEFAULT_BUDGET counts objects and is meant to bound bytes. Measured 29 412 B per emitted object on bifrost, 627 on a large repo — the ratio moves 47× across the range, so an object count cannot bound memory. The comment "on the order of a gigabyte" is 6× under: 10M objects ≈ 6.3 GB. Clearing the kernel needs ≥14.7M ≈ 9.2 GB per in-flight clone (inferred, 39× extrapolation). Moving the ceiling into Limits as a bytes ceiling is a fetch-path design call.
• LFS batch error semantics. RESOLVED 2026-08-05, and the reading was right: the SSH grammar genuinely cannot carry a per-object error. batch-oid-line = PKT-LINE(oid size action *(key=[value]) LF) (git-lfs/docs/proposals/ssh_adapter.md, where the pure-SSH protocol is specified — docs/api/batch.md governs HTTP only), there is no error production, and "unknown key-value pairs in the oid-line production" are specified to be ignored, so no extension can work either. The HTTP API's per-object {"error":{"code":404}} with the batch still 200 has no counterpart in the SSH batch response — but that is not the same as SSH having no per-object error at all, and reading it that way is what produced noop. get-object has a status-error-response; the 404 exists one command later, and is reachable only if the batch hands the client an action to ask with. Fixed without a whole-batch refusal, which would cost a client every other object in the batch: a download batch now hands back the download action for every oid and get-object answers 404 for what is absent — the one place the protocol does have a per-object error. That reaches over SSH the outcome git-lfs's own suite demands over HTTP (t/t-fetch.sh, "fetch with missing object": "should return non-zero, but should also download all the other valid files too"). MEASURED with git-lfs 3.7.1: before, a two-object fetch of which gunnar held one printed "2 objects found, done." and exited 0; now it exits 2 and names the oid. Cost: tq/ssh.go wraps every non-2xx get-object in NewRetriableError unconditionally, so an absent object walks the backoff ladder — 0.25+0.5+1+2+4+8+10+10 = 35.75 s — once for the whole batch, not per object. 507 added for ENOSPC/EDQUOT/EFBIG (it was a fatal Error::Io that dropped the channel with no status at all, not merely a 500), and the hash-algo refusal is 409, the code batch.md names for it. The 403 is declined on evidence: batch.md's 403 governs the batch endpoint, tq/ssh.go reads 403 from put-object as "the auth token expired, retry", so it would buy 8 futile round trips before the same refusal — 405 is immediate and is what the reference server sends. The protocol gap itself is filed upstream: git-lfs/git-lfs#6319.
• LFS over HTTP. A tunnr clone pulls its git objects and then fails in git lfs smudge: /info/lfs/objects/batch is not served on the HTTP listener. Batch API on HTTP, or LFS stays SSH-only.

The client is behind the server

• No shallow, no unshallow, no partial-clone filter in any client layer, while gunnar-wire serves all three. Gunnar can serve a partial clone it cannot make.
• No ls-remote verb — library only.
• fetch identifies as agent=git/oxide-0.86.0 while push says gunnar/<version>, so a server log cannot attribute a gunnar fetch.
• gunnar clone runs no .gitattributes filters, so LFS pointers check out as content.
• Error messages write the chain twice (X: X).

Operability

• gRPC has no request-id and no span. DONE 2026-08-05 (S-063). A tower_layer::Layer — not an interceptor, which cannot see a response — wraps the whole Routes table in gunnar_grpc::router, so the ungated Bootstrap and an unknown method are covered too. Every response, including every refusal, carries gunnar-request-id; gunnar control prints it on any failure from its one explain writer. The exit line adds elapsed_ms and grpc_status, neither of which an interceptor could produce. The id comes from gunnar_core::observe::RequestId, injected as a required argument because gunnar-grpc must not grow a gunnar-core edge (that would put aws-lc-rs and a C build script into the client,control graph). Costs the appliance graph 0 packages — 281 / 267 third-party, unchanged; tower-layer was already there as a direct dependency of tonic.
• No runtime log level. DONE 2026-08-05 (S-064). The filter is behind a tracing_subscriber::reload handle: SIGUSR1 walks startup → debug → trace → startup, SIGUSR2 goes straight back. Signals rather than a port or an RPC — the control plane is exactly what an operator cannot reach when the control plane is wedged, and both signals already existed as a hazard, since their default disposition is to terminate the process. The acknowledgement is emitted under the more verbose of the two filters (after the reload going up, before it coming down), or a server at RUST_LOG=error would get silence as its confirmation. Cost: reload disables tracing's callsite interest cache for that layer, so the filter is evaluated per event rather than per callsite, on every gunnar process.
• No metrics endpoint DONE 2026-08-05 (S-065). gunnar serve --metrics-bind <addr> serves the nineteen counters as Prometheus text on GET /metrics. Nothing binds by default — no flag, no port, the same shape --http-bind uses — and the flag says in its own help that there is no authentication on it, because the body is nineteen integers about the process and no repository name, principal or reference. A non-loopback bind is announced at warn and on the start-up banner. Costs the appliance graph 0 packages — 281 / 267 third-party, unchanged, MEASURED by cargo xtask guard before and after: hyper, hyper-util, http-body-util and bytes are already resolved there by tonic, and it is under server rather than the default-off http feature deliberately, because the shipped gunnar-server atom is where an operator can least attach a debugger. Asserted on what an HTTP client receives over a socket after a real git clone: gunnar_bytes_out is not allowed to be zero, and the failure says so in those words.
• The heartbeat was publishing 18 of the 19 counters. Found and fixed 2026-08-05 with S-065. connections_anonymous — written by both transports, and the answer to how much of my traffic is strangers — appeared in no line gunnar had ever printed, because emit_metrics was a hand-written list beside the struct and nobody updated it when the field arrived. Both the heartbeat line and the scrape body now expand from the same counters! arm that declares the fields, so neither can fall behind (LAW 5).
• Seventeen tracing call sites were on a target no filter selects. Found and fixed 2026-08-05. gunnar-lfs emitted on "gunnar.lfs" and gunnar-coldtier, gunnar-cli's cold-tier arm and gunnar-znippy on "gunnar.store" — a dot, not :: — so RUST_LOG=gunnar::store=info, which gunnar_core::observe's own table tells an operator to use, selected none of them. Five target guards were green above them: every one pins a named constant against another named constant and none had ever looked at a call site. every_tracing_target_in_the_shipped_source_is_one_of_ours reads the source of every shipped crate and is the guard that can.
• The clone path emitted one line and it was unfilterable. Fixed 2026-08-05. gunnar-wire's upload_pack summary was on the module-path default, so RUST_LOG=gunnar::git=debug selected the push path's seven phases and nothing a clone did. It is git.upload_pack.sent on gunnar::git now, and gunnar_wire::upload_pack came off the recorded remainder in info_is_a_usable_production_default.
• gunnar control bootstrap's second-bootstrap refusal printed no request id. Fixed 2026-08-05. That arm writes its own prose and bypassed explain, the single site that appends the id — on the refusal a first-time user is most likely to hit and the one where the caller holds no credential and can ask the server nothing else. The id is now appended by with_request_id, which both paths reach.
• midx rendered a failed stat as 0 bytes. Fixed 2026-08-05, the same shape as the pack_bytes lie: an absence rendered as a value, so a multi-pack index that could not be measured read as one that came out empty.
• gunnar control has no proto RPC for gc, token or reflog seal.

Not swept at all

• gc under load, and gc concurrent with a live push or fetch.
• Adversarial auth on the SSH side: wrong key, revoked key, principal deleted mid-session, namespace vanishing between check and use. LFS's directional grant is clean — one decision, both directions.
• refs/heads/x vs refs/heads/x/y D/F collision, and concurrent pushes to one ref. The rest of ref semantics measured clean: atomic is one RefStore::transaction, the CAS is PreviousValue::MustExistAndMatch, force/non-ff matches git's server default.

Accepted positions, written down so they cannot drift into surprises

• A refused push leaves its objects in the ODB. Measured: 3 of 3 objects, 7 781 pack bytes, and the ref never moves. There is no quarantine; stock git migrates out of a temporary object directory after fsck and hooks, gunnar accepts first. Guarded by name in a_refused_push_leaves_its_objects_in_the_object_database, so a change is a decision rather than a drift.

• gunnar gc has no prune by default — --prune exists now, but rejected and force-pushed objects survive until someone runs it.

• The ISO is not byte-reproducible. Identical payload — same binary, same .znippy, same pin — produced two ISOs of identical size and different sha256. Something in the mastering carries a clock, most likely ISO-9660's volume creation timestamp. That is edda's korp-installer, not gunnar's. It matters because an ISO's digest then cannot be its identity, which is the family of defect that produced the BUILT … 0 artifact(s) lie.

• Unpinned 32-client throughput is still 2.088× behind gitea EXPLAINED, and re-measured at 1.168×. The gap is not between gunnar and gitea, it is between two ways of spelling eight CPUs, and the explanation was written down on 2026-08-04 in serve::report_cpu_budget's own doc comment and never reached this file. Re-run of the same arm on 2026-08-06 at load1 152.9: gitea ÷ gunnar is 1.168× unpinned and 1.206× pinned — the two columns have crossed. The +32% CPU-per-operation ramp is real and current and is the same fact, not a second one. See "ANSWERED: the unpinned 32-client gap" below, including where 6.45 and 13.48 do and do not exist.

• The O_EXCL lock does not self-clear after a hard kill, and on an appliance there is no operator to run force_unlock.

• Refs.ReadObject inflates before applying a size limit. A bounded read of a delta-encoded object is not possible without materialising its base chain, so the honest fix is a header check and an explicit refusal, not a patch.

• znippy's git format tests have never been verified by us. They are on feat/git-package-format, rebased onto the rewritten history and pushed. Not merged to master for exactly that reason.

• Gunnar has never been booted as an appliance, and fs remains the only object store surviving a restart, so "no filesystem" is true of the design, not the server.

DONE: delta-as-a-chunk, instead of a second reader

Landed 2026-08-05 on znippy master — 2a65e83 (the seam move, byte-identical) and 04290b5 (the delta chunk variant). What follows is what it turned out to be worth, measured, including the part where it makes gunnar no faster.

EntryReader sat on the read side and its two implementations differed in how they combine: chunks concatenate, deltas apply. That is what forced two of them, and it put the choice per entry. It is now one struct with one reconstruction loop, and ChunkSource::{Stored, Delta { base_path }} says per chunk where the bytes come from. The seam moved to the write side as Splitter, with three implementations: ChunkSplitter (what znippy does today), DeltaSplitter (what EntryReader's delta arm could express) and RegionDeltaSplitter — a file tiled against its previous version, unchanged tiles stored as a ~6-byte COPY, changed tiles stored whole. That last one is the point, and EntryReader had no way to say it.

Byte-identity re-taken. bc191568a85e66bd67773a75ba4fbf1438762cc562bdb190befbced15fa3e8fd is unchanged, so the constant has now survived both the trait's arrival and its removal. Seen red three ways before it was trusted (drop the fdata_offset sort; flip one bit per chunk; return a size one byte too large, which the round-trip cannot see and the digest can). A fourth probe — summing uncompressed_size instead of taking the max — does not fire, because the corpus tiles contiguously and sum equals max on it. That is a real hole in the golden test and is recorded rather than papered over.

Integrity, and it is not the same property. A chunk's blake3 is over the bytes it contributes to the file: the decompressed payload for a stored chunk (what the column always meant), and the bytes the delta produces for a delta chunk — not the instruction stream. That is what result_checksum was for in the per-entry model, except per chunk, so a mixed entry is covered too, which result_checksum could not be. It is checked on the fast read for delta chunks and only under verify for stored ones.

The chunk-chain depth curve, which nobody had derived. MEASURED oden 2026-08-05, release, 256 kB payload, 20 reps, taskset-free (load 3–7):

Three things follow, and the first is the answer to the question the table left open. A chunk chain does not inherit the entry chain's curve. It is linear — 72.6 µs per marginal link, flat from depth 4 to 50 — where the per-entry arm's own measurement had a knee past 32 (104/152/323/1289 µs at 1/8/32/50, 2026-08-04). Second, 79% of the depth-50 cost is the per-link blake3, i.e. the integrity property and not the seam; with it off the curve is 56→752 µs. Third, the fan is nearly free in K, because the base is resolved once — a memo local to one reconstruct, which is the one new cost this shape introduces and is not optional: without it, RegionDeltaSplitter's output would get slower the more of the file is unchanged. a_mixed_entry_resolves_each_base_exactly_once counts the resolutions rather than timing them, and goes red at 4 when the memo is removed.

A hole closed that the per-entry model also had. MAX_DELTA_CHAIN is a policy a writer applies to its own output; it binds nothing about an index that arrives from elsewhere. A base that names itself used to recurse until the stack ended — verified, the probe aborts with SIGABRT. MAX_RECONSTRUCT_DEPTH = 64 makes it an Err.

Does it make gunnar faster? No, and here is why that was knowable in advance

Which arms could possibly move, stated before measuring. znippy enters gunnar only through gunnar-coldtier, which is in the resolved graph only under gunnar-cli/znippy, which is off by default. bench/Cargo.toml names no gunnar-coldtier and no znippy-common/-compress/-plugin-git — only znippy-zoomies, a different repository this change does not touch. So none of the 48 arms in the shipping suite can move, and re-running them either side would have measured noise twice. The arms that can move are the six in crates/gunnar-coldtier/bench, which cargo test --workspace and nornir bench run gunnar both cannot see.

And they did not move either, because there is still no delta writer: build_file_index constructs only ChunkSource::Stored, since the Arrow index has no column naming a base. MEASURED on oden 2026-08-05, both binaries built from the same gunnar tree against znippy ab9cc7b and this branch:

The noise floor for that last row, and the reason the +0.12% means nothing: two byte-identical copies of the same binary run as separate arms in the same interleaved loop differ by −0.23% on medians and −1.23% on means, with per-arm min–max spreads of 13.9–23.2% on a box whose load ran 3–135 during the session. The subject's difference is smaller than the instrument's own floor.

On its own it makes gunnar exactly as fast as it was, which is the expected result for an enabling change: it makes "part chunk, part delta" expressible and removes a stack-overflow hole. coldtier_delta_append's delta_efficiency_x of 0.0608 made the whole idea look hopeless, and that reading was wrong — it was measured on a corpus git had already deltified before znippy saw it. The case that pays is cross-generation, and the writer for it landed the same day; see below.

It does not break the release wave. release/0.9.13-prep does not contain the trait commits at all (git merge-base --is-ancestor 921261f origin/release/0.9.13-prep is false), so EntryReader, ChunkedEntry and DeltaEntry — public for three commits on master and never published — are removed from a surface the wave does not carry. gunnar's own cargo xtask guard is 280 crates / 267 third-party / 0 of 16 banned / 7 tracked before and after, and all 29+6 gunnar-coldtier and cold_tier_wired / refs_from_archive tests pass against the new znippy unchanged.

DONE: the delta writer, on superseded gc generations

Landed 2026-08-05 on znippy master ac2d4c7. Measured before it was built, on real gc generations of nornir — pushed into a bare repo and repacked, never clone --bare — with znippy's own encode_delta_against, not zstd --patch-from:

The control is what makes this a redundancy result and not a compression one: zstd -19 on the newer pack alone is 1.02×. Pack bytes are already deflated, so compression buys nothing and cross-generation redundancy buys 4–15×. znippy's minimal encoder lands within 2% of zstd --patch-from on the same pair (2.00× vs 1.96× from a 400-commit-distant base), so no new encoder is needed — the one already in the tree is fit for this.

Also measured, and not obvious: repacking the same commit twice still needs a 262 kB delta. git repack is not byte-deterministic, so a gc that adds no commits is not free.

Direction: the SUPERSEDED generation is delta'd against the LIVE one, never the reverse. The current generation is what P-001's write_pack_copy copies entries out of on every clone; putting it behind a chain would move it onto the entry-copy path, 18× the bytes and 221× the CPU. Backwards chaining keeps the live generation whole at depth 0 for ever and pushes depth onto superseded generations, which no serve path reads.

What it saves and costs — real eight-generation nornir chain, oden 2026-08-05:

The two design calls, decided with numbers rather than argued.

Per-link blake3 is kept. It is 56 % of the cold read (140.2 ms at depth 7 against 61.4 ms with it off) — not the 79 % the synthetic 256 kB case suggested, because a real link does more decompression. It is the only thing that catches a delta applied to the wrong base, and a generation chain is exactly where that is plausible: every entry is a pack of the same repository at a similar size, so a neighbour would apply cleanly and produce plausible garbage. Trading that for 80 ms on a read nothing makes is the wrong way round.

MAX_GENERATION_CHAIN = 32, half the reader's MAX_RECONSTRUCT_DEPTH of 64. Past it the writer declines and the caller keeps that generation whole — a keyframe, so every generation stays reachable in at most 32 links for ever. A weekly gc reaches 64 in fifteen months; this is designed now rather than discovered. Thirty-two links is ~0.6 s, on an archival read no fetch performs.

Format: __znippy_delta__, a reserved section of (relative_path, chunk_seq, base_path) — not a delta_base column. base_index_fields() is the schema of every archive znippy has ever written, so a column means a ZNIPPY_FORMAT_VERSION bump, a new column every writer must supply, and a changed on-disk shape for the iceberg sink and for DuckDB — all to describe a property almost no row has. A reserved section is additive by construction and needs no bump. Absent, every chunk is Stored and the read path is byte-for-byte the one bc191568… pins, which still passes.

AMENDED 2026-08-05, and the amendment matters: the 5.26× above is the archive's LIVE bytes, not its size on disk. supersede_as_delta appends the delta and drops the superseded entry's index rows — the superseded blob stays in the file, unreferenced. MEASURED: an 8 056 624-byte archive went to 8 057 731 after a supersede that replaced 4 000 000 bytes of live payload with 11. So an archive fed one generation per gc and never rewritten grows by a whole generation each time, which is exactly what storing them whole would have cost. See the next section.

DONE: gunnar gc retains the generation it supersedes

Landed 2026-08-05. ColdTier::fold — which is gunnar gc's path — now retains the pack the seal just wrote as the newest link of a chain, and delta's the one it replaced against it. GUNNAR_COLD_TIER_HISTORY=off turns it off.

It is a second archive, gunnar-cold-history.znippy, and that is not tidiness. The live archive is not opened, not read and not written by any of it, so "the live generation's read is unchanged" is a property rather than a measurement. The reason it has to be one is MEASURED, on znippy ac2d4c7:

HotColdTier is the push path and re-heats whatever archive it is given, so a delta chunk in gunnar-cold.znippy would survive exactly until the next push, and fail silently. a_hot_seal_destroys_a_delta_map asserts it so that stays known.

__znippy_delta__ is carried across an append, and as decoded ROWS (like __meta__, unlike __gunnar_refs__) — it is deliberately not in CARRIED_RESERVED_MODULES, because an append may add rows to it and the manifest cannot hold two sections under one module name. Asserted both ways.

What it saves, 19 real workspace_nordisk repositories, oden 2026-08-05

Eight generations each, built the way a server produces them: git push into a bare repository at eight checkpoints spread over its real history, repack() with the cold tier after each. raw is what those generations cost stored whole. Every generation is read back through however many links it sits behind and compared byte for byte with the packfile repack() wrote at the time; the exact column is that comparison and every row is true.

3.96× over the corpus, 2.92× – 5.48× per repository, and every generation byte-exact. Seventeen are sha1 and tunnr is sha256, so both hash kinds are in the corpus. bifrost is not measured: its checkout at /home/rickard/git/bifrost is corrupt (git rev-list HEAD → packed object … is corrupt), which is a fact about that checkout and not about this.

The saving is lower than the 5.26× the writer measured because it is a different chain: eight checkpoints spread over a repository's WHOLE history are far apart, where the writer's chain was eight consecutive gcs. Consecutive generations delta better — the writer's own table says so (4.08× at 100 commits apart, 32.32× at zero) — so 3.96× is the pessimistic end of the range a real server sits in.

Cost, stated plainly rather than buried: the history archive holds one whole generation as its keyframe, so it is roughly the size of the live archive plus one delta per superseded generation. Retention therefore costs about 2× the disk of a repository with none — and keeping those eight generations WHOLE would have cost 4.1× that again. There is no way to avoid the whole keyframe with the format as it stands: the base of a delta has to be in the same archive.

build_ms_median is the whole repack() including the fold, 85 ms (draupnir) to 1431 ms (nornir), so the supersede is inside the noise of a gc that was already doing the work.

The live generation's read is unchanged, measured on the LIVE archive with history off and on — the last column. Median of the on/off ratio over the 14 repositories above the 0.2 ms clock floor is 1.000. The one outlier, edda at 5.18 → 1.92 ms, is the off arm paying a cold page cache because it is measured first; it is called out rather than averaged away. This is the number the design exists to protect, and what protects it is the history never touching that file — which is a property, not a result, and the_live_archive_is_untouched_by_the_history is where it is asserted.

The two live archive columns are two separate bare repositories fed the same pushes, so they differ by a few hundred bytes on 10 of 18: git repack is not byte-deterministic, which the writer's own measurement recorded. Eight of eighteen came out byte-identical.

The decline is a correct outcome, not an error. Past MAX_GENERATION_CHAIN = 32 the writer refuses, the generation stays whole and becomes the keyframe the next chain is built from, so the deepest generation never passes 32 however many arrive. the_chain_declines_past_the_cap_and_the_keyframe_restarts_it drives 35 generations and asserts both halves.

Stock git 2.53.0 judges it, sha1 and sha256: after four supersedes a real git clone and a real negotiated fetch land the tip, fsck --full --strict passes, and every superseded generation — reconstructed out of the chain — is a packfile index-pack --strict accepts and fsck --full --strict passes over. Seen red by flipping one byte of the reconstruction: inflate: data stream error.

gunnar gc refuses against a live data root ("the store is locked by another writer"), and that turns out to help rather than hinder: the lock means the retention path never runs while the push path could be re-heating the live archive, so the two writers cannot meet.

FIXED: reserved sections were dropped on append

MEASURED 2026-08-04 and now fixed on znippy master 25850cd. An object-carrying push took the archive from 144 928 to 146 396 bytes with __gunnar_refs__ gone, because ArrowIpcSinkAppend::open_existing truncates the whole metadata tail and re-supplies no reserved section it was not handed — and nothing hands it one, since that sink has no with_reserved_builder at all. So a repository lost its ref log on the very next push after the one gc that wrote it.

The distinction the append path could not express is the fix. __gunnar_oid__ / __gunnar_graph__ / __gunnar_reach__ are derived from the objects and must be dropped when objects change; __gunnar_refs__ / __gunnar_secrets__ are independent logs and must be carried forward. CARRIED_RESERVED_MODULES names the second kind, open_existing reads their raw bytes before it truncates, and finish re-emits them verbatim.

Guarded at both ends, in znippy and here. znippy's an_append_carries_the_ref_log_forward_and_drops_the_derived_sections goes red both ways — drop the carry-forward and refs read back None; add __gunnar_graph__ to the carried list and the derived assertion fires. And gunnar's own a_push_reaches_the_znippy_cold_tier_and_gc_seals_its_refs now continues past the seal into a second, object-carrying push: it is GREEN against the new znippy and RED against ab9cc7b with the message "the second push ERASED gunnar_refs". That is the end-to-end proof, on gunnar's own path, and it unblocks serving refs from the archive for the rest of a repository's life.

ANSWERED: the unpinned 32-client gap, and what the number is worth

The question. gunnar wins on fetch bytes, push CPU, ls-remote and RSS and ties on clone, and one figure did not fit: at 32 concurrent clones gunnar was 1.083× behind gitea pinned and 2.088× behind unpinned, and gunnar's CPU per operation rose from c1 to c32 while gitea's fell. Both were carried in this file as unexplained.

The answer: they are one fact, not two, and it was already written down

The two configurations are podman run --cpus 8 (a cgroup cpu.max quota of eight CPUs' budget, spendable on any of oden's thirty-two) and the same quota plus a disjoint eight-CPU affinity. gunnar serves one transfer on threads that hand work to each other continuously — a hyper task on a runtime worker, and the spawn_blocking thread that runs the git state machine, bridged by a tokio::io::duplex. Under quota alone the scheduler has ~N runnable threads and thirty-two places to put them while only eight may run at a time, so a hand-off crosses a core, and a cross-core hand-off is a futex. perf record on the serving process on 2026-08-04 put 40% of the server's CPU in native_queued_spin_lock_slowpath, the kernel's futex hash-bucket spinlock.

That paragraph is not new: it is gunnar_cli::server::serve::report_cpu_budget's doc comment, and bench/examples/forge/mod.rs::pin_cpus says the same. What was missing was the connection to this file's two open figures, and the connection is arithmetic. From the raw 2026-08-04 rows (/home/rickard/scratch/ll-hist7.json, year corpus, 28 893 objects, 8-CPU allocation, 3 rounds, cpu_per_op_s from the container's own cgroup):

Pinned, gunnar's CPU per operation is flat within 7% from one client to thirty-two. The rise exists only in the quota-only column, and the throughput deficit tracks it: at load 9.31 gitea is 2.33× ahead unpinned and 1.24× ahead pinned. So "CPU/op rises 32%" and "unpinned is 2.088× behind" are the same measurement seen twice, and there is no second unexplained thing.

Two consequences worth stating plainly:

• The effect is a property of a box that can actually deliver the quota. It is 2.8× at load 9 and 1.17× at load 171 — when the host is oversubscribed the cgroup never reaches its ceiling, few of its threads are runnable at once, and the hand-offs stop crossing cores because they stop happening concurrently. Any re-measurement on a busy oden is therefore biased against finding it, which is exactly what happened below.
• --cpuset-cpus is the whole fix and it is the operator's. gunnar must not pin itself: it does not know which CPUs are its to take. It says so at start-up — MEASURED 2026-08-06 on the applied output, not on the unit test: a server in a 2-CPU scope on this 32-CPU box logs "this server may run on 32 CPUs but may only use 2 CPUs".

Where 6.45, 13.48 and 2.088 exist, and where they do not

They appear in exactly one place in the repository — the prose of ec26a87, written 2026-08-05 17:02 — and in no receipt, no .jsonl, no bench history and no performance-design.md row. Every recorded head-to-head c32 row on this machine is in /home/rickard/scratch/ll-hist{,2,3,4,5,6,7}.json, and the newest of them was written 2026-08-04 23:44. Both of the thread-reduction changes that bear on this path landed after that:

Neither was re-measured against the unpinned c32 rung. So the 2.088× in this file describes a build that no longer exists, and it should not be quoted again without a date and a load next to it (LAW 7).

The arm itself, re-run 2026-08-06: 2.088× is now 1.168×

gunnar.vs_forge_concurrent, ramp 1,32, 2 rounds, three engines interleaved, the current build named in the receipt (gunnar 1.1.0 (v1.1-120-g54a0282)), 8-CPU allocation, year corpus 28 893 objects, load1 152.9 — the box spent the whole morning between 44 and 370 because korp's planet chain and another lane's build had it, and this is the quietest window there was.

Three things this settles and one it does not.

• The +32% CPU-per-operation ramp is real, current, and belongs to the quota-only column. 0.2186 → 0.2894 reproduces the figure this file carried; the same ramp with a cpuset is +10.4%. gunnar is the only engine whose per-operation cost rises at all.
• The 2.088× does not reproduce. At load 152.9 the unpinned deficit is 1.168× and the pinned one is 1.206× — the two columns have crossed, and every engine is slower pinned (gitea 9.657 → 7.249, forgejo 9.501 → 6.704, gunnar 8.269 → 6.009), because on a box that is already oversubscribed a cpuset is a restriction that buys nothing. The pinned column at load 152.9 and the pinned column at load 9 are not the same experiment, and no conclusion should be drawn by putting them in one sentence.
• gunnar is not behind on CPU. At one client it is the cheapest of the three (0.2186 against gitea's 0.3112 and forgejo's 0.2712) and at c32 pinned it is level (0.979). It is behind on throughput at c32 in both columns while using ~1.5 of its 8 CPUs, so what limits it there is not its CPU budget.
• Not settled: what the deficit is at low load on today's build. 1.168× at load 152.9 says nothing about the 2.33× the same arm measured at load 9.31, because the effect scales with the box's ability to deliver the quota. That measurement needs a quiet oden and did not get one.

Re-measured 2026-08-06 in isolation, and what it could and could not show

A standalone probe was built to ask only the affinity question: .nornir/bench/affinity-arm.py with affinity-setup.sh / affinity-seed.sh. One gunnar serve under a cgroup quota, N concurrent git clones over smart HTTP, arms interleaved with alternating order every round, five guards, and the build under test as an arm dimension so the current binary and its 2026-08-04 parent (fdf404c) are compared inside one run on one box. The corpus is production-shaped on purpose — a real repository pushed in twelve increments, 13 packs, no repack, no bitmap — because git clone --bare repacks and would have given a single-pack corpus.

All five guards were seen red (oden, 2026-08-06), four of them driven by a real condition rather than by faking the observation: a client pointed at a repository that is not there ("1 of 4 clients did not complete"), a --depth 1 client ("landed 666 objects, the corpus holds 12370"), a scope booted with no CPUQuota ("cpu.max is max 100000"), an affinity applied to 0-3 while the arm claimed 0-7 ("the server's threads are on {'0-3': 10}"), and a synthetic zero for the CPU floor. The green control run passed immediately after.

Result: on this corpus, on this box, at these loads, there is no unpinned throughput penalty and no CPU-per-operation inflation, for either build. 90 scored batches:

This is not evidence that the two Aug-5 fixes closed the gap, and it must not be read that way: the 2026-08-04 parent shows the same null, so whatever differs, it is not the binary. What the probe does show, and reproducibly, is the mechanism's fingerprint — the quota-only arm is throttled where the pinned arm is not, at the same throughput: 131 of 159 periods throttled versus 13 of 172 (2-CPU quota, 8 clients, load 72–98). The cgroup is being stopped mid-period and the box is loaded enough that it loses nothing by it.

And it puts a number on where the effect is not. The forge arm and this probe ran on the same box within the same two hours, on the same binary, at the same 8-CPU quota and the same 32 clients, and disagree by an order of magnitude on the one quantity that matters:

So the mechanism needs something the smaller run does not have. Two candidates remain and neither is excluded: the corpus (year is 28 893 objects and 240 branches against 12 370 objects and ~30 refs, and its per-operation cost is half — 0.24 against 0.58 — so a transfer there is shorter, and thirty-two of them overlap differently), and the harness itself (three containers up, and every client's bytes crossing a counting relay that the probe has no equivalent of). Ruling between them is one more run and it was not made.

What could not be determined. The size of the deficit at low load on today's build — the only condition under which the original 2.088× was taken, and one this box did not offer today. And whether the two Aug-5 thread reductions moved it at all: they were never measured against this rung, and today's null shows up on their parent too.

Found on the way: fdf404c served a corrupt pack from a 16-pack repository

MEASURED oden 2026-08-06, and deterministic at one client. A repository grown to 27 864 objects across 16 packs — several unrelated histories pushed under separate ref prefixes, no repack, no bitmap — is served by fdf404c (2026-08-04 23:43) as a stream stock git rejects: fatal: pack is corrupted (SHA1 mismatch) / fatal: fetch-pack: invalid index-pack output, 32 of 32 clients and then 1 of 1. The current build serves the same store correctly and every client's clone carries the full object set and the right tip.

The store is not the suspect: git verify-pack -v accepts all 16 packs and git fsck is clean. The same binary serves the 13-pack corpus in this section's table without complaint, so it is not a store-format difference either — something between 13 and 16 packs, or in what those extra packs contain, crosses a line. The 13-pack corpus was built from one repository's own history and the 16-pack one from six unrelated ones, so a cross-pack delta base is the obvious hypothesis and it is not tested here.

What this is worth: it is the twelfth time the multi-pack fixture shape has caught something a single-pack corpus could not, and the arm that would have caught it does not exist — vs_forge_concurrent builds its corpus with one push. A multi-pack, multi-root corpus belongs in the bench suite, and until it is there this is a defect that was fixed by accident rather than a regression that cannot come back.

Methodological note for whoever runs vs_forge_concurrent next

The two CPU configurations are never interleaved with each other. concurrent.rs runs the whole quota-only pass — every rung, every round, every engine — and only then applies the cpuset and runs the whole pinned pass. Engines are interleaved inside a rung and rungs inside a round, precisely because "on a box whose load moved 2->218 in a day, the gap between two measurements is in their ratio"; the configuration axis is the one place that reasoning is not applied. It does not explain the 2026-08-04 result — load rises during a run, which would favour the pass that goes first, and quota-only goes first and lost — but it is the one confound in that arm that nothing controls, and affinity-arm.py alternates its arm order every round for this reason.

Rules that keep being earned

Each was learned by something breaking.

• A guard nobody has seen fail is worth nothing. The dependency guard was green while the appliance linked gix-command in a build whose specification opens "Zero shell". The exit-status assertions were green with the bug present.
• Assert against stock git, not against ourselves. A client and server that agree only with each other can be wrong in the same way with every test passing.
• Separate measured from inferred. Four spec claims died under measurement: the C-toolchain ban, "reject a blob before it lands", the dependency arithmetic behind dropping redb, and rejecting absolute paths, which would have refused every ssh:// clone.
• Do not claim a speedup nobody measured. gitoxide's own status lists "delta compression" as done; what exists is delta reuse.
• Read the benchmark before recommending the swap. Twice now, reasoning about skade's numbers without opening them produced the wrong recommendation.

2026-08-06 — the service lane: the suite, the unit on oden, and the door held open

What the suite actually says

cargo test --workspace --all-targets --no-fail-fast at 27dc3df, exit code read unpiped from the log file rather than through tail:

The nine ignored are declared, not skipped by accident: the three-binary release linkage audit, on_a_real_repository_the_window_is_the_same (needs GUNNAR_CORPUS), and the sweep_* measurements that are measurements rather than guards. --all-targets does not run doctests, so that is a number this lane did not take.

bench/ is a detached workspace with its own Cargo.toml, Cargo.lock and target, so --workspace never reaches it. xtask's bench-compile rule covers it with --no-run, and the arms themselves (nornir bench run gunnar) were not run here — a full pass builds real corpora and this box was at 351 MB free on /home when the lane started. That is stated rather than implied: the bench arms are unmeasured today by this lane.

The disk, because it nearly took the box down

df /home fell from 46 G to 351 M while this lane's own cargo test ran, on a 5.9 T md1 that five other lanes were building on. 32.3 GiB came back from cargo clean on one stale worktree target. Sizes at the time: korp 281 G, holger 180 G, gunnar 146 G, nornir 126 G, dwarves 104 G, edda 73 G, skade 47 G. / (md0) had 546 G free, which is why the service's data root is /srv/gunnar and not ~/.local/share/gunnar: a git forge whose repositories land on a filesystem at 100% is a forge that stops mid-push.

gunnar service install reported three greens it had not earned

MEASURED on oden. An install over a running gunnar, with a deliberately wrong --exec:

✓ updated gunnar systemd --user service (runs as you)
installed gunnar.service                                  exit 0
systemctl --user is-active gunnar.service   →  active
gunnar service status                       →  gunnar.service active   exit 0

and the process that was running was the old one, from before the install. The unit on disk named a path that did not exist; nothing had restarted.

The mechanism is worth writing down because it is not gunnar's alone: D-Bus StartUnit does nothing to a unit that is already active. Lifecycle::install is write → Reload → EnableUnitFiles → StartUnit, so over a live service the last step is a no-op and the new unit text is only a file. And SystemdServiceSpec::validate in Skidbladnir 0.1.12 stats nothing — it checks only that a spec carrying secrets also carries an EnvironmentFile=. Every consumer of skidbladnir::service::Lifecycle has this shape, not only gunnar.

The damage is deferred, which is what makes it dangerous: the unit only fails at the next restart, as ExecMainStatus=203 / Result=exit-code under Restart=on-failure. In practice that is the next reboot, with nobody watching.

Closed in three places, one per green (0c2ed71):

• preflight_exec stats ExecStart — absent, not-a-file and no-+x are three distinct refusals — before a unit naming it is written;
• install stops before it installs, so start cannot be a no-op;
• install then waits for the declared HealthProbe::Tcp on the SSH port and refuses to claim success without it. active is systemd's opinion about a process it spawned; a socket accepting is the service answering.

service status also exited 0 while printing inactive — measured during the 203 crash loop. It now exits non-zero, which is what systemctl is-active has done since 2010.

Seen red on the real binary, not only in tests: the fixed installer refuses a missing --exec with exit 1 and leaves the running service untouched.

The unit that is now on oden

unit          gunnar.service, systemd --user (linger on), enabled
ExecStart     /home/rickard/.local/bin/gunnar-service serve
              --ssh-bind 0.0.0.0:2222 --grpc-bind 100.115.167.95:50061
data root     /srv/gunnar   (md0, 546 G free — NOT md1)
ssh           100.115.167.95:2222, host key SHA256:QYc09pf3H1LUvxf1YuQrIy8dLEJ8aVRlqhr7vfh4ZkY
grpc          100.115.167.95:50061   (50051 is korp's; the tailscale address, not 0.0.0.0)
drop-ins      none — `DropInPaths` is empty, and nothing was hand-written

The running binary is the one that was built, and that is asserted the only way that cannot be faked: sha256sum /proc/<MainPID>/exe equals the sha256 of target/release/gunnar-service. d4467218deecf957f6d145a71199c8700666254647034261833ef3473617b792 on both sides, rebuilt from the pushed sha so the running artifact corresponds to a commit that exists on origin.

The log, and the filter that was checked against output rather than source

A unit that names no Environment=RUST_LOG= is not a unit with no opinion — the process falls through logging::init to a bare info, decided inside the process and recorded nowhere. The unit now always carries a filter, and the filter is built from observe::TARGETS rather than typed beside it (efef15e), because that list is the only thing that knows what gunnar emits on.

That is not the same as the filter working. The target a call site emits on and the target a filter names are two independent strings, and until 2026-08-05 seventeen call sites spelled theirs gunnar.lfs and gunnar.store with a dot while every document said ::. So tests/service_log_filter.rs puts the real filter in front of the real capture layer and emits on literal targets: all seven must come through, and gunnar.lfs, gunnar.store and gunnar_lfs must not. A blanket trace — which satisfies every "does gunnar's output appear" test ever written — fails it on all three.

Verified against the live journal, not the source: one Bootstrap.Status produced three gunnar::audit lines carrying a request id, and one SSH connection produced four gunnar::ssh lines. The watch command is journalctl --user -u gunnar.service -f -o cat.

The first-user door: open, and verified without being consumed

gunnar's ACL bootstrap is trust-on-first-use: the first call with no credential creates the first user as gunnar-admin, and the same compare-and-swap that creates it writes the record that shuts the window for ever. Deleting every principal does not reopen it.

Bootstrap.Status is the probe that does not consume it. It is unauthenticated by design — an operator must be able to ask an unenrolled server whether its door is open, and on an unenrolled server there is by definition no credential with which to ask anything else — and it only reads.

That it does not consume the window was proved, not assumed, on a throwaway server on its own data root and its own ports: three Bootstrap.Status calls, window_open true each time, and then Bootstrap.Enroll still succeeded. The real server has had Bootstrap.Status called on it and nothing else: principals 0, keys 0, bootstrap_window = OPEN in its own start-up WARN.

ACL state survives a restart

Also proved on the throwaway, so that the real server's window stays untouched:

• enrol aclprobe, add a second principal second as an authenticated admin;
• kill the process, confirm it is gone and the ports are free, start a new one, confirm a new pid and its own identity table ready line;
• both principals are still there, both keys still authenticate, and Bootstrap.Enroll is still refused with the original bootstrap timestamp.

The first attempt at that restart was a hollow green and was caught: the replacement process refused to start on the store lock the old one still held, so the "after restart" query answered from the old process. The store said so loudly, which is the only reason it was visible. A restart is not proved by starting something — it is proved by the old pid being gone, the port being free, and the new pid printing its own start-up line.

On the real server the same persistence is visible in the one piece of state it has: the SSH host key fingerprint is byte-identical across a systemctl --user restart, and the door is still open afterwards.

xtask ci compiled the systemd feature and ran none of its tests

The step above guard reads

cargo check -p gunnar-cli --no-default-features --features server,systemd

and its comment says why it exists: the feature was resolved by cargo tree and by tests/feature_split.rs but never COMPILED, and a resolve is not a build. The same sentence one step further on is the hole. A build is not a run. cargo check does not compile #[cfg(test)] code at all, and systemd is off by default, so cargo test --workspace never reaches gunnar_cli::service either.

MEASURED: twelve tests in that module and no command in this repository ran one of them — the unit renderer, the --data-root regression guard written on 2026-08-04 after a unit shipped that would have written /gunnar-data at the root of the filesystem, the RUST_LOG the unit carries, and the ExecStart preflight. A systemd-tests step now runs them: 52 tests in the feature resolution CI was already checking. Seen red — with one assertion in service::tests inverted, that exact argv exits 101.

This is also why the workspace number moved by only four. --workspace --all-targets takes the default features, systemd is not one of them, and the four new service.rs tests are invisible to it. 1 461 → 1 465 is observe's new test plus the three in tests/service_log_filter.rs; the other four live behind the feature and are counted in the 52.

What was run, in full

The two detached test workspaces are 12 binaries and 79 tests between them. Every exit code was read from a file, unpiped. The bench arms were not run.

Left for someone else

• bench/ arms are unmeasured by this lane. nornir bench run gunnar was not run.
• smart HTTP is not in the deployed shape. The gunnar-service cell is --no-default-features --features server,systemd; --http-bind lives behind http.
• --admin-key's doc comment is stale. It says an empty identity table makes gunnar "refuse to serve"; prepare_to_serve returns Readiness::AwaitingBootstrap and the server starts and serves, which is what the whole bootstrap design needs it to do.
• The Skidbladnir hole is upstream. gunnar now guards its own install; the same StartUnit-is-a-no-op and unstatted ExecStart are still there for every other consumer.

FIXED 2026-08-06 — -i did not name the identity, and that made a whole test shape a false green

The finding

t14s, against the live oden server as user rickard: gunnar-cli ls-remote -i <a key the server has never seen> printed the refs and exited 0. The server was correct throughout. Its own log, for that one call:

ssh.auth.refused  reason="unknown-key"  fingerprint=SHA256:ySF3ogYe…   ×4
ssh.auth.accepted                       fingerprint=SHA256:OC2Yqguq…       ← rickard's, asked NOT to be used
grpc.auth.refused reason="not-verified"

The client refused to take no for an answer. clone did not have the bug; ls-remote did. An asymmetry between two code paths in the same client.

Why it was worse than a client bug

Every authorization test written as ls-remote -i <disallowed key> passes green and proves nothing — a false green in exactly the test that is supposed to show the door is closed, and green precisely on the machine of anyone whose own key is enrolled. gunnar's whole ACL rests on the key being the identity.

The root, which was not ls-remote

gunnar-client/src/ssh.rs::candidate_key_paths() put the named key at the head of a ladder — $GUNNAR_SSH_KEY, nornir-git's deploy key, ~/.ssh/{id_ed25519,id_ecdsa,id_rsa} — with use_agent still true in front of all of it. A named key that did not exist was dropped by an exists() filter, so a typo in -i degraded to "no identity given".

Everything through SshOptions inherited it, not only ls-remote:

So plain clone was right for a reason nobody had chosen: it was the one caller that did not go through gunnar's copy. One behaviour written twice — the shape LAW 5 forbids.

The fix, by construction

ssh::resolve_identity() is now the single writer, and both paths route through it:

• -i <key> (or $GUNNAR_SSH_KEY) means exactly that key. The ~/.ssh ladder is not consulted and ssh-agent is switched off — the agent holds other identities, and offering them is the same silent substitution wearing a different hat.
• A named key that is not there is Error::Identity, its own variant, so a test about an access-control boundary can tell "the door is shut" from "the server said no". Both are exit 1 and only one proves anything.
• SshOptions::auth() returns Result. There is no longer a way to obtain an SshAuth that has quietly substituted an identity.
• fetch::clone_or_fetch_with resolves through the same function, so plain clone stops being right by accident.
• The environment is read in exactly one place, IdentityEnv::current(), so the rule can be tested without mutating process-global state — a guard about identity that was flaky under test threads would end up #[ignore]d, and then blind.

This agrees with nornir-git's own meaning of a key path rather than inventing a second one.

The test audit, and the honest headline

The brief expected hollow tests. The test class did not exist at all. Across the whole workspace on 2026-08-06 there was not one test — binary or library — that drove gunnar's own client with a disallowed key and asserted a refusal. Every real refusal proof is either in-process against gunnar-ssh (tests/auth.rs, adversarial_auth.rs, sandbox.rs, anonymous.rs — 36 of them) or driven with stock git+ssh under IdentitiesOnly=yes -o IdentityAgent=none, which has no ladder to fall down. gunnar's client was the one caller nothing checked, and that absence is why the substitution survived in it.

Two tests were hollow in the weaker sense, and both are fixed:

• gunnar-cli/src/client.rs::a_fetch_can_authenticate_with_the_agent_alone carried the comment "an explicit key must still win, or -i would be decorative" and then asserted that the field the same function had just set was set. A state round-trip, the LAW 2 shape exactly. -i was decorative, and this could never have said so, because the substitution happened afterwards in auth(). It now asserts the SshAuth handed to nornir-git.
• gunnar-client/src/ssh.rs::candidate_keys_are_deduplicated_and_must_exist asserted the defect as the rule: "a path that does not exist is never offered" — which is precisely how a typo'd -i became a fallback.

Seen red, both layers

Also proven against the live server before any code changed: with -i <stranger> the client offered three identities the caller never named (stranger, ~/.nornir/ssh/id_ed25519, ~/.ssh/id_ed25519). None is enrolled on oden, so the call failed here; on t14s's machine the second rung was enrolled, which is the whole difference between a red and t14s's exit 0.

crates/gunnar-cli/tests/named_identity.rs is the missing class: ls-remote, fetch, clone and clone --depth 1, each with an un-enrolled key named while a working one sits in $HOME/.ssh.

FIXED 2026-08-06 — every healthy transfer logged three things an operator greps for as errors

The finding

t14s: 9 successful transfers produced 12 × ssh session ended with an error error=early eof, 6 × ssh.teardown.failed step="channel-close" err=Channel send error, and git: the client was gone before the exit status channel=ChannelId(2) status=0.

Confirmed from the live journal on oden at 10:55:10, one ls-remote that succeeded:

INFO  served git-upload-pack  event="git.upload_pack.served" … bytes_out=265
DEBUG ssh session ended with an error  error=early eof
WARN  the client was gone before the exit status  channel=ChannelId(2) status=0
DEBUG a session teardown step did not complete  event="ssh.teardown.failed" \
      step="channel-close" err=Channel send error
INFO  transfer finished  event="git.transfer.finished" status=0 elapsed_ms=106

Nothing is damaged — status=0, transfer finished. But three of five lines carry error, err or failed on a completely healthy call, so grep -i error in gunnar's journal returns a wall of working traffic. That is the first command an operator runs.

All three lines are the same fact said three times: the client took what it asked for and hung up before the server's exit-status and close. That is how ls-remote and fetch end.

The fix

One classifier, server.rs::peer_hung_up() (LAW 5) — three places used to answer this question independently and all three answered "no". It matches on the concrete cause, never on the message text: io::ErrorKind::{UnexpectedEof, BrokenPipe, ConnectionReset, ConnectionAborted, NotConnected} and russh::Error::{SendError, HUP, Disconnect}. (early eof is tokio read_exact's UnexpectedEof; Channel send error is russh::Error::SendError. Neither string appears in the code — a contains("early eof") would stop matching the day tokio rewords it, and would fail in the quiet direction, back to noise, invisibly.)

Seen red

observability.rs::a_healthy_transfer_logs_nothing_an_operator_would_grep_as_an_error states the rule as the operator states it — no line from a healthy transfer may match error, err= or fail in its level, message, or any field name or value — and fails against the pre-fix server naming both lines it can see.

gunnar-ssh::server::tests::a_real_failure_is_still_reported_as_one is the half that matters more: 6 benign shapes must be classified quiet, and 7 real ones (Kex, PacketAuth, DecryptionError, Inconsistent, WrongChannel, PermissionDenied, Other) must still say ssh.teardown.failed and still carry err=. A classifier that answered true for everything would give a beautifully quiet journal for a server on fire, and no other test would notice.

Known, not fixed

• ssh session ended with an error carries no conn. russh calls Server::handle_session_error on the server, not the session, so the line cannot be attributed to a connection — the only line in a connection's story that cannot. Every other line inherits conn, peer, principal, repo from the span. Fixing it needs a russh API that does not exist in 0.61. Consequence for testing: the e2e guard above sees two of the three lines, and the third is guarded by gunnar-ssh::server::tests::a_hung_up_session_is_not_logged_as_an_error instead. Stated rather than glossed.
• The client still hangs up early. push already drains its transport to EOF (push.rs::wait_for_remote_to_finish); ls-remote and fetch do not, which is what creates the race. Making them drain would remove it for gunnar's own client but not for any other, and after only reading an advertisement a drain risks blocking on an upload-pack that is waiting for the next request. The server-side classification is the durable fix; the client-side one is a separate, optional tidy.

FIXED 2026-08-06 — one human, two machines, two principals. It was a missing verb, not a schema

The finding

t14s, on the live oden server: oden's ssh key is not t14s's, and there was no way to add a second key to an existing principal while the server was up. So the same human became two principals — rickard and rickard-oden (SHA256:Bw3OsBmc…).

The worry that follows is the right one: if a principal cannot hold a second key then the ACL models keys as identities rather than principals holding keys, every grant, ownership and revoke fragments per machine, and "revoke rickard" stops meaning what an operator reads it as.

What the model actually is — read, not inferred

It is already principals-hold-keys, and always was:

So this is neither a schema change nor a design call. It was one missing CLI verb. gunnar control user add does reach AddKey — but it calls AddPrincipal first, so it could only ever key a brand-new principal. Nothing could call AddKey for somebody who already existed. The offline gunnar key add <principal> <key> writes the data root directly and is not usable against a running server.

What landed

gunnar control user add-key <name> --key @<path>   # enrol another machine's key
gunnar control user keys [<name>]                  # what keys does this human hold?

Both are thin adapters over RPCs that already existed. add-key reads back the whole key set and prints it, because "one key added" is a round-trip of the request and the set is the applied output — and the set is also the answer to the question the verb exists for.

Seen red, three ways, all against the workaround t14s had to use

The guard is crates/gunnar-cli/tests/second_key.rs, and its red probe is literally control user add rickard-oden in place of control user add-key rickard:

And red first, before anything is enrolled: the second key must be refused at the SSH door, with the first key proved to work in the same breath, or every later assertion passes on nothing.

The restart is asserted on disk, not in memory: the server is killed and restarted on the same data root, and the second key must still open the SSH door and still list under rickard. An identity table that lived only in that process passes every earlier step and fails there.

Left as a design call for Rickard, deliberately not decided here

• Access.AddKey needs server-wide Admin. So a non-admin cannot enrol their own second machine; someone with gunnar-admin has to. Self-service — "a principal may add a key to itself" — is a policy change to the RPC's authorisation, not a CLI gap, and it widens what a single stolen key can do (one key could mint a second, and revoking the first would then not lock the holder out). It is not slipped in beside an auth fix.
• revoke rickard still has no single meaning. RevokeKey is per fingerprint and RemovePrincipal takes the human whole. Both are right; there is no verb between them ("revoke this machine, keep the human"), and user keys is now the thing that makes the gap visible. Naming it is a design call.
• The live oden server still holds rickard and rickard-oden as two principals. Nothing in this lane touched the running service. Merging them is an operator action once this build is deployed: control user add-key rickard --key @<oden pub>, then control user remove rickard-oden.

STRESSPLANEN — git-funktionaliteten ar A och O

Rickards riktning 2026-08-06. Git-funktionaliteten gar fore allt annat. Flertradigheten ar parkerad, inte avbruten — han vill ha den ocksa, men efter att servern bevisligen gor ratt saker under press.

Metoden ar inte att resonera fram fel. Den ar att stressa tills nagot gar sonder och sedan laga det som gick sonder.

Ordningen

1  git-funktionaliteten haller under press      HAR ligger tyngden
2  varje defekt lagad med en vakt sedd rod
3  flertradigheten                              parkerad, ska goras, inte nu

Testmatrisen

A. Manga sma — dar per-item-kostnad och las-kontention syns

tusentals sma objekt i ett pack
tusentals refs i ett repo
manga repon pa samma server
snabba upprepade pushar mot samma repo
manga sma pushar i stallet for fa stora

Skalet: git-objekt ar 393,8 till 1 572,9 byte i snitt, matt over flera korpusar och satt av innehall snarare an antal. Sma objekt dominerar i ANTAL medan fa stora bar merparten av byten. Allt som har en fast kostnad per objekt multipliceras darfor med ett stort tal.

B. Fa enorma — dar buffring, stromning och minnestak brister

blobbar pa hundratals MB
djupa delta-kedjor
ett repo storre an RAM
en klon vars arbetsmangd overstiger minnesborsen

Skalet: en kernel-klon ar 14 042 000 objekt, VmHWM 5,59 GB, och en tidigare extrapolering var 1,6x for hog. Tal fran sma korpusar gar inte att rakna upp.

C. Fientliga former — dar felvagarna aldrig kors

tva klienter pushar samma gren SAMTIDIGT
en push som nekas mitt i overforingen
en klient som kopplar ner mitt i packet
ett tomt repo
ett repo vars default-gren aldrig skapades      <- FYND 4, redan en oppen bugg
unicode i ref-namn
mycket langa ref-namn
en push som inte sluter sig (saknad bas)

Skalet: varje bugg vi hittat i dag satt i en felvag eller i en form ingen fixtur byggde.

FIXTURREGELN — villkor, inte rad

Bygg korpus genom att PUSHA cirka tolv ganger. Aldrig git clone --bare.

clone --bare packar om till ETT pack och skriver ingen bitmap. Produktionsformen ar tolv pushar, tolv pack, ingen ompackning, ingen bitmap. Det ar precis darfor hela sviten missade en korruptionsbugg vars golv visade sig vara TVA pack och inte sexton — och darfor 13-pack-korpusen sag "ren" ut trots att den redan holl 130 av 240 objekt i tva pack.

Tolv ganger har det har tradet gatt i den fallan.

Vad "klart" betyder per defekt

1  reproducerad         deterministiskt, med kommandot nedskrivet
2  lagad
3  en vakt SEDD ROD     som namnger sitt eget amne
4  raden i PLAN.md      vad det var, inte bara att det ar lagat

En stresskorning som producerar en lista ingen atgardar ar vard noll.

Var det ska skrivas

Bencharmar i nornir/dwarves bencher-ramverk, verktyg i xtask. Inga nya shell- eller python-script — Rickards regel, och den galler framat aven om gamla script far ligga kvar.

Anteckna lasten vid varje siffra. 1,083x och 2,088x skilde sig enbart pa att burken lag pa 23,7 respektive 85,2.

Parkerat: flertradigheten, och det vi redan vet sa det inte tappas

Han vill ha den. Den kommer efter git-funktionaliteten.

SERVERN   copy.rs thread_limit: Some(1), seriell INOM en overforing,
          parallell MELLAN dem. P-020 sager att raden inte ar ett tuning-val
          och xtask/tests/pack_pipeline.rs faller bygget om den tappas.
          P-004: serving tier stannar seriell tills det finns en
          ADMISSION-GATE som vet att burken ar ledig. Gaten byggdes aldrig.
KLIENTEN  write_pack_deltified -> delta.rs thread_limit: None,
          alltsa gitoxide pa alla karnor.
MATT      3 tradar per overforing korsar karnor, varje korsning en futex,
          40 % av CPU i native_queued_spin_lock_slowpath utan cpuset.
          CPU/op c1->c32: 2,85x opinnad mot 1,07x pinnad.
t14s      8 klonar 16,42 s · 16 -> 30,94 s · 32 -> 60,56 s, 56/56 ok.
          EXAKT linjart, alltsa noll parallellvinst pa 32 karnor.
OKLART    serve.rs forklaringen bygger pa en container med --cpus 8.
          Tjansten pa oden kor UTAN kvot och fick anda linjar skalning.
          Antingen galler futex-forklaringen anda, eller sa ar det en
          annan orsak som rakar se likadan ut. INGEN HAR AVGJORT DET.
FORSLAG   en global permit-pott, samma form som minnesborsen som tog
          32 samtidiga bitmapslosa kloner fran 4,66 GB till 1,09 GB.
          En overforing kop 1 permit for att borja och far kopa fler bara
          om potten har over. VARNING ur samma bors: en snav bors SVALTER
          en fan-out -- fyra kernel-kloner mot en bors for tva serverade
          NOLL. CPU-grinden far darfor aldrig neka, bara degradera till 1.
IMMUTABLE ett git-pack skrivs en gang och andras aldrig, sa lasningar
          behover ingen uteslutning. Sparren ar CPU-budget, inte korrekthet.

FIXED 2026-08-06 — HEAD was not advertised, because the default branch lived in two places

t14s's roundtrip harness (FYND 4) pushed foot into a fresh gunnar repository and could not clone it back. The push was clean — unpack ok, ok refs/heads/master, ls-remote listed the branch — and the clone produced nothing. ls-remote had returned one line and no HEAD line at all. Every master-default donor is this shape: foot, ripgrep, mdBook, which is t14s's whole bench corpus.

The hypothesis, and it held

Two places held a repository's default branch, and only one of them ever moved.

UploadPack::head_ref resolves the repository's HEAD, not the catalog. So a repository whose default branch is main and whose only branch is master has a HEAD with no object behind it, the entry is dropped from the advertisement, and there is no HEAD line. set-default-branch master then moved the authority and left the projection where it was: the RPC answered refs/heads/master, repo get read refs/heads/master back, and the advertisement still resolved refs/heads/main.

MEASURED, own server on 127.0.0.1:2299 / :50099, the live oden service untouched:

create                  branch refs/heads/main
push master             unpack ok · ok refs/heads/master
ls-remote               40920e3…  refs/heads/master          ← one line, no HEAD
set-default-branch      branch refs/heads/master             ← answered right
repo get                branch refs/heads/master             ← read back right
cat …/rt.git/HEAD       ref: refs/heads/main                 ← never moved
clone                   FAILED

The count

refs/heads/main was spelled in five places as a decision, not as prose:

1. gunnar-core/src/model.rs DEFAULT_BRANCH — the catalog's default
2. gunnar-store/src/fs.rs HEAD_CONTENTS — what init_bare writes
3. gunnar-grpc/src/seam.rs:398 DEFAULT_BRANCH — MemoryControlPlane's own
4. gunnar-client/src/clone.rs — the client's fallback when a remote advertises no symref
5. gunnar-client/src/fetch.rs — the same fallback, again

(1) and (2) were the pair on the live path and are now one: the constant moved down to gunnar_store::refs::DEFAULT_BRANCH and gunnar-core re-exports it. (3) is a test double in a crate with no dependency edge to either — left, and said out loud rather than guarded. (4) and (5) are client-side and are twins of each other; not touched by this lane.

The fix, by construction

The catalog is the authority; a repository's HEAD is a projection of it, and there is one writer of the projection:

gunnar_core::point_head_at(&dyn RefStore, default_branch) -> Result<bool /* moved */>
        ▲
ServerStores::point_head(store, repo)              ← the server's only caller
        ├── repo_store()  cache miss                 create, and every open after a restart
        ├── repoint_head()                           ControlPlane::set_default_branch
        └── adopt_default_branch_if_unborn()         a push into an unclonable repository

Three consequences worth naming:

• set-default-branch now moves the advertisement. That is the reported bug.
• Repositories already broken on disk repair themselves the next time the server opens them. No migration step; the ones t14s already pushed are fixed by a restart.
• A master donor no longer needs a second command at all. If a push leaves HEAD resolving to nothing, the first refs/heads/… it created is adopted as the default branch and logged at warn. It fires only on a repository that is unclonable as it stands, never on a delete, never on a tag, and set-default-branch overrides it. This is what every hosted forge does, and it is why xtask roundtrip's step 3b workaround is no longer load-bearing.

The proof, and the reds

foot cloned all the way back: 6954 commits, 25643 objects, 38 363 676 B pushed, diff -r against the donor's worktree is 0 lines over 249 files.

crates/gunnar-cli/tests/head_advertisement.rs — four guards, stock git as the client, and the assertion is on a file in the worktree, not on an exit code: git clone of a repository with no usable HEAD prints a warning, writes an empty tree and exits 0. A test that checked the clone's exit status would have passed against this bug.

Six reds seen, each on this lane's own code, each failing only the test that owns it:

Left open, honestly

• The gunnar clone error on a HEAD-less remote is wrong. nornir-git fails with "advertised no HEAD", remote_is_unborn matches that string, and init_empty_clone then finds refs and reports "advertised references after all … the clone failed for another reason" — which names nothing. Against a fixed gunnar this is unreachable; against an older gunnar or any other server it is the message you get, and it cost this lane the first twenty minutes of diagnosis. Not fixed here.
• The two client-side refs/heads/main fallbacks are still twins (clone.rs, fetch.rs). One pub const in gunnar-client would collapse them.
• gunnar-grpc/src/seam.rs:398 still holds its own copy of the name. gunnar-grpc depends on neither gunnar-core nor gunnar-store, so there is no edge to re-export along, and a guard asserting two copies agree is what LAW 5 says not to build.

CHAOS_MONKEY_III, bana INIFRAN — ref-namn (2026-08-06)

Formerna som en val uppfostrad klient aldrig skickar. Allt nedan ar matt pa oden 2026-08-06, load1 antecknad vid varje siffra, med git 2.53.0 som sanning. Ingen last lades pa den levande tjansten; matningarna gar mot egna fixturer och git receive-pack startad for hand.

FYND 1 — FullName hade ingen langdregel alls, och de fyra backendarna var oense

Samma sjukdom som check_no_name_conflict skrevs for, i andra riktningen, och lika osynlig av exakt samma skal: varje fientligt ref-namntest i tradet kor pa mem.

MATT, ext4, load1 36,6:

Tva oberoende defekter i den tabellen.

(a) For langa komponenter togs av tre backendar av fyra. gunnar kunde alltsa halla ett repo som ingen loose-ref-backend och ingen stock git nagonsin kan halla — och ingenting pa traden sa det. Golvet ar inte NAME_MAX 255 utan 250, darfor att bade gix-lock och git skriver <namn>.lock bredvid: git update-ref tar 250 och vagrar 251 med "Unable to create … File name too long", och gunnars egen fs drar strecket pa exakt samma stalle. Regeln bor nu i validatorn som gunnar_store::refs::MAX_REF_COMPONENT_BYTES, dar den binder alla fyra genom konstruktion.

(b) fs klarade inte ett djupt namn som de andra tre OCH git klarar. Orsaken ar inte filsystemet: gix-ref tar sitt las genom gix_fs::dir::create::all med Retries::default(), vars on_create_directory_failure: 25 upstream beskrivs som "the maximum subdirectory depth we allow to be created". fs tog 20 komponenter och vagrade 30; git tog 2000. FsRefStore::create_parent_directories skapar katalogen sjalv fore prepare, och std::fs::create_dir_all har ingen djupbudget.

Vakt: gunnar_store::conformances ref_name_size, som kor mot alla fyra backendarna och bada hasharna, med acceptans-halvan forst sa en backend som vagrar allt inte kan passera. Sedd rod tva ganger, oberoende: utan langdregeln foll den pa alla fyra ("a 251-byte path component was accepted"); utan create_parent_directories foll den bara pa fs ("a 130-byte reference name in 62 components was REFUSED; git 2.53.0 takes it").

Den blinda vakten den ersatter: gunnar-wire/tests/hostile.rs a_very_long_reference_name_is_answered_consistently. Dess egen doc-kommentar namngav filsystemet — "a path component over 255 bytes cannot be created on ext4" — och dess fixtur ar MemObjectStore + MemRefStore. mem tog namnet, rapporten och lagret var overens, och ok == holds gick igenom. Den kunde aldrig ha gatt rod av sitt eget angivna skal.

FYND 2 — ett daligt ref-namn dodade HELA sessionen

Det har ar det starkare fyndet, och det hittades av att FYND 1:s vakt stallde en fraga ingen hade stallt: nadde vagran fram till klienten?

read_commands anropade FullName::new med ?. Ett enda ogiltigt namn gav alltsa Err ur serve innan packet lasts: klienten star och skriver in i ett ror ingen tommer, far EPIPE, skriver the remote end hung up unexpectedly, och far ingen report-status alls. Varje annan ref i samma push forsvinner med den, utan en rad som sager vilken som var fel.

git gor inte sa, och det ar matt och inte antaget. git receive-pack 2.53.0, matad med en rak tvakommandos-push over en anslutning:

unpack ok
ok refs/heads/goodone
ng refs/heads/bad~name funny refname

exit 0, och refs/heads/goodone skapad pa servern. Den bra referensen landar, den daliga namnges, sessionen avslutas rent.

Fixen ar typen UnnameableRef: ett namn validatorn inte vill ha blir en uppskjuten vagran i stallet for ett fel. Packet lases anda (annars ligger det kvar i roret och rapporten skrivs mitt i det), rapporten flatas ihop i den ordning klienten skickade, och --atomic faller fortfarande hela knippet.

Ordningen ar inte kosmetisk: kollapsen for atomic gors inne i apply, inte efterat. En kollaps i efterhand skriver om statusen for referenser transaktionen redan applicerat, och rapporterar som vagrad det som ligger pa disken. an_atomic_push_with_one_bad_name_lands_nothing ar den vakten och den ar sedd rod pa precis den lognen.

Vad den gamla sviten inte kunde se, och varfor. hostile_ref_names_are_refused_and_legal_ones_are_not kor tjugo fientliga namn och havdar !ok for vart och ett. Testhjalparens ok ar falsk bade nar servern svarar ng over en levande session och nar serve returnerar Err och samtalet tar slut — den viker ihop dem till samma boolean. Alla tjugo tog den andra vagen och tabellen kunde inte skilja dem at. Nar den gamla hardfels-vagen aterstalldes som rod probe gick de tre nya vakterna rod och hostile_ref_names stod kvar gron. Det ar blindheten, matt.

FYND 3 — vagran-vagen hade sitt eget satt att tappa sessionen

Foll ut ur FYND 2 direkt: Error::InvalidRefName renderar namnet inuti sitt eget meddelande, sa format!("{err}") la namnet i ng-raden tva ganger. En ng-rad ar en pkt-line. Ett 60 000-byte namn gav ett 120 105-byte paket, write_text vagrade det med TooLong, och hela rapporten gick inte att skriva. Vagran-vagen dodade alltsa sessionen — exakt det FYND 2 fanns for att ta bort. Nu bars bara regeln over i skalet, inte hela felet.

echoable_name kapar dessutom ett namn som anda inte far plats. Och den ar inte nabar fran traden i dag, vilket sags rakt ut i stallet for att latsas: en kommandorad ar en pkt-line, sa langsta namn en klient kan skicka ar 65 516 - 82 = 65 433 B; ng-raden rymmer namnet sa lange skalet ar hogst 65 511 - 65 433 = 78 B; langsta skal parse_command kan producera ar "funny refname: " plus validatorns langsta fasta strang = 51 B. 51 < 78. Marginalen ar 27 byte sammantraffande mellan tva konstanter ingen haller ihop, sa kapningen star kvar och bevisas dar den kan bevisas — ett enhetstest, a_refused_name_too_long_to_echo_is_truncated_rather_than_costing_the_report, sett rott med ramningspastaendet forst sa den roda raden namnger foljden och inte en ombudsvariabel.

Vad som INTE ar matt

• Ingenting av detta over en riktig socket. Allt gar genom ReceivePack::serve med en Vec<u8> i bada andar; ingen gunnar-serverprocess och ingen SSH-session kordes for de tre fynden. git receive-pack-sanningen ar daremot en riktig process med riktiga byte.
• Vad langdregeln gor mot ett lager som redan haller ett for langt namn. validate() sitter ocksa pa las-vagen, sa en arrow-log som redan bar ett 300-byte namn borjar fela vid uppslag i stallet for att tyst servera nagot ingen kan klona. Det ar ratt riktning, men det ar en beteendeandring pa ett redan trasigt lager och ingen har letat efter ett sadant lager.
• Djupet mellan 20 och 30. Golvet ar inringat till det intervallet och mekanismen ar last (on_create_directory_failure: 25), men den exakta brytpunkten ar inte bisekterad.
• Kostnaden. Ingen genomstromningssiffra togs; create_dir_all ligger pa ref-transaktionsvagen och kostar en is_dir() per edit som inte ar en delete. Omatt.

cargo xtask ci ar ROD pa origin/master, och inte av mitt arbete

Sagt rakt ut sa nasta korfalt inte tror att det ar deras. fmt och clippy foll pa ba2c691 fore en enda rad av det har, och det ar verifierat mot ett rent origin/master och inte antaget:

fmt      bench/examples/nornir-bench.rs
         crates/gunnar-build-thing/src/cells.rs
         crates/gunnar-build-thing/src/emit.rs
         xtask/src/roundtrip.rs
clippy   crates/gunnar-build-thing/src/emit.rs:284  needless_question_mark

Alla fyra ar rustfmt --check-roda pa git show origin/master:<fil>, och Ok(builder.build()?) star ordagrant pa rad 284 dar. Ingen av dem ar rord av CHAOS_MONKEY_III; mina sex filer ar rustfmt-rena en och en. De ligger i korfalt som arbetar i dem just nu, sa att formatera om dem harifran skulle ge dem en konflikt utan att laga nagot — darfor lamnade jag dem, med beviset.

test-steget ar gront: 1678 passed, 0 failed over 210 sviter, oden 2026-08-06, last1 0,89 vid slutet av korningen. Det blev gront efter de tva vakter som faktiskt fangade nagot: konformanssvitens antalsspark (44 -> 46) och citeringsvakten, som pekade ut tre file:line som flyttade nar receive_pack.rs och fs.rs vaxte. Bagge hade ratt.

Nasta i den har banan, oppna

tva klienter som pushar SAMMA gren samtidigt     PLAN sager sjalv "not swept at all"
push som nekas mitt i overforingen               delvis tackt av FYND 2, inte av gaten
klient som kopplar ner mitt i packet             finns som arm, kor bara pa mem
tomt repo                                        orort
push som inte sluter sig (saknad bas)            finns, ej stressad
tusentals refs / snabba upprepade pushar         orort