nordisk/holger

Stinky 12 — Lane R15: the data plane, finished

Lane R15 takes the five items stinky-11 left and closes what can be closed. Branch lane/r15-dataplane, off origin/main @ 05d664e. All work, all builds and all measurements on oden (Threadripper PRO 3975WX, 32 cores), release, host toolchain.

Read the load stamp on every row below. oden was NOT quiet: other lanes compiled throughout, and the one-minute load average sat between 32 and 117 against 32 cores. Every A/B is therefore INTERLEAVED — before, after, before, after — over three or five rounds, so both halves meet the same box, and the load is printed beside each round. Where the spread is wider than the difference, the row says so instead of quoting a median.

The lane's own scoreboard

4 · The other writable backends: there are none

The item assumed siblings. There are not any. MEASURED by reading all 16 server/repository/znippy-*/src/lib.rs:

• znippy-rust is the only one with a writable_archive, and it is the one stinky-11 fixed.
• The other fifteen — build-thing, composer, conda, deb, docker, gem, go, helm, maven, npm, nuget, package, python, rpm — every one of them answers fn put(&self, _id, _data) with Err(anyhow!("… is read-only")).
• grep -rn append_files --include=*.rs over the workspace now finds one hit, and it is a comment (bench-scenarios/src/stress.rs:31, describing the path that used to exist).

So there was nothing to re-treat, and the two traps stinky-11 named — the second writer, and the held handle that stops noticing a swap — have no second site to repeat at. Closed as a null result rather than left open as a suspicion.

2 · The hops have arms — the blocking gap, closed

Three arms, all NEW names (bench_runs keys history by metric name, so nothing was renamed). They drive the REAL engines, over REAL writable znippy archives on disk, never a stub.

Every one carries an ANSWER floor that fails the arm rather than reporting a pretty number over a broken path: N rows all Copied then N all Skipped with the last artifact read back byte for byte; the proxy's PRIMARY must itself hold the coordinate after the miss pass, or the arm measured a forward and not a cache-and-forward; a body whose advertised sha256 does not match it must be refused with 400.

The first thing they say, which holger did not have a number for: a proxy cache MISS costs about 57× a cache HIT — median 5 157 vs 292 192 ops/s over five interleaved rounds — and the whole of that difference is the write-through put. Neither number moved measurably in this lane (fill 4 969 → 5 157, hit 292 192 → 261 197, both inside a spread that includes one 1 664 /s outlier), and that is the honest reading: the proxy's cost is the put, and the put is bounded by the destination's write lock, not by anything the seam or the handle locks touch.

What they say about the Bytes seam

Stinky-11 reported the migration flat and named these hops as where it should pay. With the arms in place: the proxy HIT arm is flat too (272–308 k ops/s before and after this lane's changes), and that is the honest answer. A proxy hit returns the primary's buffer; that buffer was already MOVED, not cloned, before the seam and after it. The seam removed the to_vec() the unary gRPC response used to pay — stinky-11's proto change — and there is nothing left on the in-process hops for it to remove. Saying so with an arm behind it is the result; there is no faster number hiding here.

3 · gatling: called once, and the double link root-caused

The engine is called

stinky-01 §1.1 MEASURED rotaryengine in holger's graph with zero calls from the server. replicate_one_pair was the clearest place to end that: ONE for loop doing source.fetch → sha256 content id → destination presence probe → dest.put per artifact, on one core, with sha256 at 0.61 GB/s.

It is two phases now, the shape skade/parquet_io.rs:131 and ljar/batch.rs:113 use: gatling_map_balanced over the entries weighted by the size_bytes list() already returns (heaviest first, no barrier, index-ordered out), then ONE serial committer for the puts, because a writable znippy destination takes one process-wide lock per write. Entries are batched into windows capped at 256 MiB of declared size so the fan-out cannot read a whole source into RAM ahead of the committer — stinky-01 §7.4 is the reason that cap exists.

MEASURED, oden, five interleaved rounds on the quietest window of the day (load 22.9-29.5), N = 2 000 × 16 KiB, both binaries built back to back from the merged tree differing only in the files under test:

Read the two columns separately, because they say different things.

RESCAN is the fan-out with nothing else in it — every artifact is already present by content id, so the pass is fetch + sha256 + probe and not one put runs. 9.6× to 11.9×, median 9.7×, and every one of the five rounds is in that band. That is the engine.

COPY is flat: 4 980 /s median before and after. It contains the serial committer, and on a quiet box the committer is the whole cost — a fan-out cannot make a destination's single write lock go faster. (On the LOADED box earlier in the day the copy pass did gain 23-45 %, because there the lock-contention fixes of §5 mattered more than the schedule. Both readings are true of their box; neither is "replication got faster" on its own.)

⚠ The first A/B of this change said the copy pass got 2.2× SLOWER, and that was a true reading of a real defect. A fan-out slower than the loop it replaced is a lock, not a schedule, and there was one left — see §5.2. Both numbers above are from after it was removed. The bench found it; no argument would have.

cargo tree -i rayon still reports package ID specification rayon did not match any packages in all four feature tiers (default, full, embedded, mvp-server). ROOT LAW #0 holds.

The double link, halved and root-caused

MEASURED from server/cli, cargo tree -e normal, before: lgz 0.3.14 twice, ldeflate 0.1.1 beside 0.1.2, znippy-zoomies 0.1.24 beside 0.1.25, rotaryengine 0.1.10 beside 0.1.12, and linflate twice with them. Two path overrides were doing it from holger's side — agent/lib named znippy-zoomies with BOTH a path and a version, znippy-helm named lgz the same way, and local lgz 0.3.14 is registry 0.3.14, so that override bought nothing. Registry-only: lgz and linflate collapse to one copy each.

The other three pairs did not, and they have ONE cause: server/lib and plugin/host path-dep znippy-common because the sign feature exists only on the unpublished local 0.9.15 (the manifest says so, with the proof command), and that path copy drags its own znippy-zoomies → rotaryengine → ldeflate. They collapse the day znippy-common 0.9.15 reaches crates.io, and not before. A [patch.crates-io] would also collapse them and must not be used: nornir removed all eight of its patches on 2026-08-23 precisely because patch_fork_blockers treats a patch target as a foreign fork and holds every crate above it from publishing — 19 of holger's, holger-traits among them.

5 · The write path under contention

5.1 Four serialisation points, removed

All four were in znippy-rust, all four on the path every read and every write of a writable archive goes through:

1. The held-open handle sat behind a Mutex. Every read passes through hot_handle() for what is, in the steady state, an Arc bump. RwLock now: N readers share the guard, and only a re-gate takes the exclusive one.
2. The swap-detector stat was inside that lock. Holding a process-wide lock across a syscall is what turns a shared read path into a queue. The stat happens before any lock is taken.
3. put and store_index_meta each took the handle TWICE — once through self.contains() → open_reader() → hot_handle(), once to append — and both while write_lock was held, so they landed on top of the readers rather than beside them. One handle serves the whole critical section now and answers the contains check itself.
4. Group commit. See below.

5.2 The stamp was the last lock, and the bench found it

HotHandle::stamp was a Mutex<(u64, Option<SystemTime>)>, compared on every handle lookup — so every read in the process still queued on a mutex to do a two-word compare, and phase A of the new replication fan-out does two lookups per artifact. Under a fan-out that is 32 threads on one futex, which is exactly what the first A/B showed.

Two atomics now (len, and mtime as nanoseconds since the epoch with u64::MAX for "the platform did not report one"), read with two relaxed loads. The two stores happen together under the archive's write guard, so a reader that catches them half-applied sees a stamp matching nothing on disk and re-gates — one extra open, never a missed swap. That is the only direction a torn pair can fail in, which is why no lock is needed to make it correct. All three format_version_gate tests still pass, including the one that copies a future-format archive over a live repo mid-flight and requires a 503 naming the version.

5.3 Writers stop taking turns and share one

The remaining cost is not the lock holger owns, it is the work inside it. HotArchive::append writes the blobs, sync_all()s them, writes the journal batch, flushes and sync_all()s that too — two fsyncs — and it takes &mut, so all of it sits inside the exclusive guard every reader is waiting on. One append per artifact is two fsyncs of read stall per artifact.

So a put no longer waits for its own turn. It posts its entry to a queue and then queues for write_lock exactly as before; whichever writer wins the lock drains EVERYTHING posted and commits it in ONE append, and the writers behind it find their work already done. N concurrent publishes cost two fsyncs instead of 2N. cargo publish halves on its own: a version is two entries, the .crate and its .idx.json sidecar, and they now ride one append.

Three details make the contract hold rather than nearly hold: the entry is posted BEFORE queueing for the lock, so any leader from that moment sees it; each entry carries the slot its poster reads the verdict out of, so a leader that commits somebody else's entry also reports to them including when the append fails (a bare "was my entry drained" check cannot tell a commit from a failure); and the leader drops an entry already in the archive and a duplicate WITHIN the batch, so two racing publishes of one coordinate still produce one row.

MEASURED, holger.mixed_read_write, oden, 32 readers against 4 publishers. Three interleaved rounds on the QUIET window (load 21-29), and three earlier on a box at load 75-91; both agree, which is why both are printed:

The write side roughly doubles, median +101 % on the quiet rounds, and the result does not depend on how busy the box is.

5.4 What is NOT fixed, stated plainly

The read collapse under write is not fixed. read_throughput_retained_pct was 2.7–4.4 % before and 2.2–3.0 % after on the quiet rounds (4.1–5.2 % → 2.5–3.3 % on the busy ones), and read p999 inflation did not improve. That is not a regression in the mechanism: the write side now does roughly twice the work per second, so it holds the archive's exclusive guard for a larger share of the window. The cause is structural and it is not in holger: HotArchive::append needs &mut self and does two fsyncs inside it, and every reader of that archive is excluded for the whole call. Nothing holger can write shortens that. The fix belongs in znippy — an append that takes &self with the blob write and the journal commit separately guarded, so readers are only excluded for the index swap and not for the fsyncs.

Two honesty notes on these numbers. read_p999_ms_quiet — the arm's own control — measured 1.4 ms to 4.1 ms across these rounds against the 0.68 ms the baseline quotes, i.e. the box's "quiet" phase was already 2–6× degraded by other lanes; and read_failed_mixed was 0 in every round on both binaries, so the baseline's "3 650 reads fail outright" did not reproduce at this scale on this box. holger.concurrent_upload produced 14.8–49.9 ops/s across six runs with its own serial control swinging 1 758–5 583: unmeasurable at load 43–59, and no verdict is drawn from it.

1 · Phase 7: the second pass over the body is gone

Two of the three things stinky-01 §2.2 named were already fixed by the time this lane read the code, and saying so is part of the answer:

• the read path no longer collects the body at all (is_read ⇒ Bytes::new()), so the "one chunked GET / per connection is a gigabyte resident" hole is closed;
• the response Full::new(Bytes::from(body)) is a MOVE of the backend's Vec, not a copy, so changing that signature to Bytes buys nothing measurable.

What was left is the write path's second traversal: the door collected the whole body and THEN ran verify_upload_checksum over the finished buffer — sha256 at 0.61 GB/s, sha512 at 0.15 GB/s, single-threaded, on the request, so a 1 GiB upload held one blocking thread for 1.75 s (7 s for sha512) re-reading bytes it had just read. It is a Digest::update per frame now, built only for the digests the client actually advertised, so an upload with no X-Checksum header pays exactly what it paid before. The arithmetic is unchanged and has to be: sha256 is sequential and cannot be forked over chunks without changing the wire digest cargo names. Where holger picks its OWN digest, blake3's tree hash is the one that goes on gatling; this one cannot.

The body buffer is also sized from Content-Length (clamped to what Limited would pass anyway, so a lying header cannot make the server allocate) instead of being grown — a 4 MiB upload was doubling its way through eleven reallocations, each copying everything written so far.

verify_upload_checksum stays, rewritten in terms of the same streaming type, so its eight unit tests now drive the LIVE gate rather than a second copy that could drift from it. All eight pass, with the other 531 in holger-server-lib.

MEASURED, and the measurement does not resolve it. Five interleaved rounds, N = 60, 4 MiB bodies, oden at load 42–46: gate_cost_pct median 21.9 % before and 30.2 % after, spread 11.6–61.0 % and 3.1–139.0 %. The absolutes moved the right way in the median (verified 399 → 470 MB/s, plain 502 → 550 MB/s — the plain half being the preallocation), but the spread is wider than the difference, and a median inside that spread is not a result. The arm now exists, which is what was missing; the number is owed a quiet box.

How the measurements were taken, and one warehouse gap

Both binaries for every A/B were built back to back in the same minute, from the same target directory, differing only in the files under test — the bench arms are identical in both halves, so only the subject varies. Runs are interleaved and the load average is printed per round.

⚠ No row reached the warehouse. nornir bench run holger resolves the repo to /home/rickard/git/holger whatever directory it is invoked from, and that is the owner's checkout, which this lane does not touch — and it does not carry these arms in any case. nornir's own help text names the working alternative ("run each repo's nornir-bench example directly and keep the emitted BenchRun envelope as a file"), and that is what bench-results/r15-hops-oden-2026-09-20.json is. Every row in this document is from the harness; none is persisted. That is the nornir defect, unchanged and still owed a fix.

⚠ This lane also hit a second worktree trap worth writing down: a holger worktree at /home/rickard/git/_wt/<name> resolves its ../../edda sibling to /home/rickard/git/_wt/edda, which is ANOTHER lane's worktree — and that lane's uncommitted jera manifest pointed at a draupnir-namefix directory that does not exist, so cargo refused the whole graph. The fix is the h-lane / iso-lane convention already in use on oden: a lane directory holding the holger worktree beside its siblings. This lane's is /home/rickard/git/r15-lane (holger + a private edda worktree + symlinks to the clean siblings).

What is left

Lane SA, 2026-10-22 — the data-plane rows closed

Every row: red test first (seen red on sa/reds-2026-10-22), then green, one branch, merge --no-ff to main. Suites on the merged tree: holger-server-lib --features supply-chain 1008/0 (4 ignored), supply-chain,metered 1052/0, --features mirror replicate tests 24/0, update-bundle + file-oci + traits 158/0, bench-scenarios gates 3/0, --features federation checks clean, Skidbladnir pool 8/0. Built on dwarf in nordisk-cargo-toolchain:live, -j4.

Bench, dwarf (Ryzen 5 7535HS, 12 threads), bench_history.jsonl. Before = main + the new concurrent passes; after = the merged tree. ⚠ The box was NOT quiet during either run (load average 6–32 from work outside the cargo lock).