nordisk/holger

Stinky 11 — Lane R2: the data plane, restored

Lane R2 consumes stinky-01 (the data-plane audit), stinky-08 (the archaeology) and stinky-10 (the benchmark baseline) and RESOLVES them, in the order of cheapness the archaeology gave. Branch stinky/r2-dataplane, off stinky/bench @ a5ef6e6 — that merge is not optional, it carries the fix without which cargo build -p xtask --example nornir-bench dies with 10x E0063.

Every row below is MEASURED on a named box, before and after, from the benchmark harness. Builds on loki (Ryzen 9 7900, 12 cores), measurements on oden (Threadripper PRO 3975WX, 32 cores), release, host toolchain — the same road stinky-10 took, because a comparison across two roads is void.

The four, and what each one cost and bought

Commits, in order: b18627f · 4dada1a · fc671dc · 39aac92 · 5f7aed5 · 22476b7 · 49b9374.

2 · The six reqwest lines — the one that was already burning

server/lib:121 was always right and even ships the proof command. Six others forgot two words, so cargo unified reqwest's default-tls → hyper-tls → native-tls → openssl-sys across the graph, and §6c's "the ring provider is pinned process-wide, no OpenSSL" quietly stopped being true. All six date to b5a435c (2026-05-30): sloppy from birth, not degraded. A binary shipped to the live box the same morning died on libcrypto.so.3: version OPENSSL_3.3.0 not found — a C library nobody asked for, resolved against a host that had another.

MEASURED, loki, same worktree, before and after (cargo tree -p <p> -i <dep>):

What native-tls is left in the workspace comes through nornir (hf-hub, fantoccini — xtask's dev-dependency) and mannequin's cert-helper, which uses openssl on purpose. Neither is holger's reqwest.

znippy-rust also gained rustls-tls by name: its line asked for blocking + json and NO tls feature, so dropping the defaults would have left it with no provider rather than the rustls one it means.

3 · The serving path has a number again

8aa6159 (2026-05-22) benched handle_http2_request beside get_file; cd7636d (2026-05-31) kept only get_file. For the next four months nothing measured the route that answers a client — and that is the route stinky-01 measured as the slow one. Three arms, all NEW names (bench_runs keys history by metric name):

• holger.rust_crate_http_st / holger.rust_crate_http_mt32 — same corpus, same loop shape as the read twins. The difference between the twins IS the door.
• holger.rust_crate_put — N one-at-a-time puts, fixed N (HOLGER_BENCH_PUTS), because a quadratic hides inside a time-boxed loop by simply completing fewer operations. Payloads are built before the clock starts. It carries an ANSWER floor: the last put is read back through handle_http2_request, so a write path that produces an archive the read path cannot serve fails the arm.

MEASURED, oden, quick scale (the run that also serves as the before for #4):

The door costs 21 % single-threaded and 5.6 % across 32 — which agrees with the archaeology lane's 19 % on loki, from a different corpus size on a different box. The parse is not the problem, and now there is a number that will say so out loud if it ever becomes one.

4 · HotArchive: a quadratic, stated as a rate

The writable dev-mode backend (/sparring, the /cache primary) did two things per operation that grew with the archive: put re-sealed the whole Arrow index through append_files, and every fetch/contains/list re-opened the file through ZnippyArchive::open. Neither needed inventing — znippy shipped HotArchive on 2026-08-04 for exactly this, and holger's backend (2026-06-19) simply predates it. znippy-common 0.9.14 was already in the lock.

MEASURED, oden, both binaries built and run back to back in the same minute:

Read the rate columns, not the ratio. Before, the rate FALLS — 1386 → 587 → 190 — which is the quadratic stated as a rate. After, it is flat and rising — 6467 → 7554 → 7674. The ratio therefore keeps growing with N, exactly as znippy's own table (oden 2026-08-04, 26.7× wall at 8 000 appends) says it must, because only one side of it grows.

Three things came out of doing it, and all three are in the commits:

1. A second writer. cargo publish writes TWO entries per version — the .crate and a {name}-{version}.idx.json with the real deps. put moved to the handle; store_index_meta was still on append_files, re-sealing the archive the handle was appending to. The SUITE caught it, not reasoning: sparse_index_serves_real_published_deps, "deps must NOT be empty". Moving one writer onto a held handle means finding every writer, because the ones left behind do not fail — they disagree.
2. A held handle still has to notice a swap. The old per-request ZnippyArchive::open was also what made a REPLACED archive fail the request (format_version_gate.rs copies a future-format archive over a live repo and requires a 503 naming the version, with the other repos still serving). So the handle stamps (len, mtime) as it leaves the file and compares on every lookup — one stat, which is what the old memoised gate cost — and a mismatch drops the handle and re-runs the gate. The 503 still happens, for the same reason, reached by noticing instead of by re-parsing.
3. One test assertion was restating the old design ("znippy's append writer records a format version"), which a hot archive does not: its journal is its index and it records nothing until seal. Restated, not deleted, and it now asserts both halves — live ⇒ hot, sealed copy ⇒ a version this build supports.

Known limit, stated: a swapped-in archive this build CAN read is not adopted; the journal this process holds still describes the repo until it restarts. Replacing a live writable archive with a good one was never supported; replacing it with an unreadable one is refused, which is the half that mattered.

Read arms in the same run, unchanged as they must be — the static read path was never touched: rust_crate_st 420 738 → 471 932, rust_crate_http_st 332 627 → 372 451 ops/s. Shared box, not a claim. The swap detector's stat costs ~3 % of the put rate (7 554 → 7 242 at N=2 000) and is worth it.

1 · The Bytes migration, executed

81d8e24 wrote it on 2026-06-28; be80f18 deleted it on 2026-06-30 in a "prune dated/scratch notes" pass, unexecuted and unrejected — swept up because its name had a date in it. Restored verbatim (fc671dc) and then carried out (49b9374): 50 signatures, bytes named once in traits/Cargo.toml and re-exported as traits::Bytes so no repository manifest moved, one Bytes::from per backend at the point its extractor hands over.

The plan assumed one thing the tree did not have. It says the unary gRPC field "is already prost::bytes::Bytes" — only ArtifactChunk.data was. Left alone, the seam would have BOUGHT a copy rather than removed one: a Bytes from the archive to_vec()'d back into the response. cargo xtask gen-proto now declares FetchArtifactResponse.data and PutArtifactRequest.data as bytes too.

MEASURED, oden, both binaries back to back on a busy box (loadavg ~48):

Flat, and that is the honest result. These four arms drive get_file/handle_http2_request in process, where the buffer was already moved rather than cloned. The hops this seam pays off on — the proxy's cache-and-forward, replication, and the gRPC unary and stream doors — have no bench arm in holger at all. Naming that is worth more than quoting a number from the wrong workload.

What is left