nordisk/holger

Stinky 08 — Archaeology: what was fast, and what became of it

Lane A2. The question the owner asked, 2026-09-20: "I built the initial holger machinery with a trait and an impl of the packages. TLS was hard, but that was 1 year ago and it may have harmonised now. It was TERRIBLY FAST, and TLS was done in a way that things were kept up and cached."

Every claim below is marked MEASURED (a number produced in this lane or quoted from a commit that produced one) or READ (found in the tree or in history).

The verdict alphabet: (A) never there, must be built · (B) there, still there, just not wired to today's path · (C) there, and LOST.

0 · The calendar correction (READ)

There is no year of history. holger's first commit is d2f2706, 2026-05-19 — four months, 672 commits on main, 730 across all refs, and no earlier lineage anywhere (workspace_nordisk/holger is a 2026-06-14 workspace, holger-egui 2026-06-13, holger-ui 2026-08-13, private-holger-ops 2026-09-17; no dangling re-root, unlike gunnar). "A year ago" is 2026-05-19 → 2026-05-31: the first two weeks. Everything the owner remembers as the original machinery was built, benchmarked and then reorganised inside twelve days.

1 · The original trait — was it ever streaming?

VERDICT: (A). It was NEVER streaming. There is nothing to restore.

The trait was born whole-Vec<u8> on day one, 17e161b (2026-05-19), holger-traits/src/lib.rs, 88 lines (READ):

fn fetch(&self, id: &ArtifactId) -> anyhow::Result<Option<Vec<u8>>>;
fn put(&self, id: &ArtifactId, data: &[u8]) -> anyhow::Result<()>;
fn fetch_many_with_upstreams(..) -> anyhow::Result<HashMap<ArtifactId, Vec<u8>>>;
fn handle_http2_request(&self, suburl: &str, body: &[u8])
    -> anyhow::Result<(u16, Vec<(String, String)>, Vec<u8>)>;

That is today's shape, verb for verb. Today's traits/src/lib.rs is the same four signatures plus eight verbs grown around them. git log -S'RepositoryBackendTrait' over all refs returns 48 commits and not one of them ever widened fetch to a stream. git log -S'fetch_stream' across all refs returns two hits, both in prose, never in code:

• acde385 (2026-06-27), remote-cache-trait-design-2026-06-27.md — "today the trait returns Vec<u8> (whole-body); keep that for the first cut to avoid a trait-wide signature change, and add a fetch_stream(id) -> Result<Box<dyn Read>> default-None method later if large-blob streaming becomes a concern (Docker layers)."
• be80f18 (2026-06-30) — the same file deleted, three days later, in the årsstädning prune.

So the flattening the lane hypothesised never happened, because there was never anything to flatten. The whole-Vec cost is original sin, not regression.

1b · The one real loss on this axis: the Bytes migration (C, doc only)

81d8e24 (2026-06-28) landed .nornir/vec-to-bytes-migration.md: a shovel-ready Vec<u8> → bytes::Bytes plan — the deref-safe seam, the blast radius counted (26 impls, ~25 call sites, 14 znippy backends + 2 file + 3 proxy + 1 upstream + 6 mocks), a six-phase landing order, hot path first, and the observation that grpc.rs::stream_artifact's per-chunk Bytes::from(data) disappears on phase 2.

be80f18 (2026-06-30) deleted it, unexecuted, two days later, in a commit whose subject is "prune dated/scratch notes" — it was swept up with ten other files because it carried a date-shaped name, not because anyone decided against it. bytes is still in the tree via prost. The plan is recoverable verbatim: git show 81d8e24:.nornir/vec-to-bytes-migration.md.

RESTORE, do not rewrite.

2 · The TLS story — "kept up and cached"

VERDICT: (B) for the design — it was deliberate, it is written down, and it is still there. (C) for the dependency hygiene around it.

He is remembering bcf9549 (2026-06-16), "docs(design): §6c read-path perf + mTLS/OIDC auth cost notes". §6c is still in .nornir/design.md and it is exactly his sentence, spelled out (READ):

"The client-cert chain is verified once, at the TLS handshake, by rustls WebPkiClientVerifier. Per request, auth only reads the already-validated leaf cert's CN … The handshake cost itself is amortized: ALPN negotiates HTTP/2, so many requests multiplex over one long-lived connection (one handshake, N requests), and rustls' default session resumption (TLS 1.3 tickets / session cache) lets reconnects skip the full handshake. The ring provider is pinned process-wide (no OpenSSL). Net per-request mTLS cost: one cheap CN parse on writes, nothing on reads."

Still there today, verbatim mechanism, server/lib/src/exposed/tls.rs (READ): ServerConfig::builder_with_provider on a rustls::crypto::ring::default_provider() pinned once through a OnceLock, WebPkiClientVerifier for the client chain, config.alpn_protocols = [h2, http/1.1], leaf_common_name() as the only per-request work. Nothing was lost. rustls has been the choice since the first commit — 17e161b:holger-ron/Cargo.toml already pinned tokio-rustls 0.26 / rustls 0.23.

The §6c note also named its own open wound: "OIDC is the un-optimized auth path — /userinfo on every request and a fresh reqwest::Client each time; the clear optimization target: reuse one client, cache (token → sub), verify the JWT against JWKS locally." That was done, and it left the repo: 4341b6c (2026-07-18) moved the providers to edda/crates/nornir-auth-oidc, which today carries a process-global JWKS cache keyed by issuer, local jsonwebtoken verification of exp/iss/aud, a pooled client and a TTL result-cache on the /userinfo fallback. The caching story is not merely intact — it grew.

2b · What actually rotted: default-features (C, and cheap)

server/lib/Cargo.toml:121 declares reqwest correctly and even ships the proof command:

reqwest = { version = "0.12", default-features = false, features = ["rustls-tls", "json"], optional = true }

Six of the nine reqwest declarations in the tree forget default-features = false, so cargo unifies reqwest's default-tls — hyper-tls → native-tls → OpenSSL — into every crate in the graph (READ, grep '^reqwest' **/Cargo.toml):

MEASURED (cargo tree -i native-tls): nine roots reach native-tls, and holger-server-lib is one of them. So the leak is not only the benchmark crate — a repository backend is one of the doors. And MEASURED again, by accident: building nothing but -p holger-rust-znippy-repository --test bench_znippy_throughput on loki compiles openssl-macros, foreign-types, openssl-sys. Two words in six manifests are the whole fix, and §6c's "no OpenSSL" becomes true again.

Every one of those six lines dates to b5a435c (2026-05-30), the great reorganise — they were sloppy from birth, not degraded.

3 · What else was fast

3a · The kept-open archive — (B), still there on the read path

The legacy README's "How it works" is the doctrine, written 2026-05-29 (READ, legacy/README.md): "Each get_file is one pread from a shared fd plus either a zero-copy skip (pre-compressed .crate/.jar) or an openzl decompress into a reused buffer — fully parallel across cores, no shared reader thread."

The shared fd is still shared on the static read path. RustRepoZnippy::with_archive opens the ZnippyArchive once and holds it as Option<Arc<ZnippyArchive>>; try_open_reader (server/repository/znippy-rust/src/lib.rs:197) returns self.reader.clone() — an Arc bump, no open. The 15 ms-per-GET ZnippyArchive::open the data-plane lane measured is the other arm of that same function:

if let Some(path) = &self.writable_archive {
    if path.exists() {
        self.format_gate.check(path)?;
        let archive = ZnippyArchive::open(path)?;   // every call

That arm was born slow, 6280d81 (2026-06-19) "znippy-rust: writable dev-mode backend (the /sparring + /cache primary)", and its own rustdoc says so out loud: "In writable dev-mode the on-disk archive is re-opened on every call (a put invalidates any cached view)." It is not a regression; it is a dev-mode shortcut that became the primary write repo. (A) for the writable path, (B) for the static path.

3b · HotArchive — (A) in holger, but the fix is already on the shelf

git log -S'HotArchive' across all holger refs returns one commit: 178ae87, today's stinky-01. holger has never called it. But it is not something to build: znippy-common has carried HotArchive since 604ccf4 (2026-08-04, "Every append was re-sealing the whole index, so N pushes cost N squared"), it is re-exported from znippy_common (pub use hot::{HotAppendReport, HotArchive, HotSealReport, seal_hot}), and holger already depends on that crate at 0.9.14/0.9.15.

holger's put calls znippy_common::meta_sink_append::append_files — the exact function hot.rs's header names as the defect, with znippy's own MEASURED table beside it (oden 2026-08-04: 8 000 appends, 10.18 ms → 0.060 ms per append, 61.16 s → 2.29 s wall, 26.7× wall / 170× CPU, and the ratio grows with N). The data-plane lane's 2 000 puts in 18.5 s vs 24 ms batched is the same curve on a different box.

holger's writable backend (2026-06-19) simply predates the fix (2026-08-04) and was never revisited. Nothing was lost; the remedy is a dependency bump away.

3c · gatling in the server — (A), never

git log -S'gatling' over all refs: the engine entered holger once, b3b76d3 (2026-06-28), "agent/lib: parallelize push upload save-loops via znippy_zoomies::gatling::io bounded in-flight pool (~7.4x; no rayon)". The four call sites today are all in agent/lib/src/operations.rs (531, 578, 660) plus its doc comment. The server has never called it, at any commit. The 10 M ops/s number was never the server's — it was the library's ceiling, measured by a test that spawned its own std::thread fan-out, and the owner already re-labelled it as such at 0632b52 (2026-06-28, "in-process relabeled as internal ceiling").

3d · a drifted twin in the graph (READ, incidental)

Cargo.lock carries two holger-rust-znippy-repository 0.6.10: the workspace path copy and the crates.io copy (checksum d36ce278…), reached through the published holger-server-lib/holger-embed chain. The crate’s own Cargo.toml already documents the cause — "0.6.10: local source DRIFTED from the published 0.6.9 under the same version number" — so two different bodies of code answer to one version string inside one build. Not a performance finding, but it is the kind of thing that makes a measurement unreproducible, and it should be resolved before any of the restores below are measured.

4 · The benchmark that proved "terribly fast"

VERDICT: (B) for the number, (C) for the serving path.

It exists. It was 8aa6159 (2026-05-22), "Add throughput benchmark: 523k crates/s and 341k JARs/s single-thread", holger-rust-znippy-repository/tests/bench_znippy_throughput.rs, 627 lines; last refreshed at b91017f (2026-05-29) against znippy 0.8.0, and the number it produced is preserved in legacy/README.md (32-core AMD, 500 000 entries, mixed sizes):

The file was deleted at 135b8c8 (2026-05-31) — "Delete legacy tests/bench_znippy_throughput.rs (benches owned by xtask/examples/nornir-bench.rs)" — and, unusually for this kind of claim, it was true: the same day, cd7636d extracted the archive builders and bench loops into holger-bench-scenarios, and bench-scenarios/src/lib.rs:898 still carries rust_crate_st / rust_crate_mt today, registered as holger.rust_crate_st / holger.rust_crate_mt32 through the repo_bench! macro in xtask/examples/nornir-bench.rs:55-56, with the same synthetic corpus and the same bench_loop/bench_threaded shape. The headline number is still runnable.

What was lost in that move is the other half of the original file: the original benchmarked handle_http2_request beside get_file, single- and multi-threaded — the serving path, path parse plus extraction plus the whole-Vec response. bench-scenarios kept only get_file. Since 2026-05-31 there has been no bench on the request path at all, which is precisely the path the data-plane lane measured as slow. That is the gap that let the sha256-on-the-request-thread and whole-body-collect costs accumulate unobserved.

This lane restored the deleted file (Rust-crate half, verbatim, count tunable via HOLGER_BENCH_CRATES) at server/repository/znippy-rust/tests/bench_znippy_throughput.rs so both halves are measurable again.

The two numbers, side by side

MEASURED, loki, 2026-09-20, 12 cores, release, 500 000-crate synthetic archive (the original corpus and the original code), against the recorded 2026-05-29 run on a 32-core AMD box:

Scaling 1 → 12 threads is 9.13× on 12 cores: still "fully parallel across cores, no shared reader thread".

So the machinery he remembers is not slower. It is faster than its own record, per core, on a smaller box, four months later. Whatever today's slowness is, it is not in the trait + znippy impl he built — it is in the hops the lane-01 measurements point at (the writable put, the request thread's sha256, the whole-body collect, the gRPC "stream"), none of which this benchmark has ever touched. The serving route itself (handle_http2_request, path parse + extraction + Vec response) costs only 19% over raw get_file single-threaded — the parse is not the problem either.

A note from restoring it: handle_http2_request grew its method parameter at a3cd65e (2026-06-13) as the first argument, beside suburl, and both are &str. Swapping them compiles silently and every route falls through to the 404 catch-all. It cost this lane three round trips on loki. A Method newtype would have made it a compile error.

5 · Verdict table

6 · What to RESTORE rather than rewrite

1. 81d8e24's Bytes plan — git show 81d8e24:.nornir/vec-to-bytes-migration.md. Six phases, deref-safe, "build breaks: none expected". It was never rejected, only swept up.
2. default-features = false on six reqwest lines — two words each, and §6c's "no OpenSSL" is true again.
3. The request-path half of 8aa6159 — restored here; wire it into nornir-bench beside rust_crate_st so the serving path has a number that can regress loudly.
4. HotArchive under holger's put — not a rewrite, a call-site change in server/repository/znippy-rust/src/lib.rs:663, against a dependency already in the lock.