nordisk/holger

STINKY 01 — the data plane

Lane 1 of the stinky list. Audited 2026-09-20 against the house doctrine in workspace_nordisk/nornir-workspace.toml lines 1-28 (gatling never rayon, zero copy, zero allocation, no serial CPU wait, Arrow IPC as storage) and the owner's product sentence of the same day: "en mycket performant, säker produkt, zero copy, apache arrow och produkt som överlever, har job köer".

Every finding is tagged MEASURED (I ran it, on t14s, numbers in the text) or READ (I only read the code). The two are never blurred. Ranked by cost inside each theme; the five worst across all themes are the summary table at the end.

Worktree: branch stinky/dataplane off main @ 542e296. Nothing in the tree was changed but this file. Bench source is reproduced in the appendix so the numbers can be re-run; it lives in the lane's scratchpad, never in holger.

0. What gatling saturation looks like in this house (the yardstick)

READ, in skade and znippy-zoomies, before touching holger:

• skade/skade/src/parquet_io.rs:131-155 — row groups bucketed per worker, then ONE gatling_map_owned(buckets, …); results land in a pre-allocated index-ordered Vec, one out-slot per unit, no barrier, no channel. skade/skade/src/write.rs:1188-1209 fans FileIO writes through gatling::io::run_ordered(jobs, depth, …) with one sequential committer after. skade/tests/rayon_free_law.rs enforces it.
• znippy-zoomies/lgz/src/bin/lgz.rs:290-330 — prefix-sum of member sizes → offsets, output set_len once, every worker decodes into its OWN reused buf and pwrites at its offset (gatling::parwrite::write_segments_positional). No collector.
• znippy-zoomies/ljar/src/batch.rs:113-131 — LPT-weighted gatling_for_each_balanced inflating entries out of ONE preallocated slab, addressed by buf_offset..buf_offset+len.
• Documented results: .nornir/gatling-reasoning.md:106-145 ljar 3.5 → 11.5/12 cores busy; skade README 234 M rows/s at 9.5 cores, ~21/32 cores on compute-bound data.

The pattern is: units known up front → one fan-out over all cores → preallocated, offset-addressed output → one committer. Hold that picture while reading below.

Does holger-server do any of it? No. MEASURED (cargo tree, see §1.1): the engine is linked, twice, and the server calls it zero times. The only holger code that uses gatling is the agent's upload fan-out, agent/lib/src/operations.rs:531,578,660 (gatling::io::run(jobs, UPLOAD_CONCURRENCY=8, …)) — client side, and it still std::fs::reads each crate whole first (:523).

1. The engine: linked twice, called never

1.1 gatling/znippy-zoomies are in the graph twice and holger-server calls neither — MEASURED

cargo tree -e normal --prefix none from server/cli (default features, 433 unique packages) contains:

rotaryengine v0.1.12 (…/znippy-zoomies/gatling)      <- the gatling crate
znippy-zoomies v0.1.24                               <- crates.io copy
znippy-zoomies v0.1.25 (…/znippy-zoomies)            <- path copy
lgz v0.3.14  +  lgz v0.3.14 (path)                   <- twice
ldeflate v0.1.1  +  ldeflate v0.1.2 (path)           <- twice

They arrive transitively through znippy-common (path-patched) and through server/repository/znippy-helm/Cargo.toml:32 (lgz by path). grep -rn gatling\|zoomies server/ traits/ plugin/ finds only doc comments and the helm lgz decode; not one call into gatling_forkjoin, gatling::run, parwrite or ChunkRevolver from the server. Cost: two compiled engines in every binary, zero fan-out in the server. Should be: ONE version (drop the crates.io copy via the workspace [patch] or bump helm's lgz to the path), and the server's fetch/put/verify loops (§2, §3) on it.

1.2 rayon: absent in all four tiers — MEASURED, and it is the one clean row

cargo tree -i rayon for default, --features full, --no-default-features --features embedded and --features mvp-server: package ID specification rayon did not match any packages every time. ROOT LAW #0 holds. Note the agent's integration test server/repository/znippy-rust/tests/integration_znippy_serve.rs:128 lists "rayon" as a crate NAME in a fixture, not a dependency.

2. Zero copy: the artifact is copied whole, at least four times per request

2.1 The backend contract is whole-artifact Vec<u8> in both directions — READ

traits/src/lib.rs:328-329:

fn fetch(&self, id: &ArtifactId) -> anyhow::Result<Option<Vec<u8>>>;
fn put(&self, id: &ArtifactId, data: &[u8]) -> anyhow::Result<()>;

and :392-397 handle_http2_request(&self, method, suburl, body: &[u8]) -> (u16, headers, Vec<u8>). Every one of the 21 backends, the proxy (proxy.rs:493), promotion (promote/mod.rs:699), replication (replication.rs:203), search's checksum mode (search.rs:283) and both doors inherit it. There is no Read/AsyncRead, no Bytes slice, no offset into a mapped archive anywhere on the contract. Cost: the artifact's full size in RAM per request, per hop, plus the copies below. Should be: a fetch_into(&self, id, sink: &mut dyn Write) / fetch_range pair or a Bytes-returning contract over an mmap'd store, so that a znippy blob is served as an offset+len into the archive (the store already knows both — ZnippyArchive.file_index) and a file store is served with sendfile-class streaming.

2.2 HTTP door: body collected whole, digested serially, dispatched whole, replied whole — READ + MEASURED

server/lib/src/exposed/http.rs:1408-1412 collects the entire request body (Limited::new(req.into_body(), settings.max_body_bytes).collect().await → to_bytes()), cap DEFAULT_MAX_BODY_BYTES = 1 GiB (grpc.rs:45, lib.rs:1679). :1424-1426 then runs verify_upload_checksum = Sha256::digest(body) / Sha512::digest(body) (:1469-1494) — one thread, whole body, on the request. :1432 hands the whole Bytes to spawn_blocking, the backend returns a whole Vec<u8>, :1445 wraps it in Full::new(Bytes::from(body)). The read side is the same: fetch() → Vec → Full. No StreamBody, no ReaderStream, no ranged read exists in the file (grep is empty).

MEASURED on t14s (16 cpus, release, 256 MiB pseudo-random buffer):

So a 1 GiB upload holds one blocking thread for 1.75 s of sha256 (7 s if sha512) plus the collect copy, and a 1 GiB download costs a 0.66 s fs::read copy before the first byte leaves. Fifteen of sixteen cores idle throughout. Should be: hash while the body streams in (a Digest::update per frame, no post-hoc pass), sha256 on the gatling fork-join over chunks with a tree-combine where the client allows blake3, and zero-copy serve (mmap + Bytes::from_owner, or tokio::fs::File + ReaderStream).

2.3 gRPC: the "stream" door loads the whole blob first — READ

server/lib/src/grpc/services.rs:465-600 stream_artifact: spawn_blocking(move || repo.fetch(&trait_id)) (:510) returns the WHOLE Vec<u8>, :577-596 then slices it into 64 KiB ArtifactChunks over a channel of 16. The comment at :499-503 admits it: "chunking below is a WIRE-FRAMING detail, not a memory bound". put_artifact (:339-460) takes a unary PutArtifactRequest{data} capped at 1 GiB, so the tonic decoder buffers the whole message (lib.rs:1889-1925 sets max_decoding_message_size(max_body_bytes) on every service). Cost: identical to §2.2 plus one prost copy. Should be: client-streaming PutArtifact feeding a Write that hashes as it goes; StreamArtifact reading ranges from the store.

2.4 OCI chunked upload: appended to disk, then read back whole, hashed, written again — READ

server/repository/file-oci/src/lib.rs:276-306 finalize_upload: the last PATCH body is appended to the staging file, then fs::read(&staging) loads the ENTIRE blob, sha256_hex(&contents) digests it single-threaded, write_atomic (:142) writes it a second time via fs::write(&tmp) + rename, then the staging file is removed. A 4 GiB image layer is therefore written twice and read once through user space, with a 1-thread sha256 pass in the middle. Should be: hash incrementally per PATCH (keep the Sha256 state beside the upload id), rename(staging → blob_path) on finalize — zero extra bytes moved.

2.5 Writable znippy repo re-parses the whole Arrow index on EVERY read — MEASURED

server/repository/znippy-rust/src/lib.rs:186-199 try_open_reader: when writable_archive is set, every call does self.format_gate.check(path) and ZnippyArchive::open(path) — a fresh parse of the archive's Arrow IPC footer, manifest and fst — and fetch (:642), contains (:231), list (:699), archive_files (:731), index_deps_features (:240) all go through it. The read-only sibling caches the reader (:131,149); the writable one throws it away after every request.

MEASURED (t14s, tmpfs, 1 KiB crates, release):

A crates mirror of 20 000 crates pays 15 ms of index parsing per GET before it copies a single byte, and it pays it again for the contains check inside put. Should be: one cached Arc<ZnippyArchive> swapped under an ArcSwap/RwLock on invalidate_index_cache() — the read-only path already does this.

2.6 Smaller copies on the same paths — READ

• promote/mod.rs:632-647 seal_inner: blake3::hash(data) single-thread, then fs::write(&tmp, data) — a second full copy of the artifact that promote_artifact (grpc/services.rs:911) already fetched whole from the source repo.
• replication.rs:196-232: fetch whole → content_id(&bytes) → put whole, one artifact at a time, one thread, inside one spawn_blocking (grpc/services.rs:1705).
• znippy-rust/src/lib.rs:686 put: vec![(filename, data.to_vec())] — another copy of the whole crate before it is handed to znippy.
• raw.rs:104-108 read_relpath: std::fs::read whole file.
• properties.rs:114-120 + http.rs:1899-1910: when quarantine is configured, every GET does a synchronous std::fs::read of a JSON side-file ON THE ASYNC TASK (not spawn_blocking).

3. No serial CPU wait: everything CPU-bound is one thread on the request

3.1 The digests — MEASURED (numbers in §2.2)

sha256 at the HTTP door (http.rs:1478), sha256 in OCI finalize (file-oci:296), sha256 in store_monolithic_blob (file-oci:320), blake3 in seal (promote:632), blake3 in replication (replication.rs:43), sha256 in the sparse-index synthesis of znippy-rust — all Digest::digest(whole) on one thread. The house has two multicore digest tools already: blake3's tree hash (the rayon-free way is chunk-per-worker on gatling_forkjoin + Hasher::update per chunk with the standard subtree combine — znippy-common's CAS per-chunk verify, Cargo.toml:58, is that pattern), and, for sha256 where the wire format demands it, hashing WHILE the bytes stream so the pass costs no wall time at all.

3.2 Replication and promotion are a for loop — READ

replication.rs:196 for (i, entry) in entries.into_iter().enumerate() fetch → hash → put, serial; replicate_pairs (:168) is a serial loop over pairs on top. This is exactly the shape gatling_for_each_balanced (weight = size_bytes, which list() already returns) exists for: units known up front, skewed sizes, no barrier. Cost: one core, and a mirror of N artifacts takes N × (fetch+hash+put) wall time.

3.3 The audit log is a global mutex + synchronous file write on every request — READ

audit/arrow_ipc.rs:701-719 record: self.writer.lock() (one Mutex<Segment> for the process), build a one-row RecordBatch, StreamWriter::write, flush(). Called from the async task in BOTH doors — http.rs:1250-1261 (the record closure) and grpc.rs:541-555 record_audit — not from spawn_blocking. Every request in the process serialises through one lock and one write(2). The file's own header says "audit volume is low relative to artifact serving, so the per-row batch overhead is irrelevant" — that was true when audit was rare; now every GET/PUT/HEAD records. Should be: an mpsc channel into ONE writer task that batches rows into a RecordBatch per tick (Arrow IPC is built for batches, one-row batches defeat it), flushed on a timer and on shutdown.

4. Zero allocation: nothing is pooled

READ. grep -c "Vec::with_capacity\|BytesMut::with_capacity\|pool\|Pool" server/lib/src/exposed/http.rs = 0 relevant hits; every request allocates its body Vec, its response Vec, its headers Vec<(String,String)>, its audit RecordBatch, its PropertyMap. There is no buffer pool, no slab, no ChunkRevolver. The gRPC stream at services.rs:577 is the one place that avoids a per-chunk allocation (Bytes::slice). Cost: allocator pressure proportional to concurrent request bytes; with 1 GiB bodies that is the whole heap. Should be: bounded BytesMut pool for bodies, ChunkRevolver for the digest/compress workers, preallocated response headers.

5. Apache Arrow: present where it is cheap, absent where it would pay

READ.

• Present: the audit log (audit/arrow_ipc.rs) — an append-only Arrow IPC stream, good shape, but one row per batch (§3.3). The znippy index — through the library.
• Absent: list()/search (search.rs:255-300) walk Vec<ArtifactEntry> per repo, MAX_SCAN_PER_REPO at a time, string-matching in a loop; the properties store is one JSON file per artifact (properties.rs:118); the promotion lineage is JSONL per artifact (promote/stages.rs:309-322); the job ledger is redb (jera's choice, reasonable). Nothing in holger's own metadata is columnar or queryable by DataFusion, though every znippy archive under it already carries an Arrow index that list re-reads (§2.5) instead of scanning as a RecordBatch.

6. Job queues: there are none — ANSWERED 2026-09-21, agent/job-queue

STRUCK. There is a job queue: server/lib/src/jobs.rs behind the jobs feature — the four long operations as durable rows on the SHARED jera ledger (no second ledger), dequeued by a gatling_run worker pool, with async_job on PromoteArtifact/Replicate/PutArtifact returning a JobId at once and a read-only admin-plane JobService showing the rows. A restart RESUMES a resumable job or names it interrupted with a reason; nothing is left running. See .nornir/HOLGER-PLAN.md §4 "the job queue — 2026-09-21" for the shape, the numbers and the test names. What follows is the finding as measured on 2026-09-20 and is kept, not deleted.

6.1 Long operations run inline on the request; the ledger only watches — READ

• Promotion: grpc/services.rs:911 promote_artifact → fetch whole → engine.seal (:1114) inline. promote/mod.rs:566-606 wraps the seal in a jera JobHandle::start … finish/fail. jera is a ledger (its own header, edda/crates/jera/src/jobs.rs:1-40: "one place that answers what's running / did it finish"), not a queue: nothing dequeues, nothing retries, nothing schedules.
• Replication: grpc/services.rs:1645 replicate → one spawn_blocking for the whole run (:1705), the client waits on the RPC for as long as it takes; the CLI verb cli/src/main.rs:1839 is the same loop in-process.
• Uploads: inline (§2.2/§2.3). Gate evaluation (promote/mod.rs:122 OSV scan): inline on the promote request.
• Retention: lib.rs:1373-1389 one retention-gc job record around an inline run. retention.rs:7 says it plainly: "no I/O, no scheduler, no deletion" — planning only.

6.2 A restart loses in-flight work and leaves running zombies — READ

Nothing in holger sweeps the ledger at startup (grep -rn "sweep\|resume\|zombie" server/lib/src → only unrelated hits). jera has start_or_resume_daemon (jobs.rs:1400-1415) for daemons and marks duplicates STOPPED there, but holger opens the store (lib.rs:1797-1804) and never calls any resume/mark. A promotion killed between store.start and job.finish stays running forever, and the half-written blob/ref (§7.2) stays on disk with nobody owning it. The client got a broken RPC and must guess whether to retry; seal_inner answers a retry with AlreadyPromoted if the ref landed and success if it did not — the operator cannot tell which from the ledger.

6.3 Backpressure exists only as two semaphores, on reads, on gRPC — READ

grpc.rs:68-99,228-233,433-470: read_budget (4 GiB in-flight, 1 MiB units) and read_slots (512). Good design, half deployed: grep -c read_budget exposed/http.rs = 0 — the HTTP door has none; writes have none on either door (grep write_budget|max_concurrent_writes → nothing); AcceptLimits.max_connections (http.rs:171) bounds connections, not bytes. 64 concurrent 1 GiB PUTs over HTTP is 64 GiB of resident Bytes on a box that gets OOM-killed (§7.4). Should be: the same budget on both doors and on writes, charged as the stream arrives, refused with 503 before the first byte is buffered.

6.4 What the operator can see — READ

jera::jobs::JobStore::list (jobs.rs:1522) exists; holger exposes no ListJobs RPC and no HTTP page (grep -n "ListJobs\|job_store" grpc/services.rs exposed/http.rs → nothing). The ledger is written and never shown.

6.5 What it should be — the sketch

One durable queue in the existing redb (or a znippy hot journal, §7.1): enqueue(kind, payload) from the door, return the job id at once (202 + Location: /jobs/<id>); N worker tasks per class (jera::jobs::JobClass already exists, jobs.rs:295) on a bounded mpsc with gatling::io::run fanning the CPU half; at startup every non-terminal row is either resumed (idempotent kinds: replicate, retention) or marked interrupted with a reason (promote). Retry with backoff per kind, poison after N. GET /jobs and ListJobs read the same rows the ledger already has.

7. Produkt som överlever: the store is not crash-tested

7.1 Every put to a writable znippy repo re-seals the whole index: O(N²) — MEASURED

znippy-rust/src/lib.rs:686-695 put → meta_sink_append::append_files(path, &[one file], 3). znippy-common's own module header (znippy-common/src/hot.rs:1-30) names the defect and measured it on oden 2026-08-04: re-seal 10.18 ms/append at 8 000 entries, 61.2 s for 8 000 appends vs 2.29 s on the HotArchive O(1) path that holger does not use. Re-measured here, t14s, real disk (/home/rickard/cargo-target, ext4, fsyncs real), 1 KiB crates, one put per call exactly like the server:

2 000 one-at-a-time puts: 18.5 s. The same 2 000 files as one create_archive: 24 ms — 773×. First thousand 7.5 s, second thousand 11.0 s: the quadratic signature. Every put also holds self.write_lock (:680) so all writers to that repo queue behind the re-seal, and (§2.5) every reader re-parses the index that just grew. Should be: HotArchive::open once per writable repo, append per put (hot.rs:184,301), seal_hot on a timer/size threshold or at shutdown. This is the same family of defect as the Iceberg orphaned-manifest loop-write: a write path that was never crash-tested and never sized.

7.2 Three of four stores rename without fsync; one fsyncs the root of the filesystem — MEASURED

Without fsync before rename, a power loss after the rename can leave the final name pointing at a zero-length or torn file (ext4 auto_da_alloc only rescues rename-over-existing; these are new names). The promotion "seal" is therefore not a seal: the ref file can exist with its blob empty, and is_promoted (:550) says yes.

MEASURED, 100 × 4 KiB puts into a 7-deep directory on the ext4 root disk:

So the raw store pays 24 % over the correct discipline fsyncing /, /home, … on every put (and serialises the whole disk on those root inodes), while the three stores that hold cargo crates, container images and promoted artifacts pay nothing and promise nothing. Should be: one write_durable(tmp, data) in holger-traits used by all four: sync_all on the file, rename, sync_all on the PARENT only.

7.3 Disk full: undetected, unreported, leaves litter — READ

grep -rni "enospc\|StorageFull\|available_space\|statvfs" server traits → nothing. No headroom check before accepting a body, no metric, no 507. On ENOSPC: file-rust:445 fs::write(&tmp) fails and the .pid.seq.tmp file is left in the artifact directory (only the rename branch cleans up in file-oci:150); znippy-rust append can fail mid-tail-rewrite — meta_sink_append.rs:530-533 does set_len + sync_all at the end, so a failure before that leaves a truncated tail that open_existing must "recover" (:1006 recovered_rows); the audit flush fails and is logged at warn (http.rs:1259) — the request still returns 200 with no audit row. Should be: a statvfs headroom gate at the door (refuse writes under X GiB free, 507), tmp-file cleanup on every error path, and audit failure = request failure for write verbs.

7.4 OOM: the design invites it — READ

Body cap 1 GiB × no write budget × no HTTP read budget × whole-artifact Vecs at every hop (§2) × spawn_blocking default pool of 512 threads each able to hold a body. The process has no memory accounting of its own; the only limiter is the kernel's OOM killer, which takes the whole server — and with §6.2 every in-flight promotion with it and with §7.2 possibly a torn store. Should be: the read budget (already written) applied to writes and to HTTP, sized from available_memory at start, and streaming so the budget is per-chunk not per-artifact.

7.5 What is right — READ, for balance

raw.rs durability (over-eager, but correct), the audit hash chain (arrow_ipc.rs:709), the gRPC read budget that moves with the stream (grpc.rs:104-115), the OCI would_exceed_blob_ceiling check, the create_new ref that makes double-promotion impossible, Limited body caps on every JSON door. None of it is wasted; it is just not applied uniformly.

8. Ranked: the five worst

9. Patch sketches (not applied — this lane writes the list)

• Hot archive (znippy-rust/src/lib.rs:663-696): hold Option<Mutex<HotArchive>> opened with HotArchive::open(path, LEVEL, SkipPolicy::already_compressed()); put = hot.append(&[(name, data)]); seal on invalidate_index_cache every K puts or on a 30 s tick; readers hold an ArcSwap<ZnippyArchive> refreshed after seal, never re-opened per request.
• Durable write helper in traits: pub fn write_durable(dst: &Path, data: &[u8]) = tmp beside dst, write_all, sync_all, rename, File::open(parent).sync_all(). Replace raw.rs:149-178, file-rust:431-447, file-oci:142-153, promote:643-646.
• Streaming digest at the door: replace Limited::collect + Sha256::digest with a while let Some(frame) = body.frame().await loop feeding hasher.update(&chunk) and a BytesMut from a bounded pool; refuse on mismatch before dispatch as today.
• Audit writer task: tokio::sync::mpsc::channel::<AuditEvent>(4096) → one task, RecordBatch per 100 rows or 200 ms, flush + hash-chain there; record becomes try_send and a dropped-events counter in /metrics.
• Queue: jera::jobs rows with status queued + a holger-server worker set; promote_artifact/replicate return the job id; startup marks non-terminal rows interrupted.

Appendix A — how the numbers were made

Box: t14s, nproc = 16, Linux 7.0, ext4 on nvme for disk tests, tmpfs for the open/extract test, release profile, single run each except where a window average is shown. Bench crate: scratchpad stinky-bench (path dep on /home/rickard/git/znippy/znippy-common, features host-decompressors; sha2 0.10, blake3 1). Sub-benches m1 (open vs extract, n ∈ {100, 1000, 5000, 20000}), m2 (one append_files per put, n = 2000, then one create_archive), m3 (digests and copies over 256 MiB xorshift-filled), m4 (three write disciplines × 100 files). The cargo tree runs were from server/cli in this worktree with the sibling repos symlinked beside it (edda, znippy, znippy-zoomies, draupnir, skade, facett, Skidbladnir, tunnr, knut, modgunn, korp, nornir, dwarves).