nordisk/holger

10 — Benchmark baseline, BEFORE the stinky-lane changes

Why this file exists. Four lanes are changing holger right now (feature-edge cleanup, a Gate trait, plugin ABI v2, data-plane work) and a fifth is about to make holger-bench-scenarios a dev-dependency. A baseline taken after any of those lands is worthless. This is the before.

Every number here is stamped with its box. A benchmark number without its machine is not a measurement (workspace LAW; see also FACETT.md:191, where a rotate_bench hard-coded machine = "Loki" string ran for months on oden and made every recorded figure unattributable).

0. Provenance of this baseline

Host-build caveat, stated plainly. The LAW says a build thing builds in a CONTAINER, never on the host, because otherwise the BOX decides the verdict. This baseline was compiled on loki's host toolchain, not through DraupnirCargoRunner. That is acceptable for a benchmark only because a benchmark is a measurement OF THE BOX by definition — but it means these numbers are a loki-host row, not a portable verdict, and the after run must use the same road or the comparison is void.

1. The inventory — every benchmark in holger

bench-scenarios is the single source of the holger-vs-rival workload. The driver is xtask/examples/nornir-bench.rs, which registers 20 arms via nornir::register_bench! and prints ONE BenchRun JSON line on stdout.

1.1 How to run any of them (no warehouse server needed)

# ON LOKI, from a worktree — NOT from the owner's /home/rickard/git/holger
cd ~/git/_wt/bench-b3
cargo build --release -p xtask --example nornir-bench          # ~15 min cold, 800+ crates
NORNIR_MACHINE=loki NORNIR_BENCH_VERSION=0.6.x HOLGER_BENCH_SCALE=quick \
  NORNIR_BENCH_ONLY=holger. \
  ./target/release/examples/nornir-bench > run.json

• NORNIR_BENCH_ONLY=holger. is mandatory, not cosmetic. inventory collects at LINK time, so the binary carries every register_bench! in everything it links — including nornir's own 167 arms. nornir-bench 0.1.8's own comment records nornir bench run skade executing 213 nornir.mimir.* arms and reaching ZERO skade.* arms before it was stopped: "the skade bench was measuring nornir." Omit the filter and you measure nornir, not holger.
• HOLGER_BENCH_SCALE = quick (5 000 entries / 2 s loops, the default) | full (500 000 entries / 5 s) | huge (~100 GiB per type, needs 3×100 GiB). HOLGER_BENCH_BYTES overrides the budget. HOLGER_BENCH_REAL=1 seeds from real crates.io / Maven Central / PyPI downloads (default ON at full/huge).
• Fixtures live in $HOME/holger-bench-data (override HOLGER_BENCH_DIR) and are keyed (FIXTURE_VERSION, real?, size) — never /tmp (tmpfs).
• nornir bench run holger is the persisting form (writes the Iceberg bench_runs table). The plain example run needs no server.

1.2 The 20 arms

1.3 Side doors onto the same workloads

1.4 DEAD — recommend deletion

1.5 Deleted harnesses found in history (the "damma av" targets)

2. WHAT THIS BASELINE WAS BUILT WITH — feature set AND dependency kind

Without this section the after comparison silently compares two different binaries. Recorded as of 542e296.

2.1 Every edge onto holger-bench-scenarios

So: the measurement driver is already a dev-dependency. The ruling changes only the server/cli edge — which is not a bench driver at all, it is the holger-ds CLI verb shipped inside holger-server. See §4.

2.2 What holger-bench-scenarios pins holger-server-lib with

server-lib = { package = "holger-server-lib", version = "0.6.7", path = "../server/lib",
  optional = true, default-features = false,
  features = ["mini", "online", "auth-bearer", "auth-oidc",
              "auth-oauth2", "auth-mtls", "auth-passkey"] }

default-features = false is already applied (commit 6f02ddf), and it already removed front — holger.rs's public web page — from every binary built with bench. That half of the irony is fixed.

What is NOT fixed, and what this baseline therefore ran with:

1. auth-passkey still flows from the bench crate into the shipped server. server/cli's default set is ["mini","online","bench","secure_update","front","auth-passkey"]. auth-passkey was added to that list on 2026-09-20, in 542e296 — the very commit this baseline is taken at — precisely because it had until then existed only through the bench edge. The manifest says so in as many words: MEASURED — drop bench from that set and nornir-auth-webauthn leaves the tree, taking /-/auth/passkey/* with it. "A login door whose existence depends on a benchmark crate being linked is not a declaration." Status at baseline: named explicitly, so the binary no longer depends on the bench edge for it. The irony is closed, one commit old.

2. auth-oidc + auth-oauth2 + auth-mtls are still ONLY declared by the bench crate. No other manifest in the graph names them for the default server. Cargo unifies features per GRAPH, so they reach the systemd binary. This is the live half of the irony.

3. reqwest defaults are never disabled (bench-scenarios/Cargo.toml):

reqwest = { version = "0.12", features = ["rustls-tls","json","blocking","http2"], optional = true }

No default-features = false, so reqwest's default-tls stays on → hyper-tls → native-tls → openssl-sys. The bench crate asks for rustls-tls and gets native OpenSSL anyway. CONFIRMED in Cargo.lock at this commit: hyper-tls 0.6.0 (line 8730) depends on native-tls → openssl + openssl-sys + openssl-probe; reqwest lists hyper-tls and native-tls among its deps (lines 13855, 13860). A C library reaches the shipped server because a benchmark crate did not spell one line.

2.3 Baseline stamp — record this verbatim for the after run

commit          542e296a9f97ce44a8d977167ea2b7a4ed35aa79
bench crate     holger-bench-scenarios 0.1.1
dep KIND        [dev-dependencies] of `xtask`  (path = ../bench-scenarios)
bench features  nexus, artifactory, promote-ledger, retention-planner,
                audit-append, secure-update
server-lib      default-features = false, features = mini, online, auth-bearer,
                auth-oidc, auth-oauth2, auth-mtls, auth-passkey
TLS in graph    rustls (asked for) AND native-tls/openssl-sys (inherited)
profile         release

3. Which benches depend on things that are about to leave the graph

Answering the two live rulings.

3.1 The dev-dependency ruling — "den skall vara dev-beroende, dvs aldrig in i slutbinären!!"

Nothing that measures breaks. Both bench drivers (xtask example, agent/cli twins) are ALREADY dev-dependency edges. The change is a one-line removal on server/cli.

ONE thing breaks, and it is not a bench driver:

Watch this when it lands: dropping bench from server/cli's default set also drops the bench crate's server-lib pin, which is the only declaration in the graph of auth-oidc, auth-oauth2 and auth-mtls. auth-passkey was rescued by name in 542e296; the other three have not been. If the shipped server is meant to keep OIDC/OAuth2/mTLS, server/cli must name them the same way auth-passkey was named, in the same commit. Otherwise three auth doors vanish silently — the identical accident, three more times.

3.2 The feature-edge lane — does any bench NEED a feature being removed?

5. Ranked keep-list — what is an asset, what is a liability

Ranking rule: a bench nobody can run is a liability; a bench that can go RED is a gate; a bench that beats a rival is a sales asset.

Tier 1 — the standing regression gate (run on every release)

1. holger.secure_update_{prepare,sign_finalize,verify,import} — the only arms that carry a red-when-broken ANSWER floor (the bundle must verify, digests must match, anti-rollback must reject a replay). They need no container, no network, no dataset — they mint their own PKI with rcgen. They measure holger's unique capability, which Artifactory and Nexus have no equivalent for. Cheapest to run, hardest to fake, impossible to bit-rot silently. This is the gate. And it already has the cleanest historical record in the repo: bench-results/holger-secure-update-oden-2026-07-23.json carries machine, cores, date, timestamp, bench_secs and the load average at both ends of the run — the best-attributed artifact in either repo.
2. holger.audit_append — fires on the hot path of every single request, and it re-reads the segment and asserts the row count, so a broken writer fails the bench instead of reporting a fast wrong number. No deps. Keep.
3. holger.retention_planner — asserts holger's planner and the naive O(n²) baseline produce the identical verdict before comparing speed. That assertion is what makes the 785× claim honest. No deps. Keep.

Tier 2 — the sales assets (run before any customer conversation, and on a quiet box)

1. holger.nexus_load and 5. holger.artifactory_load — the reason this lane exists. They are not bit-rotted: they are fixture-gated, which is a setup job, not a dead bench. Both images pulled clean today. They are the only evidence for the headline numbers in README.md and .nornir/README-full.md. Caveat that must travel with them: workspace_nordisk/.nornir/znippy-package-capabilities-2026-07-23.md argues throughput is holger's weakest ground and that capability is the stronger pitch — and it explicitly refuses a claim (cross-file dedup is NOT IMPLEMENTED and is an Artifactory strength). Quote the numbers, keep that honesty attached.
2. holger.latency_tail — the fair-heap G1GC comparison. Same container cost as #4; the p99 story is more persuasive than MB/s.

Tier 3 — keep, low ceremony

1. holger.pip_index_serve — the holger-ds Python journey; no container needed.
2. holger.promote_ledger — real engine + real durable ledger.
3. holger.retention_executor.
4. rust_crate_* / java_jar_* / pip_wheel_* (6 arms) — the oldest series, and the reason the _mt32 names must never be renamed (bench_runs keys regression history by metric name). Cheap. Keep.
5. holger.blake3 / holger.memcopy_bandwidth — not product measurements; they are box calibration, and that is exactly why they earn their place: they are how you tell "holger got slower" from "the box got slower".

Liabilities — delete or rehome

• bench_nexus_container.sh — DELETE. Dead, unreferenced, and shells out in violation of the LAW. Fully superseded. (§1.4)
• holger-ds as a CLI verb of holger-server — REHOME to a bench-scenarios example. (§3.1)
• art_probe — a container-startup instrumentation probe, not a measurement. Harmless, but do not mistake it for a bench.

The documentation debt this lane also uncovered

• No number anywhere in holger or workspace_nordisk names its commit. Machine and cores are usually stamped (oden · 32 cores · 2026-07-23); the tree state never is. That is why this file leads with a sha.
• README.md and .nornir/README-full.md disagree: README shows one holger row (108.43 served_mbs) while README-full shows two runs (100.57 and 108.43). The README row is the nexus_load row promoted to stand for holger overall.
• workspace_nordisk/.nornir/suite-improvement-backlog.md:284 quotes "108k ops/s vs Nexus/Artifactory". That number matches nothing measured: the real figures are 108.43 MB/s and 181 863 ops/s. It is the MB/s value re-typed as ops and scaled by 1000. A corrupted quotation in a planning document.
• workspace_nordisk/ORDERS-T14S-2026-09-03-gunnar-SUPPLEMENT.md:52 claims the numbers live in .nornir/best-results-benchmarks.md. That file has never existed in holger's history.
• .nornir/proxy-design.md:25-26 and .nornir/design.md:255 still cite server/lib/src/upstream/{nexus,artifactory}.rs as live paths; both were deleted in a5775cd5 (moved to Skidbladnir).

6. MEASURED: the suite was BIT-ROTTED at HEAD, and this lane fixed it

This is the single most important finding of the lane, and it is a measurement, not a reading.

At 542e296, xtask/examples/nornir-bench.rs — the ONLY driver for all 20 arms — did not compile.

cargo build --release -p xtask --example nornir-bench
error[E0063]: missing field `source` in initializer of `nornir::nornir_bench::BenchResult`
  --> xtask/examples/nornir-bench.rs:34:5   (and 9 more sites)
error: could not compile `xtask` (example "nornir-bench") due to 10 previous errors

nornir-bench 0.1.8 added BenchResult.source: BenchSource (issue #13 — tags a row as a timed measurement vs a static-capabilities probe). The field is #[serde(default)] and skipped on the wire when Measured, so the JSON wire format did not change at all — but the struct literal did, and holger's driver used struct-literal syntax at 10 sites.

Scope of the rot: exactly those 10 lines. 1 361 crates compiled clean, including holger-bench-scenarios itself and every scenario module. The workloads never rotted; only the driver's glue did.

Fix applied on stinky/bench: all 10 sites moved to the canonical constructor BenchResult::measured(name, metrics) — which is exactly what nornir-bench documents as "the canonical constructor for a bencher's timed numbers". Behaviour-identical, and immune to the next field addition.

What this means, and it is uncomfortable:

• Nobody has built the holger bench suite since nornir-bench gained that field. The numbers in README.md and .nornir/README-full.md are stamped oden · 32 cores · 2026-07-23 — almost two months stale, and not reproducible from HEAD until this commit.
• The rot is invisible to cargo build, cargo test and CI, because an --example of a publish = false xtask is built by nothing in the normal flow. workspace_nordisk/.nornir/bench-coverage-audit-2026-07-23.md counted 363 register_bench! arms across the constellation and marked them DESCRIPTIVE ONLY; this is the harder version of that finding — a bench that cannot even be compiled is not descriptive, it is absent.
• Recommendation (cheap, high value): add cargo build -p xtask --example nornir-bench to whatever gate already runs on holger. Compiling the driver costs one link and would have caught this the day it broke. A suite that only rots where nothing looks will rot again.

7. Exact re-run recipe — so "after" needs no archaeology

ssh loki
tmux new-session -d -s bench            # the session named `holger` belongs to another job
cd ~/git/_wt/bench-b3                   # your own worktree — NEVER ~/git/holger (owner's live checkout)
export PATH=$HOME/.cargo/bin:$PATH

# 1. build the driver (this is also the rot detector — see §6)
cargo build --release -p xtask --example nornir-bench

# 2. competitor images (already on loki as of 2026-09-20)
podman pull docker.io/sonatype/nexus3:3.72.0
podman pull releases-docker.jfrog.io/jfrog/artifactory-oss:6.23.13
systemctl --user start podman.socket    # bench_scenarios::ensure_docker_host() finds it

# 3. stamp the box BEFORE and AFTER — a number without its box is not a measurement
(date; uptime; nproc) > run-env.txt

# 4. the run
NORNIR_MACHINE=loki \
NORNIR_BENCH_VERSION=0.6.8 \
HOLGER_BENCH_SCALE=quick \
NORNIR_BENCH_ONLY=holger. \
  ./target/release/examples/nornir-bench > run.json 2> run.err

(date; uptime) >> run-env.txt

Single arms: NORNIR_BENCH_ONLY=holger.nexus_load (prefix match, comma-separated list allowed). NORNIR_BENCH_SKIP drops matching ones. Both report on stderr what they filtered — a silent narrowing would read as "we benched everything".

The no-container subset, for a box with no podman: NORNIR_BENCH_ONLY=holger.secure_update,holger.audit_append,holger.retention,holger.promote_ledger,holger.blake3,holger.memcopy

8. THE BASELINE — MEASURED 2026-09-20 on loki

Run stamp. commit 542e296a9f97ce44a8d977167ea2b7a4ed35aa79 (+ the 10-line driver fix of §6, which changes no workload) · machine loki · nproc 12 · profile release · HOLGER_BENCH_SCALE=quick · bench crate reached through a [dev-dependencies] edge of xtask · features per §2.3 · 20 of 229 linked arms · started 09:14:03 CEST, ended 09:18:11 CEST, EXIT=0.

⚠ THE BOX WAS NOT QUIET, AND THAT IS RECORDED PER ARM. loki carried another lane's KVM estate bench (2 × qemu-system-x86_64) and a second cargo build throughout. Load average 16.03 at start, 12.81 at end, on 12 cores. The harness stamps telem_cores_busy_avg on every row and several arms ran at 12/12 host cores busy. These numbers are therefore a CONTENDED loki row. They are valid as a before/after pair only if the after run is taken under comparable contention — or, better, if BOTH are re-taken on a quiet box. Treat the ratios (holger ÷ rival, measured in the same window on the same box) as the durable result and the absolute MB/s as the soft one.

8.1 The head-to-heads — both rivals answered, neither cell is empty

Publish-phase (separate, never the head-to-head): nexus_upload_mbs 0.784, artifactory_upload_mbs 0.367. Internal ceiling (never a rival column): holger_inprocess_read_ceiling_mbs 1 476.5 (nexus run) / 2 124.1 (artifactory run) — i.e. the HTTP door costs holger ~40–48× its in-process read, which is itself a finding.

The tail is the story. holger's p999 is 2.99 ms where Nexus's is 191.97 ms — that is the JVM GC pause showing up, and it is far more persuasive than MB/s.

8.2 Every other arm

Retention head-to-head (retention_planner, 5 000 records, 2 000 iters): holger 6 109 157 rec/s · 1 222 plans/s · p50 0.817 ms · mean 0.818 · 559.31 MB/s vs naive O(n²) 76 368 rec/s · 15.27 plans/s · p50 65.504 ms · mean 65.473 · 6.99 MB/s → holger_speedup_x = 80.0×, and at 50 000 records holger still does 2 526 054 rec/s (50.5 plans/s, 19.79 ms). Both arms are asserted to produce the identical keep/delete verdict, which is what makes the 80× honest.

8.3 Three numbers that are BAD, and are therefore the most valuable rows here

1. secure_update_prepare = 10.84 MB/s, 1.35 bundles/s. This is the znippy pack path for a bundle, and it is two orders of magnitude below every other holger write path in this table. Compare secure_update_verify at 679.93 MB/s and _import at 536.87 — the same bundles, read back, run 50–63× faster than they are produced. On oden 2026-07-23 the same arm measured 8.40 MB/s, so this is not a loki artifact: prepare has always been the floor. It also ran at 10.5/12 cores busy while producing 10.84 MB/s, which is the signature of work being done serially inside a busy box rather than fanned out.
2. promote_ledger = 606 seals/s at 9.47 MB/s, on 3.11/12 cores. A single promotion is blake3 + blob write + exclusive ref + one redb ledger row, all single-writer by design. 9.47 MB/s is the ceiling on how fast artifacts can be sealed, and 3 of 12 cores were busy — nine cores idle while the hot path queues. This is the "no job queue, everything inline on the RPC" shape the coordinator described, measured.
3. pip_index_serve = 24.64 MB/s served, at 12/12 cores busy. The PEP 503 door saturates the box to deliver 24.64 MB/s, where the raw repo read (pip_wheel_mt32) does 1 917.69 MB/s on the same fixtures. A 78× gap between what the store can read and what the door can serve.

These three are the before-half of the stress story. None of them is a regression — they are the standing shape of the system, now written down with a commit and a box attached for the first time.

9. STRESS: the same arms at 100× the entries (full scale)

Second run, same commit, same box, same dev-dep edge, HOLGER_BENCH_SCALE=full (500 000 entries vs quick's 5 000) with HOLGER_BENCH_REAL=0 to keep the corpus synthetic and the run bounded. Started 09:19:51, ended 09:22:49 CEST, EXIT=0, 11 arms. Fixtures grew to ~730 MB under $HOME/holger-bench-data. Raw: bench-results/holger-bench-loki-2026-09-20-542e296-full.json.

What the scale run proves

1. The READ paths scale. The WRITE path does not. Every multicore read arm holds or improves at 100× the corpus — rust_crate_mt32 nearly doubles. But promote_ledger gets slower with more work while using 2.13 of 12 cores. Ten cores idle while the seal path queues. That is not a tuning problem, it is the absence of a job queue: blake3 + blob write + exclusive ref

   • ledger row all run inline and single-writer.

2. pip_index_serve is the door, not the store. Served throughput rose from 24.64 to 65.96 MB/s — but the same wheels read out of the store at 1 445 MB/s in the same run. The PEP 503 door delivers 4.6 % of what the store can read, at 10.15/12 cores busy. The bottleneck is the serving path, and it is the single largest headroom in the product.

3. pip_wheel_mt32 is the one read arm that regresses (−25 %) and it is the one pinned at 12.00/12 cores. Worth a look: it is the only read arm whose core occupancy went UP while throughput went DOWN, which is the signature of contention rather than saturation.

4. Note retention_planner/retention_executor/rust_crate_st ran at host_busy_max = 1 — the other lane's jobs had finished by then, so those three rows are from a QUIET loki and the rest are not. Per-row honesty, recorded rather than averaged away.

10. COMPETITOR VERSIONS — the comparison is half-stale, and one claim is FALSE

This was the owner's suspicion and it is correct. Verdicts:

Sonatype Nexus — 3.72.0 (2024-09-04) → current 3.96.2 (2026-09-18) → YES-BUT

• 24 feature lines / 63 published tags / ~24.5 months behind. 3.72.0 is Sunsetted; standard support ended 2025-09-04.
• The good news: the pin is on the RIGHT side of the big cut. 3.71.0 dropped OrientDB and required H2/PostgreSQL + Java 17. 3.72.0 already runs H2 on JDK 17, so we are not secretly beating a legacy OrientDB Nexus. That was the worst-case trap and we did not fall into it.
• The bad news: 3.72.0 predates Cargo entirely. Native Rust/Cargo support landed in 3.73.0 (2024-10-10) — one release after our pin. A 3.72.0 container cannot serve cargo at all. So any claim of the form "holger beats Nexus on Rust crates" is not supported by this harness; the honest framing is "Nexus on maven/pypi, 2024 vintage".
• Subtler config drift, the trap within the trap: the pinned image hard-codes -Xms2703m -Xmx2703m -XX:MaxDirectMemorySize=2703m. The current image sets no heap flags at all and lets JVM ergonomics take ~25 % of container RAM. Also UBI8/glibc → Alpine/musl, JDK 17 → JDK 21. And 3.96.0 added gzip'd npm JSON and abbreviated packuments (−89 % payload) plus more efficient Cargo sparse-index serving. A modern Nexus would serve materially less bytes for the same request. Our fair-heap arm detects the heap from the box, so on the current image it would be setting a heap the image no longer expects.
• Licensing shift worth knowing before publishing numbers: since 3.77.0 (Feb 2025) it is Community Edition with a click-through EULA and a 40 000 components / 100 000 requests per day cap — a benchmark can hit that cap. Our 3.72.0 pin is EPL OSS with no EULA, which is the legally cleanest tag.

JFrog Artifactory — 6.23.13 (2021-02-21) → current 7.161.26 → NO. Not honest.

• Five years and one whole major line stale. 6.x is long EOL; the 6.x line itself ended at 6.23.42 (2022-03-28), so the pin is 29 patches behind inside its own dead line.
• The stated reason for the pin is FALSIFIED. artifactory.rs says 7.x OSS "won't bootstrap" because of Access/router/keys microservices. In fact 7.x ships those microservices bundled inside the one image, and JFrog's own documented install is a single docker run -p 8081:8081 -p 8082:8082. Independent proof: PlaytikaOSS/testcontainers-spring-boot ships a live embedded-artifactory module pinned to releases-docker.jfrog.io/jfrog/artifactory-oss:7.161.26. The only real cost is a ~1–3 min cold boot and 2–4 GB RAM, which means a longer wait strategy, not a different major version.
• Format ceiling, in BOTH versions: Artifactory OSS serves only Maven/Gradle/Ivy/SBT + generic. No Cargo, no PyPI, no npm in any free JFrog edition. So Artifactory can only ever be a maven/generic data point against a Rust-first product, and the README should say so.

Action, ranked

1. Bump Artifactory to 7.161.26 and lengthen the wait strategy. This is the one unambiguous honesty fix, and the "can't containerize" blocker does not exist. MEASURED prerequisite: the image pulls clean (done on oden 2026-09-20).
2. Add a SECOND Nexus arm on 3.96.2, keep 3.72.0 as the historical series. Two rows, both labelled, is honest; silently swapping the tag would orphan the bench_runs history keyed by metric name. The new arm is also the only way to get a cargo-vs-cargo comparison at all, and it needs the EULA POST to /service/rest/v1/system/eula plus awareness of the daily request cap.
3. Add Gitea (gitea/gitea:1.27.3). It is the only genuinely light container that covers cargo + maven + pypi, boots in seconds on SQLite with INSTALL_LOCK, and is MIT. Best value per line of harness code.
4. Optional fourth: Forgejo (near-identical API, independent release train). Skip Archiva (retired to the Attic), Harbor (no cargo/maven/pypi, 9-container compose) and Pulp (nginx+postgres+redis bundle).

Say this out loud in any customer-facing use of the numbers: today's headline compares holger against a 2024 Nexus that cannot serve Rust and a 2021 EOL Artifactory that cannot serve Rust or Python. The wins are real on the formats tested, and the p999 tail story is the strongest of them — but the version gap cuts both ways and must be closed before the numbers are used to sell.

11. The boxes, named — because the silicon is part of the measurement

Thermal discipline on oden. Fans are held at 100 % through the measurements (owner ruling) and turned down to 70 % only afterwards. I do not change fan settings. Every oden run carries a thermal trace sampled every 5 s for the whole run — Tctl, Tccd1, mean/min/max achieved core clock across all 32 cores from /proc/cpuinfo, load1 and runnable procs — committed beside the result as *.thermal.csv.

Why the trace is not optional: a throttled CPU makes holger look slow and a benchmark cannot tell the difference. A 3975WX under sustained all-core load can fall well below boost for reasons that have nothing to do with this code. A run whose trace shows a clock cliff must be labelled or re-run, never quietly reported. Idle reference before the run: Tctl 58.9 °C, CPU_FAN 2800–2900 RPM, NVMe ~34 °C.

12. MEASURED while dusting off: Artifactory 7.x on oden

Trying to verify the falsified "7.x won't bootstrap" claim by actually booting releases-docker.jfrog.io/jfrog/artifactory-oss:7.161.26 in ONE container:

• The image pulls clean (as does sonatype/nexus3:3.96.2). All four images — both pins and both currents — are now on oden.
• The container exits 1 in under 10 s, and not for the reason the source comment gives. There is no microservices failure, no router, no Access complaint. The log says: /opt/jfrog/artifactory/var is running with 2% free storage. Free up space or increase volume size and try again. Exiting
• That is a percentage-based free-space guard, and it is reading oden's podman graphroot — /home/rickard/.local/share/containers/storage on /home, which is 98 % full (168 GB free). 168 GB is plenty of bytes; the guard counts percent. Mounting the data volume elsewhere does not help, because the container's own root layer still lives on the 98 % filesystem.

Conclusion, and it matters for the harness: the blocker on 7.x is an oden housekeeping problem, not a containerization problem. To run the modern Artifactory arm, either free /home below ~95 % or move podman's graphroot to / (526 GB free, 72 %). Written down so the next person does not re-diagnose it as "7.x can't containerize" — which is exactly how the stale 6.23.13 pin got its justification in the first place.

13. Handover — what is done, what is next

Done and pushed (branch stinky/bench, remote origin):

1. The bench driver compiles again (10 sites, canonical constructor).
2. Baseline captured at 542e296 on loki, quick scale, 20/20 arms, both rivals answering — committed with commit + machine + nproc + profile + feature set + dependency kind.
3. Scale/stress run at 100× the corpus, 11 arms.
4. Competitor pins audited against current; one stated justification falsified.
5. Raw BenchRun envelopes committed under bench-results/ so no number in this document is unsourced.

Next, in value order:

1. Bump Artifactory to 7.161.26 (free oden's /home first) and add a Nexus 3.96.2 arm beside the 3.72.0 one. This is the honesty fix, and the 3.96.2 arm is the only route to a cargo-vs-cargo number.
2. Add Gitea — cheapest competitor per line of harness code, and the only light one covering cargo + maven + pypi.
3. Gate on cargo build -p xtask --example nornir-bench. One link, and §6 would never have happened.
4. Delete bench_nexus_container.sh; rehome holger-ds out of the shipped CLI into a bench-scenarios example.
5. Strip auth-oidc/auth-oauth2/auth-mtls from the bench crate's server-lib pin and default-features = false its reqwest — but name whatever the server actually wants in server/cli in the same commit, or three auth doors vanish the way the passkey door nearly did.
6. The stress gaps this lane did NOT close, and that no existing arm covers: concurrent uploads (all arms are read-heavy), a repo of many thousands of artifacts under sustained mixed read/write, and the edge behaviours — disk filling, a slow client, a client that disappears mid-upload. The promote_ledger and pip_index_serve rows above say where the pain will be.

14. THE AUTHORITATIVE RUN — oden, 32 cores, quiet box, thermally traced

This is the row to quote. Same commit, same quick scale, same dev-dep edge, 20/20 arms, EXIT=0, 07:28:22 → 07:32:59 UTC (4 min 37 s).

Box: oden · AMD Ryzen Threadripper PRO 3975WX, 32 cores · ASUS Pro WS WRX80E-SAGE SE WIFI · 499 GB RAM · / 526 GB free (72 %) · rustc 1.98.1. Thermal trace: 56 samples @ 5 s = full run coverage. Tctl avg 56.0 °C, min 49.4, max 64.1 · all-core clock avg 2972 MHz, min 2247 (idle between arms), max 4167 under load. No throttling. Max Tctl 64.1 °C is ~30 °C below the 3975WX limit, and the all-core clock held 4167 MHz through the loaded arms. These numbers are about holger, not about the cooling. Fans were at 100 % throughout per the owner's ruling; I did not change them. Raw: bench-results/holger-bench-oden-2026-09-20-542e296.json + .thermal.csv + .env.txt.

14.1 The head-to-heads on a quiet 32-core box

Publish phase: nexus_upload_mbs 1.178, artifactory_upload_mbs 1.266. Internal ceiling: 3 481 / 3 929 MB/s — the HTTP door costs ~33–40× the in-process read, the same shape loki showed.

Fairness is real and it is generous. FairResources::detect() gave Nexus a 191 GB JVM heap on oden (~50 % of container memory) and every core. Nobody can say the JVM was starved. holger still wins p999 by 107.8×, and that gap is GC, not bandwidth.

These oden numbers independently reproduce the README. The static table in README.md / .nornir/README-full.md says oden · 32 cores · 2026-07-23: holger 108.43 served MB/s vs Nexus 10.60, and 100.57 vs Artifactory 12.24. Today, two months and many commits later, on the same box: 105.33 vs 9.999 and 97.55 vs 12.647. Within a few percent on every cell. The published claims are sound — they were merely unreproducible until §6 was fixed.

14.2 Every other arm on oden (quick scale)

14.3 THE FINDING — the write path does not merely fail to scale, it goes BACKWARDS

Three boxes/scales, one arm:

Going from 12 cores to 32 made sealing 42 % SLOWER. It used one core on both boxes. The same run's read arms went the other way on the same hardware (pip_wheel_mt32 1 918 → 4 224 MB/s, +120 %; java_jar_mt32 1 678 → 3 972, +137 %). So the machine is fine and the reads scale beautifully — the seal path is strictly serial and is bounded by single-core speed and fsync latency, and loki's Ryzen 9 7900 simply has a faster single core than the 3975WX.

That is the clearest possible statement of the "no job queue, everything inline on the RPC" defect: adding 20 cores to the machine made the product slower at its write path. It is the single highest-value target in the system, and this table is the before-half.

Same shape, weaker: secure_update_prepare 10.84 (loki) → 7.69 MB/s (oden) while pinning 32/32 cores — busy cores, no throughput.

14.4 Cross-box summary for the honest reader

Read paths scale with cores. Every write path is single-core bound and therefore tracks single-core clock, not core count. A customer who buys a bigger box gets faster reads and slower writes.

15. THE RIVAL REGISTER — pinned tag, its age, the current tag, and what the DEFAULT changed

Extends §10, which established that the comparison was half-stale and one justification was false. This section is the register the owner asked for: for every rival, the pinned tag, its release date, the current tag, and whether the default configuration changed between them. A pinned tag whose defaults have since changed is the subtle version of the same trap — the number is measured against software that no longer exists in that shape.

Release dates below are MEASURED from the image metadata on oden (podman inspect --format {{.Created}}), not read off a changelog.

Both current-tag arms are running and measured — see §17. Getting there required two live boots to discover two defaults that no changelog reading would have caught, and each of them would otherwise have produced a false number.

15.1 Did the DEFAULT CONFIGURATION change? Yes, in both, and it is measurable

Nexus — the heap default is the trap. Read straight off the two images:

3.72.0  INSTALL4J_ADD_VM_PARAMS=-Xms2703m -Xmx2703m -XX:MaxDirectMemorySize=2703m …
3.96.2  INSTALL4J_ADD_VM_PARAMS=-Djava.util.prefs.userRoot=/nexus-data/javaprefs

The pinned image hard-codes a 2703 MB heap. The current image sets no heap flags at all and lets JVM ergonomics take its share of container RAM. Those are two different pieces of software under load, and a bench that pinned the old tag would have been measuring a fixed small heap while believing it measured "Nexus".

Our harness is not exposed to that particular error, and this is worth stating plainly because it is the one place the existing design was already right: FairResources::detect() overrides INSTALL4J_ADD_VM_PARAMS on both tags with a heap sized from the host box. On the authoritative oden run that was a 191 721 MB heap. So the heap default drift is neutralised by construction.

Nexus — anonymous read is the second, and it was found only by booting. A fresh 3.96.2 says so in its own log:

AnonymousManagerImpl - Using default configuration:
  AnonymousConfigurationData{enabled=false, userId='anonymous', realmName='NexusAuthorizingRealm'}

3.72.0 allows anonymous reads. 3.96.2 Community Edition does not. The first run of the current-tag arm therefore served 0 of 91 757 GETs, and the only reason that is not now sitting in a slide as a ~0 MB/s "result" is the empty-rival-cell guard added in the same commit. The harness enables anonymous read over REST rather than authenticating the Nexus leg, so the transport stays identical to the pinned arm and the two rows differ by the software under test and by nothing else.

Artifactory — the database requirement is the whole story. 6.23.13 runs on the bundled Derby. 7.161.26 refuses to:

DbTypeNotAllowedException: DB Type derby is not allowed:
Cannot start the application with a database other than PostgreSQL.

So 7.x is a two-container arm: the harness now boots a postgres:16-alpine companion on a per-run network and points JF_SHARED_DATABASE_* at it. After that change the image logs Configured database type: POSTGRES, answers an authenticated ping in 47.5 s, and served 40 326 successful GETs.

This corrects two claims at once — the old source comment's ("7.x's microservices crash-loop, it cannot be containerised" — false) and this lane's own first assumption ("free the disk and it is a drop-in tag bump" — also false). See §15.5 for the full sequence, because the order in which those blockers appeared is the interesting part.

What is NOT neutralised, and travels with any number from the pinned tag:

Artifactory — the restricted-mode flag is new. 7.161.26 ships JF_SHARED_RESTRICTEDMODE_ENABLED=true in the image environment; 6.23.13 has no such variable and instead carries POSTGRESQL_VERSION=9.4.1212. Different product shape, not a patch difference.

15.2 THE FORMAT CEILING — say this wherever the numbers are written down

Two sentences that change what every Artifactory and Nexus number in this document means:

1. Artifactory OSS serves Maven/Gradle/Ivy/SBT and generic repositories, and nothing else. No Cargo, no PyPI, no npm — in ANY free JFrog edition, 6.x or 7.x. Upgrading the tag does not fix this and never will. Every Artifactory row in this suite is a maven/generic data point measured against a Rust-first product.
2. Nexus 3.72.0 cannot serve Cargo at all. Native Rust/Cargo support landed in 3.73.0 (2024-10-10), one release after the pin. The 3.96.2 arm added in this lane is the only route in the harness to a cargo-versus-cargo comparison.

So the honest label for the pre-existing head-to-heads is: holger versus a 2024 Nexus that cannot serve Rust, and a 2021 EOL Artifactory that cannot serve Rust or Python, on raw/generic repositories. The wins are real on what was tested and the p999 tail story is the strongest of them. The version and format gaps cut both ways and must be stated before the numbers are used to sell.

15.3 Two rows, never one — why the old tags stay

bench_runs keys its regression history by metric name. Swapping the tag under nexus_load would have left the same series silently describing different software — the exact failure mode FACETT.md:191 records, where a hard-coded machine = "Loki" string ran on oden for months.

So the harness now carries four rival arms instead of two:

Both halves of each pair run through identical code (nexus_arm / artifactory_arm in xtask/examples/nornir-bench.rs, and nexus_load_tag / artifactory_load_tag in the scenario modules). A comparison whose two halves went through different code is not a comparison. Every row now carries nexus_tag / artifactory_tag on the row itself.

15.4 What the code had to learn to run the modern tags

15.5 The Artifactory 7.x blocker, in the order it actually appeared

Written out because the sequence is the lesson: each layer looked like the whole answer until it was removed.

Three layers, and the top one was fiction. The general lesson is the one this whole lane keeps re-learning: a justification that was never re-measured is not a justification. Steps 1 and 2 were each invisible until the one above was removed, which is precisely why "we tried it once and it didn't work" decays into folklore.

16. THE STRESS ARMS NOBODY HAD — the write side, measured

§13 listed three gaps no arm covered: concurrent uploads (every arm was read-heavy), a repository of many thousands of artifacts under sustained mixed read/write, and the edges — the disk filling, a slow client, a client that disappears mid-upload. This section closes them.

New module: bench-scenarios/src/stress.rs, behind a new stress cargo feature (stress = ["served", "dep:tempfile"]), reached through the same [dev-dependencies] edge of xtask as every other measuring arm. Three new bench ids: holger.concurrent_upload, holger.mixed_read_write, holger.upload_edges.

16.1 What door these arms drive, and why that matters

All three go over the wire through a real Holger HTTP gateway (served::start_holger_gateway) standing over a writable znippy repository, using PUT /{repo}/api/v1/crates/new — the genuine cargo publish endpoint, with cargo's own body framing ([u32 json_len][json][u32 crate_len][crate]). Nothing is measured in-process. That path is:

PUT /{repo}/api/v1/crates/new
  → RustRepoZnippy::put
  → self.write_lock.lock()                      ← one process-wide mutex
  → znippy_common::meta_sink_append::append_files ← read-modify-write of the archive
  → invalidate_index_cache()

Two properties of that path are what these arms exist to quantify: the mutex serialises every publisher, and the append re-seals the whole archive, so the cost of one upload grows with how much is already stored. Neither had a number attached before today.

16.2 Why concurrent_upload has three phases and not two

A serial-then-concurrent measurement cannot distinguish a serialising lock from an append that got dearer as the archive grew: both make the second phase look bad. So the arm runs serial → concurrent → serial, and reports two separate ratios:

• scaling_x = concurrent rate ÷ the mean of the two serial phases — did more clients buy anything?
• append_cost_growth_x = first serial rate ÷ last serial rate — did storing more make each upload dearer, on its own?

A shared atomic counter hands every upload in every phase a unique crate name. Without it the later phases would "measure" RustRepoZnippy::put's idempotent re-publish no-op, which is not an upload.

16.3 Honesty notes that must travel with these numbers

• There is no rival column, and there should not be one. Nexus and Artifactory are not driven by these arms. This is holger measured against holger, on holger's own write door.
• The disk-filling edge is an ANALOGUE, not a true ENOSPC. It makes the store un-writable via permissions rather than filling a real filesystem, because a genuine ENOSPC needs a bounded mount, which needs root, on a shared bench box. The row carries enospc_is_analogue = 1 so the limitation cannot be lost in a later quotation. What the analogue does answer is the part that matters: does a failed write get a clean refusal, and does the archive survive it. A true ENOSPC arm is an owner item — see §18.
• A bad number in this section is a good result. It is the before-half of a before/after while another lane changes the write path.

17. MEASURED — four rivals and three stress arms, quiet oden, 2026-09-20

17.0 Provenance

Box quietness, recorded rather than assumed. The owner had dwarves-server stopped for these runs, and it stays stopped. Immediately before the measured set: load average 2.16, top process valkey-server at 1.0 %, Tctl 42.6 °C, all-core clock 2809 MHz idle.

Thermal trace: bench-results/holger-r4-oden-2026-09-20.thermal.csv, sampled every 5 s straight from /sys/class/hwmon/hwmon5 (Tctl, Tccd1) plus the mean of all 32 cpu MHz lines and load1. Fans were at 100 % throughout, per the owner's standing instruction; I did not change them.

One set of numbers was taken while the box was NOT quiet and is kept separately, not averaged away. The first stress run happened while dwarves-server was still using ~287 % CPU. Those rows are preserved under bench-results/contended/ and are quoted in §17.3 beside the quiet ones, because the comparison between them is itself informative: the shape of every finding survived the contention, only the magnitudes moved.

17.1 The four rival arms — same box, same window, ~10 minutes apart

Both tags of each rival ran through identical code. Every row carries its tag.

holger's lead, per rival:

Publish phase (separate, never the head-to-head): nexus_upload_mbs 1.199 (3.72.0) / 1.031 (3.96.2); artifactory_upload_mbs 1.381 (6.23.13) / 0.218 (7.161.26). Boot to an authenticated ping: Artifactory 17.1 s on 6.x, 47.5 s on 7.x.

These reproduce the published claims. README.md says oden · 32 cores · 2026-07-23: holger 108.43 vs Nexus 10.60, and 100.57 vs Artifactory 12.24. Today, against the same pinned tags: 102.07 vs 10.580 and 98.38 vs 12.568. Within a few percent on every cell, two months and many commits later.

17.2 What the modern rivals change, and it is not all in holger's favour

Two findings, and the second one costs us.

1. The modern Nexus serves less, not more. Nexus 3.96.2 delivered 4.220 MB/s where 3.72.0 delivered 10.580 — the current Community Edition is 2.5× slower on this workload than the 2024 tag, on the same box in the same window. holger's MB/s lead therefore grows from 9.65× to 24.6×. Quoting that larger number would be technically true and substantively misleading, so the honest headline against Nexus remains the p999, which barely moved (105.9× → 99.8×).

2. The modern Artifactory's tail is twice as good, and that halves our best number. Artifactory 7.161.26's p999 is 30.525 ms against 6.23.13's 54.737 ms, and its p99 improved from 6.144 to 4.288 ms at the same throughput. holger's p999 lead over Artifactory consequently falls from 80.2× to 44.5×. 44.5× is the number to quote. The five-year-old pin was flattering us by roughly a factor of two on the single metric the pitch leans on hardest.

44.5× is still a very large lead and it is now defensible. The old one was not.

17.3 The stress arms — the write path, measured for the first time

holger.concurrent_upload — more publishers make it slower.

• scaling_x = 0.467. Thirty-two clients publishing at once achieved 47 % of what a single client achieves. Not 32×, not 1× — less than one.
• append_cost_growth_x = 2.694. A single publisher got 2.7× slower between the first and last phase, purely because the archive had grown to 3 888 files. That is the read-modify-write append cost, isolated by the third phase.
• 1 944 uploads succeeded, 0 failed; p99 1 496 ms, p999 1 581 ms; 1.89 MB/s, at 1.4 of 32 cores busy. Thirty-two clients, one and a half cores, and a p99 of a second and a half.

The cause is not mysterious and it is in the source: RustRepoZnippy::put takes one process-wide write_lock and then calls append_files, which re-seals the whole archive. Publishers serialise on the mutex, and each one's work grows with what is already stored.

holger.mixed_read_write — four publishers cost the readers 94 % of their throughput. Corpus: 5 000 crates, served over the real cargo download path.

• read_throughput_retained_pct = 5.97 %.
• read_p999_inflation_x = 254.1.
• read_failed_mixed = 3 650. Three and a half thousand reads did not return 2xx while publishing was in progress. That is not a latency number, it is reads failing, and it outranks everything else in this section.
• The publishers themselves managed only 11.26 uploads/s (45 in the window, 0 failed) — against a 5 000-artifact archive, versus 377.99/s against an empty one.

This is the single most important measurement this lane produced. holger's entire pitch is the read tail — p999 0.68 ms against Artifactory's 30.5 ms. A single CI job publishing four crates takes that same p999 to 5 054 ms.

holger.upload_edges — all three edges are clean. This is the good news, and it is measured, not assumed.

So holger handles the pathological single client well. What it does not handle is the ordinary concurrent one.

Quiet versus contended — the shape survived. The first stress set ran while dwarves-server held ~287 % CPU. Kept at bench-results/contended/:

Every conclusion holds in both. The quiet set is the one to quote.

18. WHAT NEEDS THE OWNER

Five items. None of them can be settled by measuring harder.

1. A true ENOSPC arm needs a bounded filesystem, which needs root. holger.upload_edges currently measures a write-refusal analogue (permissions), and says so on the row (enospc_is_analogue = 1). A real disk-filling arm needs a small loopback or tmpfs mount that the bench may fill and destroy. That is a mount on a shared bench box, so it is an owner decision, not something a benchmark should do on its own initiative.

2. Publishing the rival numbers. Nexus 3.96.2 is Community Edition under a click-through EULA which the harness now accepts programmatically, and CE carries a 40 000 components / 100 000 requests per day cap. The pinned 3.72.0 is EPL OSS with no EULA and is the legally cleanest tag to quote. Before any of these numbers goes in front of a customer, someone who is not a benchmark has to decide what may be published about a EULA'd product.

3. Gitea as a fourth rival — go or no-go. docker.io/gitea/gitea:1.27.3 is pulled and on oden, ready. It is the only light container covering cargo + maven + pypi at once, boots in seconds on SQLite with INSTALL_LOCK, and is MIT — the best value per line of harness code of any candidate. It is not implemented here: it needs its own admin bootstrap, token mint and package-upload path, which is a scenario module of its own rather than a tag parameter. Recommended, but it is new work, not a fix.

4. Which arms belong in the release gate now that there are seven more. The four rival arms plus three stress arms roughly double the runtime of a full pass, and the Artifactory 7.x boot alone costs minutes. §5's ranked keep-list predates all of them.

5. The write-path defect the numbers point at is one lane's work, not this one's. promote_ledger going 606 → 349 seals/s from 12 to 32 cores, and the upload serialisation measured in §17, both come from the same shape: the write path runs inline and single-writer. This lane measured it and left it alone.

19. WHERE THESE NUMBERS LIVE — and a warehouse defect found on the way

The rule is that bench_runs is the source of truth and a run.json is a receipt. This lane put its rows in the table with nornir bench run holger on oden, and doing so surfaced two defects in that path. Both are recorded here because either one silently produces a number nobody can find again.

19.1 What is in the warehouse

All seven arms — four rivals and three stress arms — now answer to nornir bench history-show holger. The verification matters and is the whole point of §19.3: 2e2c2570 and 75a24cd5 are the only runs confirmed by reading the rows back out of the table, rather than by trusting the CLI's success line.

Every arm is also committed as a compact BenchRun receipt under bench-results/holger-r4-<arm>-oden-2026-09-20-e6238c1.json, with the thermal trace and the box stamp beside them. The receipts are the quiet-box measurements and are the ones §17 quotes.

19.2 DEFECT 1 — nornir bench run cannot reach a lane worktree

On oden, with no nornir.toml present, nornir bench run holger resolves the repo to /home/rickard/git/holger — the pre-existing checkout — whatever directory it is invoked from. A lane working on a branch in a worktree therefore cannot persist its own arms: the driver it builds is the one from the other checkout, so brand-new arms are simply not linked and quietly do not appear. It is visible only as an arm count: running 2 of 229 linked arms where the lane's own binary reports 234.

Two things about the workaround are worth knowing:

• [repo.holger] path = "holger" (relative to the workspace root) works.
• [repo.holger] path = "/absolute/path" is ignored — resolution falls back to <workspace_root>/holger, despite nornir-config's own doc comment saying an absolute path is honoured. Passing that config via --config <file> also re-derives a different workspace root. So the worktree must physically sit at <workspace_root>/<repo>, which in turn means its sibling path dependencies (edda, znippy, znippy-zoomies, modgunn, facett, …) must be reachable from there too, or cargo metadata fails before the bench starts.

19.3 DEFECT 2 — adding a nornir.toml silently moves the warehouse

This is the more dangerous one, and it is the same shape as the "16 of 46 receipts with no warehouse row" incident.

A nornir.toml containing only [repo.holger] and no [storage] section makes nornir bench run write to ~/.nornir/warehouse/warehouse/nornir/bench_runs, while nornir bench history-show in the same directory, with the same config, keeps reading <workspace_root>/.nornir/warehouse/nornir/bench_runs.

Two different warehouses, one command family, one invocation directory. And the run still printed its success line:

✓ 5 result(s), 5 test(s) persisted into bench_runs as 87d15865-…

That line is true — the rows exist, in an Iceberg table, that nothing reads. history-show answered holger — 2 run(s) and listed two blake3/memcopy rows belonging to a different lane. A success message is not proof of persistence; only reading the row back is. The repair is to name storage explicitly:

[storage]
kind = "local"
local_path = ".nornir"

19.4 Contention, stated

The quiet-box receipts in §17 were taken at load average 2.16 with dwarves-server stopped. The warehouse re-run happened after the machine policy changed to "oden builds", with other lanes compiling on the box (load 9–22). The warehouse rows therefore carry higher contention than the receipts, and they say so themselves: every row stamps telem_cores_busy_avg, telem_cores_busy_max and telem_cpu_pct_avg, which is precisely the mechanism that makes a contended row recognisable instead of silently wrong. Quote §17 for the numbers; use the warehouse rows for series continuity.

19.5 Scaffolding that was created and removed

To make the persist run possible, the following were created on oden and then reverted: a git worktree at ~/git/workspace_nordisk/holger (the pre-existing stray directory holding PLAN-2026-09-17-two-vm.md was moved aside and restored), symlinks to the sibling repos beside it, a temporary rename of the facett file there, and ~/git/workspace_nordisk/nornir.toml. None of it is left behind. It is listed so that the next person knows what a persist run from a worktree costs today — and §18 carries the ask to make it cost nothing.