nordisk/holger

holger-ds — the real Python journey — design

Class: design · Function: holger-ds (uses serve for the native HTTP door, registry for PipRepoZnippy, storage for znippy extract, the agent world connector for ingest). The whole-system view is design.md; the serving door is serve-design.md.

What holger-ds is

holger-ds is Holger standing in as a real Python package index for a data-science stack: a uv/pip install and an Airflow DAG pull wheels from Holger, end to end, over the real package protocol — never in-process, never by shelling to uv/pip. It is additive: it reuses the existing PipRepoZnippy backend, the start_http_gateway serve path, the agent world PyPI ingest, and the served-bench foundation (bench-scenarios/src/served.rs). No existing registry path changes; the crate/jar/wheel arms and the just-landed consolidation refactor are untouched.

The journey has three legs, all over one immutable znippy archive:

1. Ingest — the agent world connector (agent/connector/world/src/pip.rs) parses a requirements.txt or a uv.lock, resolves files+digests against the PyPI JSON API, and lays each distribution at packages/{normalized-name}/{file} inside a .znippy archive.
2. Serve — PipRepoZnippy exposes that archive as a PEP 503 HTML simple index and a PEP 691 JSON index through Holger's native HTTP door (handle_http2_request), plus the wheel-download path.
3. Install — a real consumer (uv/pip, or the pure-Rust journey client in tests) fetches the index, resolves a wheel, downloads it, and verifies the SHA-256 it recomputes against the digest the index advertised.

The serving surface (chosen design)

PipRepoZnippy::handle_http2_request routes four shapes after stripping the /{repo} prefix:

Content-addressing: SHA-256 on the PEP-facing surface

znippy content-addresses every stored file with blake3, but pip/uv do not accept blake3 — they verify a download against a SHA-256 (or sha384/512) named in the #sha256= link fragment (PEP 503) or the hashes map (PEP 691). So the PEP-facing digest is a real SHA-256 of the served file bytes, computed once per file and memoised (PipRepoZnippy::sha_cache). This is the digest a real uv install recomputes and aborts on mismatch — the "wheel content-addressing" guarantee.

The SHA-256 is computed only on the index-detail path (/simple/{name}/[json]), never on the artifact-download hot path (/packages/…), which stays byte-identical to before — so the benched serve hot path (artifactory_load/nexus_load) does not regress.

PEP 691 negotiation — the one caveat

uv/pip select PEP 691 JSON via an Accept: application/vnd.pypi.simple.v1+json header. The RepositoryBackendTrait::handle_http2_request seam does not surface request headers today, so a header-driven content negotiation would need a cross-backend trait change (all ~13 znippy-* backends). To stay additive, the JSON representation is reachable at the explicit /simple/{name}/json route — the format is fully implemented and tested, and a real uv install still works over the valid PEP 503 HTML path (now carrying hashes). Wiring true Accept-header negotiation is the documented follow-up (see the problem log).

Reachable as a HEAVY bench

holger.pip_index_serve (registered in xtask/examples/nornir-bench.rs, picked up by nornir bench run holger) is the Python sibling of artifactory_load/nexus_load:

• builds a synthetic wheel corpus (pip_corpus, byte-budgeted on heavy tiers),
• stands up the real Holger HTTP gateway over PipRepoZnippy,
• drives the pure-Rust gatling GET pool (served::run_pip_index_fetch) over the wheel-download path (headline holger_served_mbs + ops_per_sec) and the /simple/{name}/ resolve path (holger_index_ops_per_sec),
• keeps the in-process get_file ceiling as an internal Holger-vs-Holger reference (holger_inprocess_read_ceiling_mbs), never beside a competitor,
• records host cores + memory like every heavy arm.

Why this design

• Reuse over duplication. No new server, no new protocol code paths, no second wheel builder — the same make_fake_wheel corpus feeds both the in-process pip_wheel_* arms and the served pip_index_serve arm, and the same start_holger_gateway/bombard_http_get foundation the crate/JVM arms use.
• Honest measurement. Delivery is measured over the package protocol (HTTP), exactly as the Nexus/Artifactory head-to-heads are; the in-process figure is kept separate as a ceiling.
• Real verification. The journey test recomputes SHA-256 and rejects a mismatch — it exercises the exact failure a real uv install guards against.