nordisk/holger

holger-ds — the Python journey — guide

Class: guide · Task-oriented companion to holger-ds-design.md.

holger-ds turns Holger into a Python package index a uv/pip install and an Airflow DAG pull wheels from. This guide shows how to drive each leg.

1. Ingest a Python stack into a znippy archive

The agent world connector reads a requirements.txt (pinned name==version) or a uv.lock and resolves each distribution's files + SHA-256 from the PyPI JSON API:

use agent_lib::{parse_requirements_txt, resolve_with_index, PypiJsonClient, pip_packages_to_assets};

let pkgs = parse_requirements_txt("requirements.txt")?;      // or parse_uv_lockfile("uv.lock")
let pkgs = resolve_with_index(pkgs, &PypiJsonClient::new()).await?;
let assets = pip_packages_to_assets(&pkgs);                  // → packages/{name}/{file}
// hand `assets` to transfer_to_znippy → an immutable .znippy archive

2. Serve it as a PEP 503 / PEP 691 index

Point a repository at the archive and wire it into the gateway; PipRepoZnippy answers the simple-index and download routes:

GET  /{repo}/simple/                     → root index (all package names)
GET  /{repo}/simple/{name}/              → HTML detail, links carry #sha256=<hex>
GET  /{repo}/simple/{name}/json          → PEP 691 JSON (application/vnd.pypi.simple.v1+json)
GET  /{repo}/packages/{name}/{file}      → the wheel/sdist bytes

A stock uv/pip can install from the HTML index directly:

uv pip install --index-url http://<holger-host>/pip/simple/ pandas

(uv/pip negotiate PEP 691 JSON via an Accept header; until the backend seam surfaces headers, the JSON view is at the explicit /simple/{name}/json route — the HTML PEP 503 path, now carrying hashes, is what a real install uses.)

3. Run the journey the way a consumer does

The pure-Rust journey (no shelling to uv/pip) — see server/repository/znippy-python/tests/uv_airflow_journey.rs:

1. GET /pip/simple/{name}/ → parse the wheel link + its #sha256= fragment.
2. GET the wheel → recompute SHA-256 → assert it equals the advertised digest (a mismatch is what a real uv install aborts on).
3. Airflow DAG: extract → transform → load; each task hydrates its Python env by pulling+verifying its wheels from Holger, in topological order.

4. The HEAVY bench

nornir bench run holger            # picks up holger.pip_index_serve

or the fast dev-box probe:

cargo run -p holger-bench-scenarios --example pip_probe --features served
# env: PIP_PROBE_MB (corpus MiB), PIP_PROBE_SECS

Metrics recorded (host cores + memory captured by the harness):

Tests

cargo test -p holger-znippy-python-repository        # format + content-addressing + journey
cargo test -p holger-bench-scenarios --features served served::   # HTTP-served smokes