.nornir/end-to-end-conformance-testing.md · nordisk/holger

End-to-end conformance testing — real client ⇄ real holger (containers)

Status: DESIGN, for review. The authoritative way to test that a znippy package actually serves: a real package-manager client pulling from a real holger, over the real wire protocol. mannequin is the mocked world, not the thing under test.

The shape

┌─────────────────────────┐         ┌──────────────────────────────────┐
│ client container        │  wire   │ holger container                 │
│ "pretending to be a dev"│ ──────▶ │  holger serve (real binary)      │
│  real cargo / npm / pip │ proto   │  serves from a sealed .znippy     │
│  / gem / conda          │         │  [+ mannequin inside, if needed: │
│  registry → holger      │         │   OIDC / PKI / upstream world]   │
└─────────────────────────┘         └──────────────────────────────────┘

Client container runs the real tool (e.g. rust:slim → cargo fetch, node → npm install, python → pip install, ruby → gem install, conda), with its registry/index pointed at the holger container. The test asserts the install/resolve succeeds and yields the right package/version/bytes. This exercises the full stack: .znippy → holger backend → typed view → wire protocol → the real client's parser — exactly where the view fixes matter (npm scope, maven -SNAPSHOT/classifier, conda build string).
holger container runs the real holger serve over a sealed .znippy fixture. mannequin runs inside/alongside it only when the scenario needs the outside world (OIDC token issuer, PKI CA, an upstream to proxy). For a pure airgap read test, no mannequin is needed — holger just serves the sealed archive.
Why not mannequin for this: mannequin mocks upstreams/IdP/Nexus. Package serving conformance is about holger's own wire output vs a real client, so the client and holger must be real. mannequin stays the mocked world (and the human-dev "fire it up while testing" workflow stays valid).

Who drives it

Runner	Question	Cell
test matrix (`xtask/tests/holger_matrix.rs`)	correctness — does `npm install` (etc.) actually work against holger?	heavy `#[ignore]` cell per ecosystem, in the `--ignored` pass
nornir bencher (`xtask/examples/nornir-bench.rs`)	throughput — how fast does holger serve real clients at concurrency?	already containerised vs Nexus; extend to real-client pulls

Both already use testcontainers — the Nexus comparison bench fires a real Sonatype Nexus 3 (bench-scenarios/src/nexus.rs). Same machinery spins the holger + client containers. Fixture-gated: when no Docker/Podman, record SKIP (is_skip pattern), never fail off-CI. Heavy-on in the release gate.

Layering (don't put everything in containers)

Light pass — in-process, no Docker (every run): typed-view round-trips (plugin_read_api_test), auth logic via mannequin in-process (mTLS verify, offline-OIDC JWT verify, RBAC role_for), the CPU multicore sanity cell.
Heavy pass — containers, gated (release gate): real-client ⇄ real-holger conformance per ecosystem; real Keycloak (online OIDC) / real Nexus (proxy/import parity).
No-container gate: cargo tree -p static-holger asserts no egress crate linked (airgap-provable).

Per-ecosystem client cells (the matrix grid to grow)

ecosystem	client image	command	assert
rust	`rust:slim`	`cargo fetch` w/ `source.crates-io.replace-with` → holger	resolves + downloads the crate
npm	`node:slim`	`npm install <pkg>` w/ registry → holger	installs scoped + unscoped
python	`python:slim`	`pip install --index-url holger <pkg>`	wheel + sdist
maven	`maven`	`mvn dependency:get` → holger	GAV incl. `-SNAPSHOT` / classifier
gem	`ruby:slim`	`gem install --source holger <gem>`	compact-index resolves
conda	`continuumio/miniconda3`	`conda install -c holger <pkg>`	repodata.json resolves

Each starts as one heavy gated cell; SKIP without a container runtime.

Open

Whether to bake holger+mannequin into one image or compose two; one image is simplest for the matrix (holger serve + optional mannequin sidecar process).
Fixture provenance: build the sealed .znippy in the light pass, mount it into the holger container for the heavy pass (no rebuild).