nordisk/holger

Field test: Holger on-prem, end to end (task 2)

Run 2026-11-01 by Henrik, from the crates.io 3.0.0 releases (published 2026-10-31), on two fresh VMs on PVE:

Both: Debian 13 genericcloud, 4 vCPU, 10 GB, 60 GB disk, Rust 1.99.1, rootless podman 5.4.2, Python 3.13.5. The guide followed is README.md "Three products by hand" (main f9257ddd), split over the two hosts. $G=~/holger-guide; tokens ida (Importer), bob and eve (SecurityApprover), adm (RepositoryAdmin), aud (Auditor), lab (the downstream).

The result per step

Findings

Ordered by how much they block a first-time operator.

F1. 3.0.0 wants worker image :0.1.0, the guide builds :3.0.0 (blocker, fixed on main)

The guide builds localhost/holger-scan-worker:3.0.0. The published holger-sandbox 3.0.0 hard-codes DEFAULT_IMAGE = "localhost/holger-scan-worker:0.1.0" (src/lib.rs:56 in the crate), so the guide's first scan fails:

$ holger intel build --feeds-dir feeds --policy pki/intel-trust.ron --release-cert pki/releaser.der --release-key pki/releaser.pk8 \
    --approve "security-reviewer=pki/reviewer.der,pki/reviewer.pk8" --out stick/intel.holger.znippy
Error: stick/intel.holger.znippy does not verify: pki/intel-trust.ron: [holger-sandbox-20] job e2b33f61e75ea0bf0407b9c09c66213c did not run: image: [holger-sandbox-8] the scan worker image localhost/holger-scan-worker:0.1.0 is not on this host (it is minted and carried, never pulled)

Workaround: podman tag localhost/holger-scan-worker:3.0.0 localhost/holger-scan-worker:0.1.0. Main fixed it on 2026-11-01 in 6048bf6e (the tag is now the server version). The fix needs a release (3.0.1). Until then, a cargo install user cannot follow the guide.

F2. PyPI/OCI pull-through needs online AND jobs; the docs say online

docs/v2/02-install.md:40 says the online extra "pulls from internet sources named in holger.ron: crates.io, PyPI, npm, Maven, OCI registries, …". server/lib/src/sources.rs:245 refuses a python, oci, npm or maven source unless the build has both online and jobs. In the holger crate, jobs is neither in default nor implied by online (online = ["server-lib/online", "holger-intel?/fetch"]). So cargo install holger, or cargo install holger --features online, cannot start with (name: "pypi", kind: python, url: …). The field test installed --features online,jobs,auth-passkey. Suggestion: online implies jobs, or the docs name online,jobs everywhere a pull-through is shown.

F3. The pip index of a pull-through is /production/simple/, not /production/<source>/simple/

The OCI view is per source (/v2/production/cgr/…), and CLAUDE.md says pull-through paths are /production/<source>/…. For PyPI, the files are per source but the index is not:

$ curl -sS -w " %{http_code}\n" "http://$HQ/production/pypi/simple/idna/"
'simple/idna' is not a Python distribution file 404
$ curl -sS -w " %{http_code}\n" -H "Accept: application/vnd.pypi.simple.v1+json" "http://$HQ/production/simple/idna/"
project 'idna': not yet AVAILABLE; it is being pulled into the intake chain (job holger_pull_intake-1793532233780792-5); retry in 2 s 503

I found the right path only by reading server/lib/tests/pull_through_runs_the_intake_chain.rs. Suggestion: the 404 for /production/<python source>/simple/… names /production/simple/, and the docs give the pip --index-url for a pull-through source.

F4. Only loopback works for the CLI over plain gRPC, and the guide's --at "$HQ_GRPC" fails on a LAN door

Per the guide, with the door on the LAN address in cleartext (insecure: true):

$ holger-cli --at "$HQ_GRPC" --token-file pki/adm.token downstream invite --name lab --out lab.invite.json
[holger-cli-4] the gRPC door at 192.168.68.69:50051 was not reached: transport error: received corrupt message of type InvalidContentType: received corrupt message of type InvalidContentType
$ holger-cli --at "http://$HQ_GRPC" --token-file pki/adm.token downstream invite --name lab --out lab.invite.json
[holger-cli-1] --at http://192.168.68.69:50051: the credential would travel in clear (plain gRPC) to a host that is not this machine; use https:// (with --ca for your own CA), or a loopback address

The refusal itself is right. But 192.168.68.69 is this machine, and the first message (InvalidContentType) does not say "this door is cleartext, you dialed TLS". Workaround: bind 0.0.0.0 and use --at 127.0.0.1:50051. Suggestion: holger-cli-4 says "the door answered cleartext; use http:// on loopback or give the door ron_tls". The guide works only with everything on one host, which it says in its first paragraph. A two-host guide needs the TLS door (F6).

F5. The invite names the bind address http://0.0.0.0:8080 as the door

With the door bound to 0.0.0.0 (F4), the invite names an address no downstream can dial:

$ holger-cli --at 127.0.0.1:50051 --token-file pki/adm.token downstream invite --name lab --out lab.invite.json
INVITED lab -> lab.invite.json (holger.downstream-invite/v1)
  door: http://0.0.0.0:8080

--door http://192.168.68.69:8080 fixes it. Suggestion: refuse an unspecified address (0.0.0.0, ::) as the door and name --door. Also, --out onto an existing file is refused ([holger-cli-1] lab.invite.json already exists), which is fine but cost a round trip.

F6. The console (holger-ui) cannot reach a TLS door; a second endpoint is silently not started

To give the console a loopback door beside the TLS LAN door, I declared two endpoints:

exposed_endpoints: [
    (ron_name: "local", ron_url: "127.0.0.1:50052", ron_http_url: Some("127.0.0.1:8081")),
    (ron_name: "door", ron_url: "192.168.68.69:50051", ron_http_url: Some("192.168.68.69:8080"),
     ron_tls: Some((cert: "…/hq-tls.pem", key: "…/hq-tls.key"))) ],

Only door listened. local got no message at all:

gRPC server listening on 192.168.68.69:50051 [https/h2] (gRPC-web enabled)
HTTP/OCI gateway listening on 192.168.68.69:8080 [https]
$ curl -sS http://127.0.0.1:8081/healthz
curl: (7) Failed to connect to 127.0.0.1 port 8081 after 0 ms: Could not connect to server

start_grpc_endpoint runs only for an endpoint with aggregated_routes, that is, one a repository names. Separately, holger-ui-server dials holger with a plain HttpConnector, with no TLS, so it cannot use the TLS door either. Result: the field test ran with a cleartext LAN door. Suggestions: refuse (or at least print) an endpoint that no repository names; give holger-ui https:// with a CA for HOLGER_UI_HOLGER/HOLGER_UI_DOOR.

F7. The pushed bundles verify only with holger export-push --trust-policy-out; neither guide step says so

The export: (push_to: …) block signs with the box's own push signer, minted on the first push. The guide's pki/trust-policy.ron (reviewer + releaser) refuses those bundles:

$ holger-cli verify stick/push/hq-pypi-1.holger.znippy --policy pki/trust-policy.ron
untrusted release signer: fingerprint 89c6fb811388fe464026a8a8fe78471db85964fc3db8adb4a123c01adeb100be
$ holger export-push --config hq.ron --trust-policy-out pki/push-policy.ron
TRUST POLICY pki/push-policy.ron (vendor hq product pypi: the pinned signers of this box's pushes)
$ holger-cli verify stick/push/hq-pypi-1.holger.znippy --policy pki/push-policy.ron
VERIFIED  pypi 1 (seq 1)

Chapter 8 documents it. The README guide, which is where a first-time operator is, does not. A downstream pins exactly one steps.trust_policy. So a downstream that takes both the guide's hand-made bundles and the pushed ones needs a merged policy, and nothing writes one. Also, a push lands only in push_to. For the online downstream I had to PUT each pushed file into the bundles repository by hand (curl -X PUT … --data-binary @stick/push/hq-pypi-1.holger.znippy "http://$HQ/bundles/hq-pypi-1.holger.znippy"). Unclear whether push_to naming a repository is intended.

F8. An approved pull-through package stays APPROVED until a client asks again

After bob and eve approved six, it stood at APPROVED. The export push did not carry it, and the "available" queue showed it as APPROVED. Only the next client request sealed, promoted and served it, and the push after that carried it (hq-pypi-3):

$ holger-cli … intake list --queue available | grep -A3 pypi/six
                    name: "pypi/six-1.17.0-py2.py3-none-any.whl",
            state: "APPROVED",
$ curl -sS -w " %{http_code}\n" -H "Accept: application/vnd.pypi.simple.v1+json" "http://$HQ/production/simple/six/"
{"files":[…]} 200          # now AVAILABLE; the next push is hq-pypi-3

Not documented. An approver expects "approved" to mean "on its way downstream". Suggestion: the second approval walks it to AVAILABLE, or the docs say the seal waits for the next pull.

F9. A downstream cannot serve an image it imported: podman pull through a downstream is impossible (blocker for step 2/4)

The bundle lands the image AVAILABLE (LANDED cgr/chainguard/static/sha256-fc42….oci.tar@ AVAILABLE in 'production'), but no registry serves it:

$ podman pull --tls-verify=false 127.0.0.1:8090/production/cgr/chainguard/static:latest
Error: initializing source docker://127.0.0.1:8090/production/cgr/chainguard/static:latest: reading manifest latest in 127.0.0.1:8090/production/cgr/chainguard/static: StatusCode: 404, "Not found"

The /v2/production/<source>/… view exists only for a configured oci pull source. server/downstream/src/lib.rs:124 says a downstream's source "names no url and serves what its upstream sent", which is exactly what is needed here. But declaring one is refused:

$ grep sources ds.ron
    sources: [ (name: "cgr", kind: oci) ],
$ holger-downstream start --config ds.ron
Error: config ds.ron: config `sources`: source 'cgr' has kind `oci`, a pull-through into the steps, which needs the `online` and `jobs` build (`cargo build --release --features online,jobs`)

Source::pulls_into_the_chain (server/lib/src/sources.rs:199) is true for every oci source, with or without url, while pull_intake::RemoteSource::serves_only treats a url-less source as "fetches nothing". Same for python, npm and maven. (pip works on a downstream only because the PEP 503 index over the AVAILABLE set needs no source.) Suggestion: a url-less remote source does not need online, and the downstream build carries the serve half of the OCI view. A test: a bundle carrying an image, imported into a default holger-downstream, then podman pull from it.

F10. Smaller things

• Every exported bundle prints NOTE: advisory evidence is marked NON-PRODUCTION (server/lib/src/export_door.rs:303 hard-codes non_production: true). On a production export this reads as a warning nobody can act on.
• holger-downstream --version prints holger v3.0.0 (unknown): the binary's own name is missing, and the build revision says unknown for a crates.io build.
• The guide installs the one-way downstream with cargo install --root "$G/air" holger-downstream into the same ~/.cargo/registry. Two runs of the install script both printed Replaced package holger-downstream v3.0.0, which is easy to misread as one build replacing the other. They do live in separate roots.
• The upstream index (/v2/…/tags/list) lists only latest, though .sig and .att are served (…/manifests/sha256-<hex>.sig 200). cosign finds them, so this is fine. The docs say the tags are "kept", which is true but not listed.

What was verified end to end

# upstream: pull through, two approvals, pip and podman
curl -H "Accept: application/vnd.pypi.simple.v1+json" "http://$HQ/production/simple/idna/"     # 503 Retry-After, then POLICY_CHECKED
for who in bob eve; do holger-cli --at 127.0.0.1:50051 --token-file pki/$who.token intake advance --name pypi/idna-3.20-py3-none-any.whl --reason "field test"; done
pip install --index-url "http://$HQ/production/simple/" --trusted-host 192.168.68.69 idna        # Successfully installed idna-3.20
podman pull --tls-verify=false 192.168.68.69:8080/production/cgr/chainguard/static:latest       # after the same two approvals

# cosign through Holger (upstream)
IMG=192.168.68.69:8080/production/cgr/chainguard/static@sha256:fc42af33e643f20d08baf03cddcc1c02ba57052dfb02edb0d79f3fc47bcea8d9
cosign verify --allow-http-registry --allow-insecure-registry --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  --certificate-identity-regexp "https://github.com/chainguard-images/images/.*" $IMG                                    # verified
cosign verify-attestation --allow-http-registry --allow-insecure-registry --key "http://$HQ/-/evidence-key" \
  --type https://holger.rs/attestation/scan-evidence/v1 --insecure-ignore-tlog=true $IMG                                # verified
cosign verify-attestation … --type cyclonedx …                                                                         # verified
curl "http://$HQ/v2/production/cgr/chainguard/static/referrers/sha256:fc42…"        # two application/vnd.dev.sigstore.bundle.v0.3+json

# online downstream: pushed bundles PUT into `bundles`, mirrored, imported
#   mirror hq:bundles -> inbox [TrustSize]: listed 2 … fetched 2 failed 0 bytes 870890
pip install --index-url "http://$DS/production/simple/" idna                                  # Successfully installed idna-3.20

# one-way downstream (boundary: OneWay, network: Offline): the stick
holger-downstream bundle allow-first-import --config air.ron --from-bundle stick/hq-pypi-1.holger.znippy --policy pki/push-policy.ron
holger-downstream bundle import --config air.ron --bundle stick/hq-pypi-1.holger.znippy --policy pki/push-policy.ron  # IMPORTED (new), 2 LANDED
holger-downstream bundle import --config air.ron --bundle stick/hq-pypi-2.holger.znippy --policy pki/push-policy.ron  # IMPORTED (new), the image LANDED
holger-downstream bundle import --config air.ron --bundle stick/hq-pypi-1.holger.znippy --policy pki/push-policy.ron
#   Error: [holger-import-5] bundle verification failed: [holger-update-22] sequence rollback: bundle sequence 1 is not newer than the accepted 2
cp stick/hq-pypi-3.holger.znippy air/bundle.inbox/       # while running: swept to archive/
pip install --index-url "http://$AIR/production/simple/" six                                  # Successfully installed six-1.17.0

Not done

• The console's Sync board "Enrol a downstream" and "Invite" buttons: done over HTTP and the CLI instead (POST /-/sync/downstreams/enrol, holger-cli downstream invite).
• A podman pull and cosign verify through a downstream (F9).

The VMs 910 and 911 are still up on PVE for a re-run.

Follow-up the same evening: passkey sign-in, and 3.1.0

910 was made a proper install: Tailscale on the VM, tailscale serve giving the console https://holger-field-up.tail4eceda.ts.net and the package door https://holger-field-up.tail4eceda.ts.net:8443 real certificates; holger and holger-ui as systemd user services (linger); the RON's static tokens read from their 0600 files (token_file), and passkey sign-in beside them:

Passkey(rp_id: "holger-field-up.tail4eceda.ts.net", rp_origin: "https://holger-field-up.tail4eceda.ts.net", rp_name: "Holger"),
group_roles: { "holger-admins": Admin, "holger-writers": Writer, "holger-readers": Reader },
group_duties: { "holger-admins": [RepositoryAdmin, SecurityApprover, Auditor], "holger-writers": [Importer] },

holger-cli --at 127.0.0.1:50051 --token-file pki/adm.token first-key --display henrik minted the first admin's invite (/login?invite=…, one day, once), Henrik enrolled his YubiKey through it, and the door shut ([holger-auth-5] this server already has an admin key: the first-key door is shut). pip and podman through :8443 work with the Tailscale certificate. Then both VMs were upgraded to 3.1.0 (published 2026-11-01 17:04 UTC): the worker image is :3.1.0 (F1 fixed), the flow model is holger.flow/v2, pip through both downstreams still works.

F11. holger-ui 3.1.0 on crates.io ships the 3.0.0 console (release)

The console's bundle is byte-identical between the two releases, so a cargo install holger-ui of 3.1.0 has none of the console work since 3.0.0 (the smart flow, the menus):

$ sha256sum ~/.cargo/registry/src/*/holger-ui-3.0.0/console/*.wasm ~/.cargo/registry/src/*/holger-ui-3.1.0/console/*.wasm
64c7b58526be920d9e67b09de2f5a815c14a4b00ac07fb0029539cf9de8d0684  …/holger-ui-3.0.0/console/holger-ui-web-a2dfb34484eb13c6_bg.wasm
64c7b58526be920d9e67b09de2f5a815c14a4b00ac07fb0029539cf9de8d0684  …/holger-ui-3.1.0/console/holger-ui-web-a2dfb34484eb13c6_bg.wasm

crates/holger-ui/console/ is gitignored and packed from the publisher's tree (.cargo_vcs_info.json says "dirty": true); cargo xtask console was not run before cargo publish. Workaround on 910: trunk build --release of holger-ui v3.1.0 (20b4199) and HOLGER_UI_DIR pointing at its dist/. Suggestion: the release refuses a bundle whose hash equals the previous release's, or cargo xtask console is a step of the publish.

F12. holger-ui --version of 3.1.0 says holger-ui 3.0.0

It prints holger_ui_server::VERSION, and holger-ui-server was not republished (crates.io max 3.0.0; holger-ui 3.1.0 depends on holger-ui-server = "3.0.0").

F13. Passkey sessions: session.key is never written

From the first passkey sign-in on, holger logs, on every console request:

WARN nornir_auth_policy] auth: passkey session key /home/field/holger-guide/hq/holger-passkeys/session.key is unreadable (No such file or directory (os error 2)) — the passkey method authenticates nobody

Holger::serving_auth resolves an unset session_key_file to <data_root>/holger-passkeys/session.key (both 3.0.0 and 3.1.0), but nothing creates that file. Sign-in itself works (the identity core's session answers POST /-/auth/passkey/session 200, method: passkey, role: admin). After the restart onto 3.1.0 the same browser had no session ({"error":"no session"}, 401) and had to sign in again. Not yet confirmed with a controlled restart whether every restart signs everyone out.

F14. Smaller things in the console

• The Flow page is now interactive (level 1 with unfold main, unfold lab, show the full RON; ?open=), but it does not refresh by itself: the model carries refresh_secs: 30, which only holger's own /-/flow HTML page uses (<meta http-equiv="refresh">). Measured: FlowSvg was fetched at page load and at a click, and not once in the following 90 s.
• /login?next=/flow stays on the login page after a successful passkey sign-in.
• The passkey config names https://www.<rp_id> as the canonical origin first ("rp_origin":"https://www.holger-field-up.tail4eceda.ts.net"); on a tailnet name there is no www. host. The ceremony worked at the bare name.