nordisk/holger

8. Supply chain

This chapter says what Holger checks on the way in, what it signs itself, what it exports, and what a downstream checks. Every claim names the code that does it.

On the way in: provenance

The provenance verifier is the crate holger-provenance (provenance/). It runs at the intake's SCANNED step as the scanner holger-provenance (server/lib/src/scanner/provenance.rs), inside the scan sandbox. It makes no network call. The material that came with an artifact waits beside it, by its sha256, in <data_root>/holger-provenance/<sha256>/.

DSSE and in-toto

• DSSE envelopes with the payload type application/vnd.in-toto+json.
• in-toto Statement v1 (https://in-toto.io/Statement/v1) and v0.1 (https://in-toto.io/Statement/v0.1) (provenance/src/statement.rs).
• The statement's subject must be the artifact's own sha256 or sha512.
• Keys: Ed25519 and ECDSA P-256, pinned in rules.keys.

SLSA Provenance

• SLSA Provenance v1 and v0.2. The verifier reads the builder id, the build type, the invocation, the external and internal parameters and the resolved dependencies (for v0.2: materials) into the evidence.
• The RON can require it. On the floor (rules:): require_provenance, slsa_min, builders, identities, keys. On a source: gates: (provenance: require(builders: [...], keys: [...], slsa_min: N)), or prefer, or off. A source may only tighten the floor.
• A builder is written as a bare id or as (id: "…", level: N). The SLSA level comes from the policy's builder rule, not from the provenance document. A builder named without a level counts as level 2 (DEFAULT_SIGNED_LEVEL, provenance/src/verify.rs). So slsa_min: 3 refuses a builder listed without level: 3.

rules: (
    require_provenance: true,
    keys: [(id: "d24d3785e8281bcd37e8f2c40a3a85e8152d3ce2200a5c929a1a5ca636e8249e", alg: ecdsa_p256, public_key: "<PEM>")],
),
// on a source:
gates: (provenance: require(
    builders: [(id: "https://git.gunnar.rs/nordisk/dwarves/slsa/builders/dwarves-server/v1", level: 2)],
    slsa_min: 2,
)),

Holger's own releases are Build L2 (chapter 7), so the dwarves builder is named at level 2; a builder you trust at L3 is written level: 3 with slsa_min: 3.

Sigstore keyless, offline

For a keyless signature the verifier checks, with no network (provenance/src/verify.rs, provenance/src/tlog.rs):

• the Fulcio certificate chain to a pinned CA, the signer identity (issuer and subject) and the code-signing usage;
• the SCT from a pinned certificate-transparency log;
• the Rekor entry bound to this signature: its signed entry timestamp (SET), its inclusion proof against a signed checkpoint, or both.

The pinned root is a signed bundle of the product holger.sigstore-trust/v1 (provenance/src/trust.rs), named by rules.trust_root. It carries trusted_root.json and the pinned APK and OpenPGP keys (apk-keys/<name>, pgp-keys/<name>).

Keyless cosign signatures are verified on intake this way; real cosign 2.4.3 and 3.0.2 verify through Holger with --certificate-identity and --certificate-oidc-issuer against an offline test root.

The estate's trust root (server/lib/src/sigstore_trust.rs):

• Sigstore's public-good trusted_root.json is pinned by its TUF digest (6494e21e…). cargo xtask sigstore-trust stage fetches it by that digest, and holger sigstore-trust make signs it with the release trust identities ($HOLGER_TRUST_SIGNER_DIR).
• The signed root and its policy ride the sandbox payload and are seeded into <data_root>/holger-trust at boot. The estate RON names them: provenance: Some((default: prefer, rules: (trust_root: Some("/data/holger/holger-trust/sigstore-trust.holger.znippy")))) and trust_policy: Some("/data/holger/holger-trust/bundle-trust.ron").
• The root is refreshed only by a signed holger.sigstore-trust/v1 bundle through any import door, and takes effect at once, with no restart. An empty slot boots and verifies no keyless signature. Refusals: holger-trust-1 to -3.
• holger sigstore-trust status --config <file> shows what a box has installed.

Not checked offline, and said so in the evidence: a Rekor consistency proof, certificate revocation and RFC 3161 timestamps. If the trusted root pins no CT log, the SCT is recorded as "not checked" instead of refused.

Ecosystem proofs

The material parser (provenance/src/material.rs) reads:

Scanners in a sandbox

The server never unpacks, parses or scans an artifact itself. The scanners (server/lib/src/scanner/):

The INGEST walk opens every container it meets (holger-inspect, server/inspect/src/walk.rs, magic.rs): zip, tar and gzip; bzip2; xz (decoded in memory and held to max_nested_bytes, past it NestedTooLarge); zstd; ar (a .deb, GNU and BSD long names); rpm (the lead and headers, and the payload in gzip, xz, zstd, bzip2 or plain cpio newc); conda (.conda, a zip around tar.zst, and the old .tar.bz2). Compression around a tar is opened one level. A 7z, or anything else the walk cannot read, blocks as not-unpacked. An absolute symlink inside a deb or rpm payload is reported as Info (it is a root file system); anywhere else it blocks. Known limit: a .deb whose data.tar.xz decodes past 256 MiB blocks.

The scanners and the walk run in holger-scan-worker, one rootless podman container per artifact (server/sandbox/, image localhost/holger-scan-worker:<Holger release>, :3.1.0 for Holger 3.1.0, which must be this server's build: chapter 5): --network none, --pull never, the artifact read-only, a read-only root, a capped tmpfs, --cap-drop all, uid 65532, a deadline. ClamAV is clamd in its own container (holger-clamd). On the OS-less appliance the engine is a pinned static crun that Holger drives itself (server/sandbox/src/crun.rs). The worker returns one JSON report; the server keeps the advisory match and the evidence.

The worker's root filesystem is hashed again at every start, and a changed one is laid again (holger-sandbox-14); podman runs the worker image by its digest. A real build talks to ClamAV only over the sandboxed clamd's unix socket: a TCP or host clamd, clamscan, or no socket refuses the start (holger-sandbox-15). The sandboxed clamd runs with AlertExceedsMax yes, so a file past its scan limits is reported, not passed.

promotion.intake.rules: (sandbox: required) is the default. off exists only in a test build; a release build refuses it (holger-sandbox-6). No engine means every scan is refused (holger-sandbox-1), never an in-process fallback.

Signed scanner signatures

ClamAV databases and YARA rules reach Holger as a signed bundle of the product holger.scanner-sigs/v1 (intel/src/sigs.rs). It holds ClamAV database files (payload/clamav/<name>) or YARA files (payload/yara/*.yar, *.yara) and nothing else (holger-sigs-1).

• It comes in through every bundle door: holger bundle import, holger-cli bundle import, POST /-/bundle/import, AdminService.ImportBundle and the inbox bundle.inbox.
• The import verifies it, checks its floor (the first one needs the admin's grant, below), writes the audit row scanner-sigs-import, moves the floor, installs the files into <data_root>/sandbox/signatures/ and reloads clamd. Once a bundle is installed, the sandboxed clamd mounts signatures/clamav read-only over /var/lib/clamav. A failed install is holger-sigs-2; a build without scanners refuses it (holger-sigs-3).
• A hotfix is the next sequence with one urgent file and needs no new grant; an older bundle is a rollback and is refused.
• holger scanner-sigs build makes a bundle (online side); holger scanner-sigs status shows what is installed. The operator guide is v1/scanner-signature-bundles.md.

Coming: a bundle that withdraws a file; the scanners reading the installed YARA rules.

Untrusted work runs as jobs

Every piece of untrusted work is a job: every scan, and every bundle import (the inbox bundle.inbox, POST /-/bundle/import, gRPC ImportBundle, holger bundle import, holger bundle allow-first-import --from-bundle). The job API is Heimdall's nordisk-heimdall-core 0.2 from crates.io; the runners are holger-job-runner (jobs/runner).

• The verify-bundle job checks the release signature before anything is decompressed, then every member against the signed manifest, then an export's DSSE evidence, and writes the verified members back. The server hard-links the bundle into the job and never opens it itself.
• What the server believes. Only a signed result that verifies with the runner's per-boot key (pinned at start), names the job's id and nonce, and lists outputs that match by sha256 and size, with nothing extra. A result seen twice is refused as a replay.
• Heimdall's runner (engine: heimdall, feature heimdall of holger; the estate declaration names it): the scans and the bundle verifies run as jobs of Heimdall's protected runner (nordisk-heimdall-container 0.2.1: Landlock, Heimdall's seccomp profile, the worker image verified at every start). Where Heimdall cannot start, the jobs run in youki-plain, then in the interim crun runner over rootfs, and the start says which by name; with no rootfs either, every scan is refused (holger-sandbox-1) (jobs/runner/src/heimdall.rs, start_as; server/lib/src/job_runner/mod.rs).
• Host ids. When the server runs as root, a job's root maps to the host id range 100000 to 165535 (userns_base, host_id_count), never to the host's root (jobs/runner/src/heimdall.rs, host_ids_for; test as_root_with_require_rootless_a_jobs_root_maps_to_the_host_id_range in jobs/runner/src/heimdall_tests.rs). The job's inputs are then copies, not hard links.
• The interim runner is ContainerRunner over the sandbox above (runner names holger-podman, holger-crun, holger-sidecar): a fresh container per job, no network, a read-only root, the input read-only at /in/i, one writable /out, the job's own limits and the engine's default seccomp profile. It has no generated syscall profile, no syscall broker, no Landlock and no wasm. The console's Security page (section runner, "Where untrusted work runs") and holger jobs --config <ron> show the pinned key id and one yes/no row per protection. A boundary: OneWay Holger prints a warning at start that it runs on this interim runner.
• The downstream verifies its bundles in jobs too, so it needs a container engine and the worker image to import. sandbox: off exists only in test builds (feature sandbox-off); no shipped build carries it.
• Refusals: holger-sandbox-16 (an output the job did not declare), -17 (the job's answer is not valid), -18 (the signed result is not believed: wrong key, changed, or replayed), -19 (the outputs do not match), -20 (the runner could not run the job).

OpenZL bundles verify inside the Heimdall guest

In Heimdall (heimdall main 6579b33, docs/OPENZL-WASM.md), the verify-bundle guest carries OpenZL's C decoder compiled to wasm32: decoder only, vendored from openzl-sys-rs 0.4.1+openzl.0.3.0 with one patch. The guest imports nothing but the WIT emit-file.

• Real Holger-packed OpenZL bundles are accepted with the same report as Holger's native verifier. Bit flips, truncation, size lies and bombs are refused with Holger's codes (a decode failure is holger-update-28).
• Tested with 173 guest tests; the OpenZL decode was fuzzed for 47.9 M runs under ASan.
• Cost: the guest grows by 1.09 MB, and an 8 MB bundle takes about 188 ms in wasm against about 32 ms native. Bundles above about 8 MB need more fuel in the job policy.

Multi-core bundle verification in Heimdall

In Heimdall since 0.2 (docs/GATLING-WASM.md in the heimdall repository); Holger vendors the released guests (heimdall-guests-0.2.1) under guests/heimdall/0.2.1/, with their pins, SLSA statement and signatures.

• The verify-bundle job runs split: one verifier instance (no C) and decoder instances (OpenZL and BLAKE3, importing nothing), each with its own memory and fuel.
• GATLING on the host (gatling::ordered, gatling_forkjoin; no rayon) drives them, and the host checks the count, length and BLAKE3 of every decoder answer natively.
• A lying decoder is refused (heimdall-wasm-13; heimdall-bundle-7 when the verifier refuses an outside answer). The verdicts are the same as the single-instance guest's on every bundle test.
• Speed (dwarf, loaded): 64 MB in 672 ms instead of 968 ms, 192 MB in 1,482 ms instead of 2,797 ms (1.89×). Big bundles need job fuel of at least 128 × the content bytes.

With engine: heimdall and its guests: (the estate and the appliance medium), the import door verifies every bundle in this split guest inside the worker (server/lib/src/job_runner/verify_bundle.rs, feature heimdall-wasm). Without it the verify job runs Holger's native verifier inside the container.

Coming: the fetching downstream's fetch as a job; intel import as a job.

SBOM

The SBOM is CycloneDX 1.5 (server/lib/src/sbom.rs). It is filed as a signed DSSE in-toto statement with the predicate https://cyclonedx.org/bom, beside the scan evidence (server/lib/src/scanner/attestation.rs).

Two kinds of artifact get a generated SBOM:

• Python wheels (holger-wheel-sbom, server/lib/src/scanner/wheel.rs).
• Container images (holger-image-sbom), built in the scan worker from the image's layers: the apk, dpkg and rpm databases, Python, npm, Java archives (pom.properties), gem specifications, Composer, .NET deps.json, Go build info and cargo-auditable, one purl per package. It is served as an OCI referrer of the image and as a layer of its .att tag (below).

Other formats have no generated SBOM yet.

Holger signs its own evidence

Scan results (predicate HOLGER_SCAN_EVIDENCE_V1) and the SBOM are DSSE envelopes signed by Holger's attestation signer (server/lib/src/attest.rs, AttestSigner). The key is Ed25519 (the default) or ECDSA P-256, given by the operator as PKCS#8 with an X.509 certificate: attestation: (signer_key, signer_cert, signer_alg), or held on a hardware token over PKCS#11 (attestation.signer_hsm, chapter 10). Holger mints no key of its own. The same signer signs the checkpoints of the hash-chained audit log.

The export: one signed bundle

holger export (gRPC BundleService.Export) takes the AVAILABLE set of the production repository and writes one bundle (server/lib/src/promote/airgap.rs, update-bundle/):

• The container is a znippy archive, .holger.znippy, of the format holger.update-bundle/v1: manifest.json, core.json, payload/, evidence/, approvals/, signatures/.
• The index evidence/holger-export.json (holger.airgap-export/v1) lists per artifact its sha256, blake3, size, its DSSE attestations, the signer_cert, the sbom, the intake_record, the approvers and the carried provenance.
• Every member carries a SHA-256 and a BLAKE3 digest.
• Signatures are Ed25519 with X.509 certificates, not CMS. Two approvers sign the candidate (holger-cli bundle approve), then a release signer, distinct from them, signs signatures/release-manifest.json (signatures/release.sig, holger-cli bundle finalize). The release manifest covers every member, so one changed byte fails.
• The signed sync manifest (evidence/holger-sync.json, Ed25519, sync/) rides in the same bundle: what was added and every invalidation.

What crosses: the export profile. Every way out (holger export, BundleService.Export, the diode push) carries only what decides (server/lib/src/export_profile.rs, a guard over 15 structures). There is no RON switch.

• Kept: the verdicts, the digests, each evidence item's kind, source and verdict, core.json, and the approver on the APPROVED line.
• Stripped: who exported and when, the production repository name, the intake record's repository, times, policy, reason, origin and effective policy, every actor but the approver (written online), free-text evidence references (they cross as sha256:<hex>), the source provenance's source_repository and fetched_at, and a sync invalidation's issued_by.
• Signed scan evidence crosses as signed.

The same bundle is carried on a USB stick, pushed through a one-way diode, or fetched online:

• Push (feature export-push, in the default). With export: Some((push_to: Some("<dir>"), every: "5m", vendor: "holger", product: "packages", expires_days: 30)) in the RON, the job export.push writes every new signed bundle into push_to as <vendor>-<product>-<seq>.holger.znippy (as .<name>.part, then renamed); a second push carries only what is new. holger export-push --config <file> pushes once, and --trust-policy-out <file> writes the pinned trust policy the downstream verifies the pushes with. The push signer's keys are kept in <data_root>/holger-export-push/signer (mode 0600). A failed push leaves push_to unchanged (holger-export-2).
• Fetch (feature fetch, not in the one-way downstream): over mutual TLS (a mirror: source with ron_kind: Holger) or over WireGuard (via: Wireguard(...), feature downstream-wg) (server/downstream/src/via.rs).

The downstream

The downstream (on-prem) runs the same checks whether the bundle came over the network or on a stick:

• It verifies the bundle's signatures against its own trust policy (update-bundle/src/policy.rs): either a rooted chain (trust_mode: RootedChain, trusted_roots) or pinned signers (PinnedSigners, approved_signers), and the two-person rule. It refuses a rollback (a lower sequence, a replay, a reused bundle id).
• It checks each artifact's sha256, BLAKE3 and size against the index and against the signed core.json (verify_import, server/lib/src/promote/airgap.rs), inside a verify-bundle job (above).
• It verifies every DSSE attestation inside against its own attestation store. With no attestation: block the import is refused.
• It never fetches new trust. Its roots and signers come from its own config.
• One-way (cargo install holger-downstream, boundary: OneWay, the default): packages come in only through the inbox (<data>/bundle.inbox) or holger bundle import, and it dials nothing (server/lib/src/boundary.rs). It refuses to start, naming <file> line <n>, on anything that would reach out or take a receipt: a mirror: or export: block, sync.downstreams, a sync upstream with url, token_file or invite, network: Online, a *-remote repository, a twin role, recharge_from, intel: (offline: false), a pull-through url, ESTATE_CLOCK sources, an estate build (holger-boundary-1). boundary: Fetch in a build without fetch is refused (holger-boundary-2). A one-way sync apply holds the invalidated packages and signs no receipt.
• Fetch (cargo install holger-downstream --features fetch, boundary: Fetch): it dials only its upstream (Plan::may_dial, server/downstream/src/lib.rs; refusal holger-downstream-2).
• A config that asks for online intel, a twin, remote repositories or scanners is refused (holger-downstream-1).

It does not re-check the original package's SLSA or Sigstore proofs, and it does not scan again. The upstream did those checks, and the import records the upstream-signed evidence. The proofs do travel: an export from a Holger with the provenance scanner puts each artifact's provenance material in the bundle (<artifact>/provenance/<file>, the index's provenance field), and the downstream build links the verifier (holger-provenance, through server-lib intake). But the downstream has no scanners, so its judge_carried_provenance returns nothing (server/lib/src/import_door.rs). A full Holger with scanners that imports a bundle does verify the carried provenance again, under its own policy, and re-runs the malware scans.

The first import

A downstream with no accepted bundle yet has no rollback floor, so its first import is a secure-side admin's decision:

• holger bundle allow-first-import --config <file> --vendor <v> --product <p> --sequence <n>, or --from-bundle <file> [--policy <file>] (the bundle is verified first), or POST /-/bundle/first-import?vendor=&product=&sequence= (Repository Admin only), or holger-cli bundle allow-first-import --vendor <v> --product <p> --sequence <n> (gRPC holger.doors.v1.BundleService.AllowFirstImport, admin only), or the console's "First import" form (Admin → Sync).
• The grant is audited, kept in <data_root>/holger-first-import/grants.json, and used once; setting the floor is audited again. It works for update, airgap and intel bundles.
• Refusals: holger-import-9 (no grant for this vendor, product and sequence), holger-import-10 (a first-import flag from the caller, or a FIRST-IMPORT marker in the inbox), holger-import-11 (the grant itself refused: not an admin, blank, sequence 0, a floor already stands, audit off or failed).

Signatures through Holger: OCI referrers

Signing tools find an image's signatures, attestations and SBOMs through Holger as a registry (server/lib/src/referrers.rs, server/lib/src/pull_intake/):

• The oci registry (file-oci) serves OCI 1.1 GET /v2/<name>/referrers/<digest>, with the ?artifactType= filter (and OCI-Filters-Applied when it filters) and an empty index when nothing is attached. A push with a subject is answered with OCI-Subject. cosign's tag scheme (sha256-<hex>.sig, .att, .sbom) is kept as plain tags.

• The pull-through (/v2/production/<source>/<image>) keeps each image's .sig, .att and .sbom tags and its OCI 1.1 referrers (or the referrers tag schema), with their blobs, inside the sealed layout. They walk the intake chain as part of the image, are never approved on their own, and are served once the image is AVAILABLE.

• Holger's own evidence. The signed scan evidence (https://holger.rs/attestation/scan-evidence/v1) and the image's CycloneDX SBOM name the image's digest. They are served as referrers (Sigstore bundle v0.3) and as layers of the .att tag. Check them with:

cosign verify-attestation --key https://<holger>/-/evidence-key \
  --type https://holger.rs/attestation/scan-evidence/v1 --insecure-ignore-tlog=true <image>

• The evidence key door (server/lib/src/evidence_key.rs), open on every box: GET /-/evidence-key (and .pem) serves the public half of attestation.signer_cert as PEM, .jwks a JWK Set (kid = the DSSE keyid, with x5c), and .crt the certificate.

• Pushes are scanned. On a Holger with an intake chain, a push to an oci repository walks the intake chain before anything is served (chapter 5). A signature pushed before the tag is verified on intake; one pushed after is served once the image is AVAILABLE.

• The downstream serves all of it after a bundle import, with no fetching.

Tested with real clients: cosign 1.13.6, 2.4.3 and 3.0.2, notation 1.3.2, oras 1.2.3 and kyverno 1.13.4 (kyverno apply with verifyImages) (server/lib/tests/real_signing_clients_verify_through_holger.rs, server/cli/tests/downstream_serves_the_build_systems.rs; CI steps real-clients and real-clients:downstream).

Gone since the lean cut

The lean cut of 2026-10-17 (holger 9521bf06) removed:

• The OCI referrers API. It is back since 2026-10-30 (above).
• The Arrow audit query backend. It is back since 2026-10-30, as an index of the hash-chained log (chapter 10).
• korpsig.

Open questions

1. Should the downstream re-check the original SLSA and Sigstore proofs? The material is already in the bundle and the verifier is already linked; it needs the provenance scanner turned on in the downstream build, and a trust root and policy on the downstream.
2. A USB stick from a source that is not Holger: accept only Holger-signed bundles, or also verify raw SLSA and DSSE files on import?

Moved from v1

• v1/SUPPLY-CHAIN.md, all of it, corrected against the code (signatures are Ed25519, not CMS; the SBOM is generated for some formats only; the proofs already travel in the bundle).
• The provenance and secrets paragraphs of v1/README-v1.md ("The default floor").
• "With Holger itself" of v1/SLSA.md.

Still in v1

• v1/secure-update-gateway-mvp.md: the bundle format and the two-person rule in full.
• v1/demo-airgap.md: the export and import, step by step.
• v1/secure-update-gateway-threat-model.md: the supply-chain threats.