nordisk/holger

Holger — the feature-complete checklist (the long gap list)

Purpose. The single driver that answers "what does feature-complete Holger mean, and what is still missing?" — mined from the git history, PLAN.md, artifactory-parity-design.md, holger-tiers.md, static-dynamic-profile-design.md, the shared-infra charter (workspace_nordisk/.nornir/shared-infra-index.md), and the modgunn/master-plan arc. artifactory-parity-design.md is the external parity view (vs Artifactory/Nexus); this doc is the internal completeness view across the three shipped executables and the shared-infra seams they must consume. Kept at altitude; each item links to where the detail lives.

Legend: ✅ shipped & tested · 🟡 partial (exists, not in the packaged/enforced shape) · 🧭 designed, not in code · ❌ absent. Every ❌/🟡 is a candidate for a future round — do NOT invent churn against a ✅.

Last synced against the tree: 2026-07-14.

0. The three executables (build-space, task #36) — the completeness frame

One source tree (server/cli/src/main.rs), three distinctly-named binaries selected by a mutually-exclusive fatness feature, materialized by nornir-build-thing from .nornir/build-variants.toml and consumed by dwarves' pipeline. Each column below is a tier; a feature is only "complete" when it lands in the tier(s) that should carry it.

Detail: .nornir/holger-tiers.md, .nornir/build-variants.toml, .nornir/static-dynamic-profile-design.md.

Tier-completeness gaps

• 🧭 The crate-split floor (static-dynamic-profile-design.md, PLAN.md §spine). Today the three tiers differ by Cargo feature only — #[cfg], not a crate boundary. Feature unification means dynamic code behind a feature in a shared crate can leak into holger-mikro. Feature-complete mini needs the BackendProvider trait + core/serve vs dynamic crate split so "the dynamic code isn't in this binary" is provable from cargo tree, not from trusting cfg.
• ✅ Airgap egress proof as a test. The online/reqwest feature-gating shipped (commit 07a9d1a); the wiring hole (mini/embedded still linked reqwest via server-lib default features, and ureq/bench-scenarios unconditionally) was closed (commit e457db3: server-lib dep takes default-features=false; ureq + the upstream_rust rust-remote arm ride behind online; the bench verb behind bench). The automated gate now exists in code: build-space/src/lib.rs::airgap_tiers_link_no_outbound_http_client runs cargo tree and asserts embedded+mini link NO reqwest/ureq while full links both (proving it discriminates); it emits a testmatrix cell.
• ✅ Tier mutual-exclusivity enforced in the Cargo graph. A mini+full (or any two-tier) union is now a hard compile_error! in the holger binary, and the full variant builds --no-default-features so it no longer silently drags the default mini into its closure. The guard lives in the binary only — a shared-lib guard would misfire under feature-unification (the bench-scenarios edge pulls server-lib's default mini while the CLI edge pulls full). xtask/tests/build_verb.rs pins the materialized full argv.
• 🧭 The crate-split floor still stands — the tiers differ by Cargo feature, now provably stripped (the cargo tree gate above), but a BackendProvider + core/serve vs dynamic crate boundary remains the deeper "dynamic code is not in this binary at all" guarantee (see the 🧭 floor item above).
• ✅ build-thing declaration emits a testmatrix cell (commit 8ddb6c7) and a press round-trip example (build-space/examples/press_build_thing.rs).

1. Formats / protocols (parity §A)

server/repository/: znippy-{maven,python,rust,npm,nuget,go,gem,deb,rpm,helm,docker, conda,composer,package} + writable file-rust (Cargo sparse) + file-oci.

• ✅ Real protocol: Cargo, PyPI, npm, NuGet, Go, Docker/OCI, Helm, Composer, Gem (RubyGems compact-index), Conda (repodata.json).
• ✅ Deb/RPM native index (Deb Packages+Release, RPM repomd.xml+primary.xml, real SHA256) + native control-field parse (ar+control.tar / RPM header tag table).
• 🟡 Maven — passthrough/partial: no generated maven-metadata.xml, no SNAPSHOT timestamp resolution, no checksum sidecar (.sha1/.md5) synthesis.
• 🟡 Conda — .conda (zip+zstd) format read is deferred/documented, not faked.
• ✅ Generic / Raw writable repo (arbitrary blobs, path-addressed) — shipped as server/lib/src/raw.rs::RawBackend, wired by ron_repo_type: "raw" (ron_data_dir its only requirement). A disk-backed loose-file bucket: the path IS the address, one coordinate↔path bijection so the gRPC/in-process (ArtifactId) and HTTP (/<repo>/<path>) doors meet on the same bytes and a list() round-trips back into fetch/delete_artifact. Traversal-closed per-segment guard; SHA-256-verified safe delete_artifact (agrees with the GC executor); HTTP PUT/POST/GET/HEAD/DELETE with the checksum-deploy gate + RBAC at the door. Because it implements the full RepositoryBackendTrait it plugs into cross-repo search, retention/GC, replication and /metrics with zero new wiring (Law #5, no twin). 11 unit tests incl. a RED-when-broken content-id delete + traversal-refusal on both paths; gated holger-raw/http functional_status cells. .nornir/raw-repo-design.md.
• ❌ Long-tail: APK, Terraform registry, CocoaPods, HuggingFace, Vagrant, P2/Eclipse.

2. Remote / proxy / virtual (parity §B)

• ✅ Proxy fall-through with cache write-back (proxy.rs::ProxyBackend, tested).
• ✅ Virtual / group repos — ordered read-only aggregate (group.rs::GroupBackend, ron_repo_type: "group" + ron_members, tested).
• ✅ Per-upstream credentials (each remote its own auth) — wired: every remote-registry upstream (nexus-remote/artifactory-remote/azure-remote/ aws-remote) is built from its OWN ron_upstream_auth RON field (the shared UpstreamAuth enum: None/Basic/Bearer/ApiKey/Mtls), threaded through make_remote_upstream → SkidbladnirUpstream::{nexus,artifactory,azure,aws}; a proxy's ron_upstreams chain composes several such repos, each carrying its own credential. (rust-remote = public crates.io, no auth by design.) Ledger corrected 2026-07-14 — the prior "designed hook, not wired" was inaccurate.
• ✅ Negative caching (remember a miss for a TTL). ProxyBackend gains an opt-in per-repo negative cache (ron_negative_cache_secs, 0/unset ⇒ disabled, unchanged): a coordinate or GET/HEAD path that misses primary AND every upstream is remembered as absent for the TTL, short-circuiting the upstream walk on repeat. Primary is checked first so a later local publish is never shadowed; reads-only; bounded (NEG_CAP + lazy expiry). Purely in-process (std only, no egress) → no tier regression. 5 unit tests incl. 2 RED-when-broken call-counter short-circuit tests + a never-shadow-a-later-hit guard + deterministic expiry; gated holger-proxy/negcache functional_status cells. .nornir/proxy-design.md §"Negative cache". (Metadata/index-doc caching — remembering a fetched index — stays a follow-up.)
• ❌ Online/offline per-remote toggle (blocked-remote mode for the mini tier).

3. Promotion / staging / lifecycle (parity §C — the Artifactory headline)

• ✅ Promotion through the shared jera job ledger (commits 2fd2f19, 47576d6): every promotion is a bounded jera job (CAP), not a bespoke queue. promotion-design.md.
• ✅ Promotion policy gate is REAL and enforced at the boundary (commit 54b43cd) with an operator config flag (a7a5500) + OSV feed import.
• ✅ Property-based promotion requirement — "only promote when the source carries signed=true." PromotionEngine.require_property + the pure property_requirement_met (shared properties::map_matches); the promote boundary resolves the source coordinate's props and refuses (failed_precondition / 412, audited) when unmet — fail-closed (no store ⇒ refuse), runs after the CVE/license gate. Env config HOLGER_PROMOTE_REQUIRE_PROPERTY=key=value (mirrors HOLGER_PROMOTE_ENFORCE; per-repo RON surface deferred). RED-when-broken gRPC test: an unsigned artifact is refused, the same artifact promotes once tagged. .nornir/property-policy-design.md.
• 🟡 Staging repos as first-class nodes (a dev→staging→release pipeline of RON repos) — the promotion primitive exists; the multi-stage pipeline object is 🧭.
• ❌ Build-info / release bundles — the content-addressed signed release unit (a manifest of "these exact artifacts constitute build N") is designed in promotion-design.md but not materialized as a stored, servable object.

4. Retention / cleanup (parity §D) — planner + executor shipped

• ✅ Retention policies (keep-last-N per name / max-age / keep-min-per-name floor / max-total-bytes quota / protect-latest) as a pure, deterministic planner — server/lib/src/retention.rs::plan_retention, the Holger::plan_repo_retention adapter, and a holger-server retention dry-run CLI verb. 14 unit tests + 2 adapter tests; HEAVY bench holger.retention_planner (grouped planner vs naive O(n²), both agree on the reclaim set). .nornir/retention-design.md.

• ✅ GC executor. Holger::execute_repo_retention / run_repo_retention carry a RetentionPlan out — gated (dry-run default; GcOptions::execute() / CLI --execute opt-in), safe (content-id-verified delete_artifact; refuses a kept/protected generation), historized (AuditAction::Delete → the append-only Arrow warehouse / skade), and routed through the shared jera job ledger (retention-gc), reusing promotion's job seam. Loose-file backends implement the safe unlink; immutable znippy backends fail closed. 5 executor integration tests

  • a delete_artifact unit test; HEAVY bench holger.retention_executor.

• 🟡 Whole-generation znippy archive-set drop. Per-artifact reclaim ships; dropping a whole superseded znippy generation (re-seal without the reclaimed set) at the immutable-archive level is the remaining archive-level follow-up.

• 🟡 max-age over the live adapter — ArtifactEntry/list() carry no created_at, so age no-ops over the live listing (the planner honours supplied times; keep-last-N + quota already work by version + size). Surfacing archive generation / mtime as the record time lights up age end to end.

• ✅ Property-based retention pin — "keep any blob tagged keep=true." RetentionPolicy.protect_property + a pure ArtifactRecord.protected_by_property flag the planner honours as a hard protection (survives keep-last-N / max-age / quota, annotated protected-by-property); the adapter resolves it against the shared PropertyStore, the executor refuses to unlink a pinned delete row (defense in depth). CLI retention --protect-property keep=true. RED-when-broken (planner + end-to-end adapter): a tagged blob survives a sweep that reclaims its untagged sibling. .nornir/property-policy-design.md.

• ❌ SNAPSHOT rotation; Docker tag retention / delete-untagged.

• Reuse note (now realized in the planner shape): the planner is a pure function over the listing + archive metadata; it should emit jera jobs to perform the drop, and historize the decision into skade (analysis plane), never hand-roll a scheduler.

5. Integrity / CAS (parity §E)

• ✅ blake3 per-chunk verify on read (not optional); sha256 in Cargo index; OCI digest verify.
• ✅ Checksum-verify-on-deploy (X-Checksum-Sha256 / -Sha512, Artifactory's checksum-deploy convention) — the HTTP write path recomputes the advertised strong digest over the received bytes and rejects a mismatch (400) before any backend sees it. Opt-in; strong digests only (sha1/md5 deliberately unsupported). exposed/http.rs::verify_upload_checksum, .nornir/checksum-deploy-design.md.
• 🟡 Content-addressed within an archive only — no cross-archive dedup, no per-archive digest manifest.
• ❌ CAS store proper (sha256 → bytes, one copy) as the artifact-at-rest substrate (artifactory-parity-design.md DESIGN (a)); ✅ format-version pin/reject on read — traits::znippy_format refuses an archive from a newer znippy with a typed, request-facing error (503 / repo disabled) instead of crashing the reader (znippy-format-pin-design.md).

6. Security / policy gate (parity §F — the modgunn seam)

• ✅ Promotion policy gate + OSV feed (see §3).

• ✅ SBOM hosting — holger now HOSTS an SBOM attached to an artifact coordinate (repo, namespace/name/version) and serves it back byte-identical as a first-class servable artifact (server/lib/src/sbom.rs::SbomStore). Three surfaces mirror the search verb: SbomService.AttachSbom (write-gated) / .FetchSbom (open read) gRPC RPCs through the shared nornir-service serve loop, a read-only GET /-/sbom?repo=&namespace=&name=&version= HTTP door, and a holger-server sbom {attach,fetch} CLI verb — all over one on-disk store under <data_root>/holger-sboms, coordinate-keyed via promote's reused path-safe sanitizers. The SBOM data model is CONSUMED from modgunn (modgunn::sbom, re-exported; one build_cyclonedx serialization seam = modgunn's CycloneDX 1.5 verbatim), never twinned (L5). 6 store unit + 2 gRPC + 2 HTTP door tests, each with a RED-when-broken missing/wrong-coordinate not-found guard; gated functional_status emitters on every surface. .nornir/sbom-hosting-design.md. Multi-ecosystem SBOM generation (Maven/npm/PyPI purls + OSV ecosystems) stays a nornir-side extension to track — the purl already carries the ecosystem, so holger hosts a multi-ecosystem document unchanged.

• ❌ CVE/vuln gate before serve (as opposed to before promote) + license policy

  • quarantine repo.

• ❌ Signature / provenance surfacing at serve time beyond the sign (detached-CMS provenance-verify) already baked into the CLI's server-lib dep: cosign/sigstore, in-toto/SLSA attestation hosting, Helm .prov.

• Reuse note: Pass/Warn/Block verdicts must go through modgunn's Decision; the worklist flags a Decision::as_str()/Display twin across holger-lib/holger-server-ui/nornir-viz to collapse.

7. RBAC / auth / tokens (parity §G — modgunn + skidbladnir)

• ✅ mTLS CN + OIDC bearer in the request path (auth.rs); W1 pooled reqwest client + TTL token cache (blake3 key).
• ✅ RBAC write-gate: fail-closed Reader<Writer<Admin, wired into both gRPC (grpc.rs::authorize_write/authorize_admin) and HTTP (http.rs::authorize_http_write), off until a role policy is configured; per-repo ACL scoping shipped (four rbac_* tests).
• ✅ Offline OIDC (airgap auth): a bearer JWT is verified OFFLINE against a JWKS — signature (key selected by the header kid, or the sole key) + exp + iss (must equal issuer_url) + aud (when configured) — with zero network once the keys are in hand. Keys resolve offline-first: (1) in-memory cache, (2) inline jwks in the RON config (fully airgapped — keys travel with the sealed config), (3) a pre-provisioned jwks_path file, (4) [online only] a jwks_uri fetch write-through-persisted to jwks_path. A well-formed JWT that fails verification when a JWKS is resolvable is rejected (offline is authoritative — no /userinfo fallback); only an opaque token / no-resolvable-JWKS falls through to the online /userinfo path, which the airgap mini/embedded tier does not link (fails closed). auth.rs::{validate_oidc_bearer,verify_jwt_offline, resolve_jwks} on jsonwebtoken 9 (ring). 8 tests incl. RED-when-broken expired/wrong-signature/wrong-issuer/wrong-audience rejection + a fully-offline pre-provisioned-file success + fetch-once-then-cached. Ledger corrected 2026-07-14 — the prior "🧭 Not in code" was inaccurate; the offline path is implemented and tested.
• ❌ Groups (role via group membership); ❌ scoped/revocable API + deploy tokens; ❌ anonymous-read toggle; ❌ LDAP/SAML.

8. Search / metadata / REST (parity §H)

• ✅ Search (by name / checksum / GAVC coordinate / path) — a thin cross-repo search over every backend's list() + fetch() + archive_files(), shipped as a pure engine (server/lib/src/search.rs::search_repos) behind THREE surfaces: the SearchService.Search gRPC RPC (served through the shared nornir-service serve loop + auth interceptor), a read-only HTTP door (GET /-/search?name=&namespace=&version=&checksum=&path=&repo=&limit= → JSON), and a holger-server search CLI verb (drives Holger::search_all). Axes AND-compose; name/namespace are case-insensitive substring, version exact, checksum a content sha256 (the one fetch-and-hash axis), path a raw-archive-path substring. 9 engine unit tests + a gRPC RPC test + the HTTP query parser tests, each with a RED-when-broken exclusion assertion; every surface emits a testmatrix cell. .nornir/search-design.md. NOTE: the property search axis is now ✅ shipped too (see the custom-properties item below).
• ✅ Custom properties on artifacts + the property search axis — the last deferred search axis. A coordinate-keyed on-disk store (server/lib/src/properties.rs::PropertyStore, <data_root>/holger-props/<repo>/ <coord>.props.json, key→values, path-safe via promote's reused sanitize_repo/coord_key — no twin) tags an artifact (repo, ns/name/version) with key=value(s). The pure engine gains a property axis (search::search_repos_with_properties + a PropertyLookup trait the store implements; AND-composed, exact value or key-presence; fails closed with no store). Wired on all THREE read surfaces: SearchRequest.properties (proto regen, gRPC), GET /-/search?property=key:value + GET /-/properties HTTP doors, and holger-server search --property CLI. Property WRITE + read now on ALL surfaces: PropertyService.{SetProperty,GetProperties} gRPC RPCs (SetProperty write-gated, repo-scoped, fail-closed — mirrors SBOM attach), PUT/POST/DELETE /-/properties HTTP write door (write-gated) + GET /-/properties read, and holger-server properties {set,get} CLI + Holger::set_artifact_property in-process. 19 tests (7 store + 4 engine incl. 2 RED-when-broken include/exclude + fail-closed, 3 HTTP parse/read/write, 1 gRPC search axis, 4 gRPC PropertyService incl. RED-when-broken write-gate refuses + persists nothing); gated holger-*/propert* functional_status cells. .nornir/properties-design.md.
• ❌ repo CRUD over the API (config is RON-file driven — a deliberate choice, but an admin-gRPC repo-CRUD is a real gap for full).
• REST: gRPC-only by design; the native-protocol HTTP door is not a general REST API.

9. Replication / HA (parity §I)

• ✅ Repository replication (source→dest one-shot copy) — a pure engine (server/lib/src/replication.rs::replicate_pairs) over the existing RepositoryBackendTrait list/fetch/put seam: copy a repo's artifacts from a SOURCE holger store into a DESTINATION, content-id idempotent (an artifact the destination already holds by sha256 is skipped, so a re-run over an unchanged source is a no-op). Read-from-source / write-to-dest only; gated (dry-run default, --execute to write); a read-only destination fails closed. Two surfaces, one engine: the holger-server replicate --from --to CLI verb (two SEPARATE file-backed stores — laptop-testable, no network, via Holger::replicate_into, pairing repos by name) and the ReplicationService.Replicate gRPC RPC (two repos within one server, ADMIN-gated on the destination, fail-closed). 8 engine unit tests + a two-store adapter integration test + 2 gRPC RPC tests, each with a RED-when-broken assertion (a missing artifact after replicate fails; re-run must copy nothing); gated holger-grpc/replicate + holger-cli/replicate functional_status cells. .nornir/replication-design.md.
• 🟡 Continuous / event-driven replication (replicate-on-push, change feed) + multi-site topologies + conflict policy beyond last-writer-converges — the one-shot primitive ships; the scheduled/automated form is the follow-up (route through jera, historize through skade — never hand-roll a daemon). The airgap agent remains the manual cross-boundary form (ConnectorTrait build-archive-carry-import).
• ❌ HA / clustering; ❌ S3 / object backend for scale-out (znippy-iceberg + skade make this feasible — route through skade, don't hand-roll an S3 client).

10. Audit / events / webhooks (parity §J)

• ✅ Audit log — append-only Arrow IPC behind audit::AuditLog, wired into gRPC fetch/put AND the HTTP path (ident, action, artifact, source IP, status, bytes).
• ❌ Webhooks / event bus (emit on push/promote/delete); ❌ audit query surface (the log is written; a servable "who pulled X" query belongs in skade).

11. Ops / observability / misc (parity §K)

• ✅ Prometheus /metrics — a read-only GET /metrics door on the existing HTTP/OCI gateway (mirroring the GET /-/search / GET /-/sbom read doors) emitting Prometheus text 0.0.4: holger_requests_total{verb} (get/put/delete/ promote/search/sbom), holger_bytes_served_total, holger_errors_total, holger_retention_runs_total counters + a holger_repo_artifacts{repo} gauge. Counters are lock-free AtomicU64s behind a process-global metrics::global() registry, bumped at the natural call sites — the HTTP door's one exit point, the gRPC services' single record_audit choke point (so holger's primary gRPC protocol is counted too, not just HTTP — via the one shared Verb::from_audit_action classifier, no twin), and Holger::execute_repo_retention; the render is a pure function over a snapshot + a supplied repo→count list (the gauge's list() I/O runs under spawn_blocking, off the hot path, bounded at 100k/repo). 7 registry unit tests (exact-N-per-verb + parseable-and-reflects-state, both RED-when-broken) + a gated holger-http/metrics functional_status cell. .nornir/metrics-design.md.
• ✅ SBOM hosting (see §6 — SbomService gRPC + GET /-/sbom HTTP door + CLI, over <data_root>/holger-sboms). ❌ quotas / metering; ❌ multi-tenancy / projects.

12. Service, jobs, UI, warehouse, build — the shared-infra seams

• ✅ Service via Skidbladnir — holger service {add|update|delete|status} routes through the Skidbladnir systemd ship (server/cli, commit 13c9f98); default-features off so crypto/archive limbs aren't pulled into the server CLI.
• ✅ Jobs via jera — promotion runs on the shared jera job ledger, bounded (CAP), ignored runtime dir (holger-jobs/). Retention (§4) should reuse the same ledger.
• ✅ UI via facett — holger-demo (wasm + native, facett-wasm-demo harness) + holger-embed (native Box<dyn Facet> adapter, the generalization of nornir's in-process holger tab) both on the shared facett-holger panel; mannequin look-preset switcher as a remote-controllable atom + robot cell (commit 67f1a6c). 🟡 worklist: delete the vendored facett_ui.rs twin (single egui 0.35 now); the generic facett-discovery testmatrix+robot harness (facett-core) should backfill render-only panes.
• ✅ Warehouse via skade — the parity design names skade as the analysis plane (telemetry/SBOM/policy/audit historized there); retention decisions + audit query (§4, §10) should land there rather than in holger.
• ✅ Build via nornir-build-thing consumed by dwarves — holger-build-space fills BuildVariantSet/BuildSurface; the three tiers are declared as data and materialized by the shared build-thing (task #36), the first consumer.

Priority shortlist for the next rounds (crucial-first, self-contained-first)

1. Retention/cleanup planner (§4) — ✅ planner shipped (pure plan_retention + CLI + HEAVY bench). NEXT: the jera executor (drop superseded archive generations) → skade decision log, + per-artifact ingest time so max-age bites over the live listing.
2. Cross-repo search verb (§8) — ✅ shipped, and custom artifact properties (the one deferred axis) — ✅ shipped (PropertyStore + property search axis on gRPC/HTTP/CLI + PropertyService.{Set,Get}Property gRPC RPCs + PUT/DELETE /-/properties HTTP write door + holger-server properties CLI — write/read symmetric on all surfaces; .nornir/properties-design.md). NEXT here: property-based retention/promotion policy; repo CRUD over the API.
3. Prometheus /metrics (§11) — ✅ shipped (read-only GET /metrics door + process-global AtomicU64 registry + per-repo artifact gauge), and gRPC-verb counters — ✅ shipped (the gRPC plane now feeds the same holger_requests_total{verb} family via the single record_audit choke point + one shared Verb::from_audit_action classifier — the HTTP twin collapsed — plus a distinct promote verb). NEXT here: proxy hit/miss counters + request-duration histograms (both additive over the same global registry).
4. Generic/Raw writable repo (§1) — ✅ shipped (RawBackend, ron_repo_type: "raw", path-addressed loose-file bucket + HTTP door + traversal guard + SHA-256-verified delete, reusing search/retention/replication/ metrics unchanged). custom per-blob properties (the deferred §8 axis) — ✅ shipped (see #2 — PropertyStore + property search axis).
5. The crate-split floor + egress gate (§0) — makes mini provably stripped.
6. Offline OIDC (§7) — completes the airgap auth path for mini/embedded.
7. Per-upstream credentials + negative caching (§2) — proxy hardening, BOTH ✅ shipped: negative caching (ron_negative_cache_secs opt-in miss cache in ProxyBackend) + per-upstream credentials (each remote its own ron_upstream_auth — was already wired; ledger corrected). NEXT here: metadata/index-doc caching.
8. Collapse the modgunn Decision twin (§6) — Law #1 hygiene.