.nornir/artifactory-parity-design.md

Holger → Artifactory/Nexus parity — gap list + design (DESIGN ONLY)

Status: design + plan, not implemented. No code here. This captures the gap vs JFrog Artifactory / Sonatype Nexus and three focused designs: an immutable znippy artifact-at-rest (CAS), a policy gate before Holger serves, and Artifactory-style promotion. The nornir warehouse is the analysis plane (telemetry/SBOM/policy/audit historized there); the CAS is the serving plane.

Holger today (ground truth)

Config = directed graph in RON (holger.ron): ExposedEndpoint ← Repository → StorageEndpoint (+ Repository → Repository upstream proxy), resolved by wire_holger() into per-endpoint FastRoutes O(1) dispatch (server/lib/src/exposed/fast_routes.rs).
One plugin seam: RepositoryBackendTrait (traits/src/lib.rs); registry is the backend_from_config() match (server/lib/src/repository.rs).
Two front doors, one backend: native registry protocol over HTTP (handle_http2_request) for stock cargo/pip/npm/docker, + holger-native gRPC (Repository/Archive/Admin services, server/lib/src/grpc.rs).
Formats present (server/repository/): znippy-{maven,python,rust,npm,nuget,go, gem,deb,rpm,helm,docker,conda,composer,package} + file-rust (writable Cargo sparse) + file-oci (writable OCI v2). Depth varies (Python/npm/NuGet/Go/ Docker/Helm/Composer real protocol; Maven/Deb/RPM/Conda passthrough; Gem skeleton).
Storage: immutable znippy archives (read-only, blake3 content-addressed, random-access extract_file), filesystem (file-rust, writable), disk OCI.
Proxy: ProxyBackend does upstream fall-through but no cache write-back.
Auth: mTLS CN + OIDC bearer in the request path (auth.rs); enforcement is binary (any valid identity may write — no role/ACL mapping).
Agent: internet-side ConnectorTrait (crates.io/PyPI/Nexus) builds archives, carries them across the airgap. Mannequin mocks PKI/OIDC/Nexus for tests + bench.

nornir security surface Holger reuses (exact functions, `nornir/src/security.rs`)

scan(repo) -> SecurityReport; components(repo) / components_warehouse_first; query_vulns(&[Component]) (airgap-first: local rustsec mirror else OSV) / advisory_db_query (offline) / osv_query (online); SecurityReport::{to_cyclonedx (CycloneDX 1.5), license_tally, vuln_count}; warm(wh, repo, scan_root); prepare_path_deps; advisory_db_path / update_advisory_db (NORNIR_ADVISORY_DB, NORNIR_ADVISORY_DB_URL — can point at Holger itself, which serves the advisory-db repo). Surfaced by nornir as gRPC Mimir.SecurityScan, CLI nornir security, viz Security tab; warehouse tables sbom_components, vuln_findings. Integration: library mode (vendor security.rs or a nornir-security crate) for the airgap secure side; RPC only where a nornir server is reachable. NB security.rs is cargo-only today — multi-ecosystem (Maven/ npm/PyPI purls + OSV ecosystems) SBOM/vuln is a nornir-side extension to track.

znippy as immutable-artifact substrate (`znippy`)

Arrow-IPC framed multi-index archive (znippy-common/src/index.rs v0.7: [sub-indexes][manifest][ZNPYMIDX magic][manifest_offset]); ManifestEntry {pkg_type, repo, module_name, index_offset/len, row_count}; reserved __znippy_lookup__ + __znippy_trie__ give O(log n) single-file random access; blake3 per-chunk verify on read (not optional); GATLING parallel encode (~2.4 GB/s compress, ~9–10 GB/s decompress; SAVE stage GATLING too); DuckDB/Polars readable (the index is Arrow IPC); selective extract by pkg_type/repo. CLOSED (2026-07-22): the format version IS enforced, holger-side. traits::znippy_format pins the on-disk format holger parses (MAX_SUPPORTED_ZNIPPY_FORMAT) and refuses a newer archive with a typed ArchiveFormatError naming archive · recorded version · ceiling — before the reader sees it. A static archive is refused at open (that repo disabled, the rest serve); a writable repo's live archive is re-gated per request and answers 503 with the reason. Equal/older/absent versions read exactly as before. Deliberately holger's own pin, not a re-export of znippy's internal check, since a released holger may link a znippy that has none. See znippy-format-pin-design.md; RED-when-broken cover in server/repository/znippy-rust/tests/format_version_gate.rs

server/lib/tests/graceful_backend_init.rs.

GAP LIST — Holger vs Artifactory/Nexus (✅ have · 🟡 partial · ❌ missing)

A. Formats/protocols. ✅ Cargo, PyPI, npm, NuGet, Go, Docker/OCI, Helm, Composer. 🟡 Maven (no maven-metadata.xml/SNAPSHOT resolve/checksum sidecars), Deb/RPM (no signed metadata), Conda (passthrough), Gem (skeleton). ❌ Generic/Raw writable repo; long-tail (APK, Terraform registry, CocoaPods, HuggingFace, …).

B. Remote/proxy/virtual. ✅ proxy fall-through and cache write-back — an upstream hit is written through into the writable primary, best-effort (server/lib/src/proxy.rs::ProxyBackend::fetch, tested by fetch_caches_upstream_hit_into_writable_primary). ✅ virtual/group repos shipped: a read-only aggregate over ordered member backends (server/lib/src/group.rs::GroupBackend, ron_repo_type: "group" + ron_members in repository.rs), tested by group_repo_wires_and_resolves_across_members. Still ❌ per-upstream credentials, negative/metadata caching, online/offline toggle.

C. Promotion/staging/lifecycle (the Artifactory headline). ❌ promotion pipelines (dev→staging→release); ❌ staging repos; ❌ build info; ❌ release bundles.

D. Retention/cleanup. ✅ retention planner (keep-last-N/max-age/keep-min/ quota/protect-latest) as a pure deterministic function + holger-server retention dry-run CLI (server/lib/src/retention.rs, .nornir/retention-design.md). 🟡 GC executor (jera job dropping superseded archive generations + skade decision log) deferred; ❌ SNAPSHOT rotation, Docker tag retention, delete-untagged (complicated by immutable archives — retention is archive-set level).

E. Integrity/CAS. ✅ blake3 per-chunk verify; sha256 in Cargo index; OCI digest verify. ✅ checksum-deploy — the HTTP write path recomputes a client-advertised X-Checksum-Sha256/-Sha512 over the received bytes and rejects a mismatch (400) before any backend sees the write (server/lib/src/exposed/http.rs::verify_upload_checksum, opt-in, strong digests only, 8 checksum_* unit tests; .nornir/checksum-deploy-design.md). 🟡 content-addressed within an archive — no cross-archive dedup, no per-archive digest manifest. ❌ gRPC-path checksum field; ❌ checksum sidecar synthesis (.sha1/.md5).

F. Security/policy gate (the nornir hook). ❌ CVE/vuln gate before serve or promotion; ❌ license policy; ❌ signature/provenance (cosign/sigstore, in-toto/SLSA, Helm .prov); ❌ quarantine. 🟡 SBOM generated in nornir but not hosted/attached.

G. RBAC/auth/tokens. ✅ mTLS+OIDC in path. ✅ roles enforced: fail-closed Reader<Writer<Admin write-gate wired into both gRPC (server/lib/src/grpc.rs authorize_write/authorize_admin) and HTTP (server/lib/src/exposed/http.rs authorize_http_write), off unless a role policy is configured; per-repo ACL scoping shipped too (tested by rbac_on_reader_cannot_write_or_delete, rbac_on_writer_uploads_admin_deletes, rbac_on_anonymous_write_fails_closed, rbac_per_repo_acl_scopes_the_write_gate). Still ❌ groups, scoped/revocable API+deploy tokens, anonymous-read toggle, LDAP/SAML.

H. Search/metadata/REST API. ❌ no REST API (gRPC-only by design); ❌ search (name/checksum/property/GAVC), custom properties, repo CRUD over API.

I. Replication/HA. ❌ replication (push/pull/multi-site), HA/clustering, S3/object backend for scale-out (note: znippy-iceberg + skade make S3 feasible).

J. Audit/events/webhooks. ❌ audit log (who pulled/pushed/promoted what — critical given the "auditability is the point" pitch); ❌ webhooks/event bus.

K. Misc. ❌ SBOM hosting; ❌ quotas/metering; 🟡 only /healthz,/readyz (no Prometheus metrics); ❌ multi-tenancy/projects.

DESIGN (a) — Immutable znippy archive = artifact-at-rest (CAS)

Goal. Every artifact is content-addressed (sha256) bytes inside an immutable, blake3-verified, signed znippy archive; identical bytes stored once; reads verify; the archive is the airgap-carriable unit.

Layout.

<root>/holger/cas/
  blobs/<sha256[0:2]>/<sha256>            # raw artifact bytes, write-once
  archives/<repo>/<archive-uuid>.znippy   # one immutable znippy per repo generation
  manifests/<repo>/<generation>.json      # archive manifest (below)

Blob digest = sha256 (client-facing address + dedup key; matches OCI digests); znippy blake3 stays the on-read integrity check.
Archive manifest (the piece missing from raw znippy): {schema, repo, generation, znippy_format_version (PINNED — reject-with-message on mismatch), created/created_by, archive_sha256, entries[{artifact_id, path, blob_sha256, size, blake3, format}], signature{ed25519/cosign over archive_sha256}, policy{scan_id, verdict, sbom_ref}}.

Write (ingest). auth → route → stream sha256 (mirror file-rust's streaming + (len,mtime) memo) → dedup if blob exists → stage → on seal GATLING-encode the set into one .znippy (SAVE stage GATLING per the law) → compute archive_sha256, write+sign manifest, fsync, atomic rename = immutable generation N → emit events + historize manifest+SBOM to warehouse.

Read (serve). unchanged hot path (fetch → trie/lookup → extract_file, blake3-verified) + verify archive_sha256+signature once on open (cache verdict). Bad-archive open is contained per-repo (fixes the WISHLIST.md server-crash).

Surface. gRPC ArchiveService: SealArchive, VerifyArchive, ArchiveManifest. CLI holger-agent pack --from <dir|crates.io|nexus> --to znippy --repo r --sign k (also closes the WISHLIST directory→znippy gap), … verify.

Warehouse. new holger_artifacts, holger_archives alongside sbom_components/vuln_findings; object backend reuses skade's S3 path.

DESIGN (b) — Policy gate before serve/promote

Goal. Before handing out (or promoting) an artifact: CVE + license + signature/ provenance policy → block|warn|pass; cached + historized; airgap-first.

Placement. (1) At promotion boundary (default) — gate runs once when an artifact crosses dev→staging→release; the release repo holds only pass-verdict artifacts, so the serve path just checks a cached flag (matches Xray). (2) Pre- serve (optional, stricter) — hook between route and fetch: cached verdict for (blob_sha256, policy_version); block+enforcing → 403+reason; absent → scan or fail-closed/open per config.

Policy (RON): PolicyGate{enforcement: Block|Warn|Off, max_critical_cves, max_high_cves, license_allow[], license_deny[], require_signature, require_provenance, fail_open_on_scanner_error}.

Flow. resolve SBOM via nornir::security::scan (or components_warehouse_first

cached vuln_findings) → vuln gate (vuln_count + per-severity vs thresholds; airgap via local rustsec mirror served by Holger) → license gate (license_tally vs allow/deny) → signature/provenance verify (manifest sig + in-toto/SLSA + Helm .prov) → persist to_cyclonedx() SBOM + host it → verdict {pass|warn|block, scan_id, reasons[]} cached on (blob_sha256, policy_version, advisory_db_rev), historized to warehouse policy_verdicts. Idempotent / re-runnable on policy change. Emits policy_pass|policy_block.

Surface. gRPC ScanArtifact, GetSbom, GetVerdict; CLI holger policy scan, holger sbom get.

DESIGN (c) — Artifactory-style promotion (dev→staging→release)

Goal. Move an immutable artifact through stages with no rebuild — promotion is a manifest/pointer move on the content-addressed blob, gated at the boundary.

Model. stages = repos (rust-dev writable, rust-staging quarantine, rust-prod immutable znippy read-only). Promotion = include the existing blob_sha256 in the target repo's next generation manifest (no byte copy — bytes already in cas/blobs/). Promotion record {artifact_id, blob_sha256, from, to, promoted_by, at, gate_verdict, previous_generation, new_generation}.

Flow. CI → dev (fast, no gate). holger promote <id> --from rust-dev --to rust-staging → gate (b); block keeps it in dev. soak. holger promote … --to rust-prod → gate re-checks (sig + fresh advisory-db) → seals a new immutable prod generation (a). Each promotion: audit-logged + warehouse-historized + emits artifact_promoted. Release bundle: ReleaseBundle{name, version, [blob_sha256…], signature} promoted atomically (one gate run, one generation/target).

Gate interaction. gate runs at the promotion boundary (config which boundaries enforce — typically staging→prod Block, dev→staging Warn); serve path stays hot.

Surface. gRPC PromotionService: Promote, ListStaging, PromotionHistory; CLI holger promote <id> --from --to [--bundle name@ver], holger staging list, holger promote --history. Requires admin/promoter role (closes RBAC for promote).

Warehouse. holger_promotions, policy_verdicts, holger_release_bundles = full audit + lineage (seeds Gap J).

ROADMAP

graph TD
  ROOT["Holger → Artifactory/Nexus parity"]
  ROOT --> P0["P0 · Foundations + airgap gate"]
  ROOT --> P1["P1 · Lifecycle + integrity + protocol parity"]
  ROOT --> P2["P2 · Enterprise scale + breadth"]

  P0 --> A0["(x) Contain bad-archive open per-repo + pin znippy format version"]
  P0 --> A1["( ) holger-agent directory→znippy pack + re-pack"]
  P0 --> A2["( ) Immutable znippy CAS: sha256 blobs + dedup + archive manifest (a)"]
  P0 --> A3["( ) Policy gate v1: CVE+license via nornir::security, airgap rustsec, CycloneDX SBOM (b)"]
  P0 --> A4["(x) DONE — RBAC enforcement: mTLS CN / OIDC claims → reader/writer/admin (grpc.rs + exposed/http.rs)"]
  P0 --> A5["( ) Audit log → warehouse"]

  P1 --> B0["( ) Promotion dev→staging→release + staging + gate-at-boundary (c)"]
  P1 --> B1["( ) Archive signing (ed25519/cosign) + verify + Helm .prov"]
  P1 --> B2["(x) DONE — Proxy cache write-back (proxy.rs) + virtual/group repos (group.rs); remote creds still open"]
  P1 --> B3["( ) Generic/Raw repo; Maven metadata+checksums; Deb/RPM signed metadata"]
  P1 --> B4["( ) Release bundles + retention/cleanup + archive compaction"]
  P1 --> B5["( ) Prometheus metrics + webhooks/events"]

  P2 --> C0["( ) REST admin+search API (AQL-like) + custom properties + repo CRUD"]
  P2 --> C1["( ) Replication + S3/object backend + RWX/HA"]
  P2 --> C2["( ) SLSA/in-toto provenance verify; quotas; multi-tenancy"]
  P2 --> C3["( ) Scoped/revocable API+deploy tokens; LDAP/SAML"]
  P2 --> C4["( ) Long-tail formats (APK, Terraform, CocoaPods, HuggingFace, …)"]

  A2 --> B0
  A3 --> B0
  A2 --> B1
  A3 --> A5
  B0 --> B4

Key notes

Because znippy archives are already content-addressed + immutable, Artifactory promotion becomes a manifest/pointer move, not a copy — the hard part (CAS) is ~80% there.
The airgap story is real and wired: nornir::security::query_vulns is airgap-first against a local rustsec mirror, and that mirror can be served by Holger itself (NORNIR_ADVISORY_DB_URL → Holger). Use library mode on the secure side.
The two P0 robustness bugs that gated everything are BOTH CLOSED: a corrupt archive disables only its own repo (graceful_backend_init.rs), and the znippy format version is pinned + rejected-with-message on read (znippy-format-pin-design.md). The "immutable archive at rest" claim now has its prerequisites.
Holger's strongest area is formats (13 ecosystems); the table-stakes gaps are promotion, staging, retention, audit, RBAC enforcement, REST/search, replication, policy gate, signing.