.nornir/holger-artifact-store-design.md · nordisk/dwarves

holger artifact store — design

Class: design · Function axis: f0 the artifact-store seam (the whole) · f1 the ArtifactStore trait · f2 the holger-backed impl · f3 the incremental wiring · f4 configuration / no-op gating Strict: this file states only what the artifact store IS — its parts and why they are shaped this way. Sibling how-to: holger-artifact-store-guide.md; zero-knowledge walkthrough: holger-artifact-store-for-idiots.md.

The artifact store is the byte half of dwarves's action cache. The cache records a step's decision (input_id → result + artifact_id), but an artifact_id is only a sha256: label. The bytes that label points at must live somewhere the forge can retrieve on a fresh workspace or a second machine — so dwarves pushes them into holger, the constellation's artifact registry, whose storage is immutable, content-addressed, and sealed via znippy. dwarves keeps NO blob store of its own: reuse the shared registry (Law L5), never twin a hand-rolled CAS.

The system as one — the function map (f0)

part	lives in	what it is
`f1` `ArtifactStore` trait	`crates/dwarves/src/artifact_store.rs`	the seam: `put_bytes`/`put_file` → content id, `get`/`has` by content id; `put_artifacts` + `ArtifactManifest`/`ArtifactRef` capture a multi-artifact set behind one id
`f2` `HolgerArtifactStore`	`crates/dwarves/src/artifact_store.rs`	in-process holger-backed impl (writable znippy repo); behind the `holger` feature
`f3` incremental wiring	`crates/dwarves/src/incremental.rs`	`IncrementalOptions::artifact_store` + `with_artifact_store`; a RAN step's every produced file pushes (`StepOutcome::artifact_files`), its recorded `artifact_id` becomes the true content id (or, for N>1, the manifest's)
`f4` `configured()` gate	`crates/dwarves/src/artifact_store.rs`	resolves the process store from `DWARVES_HOLGER_DIR`, or `None` (no-op)

Why it is shaped this way

A trait, not a concrete store. The forge's engine depends on Option<Arc<dyn ArtifactStore>>, so the storage backend is swappable and testable (an in-memory double stands in for holger in the incremental-wiring test). The one live impl pushes to holger.
holger owns the bytes + the seal. dwarves calls holger's in-process API (read_ron_config → instantiate_backends → a writable-znippy RustRepoZnippy backend's put/fetch). holger seals the bytes into an immutable, blake3-addressed znippy archive. dwarves never hand-rolls hashing or a blob layout — brokkr::artifact::content_id (sha256:<hex>) stays the id discipline, mapped 1:1 onto a holger ArtifactId.
The index stays; the bytes move. The action cache keeps its input_id → artifact_id INDEX exactly as before — only the bytes those ids name now live in holger instead of a local dir. This is what turns a same-machine-only cache into a cross-machine one.
Every artifact, not just the first. A variant build can emit several binaries (holger's embedded/mini/full). StepOutcome::artifact_files carries the whole set and put_artifacts pushes all of them: one file records its own content id (byte-identical to the single-artifact path); more than one additionally seals a small ArtifactManifest — the sorted set of member name → content_id — whose content id becomes the step's artifact_id. The index thus carries the entire set behind one id: resolve the manifest, then resolve each member. The sort makes the manifest id deterministic, so a byte-identical rebuild still hits.
Cross-machine hits are the whole point. A local file store could never satisfy a hit on a second machine; holger, a shared registry, can. That is the reason the bespoke local CAS was ripped out.
Additive + gated + fail-open. With no holger endpoint configured (DWARVES_HOLGER_DIR unset) the store is None — a NO-OP byte-for-byte identical to the pre-store forge (the label is fed forward, no bytes pushed). The holger feature is OFF by default, so the default/published forge links none of holger/tonic/znippy. A push that fails (registry unreachable) falls back to the runner's own artifact_id: a store miss never fails a build.

Place in the nordisk constellation

holger is the SHARED artifact registry (../holger, holger-server-lib). dwarves is one of its in-process consumers (like nornir's embedded registry pane). The seal mechanism is znippy — the same immutable, content-addressed pack format holger uses for every artifact type. This mirrors the backlog's Skidbladnir SKIDCAS1 direction ("replace hand-rolled with real znippy/registry"), applied to the forge.

Deferred

skade/Iceberg ledger. Recording the index/run facts in the warehouse (skade/Iceberg) as the immutable ledger is deferred; the existing action-cache index is retained. See workspace_nordisk/dwarves_agents_problems.md.
gRPC / remote holger. The current impl is in-process (a local writable znippy archive per data dir). A true remote push over holger's gRPC client (a shared, network holger for a real multi-machine fleet) is the next step; the in-process seam is the testable insertion point for it.