.nornir/auto-materialize-design.md · nordisk/dwarves

cache-hit auto-materialize — design

Class: design · Function axis: f0 the auto-restore seam (the whole) · f1 the materialize primitive · f2 the IncrementalOptions::materialize_dir knob · f3 the reuse-branch wiring · f4 observability (materialized / state_json) Strict: this file states only what auto-materialize IS — its parts and why they are shaped this way. Sibling how-to: auto-materialize-guide.md; zero-knowledge walkthrough: auto-materialize-for-idiots.md. Builds on the byte half: holger-artifact-store-design.md.

Auto-materialize is the inverse of the artifact-store push. When a step is skipped as a cache HIT or change-proven UNCHANGED, its artifact_id is fed forward to satisfy downstream deps — but its bytes are never re-produced this run. On a fresh workspace or a second machine, a downstream real step that consumes the upstream output finds nothing on disk. Auto-materialize closes that gap: on reuse, it materializes the reused artifact's bytes back onto disk in a workspace dir, so the downstream step finds the upstream output even though the upstream step was skipped.

The system as one — the function map (f0)

part	lives in	what it is
`f1` `materialize` primitive	`crates/dwarves/src/artifact_store.rs`	`materialize(store, artifact_id, dir, single_name) → Vec<(name, path)>`: fetch by id, restore to disk — a manifest id restores every member under its name; a single-object id restores under the caller's basename; an unresolvable id (a runner label / miss) restores nothing without erroring
`f2` `materialize_dir` knob	`crates/dwarves/src/incremental.rs`	`IncrementalOptions::materialize_dir` + `with_materialize_dir`; the workspace dir reused bytes land in. `None` (default) ⇒ id-only reuse, no disk write
`f3` reuse-branch wiring	`crates/dwarves/src/incremental.rs`	`auto_materialize()` called in BOTH the `CacheHit` and `Unchanged` branches of `run_pipeline_cached`; best-effort, never fails the run
`f4` observability	`crates/dwarves/src/incremental.rs`	`CachedRunReport::materialized` (step → restored names), `materialized_files()`, and `state_json.materialized_files` / `.materialized`

Why it is shaped this way

A primitive on the existing seam, no new infra. materialize is the mirror of put_artifacts on the SAME ArtifactStore trait — it calls store.get. No new store, no new hashing, no new format. The bytes were sealed by holger on the way in; they resolve from holger on the way out. (Law L5: reuse the seam, don't twin it.)
Manifest vs single, disambiguated safely. A multi-artifact step recorded an ArtifactManifest (member name → content_id) as its id; a single-artifact step recorded its own file's content id. materialize recognises a manifest by parsing the fetched bytes and requiring an exact round-trip (m.to_bytes() == bytes), so an ordinary JSON artifact can never be mistaken for one. A manifest restores every member under its recorded name; a lone artifact — which has no recorded filename — restores under a caller-supplied basename (the step name).
Additive + off by default. materialize_dir defaults to None; every existing caller is byte-identical to before (id-only reuse, no disk write). It is also a no-op without an artifact store — the bytes live in the store, so with no store there is nothing to restore.
Fail-open, never blocks a build. A restore error (store unreachable, a member absent) is logged and skipped; the downstream step simply rebuilds. A store the forge can't restore from must never fail the run — the same fail-open discipline as the push leg.
Observable. Every reused step whose bytes were restored is recorded in CachedRunReport::materialized and surfaced in state_json (materialized_files + a per-step materialized list), so the viz / status seam — and the tests — can assert the restore actually happened.

Place in the nordisk constellation

This completes the cache round-trip: put_artifacts (produce → seal into holger) on a RAN step, materialize (resolve → restore to disk) on a reused step. Together they make the action cache a true cross-machine cache: a hit on machine B not only carries the id forward but puts the real bytes where B's next step needs them. Wired into the real forge via DWARVES_MATERIALIZE_DIR (opt-in, a no-op without a store).

Deferred

Single-artifact filename is the step name. A lone artifact records no filename (only its content id), so it restores under the caller's basename (the step name), not its original path. Multi-artifact members restore faithfully (the manifest carries names). Recording a filename for the single case would change the recorded artifact_id (a regression on existing hits) — deferred.
Materialize dir is a flat drop. All restored files land directly under materialize_dir; per-step subdirs / a path map are a later refinement if two steps emit same-named files. See workspace_nordisk/dwarves_agents_problems.md.