.nornir/auto-materialize-guide.md

cache-hit auto-materialize — guide

Class: guide · How to USE auto-materialize (what the design doc describes). What it IS: auto-materialize-design.md. Zero-knowledge: auto-materialize-for-idiots.md.

Turn it on

Auto-materialize needs two things: an artifact store (holger — where the bytes live) and a materialize dir (where reused bytes are restored). Both are off by default.

# 1. Build with the holger link and point at a holger data dir (the byte store).
cargo build -p dwarves --features holger
export DWARVES_HOLGER_DIR=/var/lib/dwarves/holger

# 2. Point the forge at a workspace dir for reused outputs to land in.
export DWARVES_MATERIALIZE_DIR=/var/lib/dwarves/work

dwarves run <repo>   # a cache HIT now restores the reused artifact bytes into the work dir

With DWARVES_MATERIALIZE_DIR unset, the forge behaves exactly as before: a hit feeds the artifact id forward, no bytes are written to disk. With no store configured, the knob is a no-op (there is nothing to restore).

Use the seam in code

use dwarves::{IncrementalOptions, run_pipeline_cached};

let mut opts = IncrementalOptions::cached();
if let Some(store) = dwarves::artifact_store::configured() {
    opts = opts.with_artifact_store(store);              // the byte store (push leg)
}
opts = opts.with_materialize_dir("/var/lib/dwarves/work"); // the restore leg

let out = run_pipeline_cached(&pipeline, &dag, &runner, &req, &run_id, &mut cache, &opts);
println!("restored {} files on reuse", out.materialized_files());

On a cache hit / unchanged skip, every reused step's bytes are restored under /var/lib/dwarves/work:

a multi-artifact step restores each member under its recorded name (holger-embedded, holger-mini, …);
a single-artifact step restores under the step name (a lone artifact has no recorded filename).

Call `materialize` directly

use dwarves::artifact_store::materialize;

// `artifact_id` is what the action cache fed forward; `dir` is where to restore.
let restored = materialize(store.as_ref(), &artifact_id, dir, "compile")?;
for (name, path) in restored {
    println!("restored {name} -> {}", path.display());
}

A manifest id restores the whole set; a single id restores under the 4th arg (the basename).
An id the store can't resolve (a runner label, a miss) returns Ok(vec![]) — never an error.
A restore failure inside run_pipeline_cached is logged and skipped; the run stays green.

Observe it

let js = out.state_json();
assert_eq!(js["materialized_files"], 2);      // total restored this run
// js["materialized"] = [{ "step": "compile", "files": ["holger-mini", ...] }, ...]

Test it

cargo test -p dwarves --lib                    # the in-memory double proves the round-trip + no-ops
cargo test -p dwarves --lib --features holger  # + the real holger seal→resolve leg

Key tests: a_cache_hit_auto_materializes_the_reused_output_to_a_fresh_workspace, a_hit_on_a_multi_artifact_step_restores_every_member_by_name, materialize_dir_without_a_store_is_a_noop, materialize_restores_single_manifest_and_no_ops_on_a_label.