nordisk/dwarves

holger artifact store — guide

Class: guide · How to USE the artifact store (what the design doc describes). What it IS: holger-artifact-store-design.md. Zero-knowledge: holger-artifact-store-for-idiots.md.

Turn it on

The holger-backed store lives behind the off-by-default holger feature and is wired only when an endpoint is configured:

# 1. Build the forge with the holger link (pulls holger-server-lib + znippy in-process).
cargo build -p dwarves --features holger

# 2. Point the forge at a holger data dir. Absent ⇒ the store is a NO-OP.
export DWARVES_HOLGER_DIR=/var/lib/dwarves/holger
dwarves run <repo>      # a RealRunner's produced artifacts now seal into holger

With DWARVES_HOLGER_DIR unset (or the holger feature off), the forge behaves exactly as before: the action cache carries the artifact label only, no bytes are pushed. This is safe by default.

Use the seam in code

use std::sync::Arc;
use dwarves::{IncrementalOptions, run_pipeline_cached};

// Resolve the process-configured store (Some(holger) when DWARVES_HOLGER_DIR is set
// and the `holger` feature is compiled; None otherwise — a no-op).
let mut opts = IncrementalOptions::cached();
if let Some(store) = dwarves::artifact_store::configured() {
    opts = opts.with_artifact_store(store);
}
let out = run_pipeline_cached(&pipeline, &dag, &runner, &req, &run_id, &mut cache, &opts);

A step that RANs and reports an artifact_path has its bytes pushed to holger; its recorded artifact_id becomes the true content id of those bytes. A later cache hit — on any machine sharing the holger — feeds that id forward and can resolve the real output with store.get(&artifact_id).

Direct store calls

# #[cfg(feature = "holger")]
# fn demo() -> anyhow::Result<()> {
use dwarves::artifact_store::{ArtifactStore, HolgerArtifactStore};

let store = HolgerArtifactStore::open(std::path::Path::new("/var/lib/dwarves/holger"), "artifacts")?;
let id = store.put_bytes(b"a produced binary")?;   // seals via znippy; returns sha256:…
assert!(store.has(&id));
let bytes = store.get(&id)?.expect("resolvable hit");
# Ok(()) }

• put_bytes / put_file return the sha256:<hex> content id (the brokkr discipline). Idempotent: re-pushing a sealed id is a no-op.
• get returns Ok(None) for an unknown id — absence is never an error.
• Every push seals immutably via holger's znippy store, so the bytes are content-addressed and reboot-safe.

Test it

cargo test -p dwarves                    # default: in-memory double proves the wiring + no-op
cargo test -p dwarves --features holger  # adds the real holger znippy push→resolve roundtrip

Bench it (Loki, authoritative)

cd bench && cargo run --release --example nornir-bench --features holger
# arms: dwarves.holger_put (seal throughput), dwarves.holger_get (resolve throughput)