cache-hit auto-materialize — for idiots
Class:
guide(beginner) · The zero-knowledge walkthrough. What it IS: auto-materialize-design.md; how to use it: auto-materialize-guide.md.
The problem, in one picture
The forge is smart: if it already built something, it doesn't build it again. It says "cache hit — I know the answer, here's the id tag sha256:abc…" and moves on.
But the NEXT step in the pipeline often needs the actual file the skipped step made. On the machine that built it the first time, the file is still lying around. But on a fresh checkout or a different machine, the skipped step never ran here — so the file was never written. The next step reaches for it and finds an empty folder.
"I skipped compiling (cache hit) — but the test step needs the compiled binary on disk, and it isn't there."
The fix
We already push every built file's bytes into holger (the shared store) when a step runs — that's the other half of this system. So the bytes DO exist somewhere reachable. Auto-materialize just brings them back:
- A step is a cache hit — skipped, not re-run.
- Auto-materialize asks holger for that step's bytes (by the id tag).
- It writes them onto disk in the work folder.
- The next step reaches for the file — and now it's there.
The skipped step's output is restored as if it had run, without actually re-running it. Fast (no rebuild) AND correct (the file is on disk).
Two flavours of "the file"
- Some steps make one file → restored under the step's name.
- Some steps make several files (like holger's embedded / mini / full binaries) → each restored under its own real name, because we kept a little list (a "manifest") of them.
What if it isn't set up?
Nothing changes. If you don't set the work folder (DWARVES_MATERIALIZE_DIR), a cache hit just carries the id tag forward like before — no files written. If there's no store to fetch from, there's nothing to restore, so it quietly does nothing. And if a fetch ever fails, the build doesn't break — that step just gets rebuilt.
The three words to remember
- skip — a cache hit means the step didn't run this time.
- restore — bring the skipped step's file back from holger onto disk.
- no-op — with nothing configured, the whole thing quietly does nothing (safe default).