crates/nornir-build-thing/tests/toolchain_images_live.rs
//! **The toolchain images the containers rule rests on — built through their own
//! door, and their digests measured against what a repo actually pins.**
//!
//! ## Why this file exists
//!
//! Measured on oden 2026-08-27, and it is the whole reason. Both halves of the
//! "containers build things — never the host's libs" law had a complete engine,
//! a committed Containerfile, and **no way for an operator to produce the image**:
//!
//! * `cargo_container.rs` is 91 KB on main and
//! [`DraupnirCargoRunner::build_toolchain_image`] writes the crate's OWN
//! embedded Containerfile into a context and builds it — with ZERO callers
//! anywhere in the constellation. No CLI verb, no test, nothing.
//! `gunnar/.nornir/build-variants.toml` pins `nordisk-cargo-toolchain` by
//! digest `f29e517c…`; the only image on the box was `nordisk-rust-toolchain`
//! (`f7388c7b…`), a leftover from the superseded `cargo-container` branch — a
//! different name under a different digest, so no declaration could resolve it.
//! * The kernel kind was worse. `KERNEL_TOOLCHAIN_CONTAINERFILE` was
//! materialized in exactly one unit test and nowhere else, and
//! `DraupnirKernelRunner` had no image-build method at all. So a
//! `[[build.kernel]]` that names no image falls back to
//! [`DEFAULT_TOOLCHAIN_IMAGE`](nornir_build_thing::kernel::DEFAULT_TOOLCHAIN_IMAGE) —
//! bare `docker.io/library/debian:trixie` — and has to `apt-get install` its
//! compiler at build time. A real chained build died exactly there after
//! 263.2 s: `Temporary failure resolving 'deb.debian.org'`, then `Unable to
//! locate package gcc / make / flex / bison / bc / libelf-dev / libssl-dev`.
//!
//! A law nobody can execute reads exactly like a law being obeyed. That is what
//! these two gates retire.
//!
//! ## What they assert, and what they refuse to assert
//!
//! They assert the ARTIFACT: after the build, the container engine is asked for
//! the image's manifest digest and that string is what the test reasons about —
//! never the builder's exit status (LAW 2). An unreadable digest is a FAILURE
//! here, not a quiet pass, because two unread digests trivially "agree".
//!
//! They do NOT assert that a digest equals a constant, and until 2026-08-30 the
//! reason given here was a BELIEF ("a container image built from the same
//! Containerfile does not generally reproduce byte-for-byte"), asserted in prose
//! and never measured — the exact shape LAW 2 exists to catch. It is now
//! measured, by the third gate in this file
//! ([`two_mints_of_the_kernel_toolchain_are_measured_against_each_other`]), and
//! the belief was right for a reason narrower and more actionable than the
//! sentence claimed — narrow enough that on 2026-08-30 the recipe was changed
//! until it stopped being true of THIS image, and that gate now holds the
//! kernel toolchain's id constant across mints. See its own doc for the numbers.
//! These two gates still only REPORT, because they are about the image existing
//! and carrying its tools. They REPORT the
//! digest beside every declared pin in the constellation, and enforce the match
//! only when the operator asks (`NORDISK_TOOLCHAIN_EXPECT_PIN=1`). The number is
//! the deliverable.
//!
//! The cargo half asks a second question the kernel half does not have: not
//! "is there a compiler" but "is there a rust-std for every road this image
//! claims". It cannot be answered by asking rustc, because
//! `rustc --print target-list` prints the 322 triples it was COMPILED TO KNOW —
//! measured in `sha256:035814c4…` on oden 2026-08-27, that list contained
//! `wasm32-unknown-unknown` while the target's own libdir did not exist. So the
//! gate COMPILES, once per declared triple, and then reads the wasm module's
//! first four bytes: `00 61 73 6d`. The same script against the gnu target
//! answers `7f 45 4c 46` (ELF), which is what makes the magic an assertion
//! rather than an identity value.
//!
//! ⚠ The finding that came out of running them, 2026-08-27: a locally REBUILT
//! image cannot satisfy a digest pin, because the digest is per-build; a pinned
//! digest needs an image PULLED from a shared registry; `image_sha256` in a
//! declaration is a per-box fact, not a constellation-wide one.
//!
//! ⚠⚠ And the follow-up, measured 2026-08-30 (lane AT): the **image id** is
//! per-build too. Lane AO proved the id — unlike the manifest digest — survives
//! a `podman save`/`load` courier, which is what makes an offline toolchain
//! possible at all; it does NOT follow that the id is a property of the recipe.
//! Two mints of THIS Containerfile, 90 s apart on one box with the mirror state
//! frozen, produced two different ids. So re-pinning a declaration from a
//! manifest digest to an id changes the pin from *a per-box fact that cannot
//! travel* to *a per-MINT fact that can*; it does not make the pin derivable.
//!
//! ⚠⚠⚠ **Both of those are RETIRED for the KERNEL toolchain image on
//! 2026-08-30 (lane AV), and the reasoning under them was wrong in an
//! instructive way.** "A digest is per-build" is not a law of container images;
//! it is a property of a recipe that leaves its inputs and its clock unpinned.
//! Pin both and a local rebuild lands on the same 64 hex characters as everybody
//! else's: four mints of this Containerfile into four fresh empty stores over
//! fifteen minutes gave one image id (`3f632569f198…`) AND one manifest digest
//! (`78eaa55a651d…`), and the bollard door agreed with the CLI door. A registry
//! is now a way to skip 50 s and 1.2 GB, not the only way to make a pin true.
//! Both claims still stand for the CARGO toolchain image, which has had none of
//! that work.
//!
//! ## How to run
//!
//! ```bash
//! # a reachable podman/Docker socket:
//! # systemctl --user enable --now podman.socket
//! cargo test -p nornir-build-thing --features cargo-container \
//! --test toolchain_images_live -- --ignored --nocapture
//! cargo test -p nornir-build-thing --features kernel-build \
//! --test toolchain_images_live -- --ignored --nocapture
//! ```
//!
//! Each pulls a base image and installs a toolchain — minutes, and GBs on disk.
//! They join the `--ignored` heavy pass beside the kernel live gates and are
//! never part of a default `cargo test`.
#![cfg(any(feature = "cargo-container", feature = "kernel-build"))]
use std::collections::BTreeMap;
use std::path::{Path, PathBuf};
use std::process::Command;
/// Ask the engine for an image's manifest digest. `None` when the image is not
/// there or the tool cannot be run — a caller must treat that as a failure,
/// never as agreement.
fn image_digest(tag: &str) -> Option<String> {
for engine in ["podman", "docker"] {
let Ok(out) = Command::new(engine)
.args(["inspect", "--format", "{{.Digest}}", tag])
.output()
else {
continue;
};
if out.status.success() {
let d = String::from_utf8_lossy(&out.stdout).trim().to_string();
if !d.is_empty() {
return Some(d);
}
}
}
None
}
/// Every `[build.toolchain]` pin checked in beside this checkout, as
/// `(repo, image, Option<sha256>)`. A tiny line scan on purpose: this file must
/// keep working when the schema grows, and it is reporting, not parsing.
fn declared_pins(constellation: &Path) -> Vec<(String, String, Option<String>)> {
let mut out = Vec::new();
let Ok(entries) = std::fs::read_dir(constellation) else { return out };
let mut repos: Vec<PathBuf> = entries.filter_map(|e| e.ok().map(|e| e.path())).collect();
repos.sort();
for repo in repos {
let decl = repo.join(".nornir").join("build-variants.toml");
let Ok(text) = std::fs::read_to_string(&decl) else { continue };
let Some(start) = text.find("\n[build.toolchain]") else { continue };
let section = &text[start + 1..];
let end = section[1..].find("\n[").map(|i| i + 1).unwrap_or(section.len());
let section = §ion[..end];
let field = |key: &str| -> Option<String> {
section.lines().find_map(|l| {
let l = l.trim();
let rest = l.strip_prefix(key)?.trim_start();
let rest = rest.strip_prefix('=')?.trim();
Some(rest.trim_matches('"').to_string())
})
};
let Some(name) = repo.file_name().map(|n| n.to_string_lossy().to_string()) else {
continue;
};
if let Some(image) = field("image") {
out.push((name, image, field("image_sha256")));
}
}
out
}
/// The constellation root — the directory the sibling checkouts live in.
fn constellation() -> PathBuf {
Path::new(env!("CARGO_MANIFEST_DIR"))
.ancestors()
.nth(3)
.expect("crates/nornir-build-thing/../../.. is the constellation root")
.to_path_buf()
}
/// Report the built digest against every declared pin, and enforce only when
/// asked. Shared by both gates so the two cannot drift into two policies.
fn report_against_pins(digest: &str, require_declared_by: Option<&str>) {
let pins = declared_pins(&constellation());
if let Some(who) = require_declared_by {
assert!(
pins.iter().any(|(_, image, _)| image.contains(who)),
"no repo beside {} declares an image containing {who:?} — then nothing consumes \
this image and the containers rule is decorative here",
constellation().display()
);
}
let mut mismatched = Vec::new();
for (repo, image, pin) in &pins {
match pin {
Some(p) if digest.ends_with(p.as_str()) => {
eprintln!("[pin] {repo}: {image} @ {p} — MATCHES this box");
}
Some(p) => {
eprintln!("[pin] {repo}: {image} @ {p} — this box built {digest}");
mismatched.push(format!("{repo} pins {p}"));
}
None => eprintln!(
"[pin] {repo}: {image} — pinned by TAG only, which is a name someone can move"
),
}
}
if std::env::var("NORDISK_TOOLCHAIN_EXPECT_PIN").is_ok() {
assert!(
mismatched.is_empty(),
"NORDISK_TOOLCHAIN_EXPECT_PIN was set and this box cannot produce the pinned \
digest(s): {}. Either re-pin to {digest}, or find out why the committed \
Containerfile no longer builds what was pinned.",
mismatched.join(", ")
);
}
}
/// **Ask the IMAGE, with the network OFF.** The one place either gate runs
/// something inside a built image, so the two cannot drift into two policies
/// about what "the image contains it" means.
///
/// `--network none` is the whole point and not a detail: an image that has to
/// reach a package index or crates.io to answer is an image that fails in a real
/// build, which is exactly what both halves of this file were written after.
/// A tool that cannot be run at all is a FAILURE, never a shrug — an unread
/// answer and an absent one look identical.
fn in_image(tag: &str, what: &str, script: &str) -> String {
let out = Command::new("podman")
.args(["run", "--rm", "--network", "none", tag, "sh", "-c", script])
.output()
.unwrap_or_else(|e| panic!("ask {tag} for {what}: {e}"));
assert!(
out.status.success(),
"{tag} cannot {what} with the network OFF — this image would have to fetch it, \
which is the failure this gate exists for.\n--- stdout ---\n{}\n--- stderr ---\n{}",
String::from_utf8_lossy(&out.stdout),
String::from_utf8_lossy(&out.stderr)
);
String::from_utf8_lossy(&out.stdout).trim().to_string()
}
/// Assert the digest is a real manifest digest, then report it. `None` is a
/// failure: an image that reports nothing after a build that claimed success is
/// the exact false-green this file exists for.
fn assert_digest(tag: &str) -> String {
let digest = image_digest(tag)
.unwrap_or_else(|| panic!("{tag} reports no manifest digest after a successful build"));
assert!(
digest.starts_with("sha256:") && digest.len() > 20,
"the digest must be a real manifest digest, got {digest:?}"
);
eprintln!("[toolchain] {tag} digest {digest}");
digest
}
// ── the cargo kind ────────────────────────────────────────────────────────────
/// ★ **The cargo toolchain image gets a door, and the door is measured.**
#[cfg(feature = "cargo-container")]
#[test]
#[ignore = "builds a ~1.8 GB toolchain image through podman/Docker"]
fn the_declared_cargo_toolchain_image_can_be_built_and_names_its_digest() {
use nornir_build_thing::cargo_container::{
cargo_toolchain_image_targets, DraupnirCargoRunner, CARGO_TOOLCHAIN_CONTAINERFILE,
CARGO_TOOLCHAIN_IMAGE,
};
// Prove the embedded Containerfile is really there before spending minutes:
// an empty const would produce a trivially-passing "built nothing".
assert!(
CARGO_TOOLCHAIN_CONTAINERFILE.contains("FROM"),
"the embedded cargo Containerfile is empty or has no FROM — nothing would be built"
);
// …and that it names a target set at all. An empty list would make the whole
// std loop below iterate zero times and pass, which is the same defect as a
// check that cannot go red.
let carried = cargo_toolchain_image_targets();
assert!(
carried.len() >= 3 && carried.iter().any(|t| t == "wasm32-unknown-unknown"),
"the Containerfile must declare the WASM road the chained build refused BY NAME \
on 2026-08-27, plus the two it already had — got {carried:?}"
);
let tag = std::env::var("NORDISK_TOOLCHAIN_TAG")
.unwrap_or_else(|_| format!("{CARGO_TOOLCHAIN_IMAGE}:live"));
let ctx = std::env::temp_dir().join(format!("nordisk-cargo-toolchain-ctx-{}", std::process::id()));
let built = DraupnirCargoRunner::new()
.build_toolchain_image(&ctx, &tag)
.unwrap_or_else(|e| panic!("build {tag} from the embedded Containerfile: {e:#}"));
eprintln!("[toolchain] built under tag {built}");
let digest = assert_digest(&tag);
// ── the image's own contents, network OFF ─────────────────────────────────
//
// The kernel half asks for compilers; the cargo half has to ask for STDs,
// and it cannot ask rustc what it knows. MEASURED in `sha256:035814c4…` on
// oden 2026-08-27: `rustc --print target-list` printed 322 triples including
// `wasm32-unknown-unknown`, while `rustc --print target-libdir --target
// wasm32-unknown-unknown` named a directory that did not exist. So the
// question is asked by COMPILING, once per declared triple, a crate that
// needs rust-std (String is std, not core), and the assertion is on the
// artifact — not on rustc's opinion and not on an exit status.
for t in &carried {
let script = format!(
"printf 'pub fn probe() -> String {{ String::from(\"std\") }}\\n' > /tmp/p.rs; \
rustc --edition 2021 --crate-type=lib --target {t} -o /tmp/p-{t}.rlib /tmp/p.rs \
&& wc -c < /tmp/p-{t}.rlib"
);
let bytes: u64 = in_image(&tag, &format!("compile rust-std for `{t}`"), &script)
.parse()
.unwrap_or_else(|e| panic!("{tag}: byte count for {t} is not a number: {e}"));
assert!(bytes > 0, "{tag}: {t} compiled to an EMPTY rlib — a zero-byte green");
eprintln!("[cargo-toolchain] std {t} → {bytes} B rlib");
}
// …and for the WASM road specifically, the artifact's OWN FIRST BYTES. An
// rlib is rustc's container format and looks the same for every target, so
// it cannot tell a wasm build from a host build. A `.wasm` module must open
// with the four-byte magic `00 61 73 6d` ("\0asm") — that is the only thing
// here a working wasm backend produces and a broken one cannot fake.
let magic = in_image(
&tag,
"emit a real wasm module",
"printf 'pub fn probe() -> String { String::from(\"std\") }\\n' > /tmp/w.rs; \
rustc --edition 2021 --crate-type=cdylib --target wasm32-unknown-unknown \
-o /tmp/w.wasm /tmp/w.rs \
&& test -s /tmp/w.wasm \
&& head -c 4 /tmp/w.wasm | od -An -tx1 | tr -s ' ' | sed 's/^ //;s/ $//'",
);
assert_eq!(
magic, "00 61 73 6d",
"{tag} produced a wasm32 artifact whose first four bytes are not the wasm magic \
(\\0asm) — got {magic:?}. This is the assertion the E0463 refusal of \
`gunnar-build-thing-wasm` on 2026-08-27 needed and nobody could make."
);
eprintln!("[cargo-toolchain] wasm32 module magic → {magic}");
report_against_pins(&digest, Some("cargo-toolchain"));
let _ = std::fs::remove_dir_all(&ctx);
}
// ── the kernel kind ───────────────────────────────────────────────────────────
/// ★ **The kernel toolchain image gets a door too — the one whose absence cost
/// 263.2 s and a DNS error in a real chained build.**
#[cfg(feature = "kernel-build")]
#[test]
#[ignore = "builds the kernel toolchain image through podman/Docker"]
fn the_kernel_toolchain_image_can_be_built_and_needs_no_package_index() {
use nornir_build_thing::kernel::{
DraupnirKernelRunner, KERNEL_TOOLCHAIN_CONTAINERFILE, KERNEL_TOOLCHAIN_IMAGE,
};
assert!(
KERNEL_TOOLCHAIN_CONTAINERFILE.contains("FROM"),
"the embedded kernel Containerfile is empty or has no FROM — nothing would be built"
);
let tag = std::env::var("NORDISK_KERNEL_TOOLCHAIN_TAG")
.unwrap_or_else(|_| format!("{KERNEL_TOOLCHAIN_IMAGE}:live"));
let ctx =
std::env::temp_dir().join(format!("nordisk-kernel-toolchain-ctx-{}", std::process::id()));
let built = DraupnirKernelRunner::new()
.build_toolchain_image(&ctx, &tag)
.unwrap_or_else(|e| panic!("build {tag} from the embedded Containerfile: {e:#}"));
eprintln!("[toolchain] built under tag {built}");
let digest = assert_digest(&tag);
// The POINT of this image: the kernel build must not reach for a package
// index. Assert it on the image's own contents, not on the Containerfile
// text — a compiler that is only mentioned in a comment builds nothing.
for tool in ["gcc", "make", "flex", "bison", "bc"] {
// `command -v` exits non-zero for an absent tool, so `in_image`'s status
// assertion IS the "no apt-get at build time" law: an image without gcc
// is the 'Temporary failure resolving deb.debian.org' that killed a real
// chained build on 2026-08-27.
let path = in_image(&tag, &format!("provide `{tool}`"), &format!("command -v {tool}"));
assert!(!path.is_empty(), "{tag}: `command -v {tool}` answered nothing");
eprintln!("[kernel-toolchain] {tool} → {path}");
}
report_against_pins(&digest, None);
let _ = std::fs::remove_dir_all(&ctx);
}
// ── is the image REPRODUCIBLE? ────────────────────────────────────────────────
/// The isolated store pair this gate mints into. Two podman `--root` stores with
/// nothing in them, so neither mint can be served by a layer cache and each one
/// really pulls its own base — the closest a single box gets to "two boxes".
struct Mint {
root: PathBuf,
runroot: PathBuf,
tag: String,
}
impl Mint {
fn podman(&self) -> Command {
let mut c = Command::new("podman");
c.args(["--root"]).arg(&self.root).args(["--runroot"]).arg(&self.runroot);
c.args(["--storage-driver", "overlay"]);
c
}
/// Build the crate's OWN embedded Containerfile into this store and return
/// the image id. A failed build is a failure here, never a skip.
///
/// `--timestamp` is the half of reproducibility no Containerfile can reach:
/// the image config's `created` field and the layer file timestamps are the
/// builder's to write. The number is NOT a constant here — the caller reads
/// it out of the Containerfile's own `ENV SOURCE_DATE_EPOCH` line (each
/// toolchain kind has its own reader over the one parser), so this mint and
/// the image agree by construction rather than by two people remembering.
fn build(&self, containerfile: &Path, ctx: &Path, epoch: i64) -> String {
let out = self
.podman()
.args(["build", "--no-cache", "--pull=always", "--timestamp"])
.arg(epoch.to_string())
.args(["-t", &self.tag, "-f"])
.arg(containerfile)
.arg(ctx)
.output()
.unwrap_or_else(|e| panic!("mint {}: {e}", self.tag));
assert!(
out.status.success(),
"mint {} failed — a gate that cannot mint measures nothing.\n{}",
self.tag,
String::from_utf8_lossy(&out.stderr)
);
self.inspect("{{.Id}}")
}
fn inspect(&self, format: &str) -> String {
let out = self
.podman()
.args(["image", "inspect", "--format", format, &self.tag])
.output()
.unwrap_or_else(|e| panic!("inspect {}: {e}", self.tag));
assert!(
out.status.success(),
"inspect {} answered nothing after a successful mint — an UNREAD identity and \
two identities that agree look the same from here (LAW 2).\n{}",
self.tag,
String::from_utf8_lossy(&out.stderr)
);
String::from_utf8_lossy(&out.stdout).trim().to_string()
}
/// Every regular file in the image, as `path -> (sha256, size, mtime)`,
/// read by running the image itself with the network OFF. `-xdev` keeps the
/// bind-mounted out-dir (a different device) out of its own measurement.
fn file_manifest(&self, out_dir: &Path) -> BTreeMap<String, (String, String, String)> {
std::fs::create_dir_all(out_dir).unwrap_or_else(|e| panic!("{}: {e}", out_dir.display()));
let script = "find / -xdev -type f -printf '%s\\t%T@\\t%p\\n' > /out/meta.tsv && \
find / -xdev -type f -exec sha256sum {} + > /out/sha.txt && \
wc -l < /out/meta.tsv";
let out = self
.podman()
.args(["run", "--rm", "--network", "none", "-v"])
.arg(format!("{}:/out:Z", out_dir.display()))
.args([&self.tag, "sh", "-c", script])
.output()
.unwrap_or_else(|e| panic!("walk {}: {e}", self.tag));
assert!(
out.status.success(),
"could not walk {} with the network off:\n{}",
self.tag,
String::from_utf8_lossy(&out.stderr)
);
let meta = std::fs::read_to_string(out_dir.join("meta.tsv")).expect("meta.tsv");
let sha = std::fs::read_to_string(out_dir.join("sha.txt")).expect("sha.txt");
let mut sizes: BTreeMap<&str, (&str, &str)> = BTreeMap::new();
for line in meta.lines() {
let mut f = line.splitn(3, '\t');
let (Some(size), Some(mtime), Some(path)) = (f.next(), f.next(), f.next()) else {
continue;
};
sizes.insert(path, (size, mtime));
}
let mut m = BTreeMap::new();
for line in sha.lines() {
// ★ These four are the ENGINE's, not the image's. Podman writes
// them into every container at run time (`/etc/hostname` becomes
// the container id; `/run/.containerenv` is podman's own marker
// file, empty and stamped with the moment the container started),
// so two walks of two images disagree about them no matter what
// the images hold — the walk's own artifact.
// MEASURED 2026-08-30: the first run of this gate went red on
// exactly `/etc/hostname` and `/etc/hosts`, and reading the two
// mints' LAYER TARS directly showed no such divergence — layer 1
// differed in 5 files, all of them apt/dpkg logs. The FIRST run
// after the image became reproducible went red the same way on
// `/run/.containerenv`, which had been invisible until then
// because it is empty: it differs only in its mtime, and an mtime
// was noise while the ids were expected to differ anyway. That is
// the shape to distrust — a reading nobody looks at until it is
// the only one left.
if line.ends_with(" /etc/hostname")
|| line.ends_with(" /etc/hosts")
|| line.ends_with(" /etc/resolv.conf")
|| line.ends_with(" /run/.containerenv")
{
continue;
}
// `<64 hex> <path>`; coreutils prefixes a `\` and escapes when the
// path contains a backslash or newline — those never reach a real
// toolchain image, and a malformed line must be seen, not skipped.
let Some((h, p)) = line.split_once(" ") else { continue };
let (size, mtime) = sizes.get(p).copied().unwrap_or(("?", "?"));
m.insert(p.to_string(), (h.to_string(), size.to_string(), mtime.to_string()));
}
assert!(
m.len() > 10_000,
"{} walked only {} files — a toolchain image with gcc, clang, llvm and \
qemu in it has far more, so this manifest is truncated and every \
comparison below would be vacuously clean",
self.tag,
m.len()
);
m
}
}
/// Which known source of per-mint noise a differing file belongs to. A file that
/// fits none of them is the interesting case: real content drift.
fn divergence_class(path: &str) -> Option<&'static str> {
if path.ends_with(".pyc") {
// PEP 552: a pyc's 16-byte header carries the SOURCE mtime, so every
// module byte-compiled at install time differs between two mints while
// its bytecode body is identical.
Some("pyc-header (PEP 552 source mtime)")
} else if path.starts_with("/var/log/") || path == "/var/cache/ldconfig/aux-cache" {
Some("package-manager log / ldconfig cache")
} else if path.starts_with("/root/.cache/pip/") {
Some("pip http cache")
} else if path.ends_with(".so") || path.contains(".so.") {
// A C extension compiled at install time: GNU build-id, plus pip's
// random `/tmp/pip-install-XXXXXXXX/` build directory in `.debug_str`.
Some("ELF build-id + build path")
} else if path.ends_with("/RECORD") {
// dist-info RECORD lists the hashes of the pycs above.
Some("dist-info RECORD (hashes of the pycs)")
} else {
None
}
}
/// ★ **The kernel toolchain image IS reproducible — this is the guard that
/// keeps it so.**
///
/// The question is not academic: every `[[build.kernel]]` in the constellation
/// pins the toolchain by a 64-hex number, and whether that number is *a property
/// of the recipe* or *the identity of one minting event* decides whether
/// re-pinning is a mechanical edit or a decision about which box is
/// authoritative for everyone.
///
/// # It used to be a minting event, measured
///
/// Lane AT, oden 2026-08-30: six mints of the then-current Containerfile into
/// six fresh empty stores gave six different image ids. Two of them, 90 s apart:
///
/// ```text
/// image id 9f1018555472… vs a2d62d1d40aa… DIFFERENT
/// layer 0 (base) 31631a95db7c… == 31631a95db7c… identical
/// layer 1 (apt) d08c27f5e8cb… vs 6617a2946aae… DIFFERENT
/// layer 2 (pip) 8058e1dc12db… vs 7acf659485d9… DIFFERENT
/// ```
///
/// with **0 files added, 0 removed, 0 sizes changed**: 1 423 `.pyc` differing
/// only in their 16-byte PEP 552 header, 8 `kextractor*.so` differing in 132 of
/// 426 528 bytes (a GNU build-id plus pip's random `/tmp/pip-install-XXXXXXXX/`
/// path in `.debug_str`), 5 apt/dpkg/ldconfig log files, and 3 025 files whose
/// only difference was an mtime. Two independent axes: the **world** moved
/// (`FROM debian:trixie` is a tag — the base layer of the 08-27 mint is
/// `102f96b3f4ce…`, of an 08-30 mint `31631a95db7c…`) and the **clock** moved.
///
/// # It is now a property of the recipe, measured
///
/// Lane AV, oden 2026-08-30. Both axes closed — base by digest, apt from the
/// Debian snapshot the base image itself records, exact pip versions, plus
/// `SOURCE_DATE_EPOCH`, the logs and caches deleted in the layer that makes
/// them, kmax's C extension built in a fixed directory with `-g0`, and the
/// builder handed the same instant (`--timestamp`, or podman's `timestamp`
/// build parameter through `build_image_reproducible`):
///
/// ```text
/// image id 3f632569f1980e96744eef4ab258341fc77c55cb17c2dabe79b243e00d6e0825
/// manifest digest 78eaa55a651d5bd76aae36db0f3307e267a92d2a547ee338a17822a103e0320e
/// ```
///
/// — four mints, two fresh empty stores each time, spread over fifteen minutes.
///
/// **Every knob is load-bearing, and each was removed on its own** (two mints
/// per row, fresh empty stores, everything else held):
///
/// | removed | mint 1 | mint 2 | |
/// |---|---|---|---|
/// | nothing (control ×4) | `3f632569f198…` | `3f632569f198…` | SAME |
/// | `--timestamp` | `359d4c48b6f8…` | `8086d7ee318d…` | DIFFERENT |
/// | `ENV SOURCE_DATE_EPOCH` | `30af41ac7b5c…` | `6e03374daf31…` | DIFFERENT |
/// | the log + aux-cache scrub | `90342d2b0188…` | `d750f1a0d167…` | DIFFERENT |
/// | kmax's fixed build dir + `-g0` | `730bd56428b6…` | `36545c9ec46c…` | DIFFERENT |
///
/// # What this gate asserts, and why in that order
///
/// 1. The two mints share a base layer — otherwise the `FROM` moved and nothing
/// below is a comparison of the same recipe.
/// 2. Each mint's `created` really is the instant the Containerfile declares.
/// podman's `timestamp` is a podman extension; a builder that ignored it
/// would be a **silent** no-op, and this is the different door that catches
/// it before the id comparison can be blamed on the recipe.
/// 3. The two image ids are EQUAL. That is the deliverable.
/// 4. If they are not, the file-by-file walk runs anyway and every divergence is
/// classified, so the failure says *which files* and *what kind* — real
/// content drift (a package version moved, an input came unpinned) can never
/// be reported as the same fact as an mtime.
///
/// The walk excludes `/etc/hostname`, `/etc/hosts` and `/etc/resolv.conf`:
/// podman writes those into every container at RUN time, so two walks disagree
/// about them no matter what the images hold. Lane AT's first run of this gate
/// went red on exactly those two, and the layer tars showed the images did not
/// differ there at all — a gate reporting its own artefact is a gate nobody
/// reads.
///
/// ```bash
/// NORDISK_TOOLCHAIN_REPRO_DIR=~/scratch/laneAV/repro \
/// cargo test -p nornir-build-thing --features kernel-build \
/// --test toolchain_images_live -- --ignored --nocapture \
/// two_mints_of_the_kernel_toolchain
/// ```
#[cfg(feature = "kernel-build")]
#[test]
#[ignore = "mints the kernel toolchain image TWICE into two fresh empty stores (~2.5 GB, ~3 min)"]
fn two_mints_of_the_kernel_toolchain_are_measured_against_each_other() {
use nornir_build_thing::kernel::{kernel_toolchain_source_date_epoch, KERNEL_TOOLCHAIN_CONTAINERFILE};
let epoch = kernel_toolchain_source_date_epoch()
.expect("the kernel Containerfile declares the instant it is minted at");
two_mints_are_measured_against_each_other(
KERNEL_TOOLCHAIN_CONTAINERFILE,
epoch,
"nordisk-kernel-toolchain",
"kernel",
);
}
/// ★ **The cargo toolchain image IS reproducible too — since 2026-09-05, and
/// this is the guard that keeps it so.** The same gate as the kernel one, over
/// the cargo Containerfile and its own epoch reader; nothing about the
/// measurement is kernel-shaped, which is why they share a body.
///
/// SEEN RED before the recipe was fixed (lane bevis, oden 2026-09-05): the cargo
/// file had no `ENV SOURCE_DATE_EPOCH`, a moving `FROM debian:trixie` and
/// unpinned apt, and `build_toolchain_image` called the plain `build_image`.
/// `podman images` on that day showed the kernel toolchain's `created` as its
/// declared 2026-07-13 instant and the cargo toolchain's as "5 hours ago" —
/// the wall clock, i.e. a minting event. gunnar re-pinned twice chasing it.
///
/// ```bash
/// NORDISK_TOOLCHAIN_REPRO_DIR=~/scratch/cargo/repro-cargo \
/// cargo test -p nornir-build-thing --features cargo-container \
/// --test toolchain_images_live -- --ignored --nocapture \
/// two_mints_of_the_cargo_toolchain
/// ```
#[cfg(feature = "cargo-container")]
#[test]
#[ignore = "mints the cargo toolchain image TWICE into two fresh empty stores (~3.6 GB, rustup fetch)"]
fn two_mints_of_the_cargo_toolchain_are_measured_against_each_other() {
use nornir_build_thing::cargo_container::{
cargo_toolchain_source_date_epoch, CARGO_TOOLCHAIN_CONTAINERFILE,
};
let epoch = cargo_toolchain_source_date_epoch()
.expect("the cargo Containerfile declares the instant it is minted at");
two_mints_are_measured_against_each_other(
CARGO_TOOLCHAIN_CONTAINERFILE,
epoch,
"nordisk-cargo-toolchain",
"cargo",
);
}
/// The shared body: mint `containerfile` twice into two fresh empty stores at
/// `epoch`, and measure the two results against each other. `image` names the
/// repository the mints are tagged under, `kind` is the word the verdict uses.
fn two_mints_are_measured_against_each_other(
containerfile: &str,
epoch: i64,
image: &str,
kind: &str,
) {
// A real disk, named by the operator. `std::env::temp_dir()` is tmpfs on
// oden and this writes ~2.5 GB of image store twice — a house rule that has
// already cost this constellation a lane.
let base = PathBuf::from(std::env::var("NORDISK_TOOLCHAIN_REPRO_DIR").expect(
"set NORDISK_TOOLCHAIN_REPRO_DIR to a directory ON A REAL DISK (not /tmp, which is \
tmpfs here): this gate writes two ~1.2 GB image stores",
));
assert!(
containerfile.contains("FROM"),
"the embedded {kind} Containerfile is empty or has no FROM — nothing would be minted"
);
let ctx = base.join("ctx");
std::fs::create_dir_all(&ctx).unwrap_or_else(|e| panic!("{}: {e}", ctx.display()));
let cf = ctx.join("Containerfile");
// The crate's OWN embedded text, not a file from a repo — the same rule
// `build_toolchain_image` follows, so the two mints and a real build are
// minting the same recipe.
std::fs::write(&cf, containerfile).expect("write Containerfile");
let mints: Vec<Mint> = ["a", "b"]
.iter()
.map(|s| {
let m = Mint {
root: base.join(format!("store-{s}")),
runroot: base.join(format!("runroot-{s}")),
tag: format!("{image}:repro-{s}"),
};
for d in [&m.root, &m.runroot] {
let _ = std::fs::remove_dir_all(d);
std::fs::create_dir_all(d).unwrap_or_else(|e| panic!("{}: {e}", d.display()));
}
m
})
.collect();
let mut ids = Vec::new();
let mut manifests = Vec::new();
let mut layers = Vec::new();
for m in &mints {
let id = m.build(&cf, &ctx, epoch);
assert_eq!(
id.len(),
64,
"{} reported {id:?} as its image id — not 64 hex characters, so nothing below \
is comparing image identities",
m.tag
);
assert!(id.chars().all(|c| c.is_ascii_hexdigit()), "{} id {id:?} is not hex", m.tag);
let l: Vec<String> = m
.inspect("{{range .RootFS.Layers}}{{.}}\n{{end}}")
.lines()
.map(str::to_string)
.filter(|s| !s.is_empty())
.collect();
assert!(!l.is_empty(), "{} reports no layers", m.tag);
// The IMAGE ID is the number a declaration pins (KTOOL-W2 fix 4,
// 2026-11-03: `kernel.rs` `toolchain_image_ref` runs `sha256:<id>`, and
// only the id survives `podman save`/`load`). Measured 2026-09-05, under
// the OLD rule: the id got pasted into gunnar's pin, the ref builder
// made it `name@sha256:<id>`, and the compose refused in 1.5 s with
// "pull image …@sha256:<id>" — the ref builder was wrong, not the pin.
// Both are 64 hex characters; only the id is the pin. The gate still
// reads both and asserts both: a recipe whose manifest moves while its
// id holds is worth knowing about.
let manifest = m.inspect("{{.Digest}}");
let manifest = manifest.strip_prefix("sha256:").unwrap_or(&manifest).to_string();
assert_eq!(
manifest.len(),
64,
"{} reported {manifest:?} as its manifest digest — not 64 hex characters",
m.tag
);
eprintln!("[repro] {} id {id} <- the pin", m.tag);
eprintln!("[repro] {} manifest {manifest}", m.tag);
for (i, d) in l.iter().enumerate() {
eprintln!("[repro] {} layer {i} {d}", m.tag);
}
ids.push(id);
manifests.push(manifest);
layers.push(l);
}
assert_eq!(
layers[0].len(),
layers[1].len(),
"the two mints produced different LAYER COUNTS from one Containerfile — the recipe \
changed under the measurement"
);
assert_eq!(
layers[0][0], layers[1][0],
"the two mints do not even share a BASE layer, so the `FROM` tag moved between them \
and nothing further is a comparison of the same recipe"
);
// ── the DIFFERENT door: was the builder's `timestamp` honoured at all? ──
// podman's `timestamp` is a podman extension to the `/build` endpoint. A
// builder that does not know it ignores it SILENTLY, and then the ids below
// differ for a reason that has nothing to do with the recipe. Asking the
// built image what instant it claims — rather than trusting that the flag
// was passed — is what separates "the recipe drifted" from "the flag was
// dropped".
for m in &mints {
let created = m.inspect("{{.Created.Unix}}");
assert_eq!(
created,
epoch.to_string(),
"{} was committed at {created}, but its Containerfile declares SOURCE_DATE_EPOCH \
{epoch}. The builder ignored the timestamp it was handed — every image id below \
is then the identity of a minting event no matter what the recipe says.",
m.tag
);
}
let reproducible = ids[0] == ids[1];
eprintln!(
"[repro] VERDICT: the {kind} toolchain image is {} — {} vs {}",
if reproducible { "REPRODUCIBLE on this box" } else { "NOT reproducible" },
ids[0],
ids[1]
);
if reproducible {
// One id must mean one manifest: the manifest is a digest over the
// config (whose digest IS the id) plus the layer descriptors, and the
// layers were just asserted equal. If this fails, podman synthesised two
// manifests for one image — the pin would move while the id stood still.
assert_eq!(
manifests[0], manifests[1],
"one image id, TWO manifest digests — the number a declaration pins is not a \
property of the recipe even though the id is"
);
eprintln!("[repro] PIN: toolchain_image_sha256 = \"{}\"", manifests[0]);
}
let fa = mints[0].file_manifest(&base.join("walk-a"));
let fb = mints[1].file_manifest(&base.join("walk-b"));
eprintln!("[repro] files: a={} b={}", fa.len(), fb.len());
let mut unclassified: Vec<String> = Vec::new();
let mut classes: BTreeMap<&str, usize> = BTreeMap::new();
let mut clock_only = 0usize;
for (path, (ha, sa, ma)) in &fa {
match fb.get(path) {
None => unclassified.push(format!("{path} — only in mint a")),
Some((hb, sb, mb)) => {
if ha == hb {
if ma != mb {
clock_only += 1;
}
continue;
}
match divergence_class(path) {
Some(c) => *classes.entry(c).or_default() += 1,
None => unclassified
.push(format!("{path} — sha {ha} ({sa} B) vs {hb} ({sb} B)")),
}
}
}
}
for path in fb.keys() {
if !fa.contains_key(path) {
unclassified.push(format!("{path} — only in mint b"));
}
}
eprintln!("[repro] identical content, different mtime: {clock_only} files");
for (c, n) in &classes {
eprintln!("[repro] differing content, classified: {n} × {c}");
}
if reproducible {
// Second reader, and it disagrees with the first when the walk is at
// fault: the LAYER DIGESTS above are the images themselves. If those are
// identical and the walk still reports a difference, the difference was
// written by the container runtime during the walk — name the file in
// `file_manifest`'s exclusion list above rather than believing it.
let same_layers = layers[0] == layers[1];
assert!(
classes.is_empty() && unclassified.is_empty() && clock_only == 0,
"the two mints report the SAME image id and yet their filesystems differ — one of \
the two readings is wrong, and an id that does not track content is worse than no \
id at all: {} classified, {} unclassified, {clock_only} clock-only.\n\
Their layer digests are {}. {}\n\
{}",
classes.values().sum::<usize>(),
unclassified.len(),
if same_layers { "IDENTICAL" } else { "DIFFERENT" },
if same_layers {
"So the images do not differ, and whatever is listed below was written into \
the container by podman while it was being walked — add it to the exclusion \
list in `file_manifest`, with the measurement that says why."
} else {
"So this is a real disagreement between the id and the content, and the id is \
the thing that must not be trusted."
},
unclassified.iter().take(20).cloned().collect::<Vec<_>>().join("\n ")
);
eprintln!(
"[repro] the id IS a property of the recipe. Re-pinning a declaration is \
mechanical, and `which box mints` is a non-question."
);
return;
}
// ── RED, and it says WHICH KIND of red ───────────────────────────────────
// The image was reproducible when this gate was written (four mints, one
// id). So a difference here is a REGRESSION, and the two kinds of it want
// two different people:
//
// * every divergence classifiable → a normalisation came undone (a knob
// was dropped from the Containerfile, or the builder stopped being
// handed its instant). The table in this test's doc comment says which
// knob produces which shape.
// * anything UNCLASSIFIED → an input moved. The base digest, the
// Debian snapshot, or a pip version is no longer pinned to what the
// recipe says, and the two images contain different software.
//
// The unclassified list is printed FIRST and in full, because it is the one
// that is not merely untidy.
let named: Vec<String> =
classes.iter().map(|(c, n)| format!("{n} × {c}")).collect();
panic!(
"the {kind} toolchain image is NOT reproducible any more.\n\
\x20 mint a {}\n\
\x20 mint b {}\n\
\x20 {} file(s) differ and are NOT clock noise — an INPUT moved (base digest, \
Debian snapshot, or a pinned version):\n {}\n\
\x20 {} file(s) differ in a known-noisy way: {}\n\
\x20 {clock_only} file(s) have identical content and a different mtime.\n\
The four normalisations and the id each one's absence produces are tabulated in \
this test's doc comment; `KernelBuild::toolchain_repin_hint()` says what a pin \
means once this is red.",
ids[0],
ids[1],
unclassified.len(),
if unclassified.is_empty() {
"(none)".to_string()
} else {
unclassified.iter().take(40).cloned().collect::<Vec<_>>().join("\n ")
},
classes.values().sum::<usize>(),
if named.is_empty() { "(none)".to_string() } else { named.join(", ") },
);
}
// ── does the BOLLARD mint agree with the CLI mint? ───────────────────────────
/// Start a `podman system service` over its own fresh empty store and return
/// `(child, socket path)`. The service is what `draupnir`'s bollard engine talks
/// to, and giving it a `--root` of its own is the only way an in-process mint
/// can be measured against a CLI mint on equal terms: two empty stores, no
/// shared layer cache, one recipe.
#[cfg(feature = "kernel-build")]
fn podman_service(base: &Path, name: &str) -> (std::process::Child, PathBuf) {
let root = base.join(format!("store-{name}"));
let runroot = base.join(format!("runroot-{name}"));
for d in [&root, &runroot] {
let _ = std::fs::remove_dir_all(d);
std::fs::create_dir_all(d).unwrap_or_else(|e| panic!("{}: {e}", d.display()));
}
let sock = base.join(format!("{name}.sock"));
let _ = std::fs::remove_file(&sock);
let child = Command::new("podman")
.args(["--root"])
.arg(&root)
.args(["--runroot"])
.arg(&runroot)
.args(["--storage-driver", "overlay", "system", "service", "--time=0"])
.arg(format!("unix://{}", sock.display()))
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.spawn()
.unwrap_or_else(|e| panic!("spawn podman service for {name}: {e}"));
for _ in 0..300 {
if sock.exists() {
return (child, root);
}
std::thread::sleep(std::time::Duration::from_millis(100));
}
panic!("podman service socket {} never appeared", sock.display());
}
/// ★ **The two ways this constellation mints an image must produce the SAME
/// image id.**
///
/// Lane AT's reproducibility measurement ran through the `podman build` CLI.
/// Nothing in the shipped constellation does: `[[build.kernel]]` mints through
/// [`DraupnirKernelRunner::build_toolchain_image`], which goes to podman's
/// `/build` endpoint over bollard — a different client, a different request, and
/// nobody had ever compared the two. "Reproducible through the path nobody
/// uses" is worth almost nothing, and worse, it reads exactly like
/// "reproducible".
///
/// It is a real question and not a formality, and running it proved the point
/// three times over. bollard's `BuildImageOptions` is generated from **Docker's**
/// API spec, so the bollard path could not spell the flag the CLI path needed;
/// draupnir now appends it through bollard's own `with_request_modifier` hook
/// (`ContainerBoot::build_image_reproducible`). This gate is what says the hook
/// works — not the fact that the code exists. **Three separate defaults nobody
/// had ever compared moved the id, and every one was found by running this**:
///
/// | red | CLI | bollard | the difference |
/// |---|---|---|---|
/// | 1 | `3f632569f198…` | `e255092edde0…` | manifest FORMAT — `podman build` defaults to OCI, the Docker-compat `/build` endpoint to Docker schema2. Layer diff ids were IDENTICAL |
/// | 2 | `3f632569f198…` | *404 Not Found* | `outputformat` is honoured only on podman's **libpod** route (the compat route accepts it and ignores it), and libpod routes are registered under `/v5.x/`, not under the Docker API version bollard negotiates |
/// | 3 | `3f632569f198…` | `f3dc2997b0f4…` | LAYER COUNT — the CLI puts one layer per instruction, the libpod endpoint squashes the build into one (3 layers vs 2) |
///
/// MEASURED GREEN, oden 2026-08-30, both doors, two fresh empty `--root` stores:
/// **`3f632569f198…` and `3f632569f198…`.**
///
/// The moral is not "bollard is awkward". It is that *the same recipe through two
/// clients is two builds*, and the constellation had been measuring the one it
/// does not ship.
///
/// ```bash
/// NORDISK_TOOLCHAIN_REPRO_DIR=~/scratch/laneAV/repro \
/// cargo test -p nornir-build-thing --features kernel-build \
/// --test toolchain_images_live -- --ignored --nocapture \
/// the_bollard_mint_and_the_cli_mint
/// ```
#[cfg(feature = "kernel-build")]
#[test]
#[ignore = "mints the kernel toolchain image through BOTH doors into two fresh empty stores (~2.5 GB, ~3 min)"]
fn the_bollard_mint_and_the_cli_mint_agree_on_the_image_id() {
use nornir_build_thing::kernel::{
DraupnirKernelRunner, KERNEL_TOOLCHAIN_CONTAINERFILE, kernel_toolchain_source_date_epoch,
};
let base = PathBuf::from(std::env::var("NORDISK_TOOLCHAIN_REPRO_DIR").expect(
"set NORDISK_TOOLCHAIN_REPRO_DIR to a directory ON A REAL DISK (not /tmp, which is \
tmpfs here): this gate writes two ~1.2 GB image stores",
));
let epoch = kernel_toolchain_source_date_epoch().expect("the Containerfile declares one");
// ── door 1: the `podman build` CLI, the path lane AT measured ────────────
let cli_ctx = base.join("ctx-cli");
std::fs::create_dir_all(&cli_ctx).unwrap_or_else(|e| panic!("{}: {e}", cli_ctx.display()));
let cf = cli_ctx.join("Containerfile");
std::fs::write(&cf, KERNEL_TOOLCHAIN_CONTAINERFILE).expect("write Containerfile");
let cli = Mint {
root: base.join("store-cli"),
runroot: base.join("runroot-cli"),
tag: "nordisk-kernel-toolchain:door-cli".to_string(),
};
for d in [&cli.root, &cli.runroot] {
let _ = std::fs::remove_dir_all(d);
std::fs::create_dir_all(d).unwrap_or_else(|e| panic!("{}: {e}", d.display()));
}
let cli_id = cli.build(&cf, &cli_ctx, epoch);
eprintln!("[doors] CLI {cli_id}");
// ── door 2: bollard, through the verb a `[[build.kernel]]` really calls ──
// `ContainerBoot` resolves its socket from DOCKER_HOST, so the service this
// spawns — over a store of its own — is what the in-process mint talks to.
// SAFETY: this is an `#[ignore]`d live gate that mints two container images;
// it is run on its own, never inside a parallel default `cargo test`.
let (mut svc, api_root) = podman_service(&base, "api");
unsafe { std::env::set_var("DOCKER_HOST", format!("unix://{}", base.join("api.sock").display())) };
let api_ctx = base.join("ctx-api");
let _ = std::fs::remove_dir_all(&api_ctx);
let api_tag = "nordisk-kernel-toolchain:door-api";
let built = DraupnirKernelRunner::new().build_toolchain_image(&api_ctx, api_tag);
let api_id = built.as_ref().map(|_| {
let out = Command::new("podman")
.args(["--root"])
.arg(&api_root)
.args(["--runroot"])
.arg(base.join("runroot-api"))
.args(["--storage-driver", "overlay", "image", "inspect", "--format"])
.args(["{{.Id}}", api_tag])
.output()
.expect("inspect the bollard-minted image");
assert!(
out.status.success(),
"the bollard mint reported success and the store has no such image:\n{}",
String::from_utf8_lossy(&out.stderr)
);
String::from_utf8_lossy(&out.stdout).trim().to_string()
});
let _ = svc.kill();
let _ = svc.wait();
let api_id = api_id.unwrap_or_else(|e| panic!("the bollard door could not mint at all: {e:#}"));
eprintln!("[doors] bollard {api_id}");
assert_eq!(
api_id.len(),
64,
"the bollard door reported {api_id:?} as an image id — an UNREAD id and two ids that \
agree look identical from here"
);
// The different door again: ask the bollard-minted image what instant it
// claims. If the `timestamp` parameter never reached podman it is a SILENT
// no-op, and the id mismatch below would look like a recipe problem.
let api_created = {
let out = Command::new("podman")
.args(["--root"])
.arg(&api_root)
.args(["--runroot"])
.arg(base.join("runroot-api"))
.args(["--storage-driver", "overlay", "image", "inspect", "--format"])
.args(["{{.Created.Unix}}", api_tag])
.output()
.expect("read the bollard-minted image's created time");
String::from_utf8_lossy(&out.stdout).trim().to_string()
};
assert_eq!(
api_created,
epoch.to_string(),
"the bollard-minted image was committed at {api_created}, not at the {epoch} its \
Containerfile declares — podman's `timestamp` build parameter did not reach the \
daemon, and bollard has no field for it, so `with_request_modifier` in draupnir's \
`build_stream` is what must be broken"
);
assert_eq!(
cli_id, api_id,
"the two doors this constellation mints images through disagree.\n\
\x20 podman build CLI {cli_id}\n\
\x20 DraupnirKernelRunner (bollard) {api_id}\n\
A pin is reproducible through ONE of them, which reads exactly like reproducible and \
is not: `[[build.kernel]]` mints through the bollard door, so that is the one that \
has to agree with the recipe. The likeliest cause is that podman's `timestamp` build \
parameter did not reach the daemon — draupnir appends it in `build_stream` via \
bollard's `with_request_modifier`, because bollard's own options struct is generated \
from Docker's API spec and has no such field."
);
eprintln!("[doors] both doors agree — the id is a property of the recipe, not of the client");
}