nordisk/edda

The kernel build kind — guide

Class: guide · audience: a techie declaring or driving a kernel build. The why is in workspace_nordisk/.nornir/kernel-trim-design.md; the field-by-field authority is the module doc of crates/nornir-build-thing/src/kernel.rs. Sibling: kernel-for-idiots.md.

What it is

Where an [[build.iso]] masters a medium around a kernel and a [[build.vm]] boots one, a [[build.kernel]] is where that kernel comes from: fetch the pinned kernel.org tarball (host-side, pure Rust), merge a committed kconfig fragment set over defconfig or tinyconfig, optionally run a SAT pass over the Kconfig DAG, and drive make bzImage inside a bounded toolchain container. Out come three files in <repo_root>/.kernel-out/<name>/ — bzImage, kernel.config, and <name>.kernel-pin.json — plus a fourth, deps-<version>.config, when the minifier ran.

The host contributes exactly two things: a container-engine socket and a workdir. No host gcc, no host python, no host qemu.

The four presets — what is IN the kernel

preset is the contents axis; profile (vanilla | hardened) is the orthogonal hardening overlay, so every preset × profile pair composes.

Measured, linux 6.12.104, airgapped

podman rootless on dwarf, 12 cores, -j8, 8192 MiB cap, from the cached tarball. All four presets have now been built and probe-booted:

full boots and is not an appliance kernel. On plain defconfig, CONFIG_TUN is not built in — it is a module, because defconfig sets MODULES=y — while VIRTIO_NET, VIRTIO_BLK and VFAT_FS happen to be =y. An appliance initramfs carries no modules, so a full kernel has no TUN device and therefore no userspace-WireGuard data plane. That is the lesson base.config exists for, and it is now measured rather than argued. (A second full receipt came in at 230 s; the 246 s row is the first.)

The gap between full and normal is 20 480 bytes — that is the entire boot contract, and it buys a machine that can actually run the appliance. The interesting comparison was never full-vs-normal; it is normal-vs-slim (7.6×), which is what tinyconfig buys, and slim-vs-slim+minify (−23.3 %), which is what the solver buys.

slim and nuked sit 86 016 bytes apart — a real difference in what the appliance can do at all, which is why a picker cell that shows the kind and hides the preset shows the half that decides nothing.

Every contract-carrying arm's built .config is asserted to hold VIRTIO_BLK=y VIRTIO_NET=y TUN=y VFAT_FS=y. A small kernel that quietly dropped virtio is the silent degrade the canary exists to prevent.

Declaring one

[[build.kernel]]
name = "korp-kernel-hardened"
version = "6.12.104"
# REQUIRED — a kernel fetched unpinned is refused BY NAME:
tarball_sha256 = "949d166a…"        # 64 lowercase hex, from kernel.org's signed sums
preset = "slim"                      # full | normal (default) | slim | nuked
profile = "hardened"                 # vanilla (default) | hardened
# base = "tinyconfig"                # optional override; unset = whatever the preset implies
# source_url = "…"                   # unset = the kernel.org CDN derived from `version`
# kconfig_fragment = "kernel/extra.config"   # repo-relative, merged LAST (it may veto)
toolchain_image = ""                 # empty + a digest = the LOCAL image id
toolchain_image_sha256 = "388a7816…" # 64 hex
toolchain_packages = []              # self-contained image ⇒ the build runs --network none
# MINIFY — the SAT pass over the Kconfig DAG (OFF by default):
# solver = "klocalizer"              # unset = off, and the build is exactly what it was
# solver_version = "4.10"            # the kmax in the image; HALF the formula cache key
# solver_timeout_secs = 1800         # the solver's OWN bound, under timeout_secs
# the compiler axis:
# toolchain = "gcc"                  # gcc (default) | clang (renders LLVM=1)
# lto = false                        # full clang LTO; needs toolchain = "clang"
# minimal = false                    # REFUSED on x86_64 — see "the third knife"
# timeout_secs = 7200                # the build bound; overrun is a loud refusal
# probe_timeout_secs = 120           # a boot that has not printed its marker is a HANG
# mem_limit_mb = 8192                # podman --memory
# cpus = 8                           # podman --cpus, AND the `make -j` width
# min_free_disk_mb = 6144            # statvfs floor, checked BEFORE the fetch; 0 disables
# cache_dir = "/big/disk/nordisk-kernel"     # tarballs AND kclause formulas live here

kind = "kernel" on a [[build.variant]] row is the equally-legal spelling; [[build.kernel]] is permanent sugar for it, exactly like iso / installer.

Every field, its default, its refusal

The bounds are real, not decorative: mid-compile the live run measured 759.82 % CPU (≈7.6 of the 8-core cap) and 2.508 GB / 8.59 GB.

MINIFY — the second knife

solver = "klocalizer" runs kmax's SAT solver in the same airgapped bounded container, between the fragment merge and olddefconfig, and hands olddefconfig a .config whose depends on / select chains are already closed. It is off by default: without it the build is exactly what it always was, and the canary is the only oracle.

Measured 2026-08-23, linux 6.12.104, airgapped, -j8 / 8192 MiB:

The formula, and why it is cached

The expensive half is reading the kernel's own Kconfig: kextract + kclause turn the tree into a z3 formula over 17 704 symbols — 19 949 986 bytes, about 46 s. That is content-addressed in the same shared cache the tarball uses:

<cache_dir>/kclause/<tarball_sha256>.kmax-<solver_version>/

Both halves of that key are load-bearing. A formula is a function of the tree and of the extractor that produced it, so keying on the tarball alone would let two kmax releases share one entry — a cache that lies is worse than no cache, because the build believes it. The host stages a hit into the workdir, and publishes a miss back through a .part directory and a rename, exactly as the tarball cache does.

The closed world — get this wrong and the "minifier" makes it BIGGER

klocalizer's --repair reads a .config as soft assumptions and drops only what an unsat core forces it to. Everything the .config never mentions is free, and free is how drivers walk in. So the match config handed to the solver is the merged .config closed: every symbol the tree knows and the config never names is written down as is not set.

Measured, and the reason this paragraph exists: 78 added symbols closed, against 472 open — the open run full of mlx5, i40e and reiserfs that nothing asked for. A naively-wired minifier is not a smaller kernel, it is a bigger one.

The write-back — the closure is a file you commit

After olddefconfig and before the canary, whatever the solver added, minus what the fragment already named, is written to deps-<version>.config beside the bzImage and sealed as a fourth artifact. That is the automation of src/kconfig/deps-tinyconfig.config, whose header describes the same loop done by hand. Read it, and commit it if you want the closure pinned instead of solved every time. A solver that ran but left no readable closure refuses the build: a minification nobody can review is not a minification.

Its own bound, its own refusal

Both klocalizer calls are wrapped in timeout. An overrun prints KERNEL-SOLVER-REFUSED and exits 91 — deliberately not the canary's 90 — so the orchestrator can say which step refused, name the knob that bounds it (solver_timeout_secs) and the budget it blew. Proven live: a 5 s budget refused in 19 s naming the variant, the step, the knob and the number. A timeout (124) and any other non-zero are reported as different facts: "did not answer in 1800 s" and "answered UNSAT in four seconds" must not arrive as the same sentence, and an UNSAT means the fragment asks for symbols that cannot all be on together on this arch.

The limit — a solver is a knife, not an oracle

This is the most important paragraph on this page. With the hand-written closure removed, klocalizer found 13 of its 16 lines by itself and missed CONFIG_BINFMT_ELF, CONFIG_MULTIUSER and CONFIG_UNIX — precisely the three the canary cannot catch, for the identical reason: nothing in the fragment names them. That kernel was canary-clean, 1 369 088 bytes, and a brick; the probe refused it with "did NOT reach PID 1".

MINIFY makes a declared contract minimal. It does not discover an undeclared one. Probe-boot is still the only thing that says a kernel runs.

The third knife — measured, and mostly falsified

The design named three knives: where you start, which features exist, and dead code inside what is left. On x86_64 the third one does not exist:

• minimal = true is a named refusal on this arch. CONFIG_LD_DEAD_CODE_DATA_ELIMINATION is gated on HAVE_LD_DEAD_CODE_DATA_ELIMINATION, which only arm, mips, riscv and powerpc select. The first live attempt was refused by the canary; adding its depends on EXPERT parent got it refused again, because the parent was never the problem. A flag that silently does nothing is worse than one that refuses — it reads on the receipt as a knife that was used. The near-empty fragment is kept for the day this kind wires an arch that has the option.
• CONFIG_TRIM_UNUSED_KSYMS is declared inside if MODULES, so a MODULES=n kernel cannot have it — and every preset it would help most is MODULES=n.
• clang LTO works and is the wrong tool for size. Same declaration, same container, slim:

It boots cleanly (PID 1, console + tun + virtio-blk, 2.4 s). Full LTO inlines across translation units, and on a kernel this small the inlining costs more bytes than the dead code it removes. LTO is a speed/codegen knife, not a size knife. It stays opt-in and off by default.

On x86_64 there are two knives, not three: tinyconfig and minify.

The toolchain image

apt-geting a build environment at build time needs a network, drifts, and cannot be pinned. So the kernel kind ships the text of one image — src/containerfiles/kernel-toolchain.Containerfile — and lets the ordinary container kind build it. Every package in it names the step that needs it: gcc/make/flex/bison/bc/libelf/openssl/pahole (compile), clang+lld+llvm (the LTO axis), python3 + python3-dev + kmax 4.10 in a venv (minify), qemu-system-x86 + cpio + xorriso (probe-boot).

Two named refusals apply to it and to no other image, because it is one we publish:

• it must be pinned (toolchain_image_sha256), and
• it must declare toolchain_packages = [], so the build runs airgapped.

⚠ podman image prune DESTROYS these images, and no pull brings them back

nordisk-kernel-toolchain and nordisk-cargo-toolchain are built HERE and have no registry behind them. MEASURED on oden 2026-09-05: podman pull nordisk-cargo-toolchain:live resolves to docker.io/library/… and dies on "requested access to the resource is denied", and the kernel name resolves to localhost/… and dies on "pinging container registry localhost: connection refused". Neither is fetchable; the only source is a re-mint.

And podman image prune -f removes unused-but-tagged images, not only dangling ones: MEASURED 2026-09-05, one podman container prune -f followed by one podman image prune -f took the image count 107 → 9 and took both toolchain images with it. Pruning them is destructive and is recovered only by re-minting (toolchain_images_live.rs, ~45 s each). The kernel image re-mints to the SAME digest (78eaa55a651d…) because its recipe is pinned; the cargo image does not — a re-mint gives a NEW digest and every image_sha256 pin in the constellation then names an image that no longer exists. Prune by name (podman image rm <name>) or with --filter dangling=true.

Build it, in Rust, through the path that already exists:

use nornir_build_thing::kernel::{
    kernel_toolchain_container, materialize_kernel_toolchain_containerfile,
};
// 1. write the committed text where your build context lives:
let cf = materialize_kernel_toolchain_containerfile(&repo_root.join("containerfiles"))?;
// 2. build it as an ORDINARY image cell (no second builder, no second engine):
let image = kernel_toolchain_container("containerfiles/kernel-toolchain.Containerfile");
let built = builder.build(&image, &repo_root)?;   // BuiltImage carries the digest

Pinning it — local vs published

Podman reads name@sha256:<digest> as a registry reference and tries to pull it, so localhost/nordisk-kernel-toolchain@sha256:… dies on "pinging container registry localhost: connection refused". A locally-built image is content-addressable only by its image ID. So:

• not yet published → leave toolchain_image empty and put the 64-hex local image id in toolchain_image_sha256. That is both a legal podman run argument and a genuine content address.
• published → the ordinary <name>@sha256:<digest> spelling, which becomes correct the moment the image is in a registry.

Proven live: the slim arm rebuilt with image=388a7816cde2…, no name at all, airgapped, off the cached tarball — 75 s, bzImage 1 811 456 bytes, byte-identical in size to the tag-addressed run.

The shared cache

A kernel tarball is ~150 MB (measured: 148 400 064 B for 6.12.104), and a kclause formula is another 19 949 986 B. Both live in one shared, content-addressed cache. Resolution order:

1. $NORDISK_KERNEL_CACHE
2. $XDG_CACHE_HOME/nordisk/kernel
3. $HOME/.cache/nordisk/kernel
4. <repo_root>/.nornir-cache/kernel (a machine with no home at all, e.g. CI)

…or the declared cache_dir, which overrides all four — that is how you put it on the big disk.

<cache>/tarballs/<sha256>.tar.xz
<cache>/kclause/<sha256>.kmax-<solver_version>/

Tarballs are keyed by the pin itself, so a stale entry is not expressible: a file whose bytes do not hash to its own name is not a hit, it is garbage that gets re-fetched. A download lands on a pid-suffixed .part inside the cache and is renamed on success, so a half-download can never look like an entry to a concurrent build; a formula is published the same way, through a .part directory. Entries are hard-linked into the workdir (free), or copied where the filesystem will not link.

The disk floor

min_free_disk_mb (default 6144) is checked with statvfs before the fetch, and refuses by name with both numbers. The floor is measured, not feared: a normal build's workdir peaks at 2.5 GB, a slim one at 1.9 GB, plus the 148 MB cached tarball. 6 GiB is that with room for a second variant.

A build that dies at 95 % because the filesystem filled has spent the whole compile to tell you something statvfs knew before it started. Note the honest half: setting the floor to 0, or running without the kernel-build feature that can call statvfs at all, leaves no trace on the receipt — a skipped check and a passed one look identical afterwards.

Refused BEFORE the container: a fragment that contradicts itself (lane AP, 2026-08-30)

KERNEL-CONFIG-REFUSED (below) is the canary INSIDE the container: it fires after the tarball is fetched, the toolchain image pulled and olddefconfig run. One shape of that refusal never needed the container at all — a =n that legally narrows a parent, leaving a =y on its child dangling:

kernel 'tunnr-kernel': the merged fragment contradicts itself in 1 place(s) — refused BEFORE
the container runs:
  - CONFIG_SERIAL_8250_CONSOLE=y cannot hold: it `depends on SERIAL_8250 = y`, and this same
    merged fragment says # CONFIG_SERIAL_8250 is not set. `olddefconfig` will DROP it and the
    build's canary will then refuse by name, after the container has run. Narrow the child
    with its parent, or stop declaring it

This is lane AN's 2026-08-30 canary bug, and UpCloud is such a machine (consoles = [vga]). The check reads the donor's own Kconfig tree, so it needs the source EXTRACTED — that means every re-build and every ISO compose takes this door, and a cold first build still leans on the canary. It reads the fragment as an OPEN world: a symbol the fragment does not mention is unknown, never off, so only a contradiction the fragment states about itself is reported. Zero false positives over the shipped fragment's 84 declarations.

The kernel workdir now keeps one more file

<donor>/.kernel-out/<kernel>/kernel.base.config — the starting .config (tinyconfig/defconfig) as the container saw it, before the fragment was merged. It is the kconfig solver's base input: without it, "this symbol is residue from the preset" and "something pulled it in" cannot be told apart. A kernel built before 2026-08-30 has none and its kernel.why.toml rows simply stay unsolved — a description must never be able to veto the thing it describes.

Reading a KERNEL-CONFIG-REFUSED list

Inside the container, after merge_config.sh -m + olddefconfig, a canary loop checks every line of the merged fragment: each CONFIG_X=y must be in the final .config, and each # CONFIG_X is not set must not have crept back. It counts every failure, prints them all, and exits 90 once:

KERNEL-CONFIG-REFUSED: CONFIG_EFI_STUB=y did not survive olddefconfig
KERNEL-CONFIG-REFUSED: CONFIG_TUN=y did not survive olddefconfig
…
KERNEL-CONFIG-REFUSED-TOTAL: 12 option(s) did not hold — the list above IS the
dependency closure this fragment still owes

That list is not an error report, it is a work list. An option "did not survive" because its parent symbol is off, so olddefconfig dropped it. For each name, add the parent that enables it and run again — or turn the solver on and let it close the chain for you.

This is exactly how src/kconfig/deps-tinyconfig.config was written. The first slim run refused in 18 s naming CONFIG_EFI_STUB and nothing else (the loop exited on the first bad line back then); the fixed loop returned all twelve at once:

EFI_STUB · SERIAL_8250 · SERIAL_8250_CONSOLE · TUN · VIRTIO · VIRTIO_PCI
VIRTIO_BLK · VIRTIO_NET · VFAT_FS · ISO9660_FS · NLS_CODEPAGE_437 · NLS_ISO8859_1

Their parents — PCI, ACPI+EFI, TTY, NET/INET/NETDEVICES, BLOCK, VIRTIO_MENU, FAT_FS, NLS — became that file, and the next run was green. Two turns of 18 s, not one container run per symbol.

The canary's limit, stated honestly: it only checks what the fragment names. CONFIG_BINFMT_ELF — without which the kernel cannot exec PID 1 at all — can never be caught by it, and the solver misses it for the same reason. That gap is what the probe exists to fill.

The probe boot

probe_kernel_with boots the collected bzImage under QEMU inside the toolchain image, with a PID 1 that is a static C binary compiled by that image's own gcc (not a shell script: there is no shell in this initramfs, and one there would test the shell instead of the kernel's ability to exec an ELF). PID 1 prints its marker, mounts devtmpfs, NAMES each contract device present or ABSENT, then powers the guest off. -no-reboot + panic=1, so a broken kernel ends — a hang and a panic are different facts and must not arrive as the same one.

Wall times: normal 4.2 s, slim 2.1 s, nuked 2.1 s, slim+minify 2.1 s, clang-LTO slim 2.4 s. The device columns are in the matrix at the top of this page.

full's ("tun", false) is the clearest thing the probe has produced: it came from PID 1 calling stat() inside the booted guest, not from reading a .config. A probe that printed only what it found would have reported a clean boot for a kernel that cannot carry the constellation's data plane.

KernelProbe is a value, not a verdict: the caller decides what a missing device means, because a preset that never declared virtio-blk is supposed to have no /dev/vda.

The probe refuses on a non-zero container exit before reading a single marker, and says so. That is not caution — the first live run reported booted: true for a kernel that was never started, because gcc failed on the PID 1 source and gcc's diagnostics quote the offending line, which contains the marker strings. A transcript is evidence only if the step that produced it succeeded.

The receipt

<name>.kernel-pin.json carries every sha256 that identifies the build — the tarball, the merged fragment, the bzImage, the .config — and, since 2026-08-23, the axes that decide which kernel it is: preset, the resolved base, toolchain, lto, solver and solver_version beside the older profile. Before that an auditor holding a pin could not tell a 13.7 MB kernel from a 1.7 MB one. The same axes are in the content id, so two presets can no longer share one content address; the run knobs (cpus, memory, disk floor, cache dir) are deliberately kept out of it, because they change how a build is run and not what it produces.

Running the heavy live tests

Both are #[ignore]d and feature-gated: they never ride a default cargo test. Missing env or a missing socket makes them SKIP loudly, never fake a green.

# ── the build (real download + real compile) ─────────────────────────────────
export KERNEL_LIVE_VERSION=6.12.104
export KERNEL_LIVE_SHA256=<64-hex from https://cdn.kernel.org/pub/linux/kernel/v6.x/sha256sums.asc>
export KERNEL_LIVE_PRESET=slim              # full|normal|slim|nuked   (default normal)
export KERNEL_LIVE_PROFILE=vanilla          # vanilla|hardened          (default vanilla)
export KERNEL_LIVE_BASE=tinyconfig          # optional explicit override
export KERNEL_LIVE_NAME=e2e-slim            # default live-vanilla
export KERNEL_LIVE_TOOLCHAIN=""             # empty name + the id below = the LOCAL image
export KERNEL_LIVE_TOOLCHAIN_SHA256=388a7816…
export KERNEL_LIVE_TOOLCHAIN_KIND=clang     # the COMPILER axis (gcc default)
export KERNEL_LIVE_LTO=1                    # set at all = lto = true (needs clang)
export KERNEL_LIVE_MINIMAL=1                # set at all = minimal = true (refused on x86_64)
export KERNEL_LIVE_SOLVER=klocalizer        # MINIFY on; unset = off
export KERNEL_LIVE_SOLVER_TIMEOUT=1800
export KERNEL_LIVE_SOLVER_VERSION=4.10
export KERNEL_LIVE_CACHE=/big/disk/nordisk-kernel   # tarballs AND kclause formulas
export KERNEL_LIVE_CPUS=8
export KERNEL_LIVE_MEM_MB=8192
export KERNEL_LIVE_KEEP=/big/disk/kernel-e2e   # keep the workdir instead of a tempdir
cargo test -p nornir-build-thing --features kernel-build \
    --test kernel_build_live -- --ignored --nocapture

# ── the probe (boots what the build left on disk) ────────────────────────────
export KERNEL_PROBE_ROOT=/big/disk/kernel-e2e   # the repo_root that holds .kernel-out/
export KERNEL_PROBE_NAME=e2e-slim
export KERNEL_PROBE_TOOLCHAIN=localhost/nordisk-kernel-toolchain:latest
export KERNEL_PROBE_TOOLCHAIN_SHA256=388a7816…   # optional
cargo test -p nornir-build-thing --features kernel-build \
    --test kernel_probe_live -- --ignored --nocapture

Naming a KERNEL_LIVE_TOOLCHAIN also clears toolchain_packages, so the run is airgapped — the live run reports network=false. The tarball pin is deliberately not baked into the test file: a committed hash would rot the day a mirror re-rolls, and a test that fetches whatever the network answers is exactly what this kind refuses by design.

The #[ignore] message still says "30+ min". That estimate was written from fear; the measured wall is 235 s for normal on this hardware with a warm cache.

The pane

A [[build.kernel]] row is reachable from dwarves: the matrix cell shows preset · version · resolved base · profile, and a Kernel tab mounts a rotating 3D feature graph over the presets' committed fragments, with a 14-step scenario in the tab library (an unreachable pane is a defect this constellation has already paid for once). Rotation re-projects a cached layout — the build counter is asserted flat across spin frames and to rise by exactly one when the preset genuinely changes.

One honest limit is written on the pane itself: its edges are the name hierarchy at an underscore boundary (VIRTIO → VIRTIO_PCI), not the real depends on / select DAG, which lives in the tarball. A kernel cell deliberately opens no run pane — a bare bzImage boots nothing without a medium or a rootfs around it.

The second limit this paragraph carried until 2026-08-27 — "the zero-hits node state needs the RUN loop, which is not built, so it is absent rather than faked" — stopped being true on 2026-08-23, with the RUN loop (below) and dwarves-gui/tests/robot_kernel_scene.rs::a_run_report_reaches_the_graph_and_zero_hits_are_not_absence, which drives the report onto the graph and holds the "absent ≠ zero" half red: with no report there are no drop candidates, because absence of evidence is never painted as evidence of absence.

An OFFLINE kernel build — BUILT 2026-08-30 (lane AO · #14; spec by lane AH)

Answer (b) below is shipped. nornir_build_thing::toolchain_archive stages the toolchain image as a file in the same content-addressed cache as the kernel tarball, verifies it before a container is created, and refuses BY NAME rather than letting a missing toolchain become a registry fetch. Jump to "(b), as built" for the procedure and the refusals; the spec below is kept because its reasoning is what decided the shape, and because one of its claims turned out to be false — see the correction.

What is ALREADY offline, measured

So two of the three fetches are already solved by a content-addressed cache that a courier can fill. The whole of the remaining gap is the toolchain image, plus the OCI runtime under it.

The gap, precisely

1. The image is per-box. toolchain_image_sha256 is the MANIFEST DIGEST of an image built locally from this crate's embedded kernel-toolchain.Containerfile, so the committed pin is true only on the box that minted it (see KernelBuild::toolchain_repin_hint). ⚠ Retired 2026-08-30 (lane AV) — this was measured (lane AT: six mints, six ids) and then FIXED; the recipe is reproducible now, so the pin is a property of it. Kept because the shape of the offline answer below was decided while it was true, and because the archive courier is still the right answer for a box with no route.
2. Minting it needs the network anyway — the Containerfile pulls debian:trixie from a registry and apt-gets its compilers. So "build the image locally" is not an offline answer; it is the online step, moved.
3. podman must exist. draupnir's container backend is the only runner KernelContainerRunner has.

The three candidate answers, and the verdict on each

(a) A vendored toolchain TARBALL — ship the compiler set as a content-addressed archive beside the kernel tarball, unpack it into the workdir, and run make against it. Rejected as the primary answer. It replaces a reproducible image with a second, unpinned way to get a compiler; it must then solve dynamic linking against whatever libc the host has, which is the problem containers exist to avoid; and it does not remove podman, because the build is still bounded (mem_limit_mb, cpus) and an unbounded kernel build took this box down at 33 GB on 2026-08-20. Worth having only as the no-OCI-runtime-at-all fallback, and then it needs its own --bound mechanism (systemd scope or a cgroup written directly), which is new code.

(b) An OCI ARCHIVE of the toolchain image — podman save/skopeo copy the image to a docker-archive or an OCI layout, content-addressed by the same 64 hex characters, kept in the same cache as the tarball, and podman loaded when the digest is absent. This is the honest fix, and it is what was built (see below). It costs no new build path (the same container, the same bounds, the same network = false), it makes the committed pin true on every box because the bytes travel rather than the recipe, and it is the same courier a <cache_dir>/tarballs/ entry already uses. Concretely: <cache_dir>/images/<toolchain_image_sha256>.oci.tar, a preflight that loads it when podman image exists says no, and a refusal naming that exact path when neither the image nor the archive is there. Publishing the image to a registry is the same change seen from the other side and should land with it: a registry is the online courier, the archive is the offline one, and both address it by the digest that is already declared.

(c) A pre-built kernel ARTIFACT in the ledger — the [[build.kernel]] output (bzImage + kernel.config + kernel.why.toml + the pin json) stored under DWARVES_HOLGER_DIR and consumed instead of rebuilt. Half of this already exists — dwarves buildthing --installer says so out loud when it is unset:

[dwarves] kernel ledger: no artifact store configured (DWARVES_HOLGER_DIR unset …)
          — a built kernel is read from the donor's .kernel-out/

Complementary, not a substitute. It is the right answer for a machine that must consume a kernel offline (a CI runner composing an ISO, an appliance factory), and it is strictly cheaper than (b) there. It is the wrong answer for a machine that must produce one, and the ledger key must be the merged fragment hash and not the preset name — the compose path already refuses on exactly that key ("the pin was built from preset 'full' (fragment e3b0c44…), the choice here merges to preset 'slim' (fragment f190ed5…)"), so a ledger keyed any other way would serve the wrong kernel silently.

The order to build it in

1. (b) first — the OCI archive plus the podman image exists preflight. It is the only item that turns "cannot build at all" into "builds", and it deletes the per-box pin problem as a side effect.
2. (c) next — the ledger read path keyed by fragment hash, for consumers.
3. (a) only if a target genuinely has no OCI runtime, and then with its own bound.

(b), as BUILT — the toolchain courier (lane AO, 2026-08-30)

crates/nornir-build-thing/src/toolchain_archive.rs. Wired into build_kernel_with_reserved as step 2c, immediately before the run request is composed, so nothing about the container changes: same bounds, same network = false, same script.

The cache gains a third drawer

<cache_dir>/tarballs/<tarball_sha256>.tar.xz     the kernel source   (since 08-22)
<cache_dir>/kclause/<tarball_sha256>.kmax-<v>/   the solver formula  (since 08-23)
<cache_dir>/images/<toolchain_image_sha256>.tar  the TOOLCHAIN       (08-30, this)

Same courier, same rule, same shape: a file whose content does not match its own name is not a hit, it is garbage.

⚠ The correction to the spec — MEASURED, oden 2026-08-30

The spec said (b) "makes the committed pin true on every box because the bytes travel rather than the recipe". That is false for the pin the format declares. podman save + podman load of nordisk-kernel-toolchain:live, into a separate --root store on the same box:

and asked of the receiving store directly:

ce66a0100cb8…                                EXISTS   (the bare image id)
sha256:ce66a0100cb8…                         EXISTS
nordisk-kernel-toolchain@sha256:f573ca0a…    ABSENT   <- the COMMITTED pin
nordisk-kernel-toolchain@sha256:5e628ed9…    EXISTS   (the RECEIVER's own digest)

Podman re-synthesises the manifest on load, so a manifest digest describes the copy, not the original, and the committed pin names nothing on the machine the archive was carried to. The image id — the config digest — survives, and is a legal reference (podman run <id>). Hence two deliberate departures:

1. docker-archive, not oci-archive. The OCI conversion rewrites the config too, so it preserves neither identity — it is the one format that makes the archive unverifiable against anything the recipe can hold. It is also the format draupnir's export_image/import_image already speak, so no second way to move an image was added (LAW 5).
2. An archive is verified against the IMAGE ID. A declaration still pinning a per-box manifest digest is refused against a staged archive, by name, with the id it should carry instead. This is the third of the three fixes the row on publishing the image already named ("publish it", "pin its inputs") — and the only one that needs no registry. Every [[build.kernel]] in the constellation is still on a manifest digest, so every one of them must be re-pinned to its id before it can use an archive; the refusal spells out the value and both files it goes in.

The procedure

On a box that has the image:

ID=$(podman image inspect --format '{{.Id}}' nordisk-kernel-toolchain)
podman save --format docker-archive -o <cache>/images/$ID.tar nordisk-kernel-toolchain
# or, through the door, so no shell is involved:
#   nornir_build_thing::toolchain_archive::stage_toolchain_archive(
#       &DraupnirKernelRunner::new(), "nordisk-kernel-toolchain", cache_dir)

Copy that one file to <cache>/images/<ID>.tar on the offline box, and set toolchain_image_sha256 = "<ID>". The staging door files the archive under the id it actually exported — never under the name the caller passed, which is how an archive ends up filed as something it is not.

The five rows, and the three named refusals

NORDISK-TOOLCHAIN-LOAD-UNCONFIRMED is the third: an engine that answers "loaded" and then answers "no such image" is two claims disagreeing, and only the second one is the one the build needs. The confirmation goes through a different door than the load.

KernelPin now records toolchain_source and toolchain_image_id, because "which toolchain this build claims" and "whether that claim was ever checked" were the same field, and unchecked and staged-archive read identically on a receipt without it.

How the airgap was PROVEN — and what the proof covers

tests/kernel_offline_live.rs, #[ignore]d. Not "we did not observe a fetch": the same build runs twice under the same denial, and the only difference is whether the archive is on the disk. One arm must fail or the other proves nothing.

The denial has to reach the container ENGINE and not merely the test process — a podman socket is a filesystem path, so a test inside a network namespace still talks to a service that may have a fine route to a registry. So:

# a podman with NO route, NO addresses, NO DNS, and an EMPTY store:
sudo unshare --net -- setpriv --reuid=$(id -u) --regid=$(id -g) --init-groups   podman --root $D/freshroot --runroot $D/freshrun   system service --time 0 unix://$D/sock/podman.sock &

# prove the denial before trusting it — this MUST fail:
podman --url unix://$D/sock/podman.sock pull docker.io/library/alpine:latest
#   -> dial tcp: lookup registry-1.docker.io: Temporary failure in name resolution

(unshare -rn without sudo does not work on oden: apparmor_restrict_unprivileged_userns=1 refuses the uid map. sudo unshare --net + setpriv back to the user is the shape that does.)

Measured 2026-08-30, oden, linux 6.12.104 nuked, fresh empty store, no network:

• RED — archive moved aside: NORDISK-TOOLCHAIN-MISSING, naming <cache>/images/ce66a0100cb8….tar; no bzImage produced.
• GREEN — archive put back, byte-identical call: the 1.16 GB image loaded into the empty store from the file, the container ran, a bzImage came out, and the pin says toolchain_source = "staged-archive", toolchain_image = "sha256:ce66a0100cb8…".

And through the CLI, the same two arms. The test above drives build_kernel_with directly — the same function dwarves' assemble_cell calls — so the verb itself was driven too, against a scratch product fixture (no committed recipe was edited) whose toolchain_image_sha256 is the image ID:

$ dwarves buildthing aokernel --variant offline-kernel     # DOCKER_HOST -> the denied,
                                                           # empty store
RED    offline-kernel [dwarves-buildthing-7] — could not start: kernel 'offline-kernel'
       FAILED and produced nothing: compile kernel: NORDISK-TOOLCHAIN-MISSING: …
       exit 1

GREEN  BUILT 'offline-kernel' in 90.2s — 4 artifact(s)
       bzImage 2 057 216 bytes · kernel.config · kernel.why.toml · the pin, which says
         "toolchain_image":    "sha256:ce66a0100cb8…"
         "toolchain_source":   "staged-archive"
         "toolchain_image_id": "ce66a0100cb8…"

⚠ The GREEN run's process exit is still 1, and that is NOT the kernel: the scratch fixture declares no cargo binaries, so buildgate refuses a bin tally whose denominator is zero ("a gate whose denominator is zero passes vacuously, which is the shape it exists to catch"). The kernel cell itself reports BUILT … 4 artifact(s). Said out loud here because "the CLI exited 1" and "the kernel did not build" are exactly the confusion this whole item exists to not have.

The honest limits. The denial is a network namespace plus an empty store. It does not simulate a box with no podman at all — that is (a), deliberately not built — and it says nothing about a declaration with toolchain_packages, which asks for a package index and reports so through needs_network().

Still open after this

• (c), the kernel ledger read path keyed by the merged FRAGMENT hash — for boxes that must consume a kernel offline rather than produce one. Unchanged by this work.
• (a), a vendored toolchain tarball — still only for a target with no OCI runtime, and still needs its own bound before it may exist.
• Re-pinning the constellation from manifest digests to image ids. Nothing is broken today (the already-present row covers the box that minted the pin), but no declaration can use an archive until its pin is an id. Since lane AV (below) this is a mechanical edit rather than a decision — the recipe is reproducible, so the number is derivable by anyone with the recipe and a podman.
• Publishing the image to a registry remains the answer for the ONLINE half. The archive is the offline courier; a registry is the online one; both address the image by a digest that is declared. They are the same change seen from two sides. It is a convenience now, not a correctness fix: a box with a route can mint the pinned image itself and get the pinned number.

The image is REPRODUCIBLE — MEASURED 2026-08-30 (lane AV)

The whole of the section above rests on a claim that was true when it was written and is not any more: "the committed pin is true on the box that minted it and false everywhere else". Lane AT measured the claim (six mints, six different image ids) and named the fix without measuring it; lane AV built and measured the fix.

image id        3f632569f1980e96744eef4ab258341fc77c55cb17c2dabe79b243e00d6e0825
manifest digest 78eaa55a651d5bd76aae36db0f3307e267a92d2a547ee338a17822a103e0320e

Four mints, four fresh empty --root stores, spread over fifteen minutes, one number — and the same number again from the bollard door (below). Before: six mints, six ids.

Two axes, and neither one alone is worth anything

AT's correction stands and is worth repeating: pinning the inputs is not enough, because an image id is a digest over both axes. So is the converse.

Every knob is load-bearing, and each was removed on its own

Two mints per row, fresh empty stores, everything else held:

The half a Containerfile cannot reach

An image id is the digest of the image config, and the config carries a created timestamp plus one per history entry. Nothing in Dockerfile syntax touches them — they are the builder's, written at commit time. So the instant has to reach TWO places, and exactly one of them may be the writer of it:

• inside the image, as ENV SOURCE_DATE_EPOCH=1783900800, and
• on the builder, as podman build --timestamp 1783900800.

The Containerfile is the writer; KernelBuild::toolchain_source_date_epoch() / kernel_toolchain_source_date_epoch() parse the number back out of that very line, and DraupnirKernelRunner::build_toolchain_image hands it to the builder. One fact, one spelling (LAW 5).

The bollard door, which nobody had measured

AT's mints ran through the podman build CLI. Nothing in the shipped constellation does — [[build.kernel]] mints through DraupnirKernelRunner::build_toolchain_image, which reaches podman's /build endpoint over bollard. And bollard's BuildImageOptions is generated from Docker's API spec, which has no timestamp parameter, so until this lane the shipped door had no way to spell the one flag the measurement depended on. Verified against podman 5.7 that the endpoint does accept timestamp=<seconds>; draupnir now appends it through bollard's own with_request_modifier hook (ContainerBoot::build_image_reproducible), which is a supported extension point rather than a second HTTP client beside the one bollard owns.

MEASURED GREEN, both doors, two fresh empty stores: 3f632569f198… and 3f632569f198…. Gate: tests/toolchain_images_live.rs::the_bollard_mint_and_the_cli_mint_agree_on_the_image_id.

Getting there took three reds, and each one was a default nobody had ever compared — the point of running the gate rather than reasoning about it:

The moral is not "bollard is awkward". It is that the same recipe through two clients is two builds, and until this gate existed the constellation had only ever measured the client it does not ship.

⚠ timestamp is a podman extension. Docker's builder ignores it silently, which is the shape this constellation refuses — so both gates ask the BUILT image what instant it claims ({{.Created.Unix}}) before comparing ids, through a different door than the build.

What it costs, and what happens when it goes away

Minting now depends on snapshot.debian.org, a Debian-project service with no uptime promise. If it goes away the mint fails LOUDLY (apt cannot reach its mirror) instead of quietly drifting; the already-minted image is unaffected; and the recipe still names the exact instant, so any mirror of that snapshot restores it. A build that stops is recoverable; a pin that quietly stops meaning anything is not.

Nothing here reaches the network at a moment the build did not before: this is the mint of the toolchain image, which was already an online step by construction (FROM pulls a base, apt-get installs compilers). A kernel BUILD on the minted image still declares toolchain_packages = [], still runs --network none, and lane AO's NORDISK-TOOLCHAIN-MISSING still refuses by name with no fallback to a registry — re-proven end to end at this lane's HEAD under a verified denial.

What follows for the pin

The pin question dissolves. toolchain_image_sha256 stops being the identity of one minting event and becomes a property of the recipe, so:

• "which box mints for everyone" is no longer a decision. Any box with a podman and a route reproduces the number.
• re-pinning is a mechanical edit: one const in the repo's *-build-thing code, then cargo run -p <repo>-build-thing -- emit.
• both spellings work — the image id 3f632569f198… (which survives a podman save/load courier) and the manifest digest 78eaa55a651d… (which does not, and is therefore still the wrong one to pin for an airgapped box).

The number to pin, if the constellation re-pins to a reproducible mint of THIS recipe, is the image id 3f632569f1980e96744eef4ab258341fc77c55cb17c2dabe79b243e00d6e0825. Lane AV did not take that decision: it is a cross-repo edit beside live emitter work, and the point of the measurement is that it can now be taken with a number in front of the person taking it.

Not yet — do not read this as working