nordisk/edda

Real metal — what must be true, and what only you can do

Written 2026-08-04 on oden (load average 79–218 all session; every timing below says so where it matters). Everything here is about the appliance ISO, which has only ever booted under QEMU/OVMF.

This is deliberately a list of your steps and unprovable claims, not a status report. The code changes that make these steps possible are landed; what is left is custody and hardware, and neither is mine to hold.

0. What is proven, and on what

The line that used to read signature-over-root=NOT-ENROLLED on every boot, taken from the e2e's QEMU console on oden 2026-08-04 (load average 11.6 at the time; the run took 111.65 s end to end and the appliance answered SSH 5.87 s after power-on):

INSTALLER: start-app root-of-trust — verifying app .znippy
INSTALLER: selftest startup root=72e0e50d361d archive-root-blake3=PASS \
           signature-over-root=PASS per-chunk-blake3=PASS
BOOTED AND LISTENING in 5.866121199s — banner "SSH-2.0-gunnar_0.1.0"

Re-measured 2026-08-05 on v1.0-4-g008d74b, with no signing key set, so the honest line is the un-enrolled one and the banner now names the build:

INSTALLER: selftest startup root=f329606c3f68 archive-root-blake3=PASS \
           signature-over-root=NOT-ENROLLED per-chunk-blake3=PASS
BOOTED AND LISTENING in 5.911152673s — banner "SSH-2.0-gunnar_0.1.0 v1.0-4-g008d74b-modified"

Forge green in 39.9 s, ISO 29 456 384 B. Booted by tunnr rather than a hand-rolled QEMU argv, cargo xtask install-test iso reaches its listener in 3.66 s (a smaller guest and a quieter box than the 5.87 s above; both are QEMU numbers and neither is metal).

That PASS was produced with KORP_ROOT_SIGNING_KEY=dev, i.e. the published development key. The mechanism is proven end to end; the anchor is not real until §1. | a pinned kernel that drifts is refused | pin_verdict, red-proved | korp-installer/src/forge.rs | | the PEs can be Authenticode-signed | pure-Rust, both images | korp-installer/tests/secure_boot.rs | | secrets are not on disk in the clear | assertions over the bytes of secrets.arrow | gunnar-core/tests/sealed_secrets.rs |

Nothing below this line has ever run on physical hardware. QEMU/OVMF is a faithful UEFI implementation and a poor model of a real machine's firmware quirks, boot order, NIC enumeration and NVRAM behaviour.

1. Enroll a real root key — only you can do this

The build now signs the payload's blake3 root and stages korp/app.sig + korp/root.pub, and the appliance verifies them and halts on a mismatch. Until you supply a key it either signs with a published development key or enrolls nothing.

# generate a real seed OFF this box, on something you trust, and keep it offline
head -c 32 /dev/urandom > korp-root.seed     # 32 raw bytes; hex also accepted
chmod 400 korp-root.seed

# build with it
KORP_ROOT_SIGNING_KEY=/path/to/korp-root.seed \
  cargo test -p dwarves --features appliance-iso --test gunnar_appliance_iso -- --ignored

The build prints the enrolled public key. Compare it, every time, against the one you recorded at the key ceremony — the whole anchor is that comparison.

• KORP_ROOT_SIGNING_KEY unset ⇒ the appliance prints signature-over-root=NOT-ENROLLED and boots. Honest: integrity, no authenticity. This is still the default.
• KORP_ROOT_SIGNING_KEY=dev ⇒ signed with the seed published in korp-installer/src/rootkey.rs, which is 32 bytes of ASCII reading korp DEVELOPMENT key NOT SECRET!. Anyone can produce that signature. It proves the path works and anchors nothing. The build says so loudly.
• I generated no production key and no file that could be mistaken for one.

To make an un-enrolled appliance refuse to boot rather than shrug, set KORP_REQUIRE_SIGNED=1 in the kernel command line. Do this only once you are enrolling for real; otherwise you brick your own demo.

2. Pin a kernel — needs a kernel you control

Today the ISO packs whatever /boot/vmlinuz points at on the build box. It is now reported (KernelProvenance::HostBoot) with its measured blake3 rather than passing silently, but it is still not a pinned kernel.

korp-installer/kernel/korp-kernel.blake3 still holds the all-zeros placeholder, and forge::tests::the_placeholder_pin_is_not_a_pin will tell you the day that changes.

KORP_KERNEL=/path/to/your/bzImage cargo test ...     # prints the measured blake3
# paste that hex into korp-installer/kernel/korp-kernel.blake3, then rebuild:
# a mismatch now REFUSES the build

The kernel must ship its config-<version> beside the bzImage. With a pin in force, a kernel whose config cannot be read is now a hard refusal rather than a warning, and that is deliberate:

the initramfs is zstd (level 19). A kernel built without CONFIG_RD_ZSTD=y boots, fails to unpack the initramfs, finds no root filesystem, and panics — and says nothing whatsoever at build time. On real metal you discover that standing in front of the rack.

Minimum config to verify by hand: CONFIG_RD_ZSTD=y (and CONFIG_DECOMPRESS_ZSTD, which follows it), plus an EFI stub, plus in-tree drivers for the NIC and storage on the target machine — an initramfs with no modules cannot load any.

3. Secure Boot — the certificate ceremony is yours

On a machine with SB enabled, an unsigned ISO does not start. The firmware says Access Denied and nothing else — no line about signatures, no hint.

The build can now produce a signed artifact:

cargo test -p korp-installer --features forge,secure-boot ...
KORP_SECURE_BOOT_KEY=/path/to/key.pkcs8.der \
KORP_SECURE_BOOT_CERT=/path/to/cert.der \
  ...

Both EFI/BOOT/BOOTX64.EFI and vmlinuz are signed, not just the first stage. korp-boot chainloads the kernel through the firmware's own LoadImage/StartImage, so the firmware verifies the kernel too; signing only the loader gives you an ISO that starts and then dies at the handoff, looking exactly like a broken kernel.

What is not done, and cannot be done from here:

1. A signature is worthless until your certificate is in the machine's db. korp_installer::secureboot::enroll_platform_keys does the offline PK/KEK/db ceremony against a varstore image, and it is proven against a real factory OVMF varstore — but enrolling into your machine's NVRAM is done at that machine's firmware setup, by you, usually with SB in Setup Mode.
2. Microsoft's UEFI CA is not involved. No shim, no MOK. That means this boots on machines where you control db, and does not boot on a locked-down OEM machine whose db you cannot change. If you need the latter, the answer is shim + a Microsoft-signed loader, which is a different project.
3. KORP_SECURE_BOOT_KEY=dev generates a throwaway self-signed identity. It proves the pipeline produces genuinely signed PEs. No firmware on earth has that certificate, so an SB-enabled machine still refuses it. The build says so.

Asking for signing without --features secure-boot is now an error rather than a silently unsigned image, because the failure mode is Access Denied with no diagnostic.

4. Secrets — one decision to make

Secrets are now sealed at rest (AES-256-GCM, D26). By default the key-encrypting key is secrets.kek (mode 0600) beside the control store.

Decide whether that is enough for the deployment. It defeats the archive leaving the machine — a cold-tier .znippy copied to object storage, a backup, a replica's secrets.arrow. It does not defeat someone holding the disk. If you need that, set GUNNAR_SECRET_KEK (64 hex chars) from a KMS/TPM at boot and no KEK is written to disk at all.

Losing the KEK loses every secret on the machine, including the SSH host key — after which every client sees REMOTE HOST IDENTIFICATION HAS CHANGED. Back it up wherever you back up keys.

5. At the machine — the list

Nothing here can be checked from oden.

• Write the ISO and boot it once with SB disabled. Confirms the kernel, the NIC and the storage controller before signing enters the picture.

• Confirm eth0 exists. pid1 exits if it finds no eth0, the kernel panics, and the box reboot-loops in a way that looks exactly like a corrupt image. Real NICs enumerate differently from virtio; a predictable name (enp3s0) is not eth0.

• Read the serial console. The three legs print as one line. You are looking for archive-root-blake3=PASS, per-chunk-blake3=PASS, and — once §1 is done — signature-over-root=PASS rather than NOT-ENROLLED.

• Boot order / removable media. The ISO relies on the fallback path EFI/BOOT/BOOTX64.EFI. Some firmware will not consider removable media without being told; some needs CSM off.

• Then enable SB and boot the signed ISO (§3). If it fails, the failure is silent by design — suspect db, not the image.

• Check the appliance's SSH host key is stable across a power cycle, not just a reboot. That is the sealed-secret path doing its job on real storage.

For the **live ISO** this is already answered, and the answer is **no**. Measured
2026-08-05 by `cargo xtask install-test iso`: two boots of one ISO answered with two
different ed25519 host keys, because the store the key lives in is the initramfs,
i.e. RAM. Every client that pinned the appliance sees `REMOTE HOST IDENTIFICATION
HAS CHANGED` after a power cycle. The same run also asserts that a repository pushed
before the cut comes back **empty**, so this is the shape of the medium and not a
bug in the secret path. The item above is therefore about the **installed** leg — a
disk the installer wrote and the machine reboots off — which is where a stable host
key is both possible and required.

• Pull the power mid-push, twice. The cold-tier fold is crash-safe by construction (stage, fsync, rename, fsync the parent) and that has never been tested against a real disk cache that lies about flushes.

6. What remains unprovable without hardware — stated plainly

• Firmware behaviour. OVMF is not an OEM firmware. Boot order, NVRAM persistence, LoadImage strictness under SB, and how a real machine treats a hybrid ISO are all untested.
• Device enumeration. Every driver question above is answered by QEMU's virtio devices today, which tells you nothing about the target machine.
• Storage durability. Crash safety is argued from the write ordering, and the ordering is right; whether the drive honours the flush is a property of the drive.
• Timing. The ~6 s boot is a QEMU number on a box under load 79–218. Real POST alone will dominate it.
• Whether the pinned kernel actually boots that machine. The build can prove the kernel matches its pin and can unpack zstd. It cannot prove the machine likes it.

The honest summary: the chain of trust is now complete and tested end to end — archive integrity, payload authenticity, kernel identity, and image signing all exist and all refuse the wrong input. What has never been tested is a machine.