nordisk/znippy

ELF binaries as a znippy format — design

Class: design · siblings: plugins-design.md · signing-design.md · signed-bundle-design.md · ../plugins/native/znippy-plugin-skidbladnir/.nornir/znippy-plugin-skidbladnir.md.

Status: §1–§5 are BUILT and green (plugins/native/znippy-plugin-elf, 53 unit tests

• an end-to-end round trip); §6–§8 are design. It states the shape we ship and, where it departs from the order that commissioned it, says so by name in §10.

1. The order, and what it is for

A built binary should be storable, searchable, verifiable and refusable by name.

Tonight nobody could answer "which artifacts were built with the wrong toolchain image" when :latest had drifted from the pin. That question is the reason this format exists. Every column below is judged against a question somebody actually had to answer and could not. A column that carries no question does not ship.

Three deliverables, in order. This note covers all three plus the follow-on the owner added:

2. The one law that shapes everything: the facts come from the binary

A path is a claim. A header is a fact.

The existing elf handler is a skeleton — znippy-common/src/plugins/native/skeletons.rs:161 — and its extract_metadata parses the filename (skeletons.rs:124, parse_coords). It declares no schema_fields(), so --format elf today writes a name and a version guessed from a file name and nothing else. That is precisely the thing this format exists to stop.

So the rule, stated once and obeyed everywhere below:

E-0 — a path may only widen candidacy; it may never assert a fact. matches_path is a cheap superset. The authority is \x7fELF in the bytes and the structures behind it. A member whose bytes are not an ELF gets no ELF columns — not a guess.

The single exception is deliberate and is itself a column: arch_claim records what the path claims, precisely so the archive can be asked which members lie (§4.6).

3. Which type_id — and why the order is challenged here

The order said "its own fresh type_id". I am taking type_id = 7 instead, and this is the one instruction in the brief I am not following. The reason:

elf is already taken. type_id 7, name "elf", aliases binary / so (skeletons.rs:161-168), and it is listed in .nornir/plugins-design.md §f3 as a reserved id awaiting promotion.

find_handler takes the first match in builtin_handlers() order, and that vector is sort_by_key(|h| h.type_id()) (znippy-cli/src/handlers.rs:57,64-69). A fresh type_id 43 named elf would therefore sit behind the skeleton at 7: --format elf would silently select the filename-only stub and seal the archive with pkg_type = 7 and no columns. That exact failure has already happened once in this repo — the image alias collision between docker (12) and media (25), written up in skeletons.rs:193-200. Repeating it knowingly would be worse than the first time.

So: promote the skeleton, do not shadow it. znippy-plugin-elf takes type_id 7, the name elf and the aliases binary / so, and the ElfPlugin skeleton is deleted in the same change. One id, one handler, no ambiguity — which is what "a fresh type_id" was asking for; only the number is different.

Tier. It needs a real ELF parser and a zlib inflate, so rule P-5 (.nornir/plugins-design.md §f3) puts it in its own crate: plugins/native/znippy-plugin-elf/, not in znippy-common. Promotion out of skeletons.rs and promotion across tiers are the same move here.

4. The columns

4.0 The two constraints that shape the schema

1. The index writer understands Utf8 and UInt32 and nothing else. build_ext_column (znippy-common/src/index.rs:222-256) builds a UInt32Builder for DataType::UInt32 and a StringBuilder for everything else; StrList and Bytes fall through to null. A List<Utf8> column would be a change to znippy's core writer, and a plugin that changes the core writer is not obeying P-2. Every list below is therefore a canonical, documented, splittable string.
2. The base index already carries relative_path, uncompressed_size and a 32-byte blake3 checksum (index.rs:45-52). Duplicating them is forbidden by the same precedent skidbladnir set ("the per-entry BLAKE3 checksum already lives in the base index columns, so it is not duplicated here").

4.1 Identity — what this thing is

elf_build_id is null for a binary built without --build-id. Null is a finding, not a shrug: WHERE elf_build_id IS NULL is "which of our artifacts can never be debugged after the fact".

4.2 The header — what it can run on

4.3 Dependencies — the libssl question

Canonical form: space-separated, each entry soname or soname(VERSION), sorted:

libc.so.6(GLIBC_2.34) libcrypto.so.3(OPENSSL_3.0.0) libssl.so.3(OPENSSL_3.0.0)

SELECT relative_path FROM elf WHERE elf_needed LIKE '%libssl.so.3(OPENSSL_3.0%';

The soname carries the ABI major (libssl.so.3); .gnu.version_r carries the minor the binary actually demands (OPENSSL_3.0.0). Neither alone answers the question, so both are in the string. It is a flat string only because the writer has no list type (§4.0.1) — that is a limitation being recorded, not a preference.

elf_rpath / DT_RUNPATH is deliberately not a column (§10).

4.4 Cost — what a fetcher pays before it fetches

total_bytes is not a column. The base index already carries uncompressed_size (index.rs:49), and the precedent against duplicating a base column is established (§4.0.2). A fetcher gets total_bytes from the base index in the same SELECT.

4.5 Recipe — read from the binary, authority elsewhere

These four are the reason the format was ordered.

Where they come from — and the section we did not know about. The estate already embeds a build receipt in an ELF section, and it is by far the best source for all four columns:

Receipt.build_thing is a Coordinate { krate, version, hash } (contract.rs:123) whose hash is always populated in a receipt — the recipe's content hash. settings are the resolved settings, sorted; inputs carry commit, toolchain and triple. It is BTreeMap on purpose, so two identical builds serialize byte-identically.

So the read order is, per column, most authoritative first, and null rather than a guess:

Two extra columns fall out of the receipt and earn their place immediately:

.note.package stays as the second source, not as a curiosity: it is the systemd/Fedora ELF package metadata note (owner FDO, type 0xcafe1a7e, NUL-terminated JSON), its spec permits additional fields, and a binary carrying it is legible to coredumpctl and to Fedora tooling outside the estate. A nordisk-built binary should carry both; a foreign binary will carry at most one; a stripped-of-notes binary carries neither and every one of these columns is null, which is itself the finding.

Spelling, so nothing new is invented:

• toolchain_image + toolchain_image_sha256 — CargoToolchain { image, image_sha256 } (edda/crates/nornir-build-thing/src/cargo_container.rs:445,448,454), whose image_ref() at :541 is exactly format!("{}@sha256:{}", image, image_sha256), and whose validate refuses an empty digest by name (test an_unpinned_toolchain_image_is_refused_by_name, cargo_container.rs:1992). Bare 64 lowercase hex, no sha256: prefix, a MANIFEST digest — the convention gunnar's xtask/src/build.rs:55 already states in so many words.
• source_commit — 40 hex, the value Inputs::vcs_rev carries (nornir-build-thing/src/provenance.rs:169).

This column already has a live disagreement to find, which is the whole point. .nornir/build-variants.toml in znippy (:60-61) and in gunnar (:2367-2368) both pin nordisk-cargo-toolchain at 3d5115baa07d03ca51eb3e44ebb86b4e64e679bd88f38e156ef9da12c35d2143. dwarves' own template constant (dwarves/crates/dwarves/src/build_thing/code.rs:855) says f29e517c1ba34b920c1234fbd628046d64c0f1072d85ae7dee52b5cce054a917 for the same image and the same rust_version. Two pins, one image, nobody's binary says which it got. That is tonight's question with a file:line, and toolchain_image_sha256 as a column is the thing that turns it into one SELECT.

We are not inventing a section or a field name anywhere in this table.

Authority. These columns are searchable copies, not the record. The authority is the build-thing provenance artifact the statement points at (§6). Two records that can disagree are worse than one that is pointed at — so this note says, and the plugin's doc comment repeats, which is which:

The .note.package fields in the index answer "which ones" in one SQL scan. The referenced build-thing attestation answers "is that true". When they disagree, the attestation wins and the refusal is elf-recipe.

4.6 The lie — the owner's extra query

SELECT relative_path, arch_claim, elf_machine FROM elf WHERE arch_lies = 1;

arch_lies is not derivable in SQL by a reader, because the triple→machine mapping lives in the plugin. That is what earns it its own column rather than leaving the reader to CASE WHEN. arch_claim IS NULL means the path claimed nothing — which is not a lie, and arch_lies is 0.

This is the same condition as the elf-machine refusal (§5), seen from the other end: the column is the archive-wide sweep, the refusal is the per-start check. One rule, two vantage points, no second implementation.

4.7 The audit root

.dep-v0 is cargo-auditable's embedded dependency tree: a section holding zlib-compressed JSON ({"packages":[{"name","version","source","kind","dependencies", "root"}]}). Again — an existing convention, read rather than reinvented. The inflate uses linflate (znippy-zoomies/linflate/src/lib.rs:236, inflate_to_vec), which znippy already depends on through znippy-common — no new decompressor, LAW-conformant reuse.

The whole tree is not a column (§10). The section's bytes are in the archive; the column is the entry point.

4.8 The full schema — 22 columns

member_kind  elf_build_id
elf_machine  elf_class  elf_osabi  elf_type  elf_interp  elf_abi_level
elf_needed
text_bytes  debug_bytes  has_debug
recipe_coordinate  recipe_variant  recipe_features  recipe_target
toolchain_image  toolchain_image_sha256  source_commit
arch_claim  arch_lies
auditable_root

UInt32: text_bytes, debug_bytes, has_debug, arch_lies. Everything else Utf8, all nullable.

logical_path is not among them. skidbladnir and rust-toolchain carry it because their logical key differs from the stored path (dokument.fil_nyckel). ELF has no such second name; the base index's relative_path is the path. Copying a column because the siblings have it is exactly the cross-linking the brief forbids.

5. The refusals

House style, verbatim from the estate (gunnar/xtask/src/isoinstaller.rs, edda/crates/korp-installer/src/installer_source.rs:257):

REFUSED [<kebab-name>] <one sentence naming BOTH sides and the fix>

The elf-* namespace is free — nothing in gunnar, holger or edda uses it today.

None of these excludes hardware. Every one of them turns a box that starts and dies into a box that says why, before it starts.

How each fact is obtained — pure Rust, no shelling (LAW):

• host machine — std::env::consts::ARCH.
• host ISA level — std::arch::is_x86_feature_detected! / is_aarch64_feature_detected!. These are std macros reading CPUID/HWCAP in-process. No /proc/cpuinfo parse, no subprocess.
• interpreter presence — std::fs::metadata on the PT_INTERP path.
• the coordinate — the archive's own index row plus the DSSE statement (§6).

Three-way, never binary — the shape installer-source-guard already proved. edda/crates/korp-installer/src/installer_source.rs:236-268 has Verdict { Match, Unchecked, Unpinned }: an unpinned build is recorded, never silent, and only a genuine contradiction refuses. elf-recipe takes the same three states, and Unchecked is printed as a named skip, because GATE-IDENTITY's rule (gunnar/deploy/upcloud/src/banner.rs:17-31) is the estate's: a check nobody can see not running is worse than no check at all.

GATE-IDENTITY's other rule applies directly here and is worth stating as the reason this whole format is built the way it is: the only trusted evidence is what the artifact says about itself. An API saying 200 OK can mean a brick. A path saying x86_64 can mean an aarch64 binary. The ELF header cannot lie about the ELF header.

Where they live. A typed enum in the plugin crate with a Display writing the operator sentence, the shape gunnar/crates/gunnar-server/src/registry.rs:39 (enum Refusal { NotAttested, ProvenanceForged(String), … }) already uses for provenance admission. The refusal check is a pure function of (ElfFacts, HostFacts, Coordinate) -> Vec<Refusal> so every arm is unit-testable without a box, a binary or a boot.

6. Provenance: reference the build-thing artifact, do not restate it

6.1 Almost all of this already exists — build the binding, not a second scheme

So: no new predicate, no new envelope, no new signer, no new store, no allowlist change. The brief said "do not add a second signing scheme"; the correct reading is stronger — do not add a second anything. What is genuinely missing is three things, and only three.

6.2 Missing thing 1 — the reference to the build-thing artifact

The answer to "is there a stable digest to point at" is: yes, but not the one you would expect, and the link is missing at one specific joint. Three candidates, judged:

(A fourth exists and is a trap: dwarves .../build_thing/code.rs:244 code_key covers the recipe crate and every in-repo path dep transitively — strictly better coverage — but it is truncated to 8 bytes / 16 hex (code.rs:267) because it is a wasm cache key. Truncated is not an identity. Not usable as a statement reference.)

So the ELF statement carries two resolvedDependencies references, neither a copy:

{ "name": "build-thing-provenance",
  "uri":  "nornir.build-provenance/v1/<BuildProvenance.id>",
  "digest": { "sha256": "<Attestation::digest_hex()>" } },

{ "name": "build-thing-recipe",
  "uri":  "nornir+buildthing://<krate>@<version>",
  "digest": { "sha256": "<Coordinate.hash>" } }

externalParameters carries only the minimum needed to find them. A verifier loads the referenced attestation (from the same archive, or a local store — §7) and re-derives the recipe instead of trusting a transcription. One record, pointed at, twice.

6.3 Two joints are missing, and both are the owner's call

Stated plainly, at once, because together they are what blocks step 3 — and neither blocks the columns, the refusals or debuginfod.

(a) BuildProvenance has no build-thing field at all. provenance.rs:182 carries schema, id, repo, variant, inputs, toolchain, host, outcome, artifacts, built_at, builder — and no Coordinate. to_slsa_statement (:297) names the variant only as a buildType string, https://nordisk.dev/nornir/build-thing/variant/{name}. The recipe→artifact link exists only inside the binary, in the .nordisk.buildthing receipt (§4.5). Today the signed statement therefore cannot reference the recipe by digest. The fix is build_thing: Option<Coordinate> on BuildProvenance, threaded through capture (:212) and to_slsa_statement (:297).

(b) The toolchain image never reaches the record either. BuildProvenance records Toolchain { rustc, cargo, host } in internalParameters. That is the rustc on the build host; it is not the container image. CargoToolchain::image_ref() already produces the exact name@sha256:hex string (cargo_container.rs:541) — and it reaches no attestation. The fix is a resolvedDependencies entry, where SLSA puts materials:

{ "uri": "<toolchain_image>", "digest": { "sha256": "<toolchain_image_sha256>" } }

Both are changes in edda, another lane's repo. I am not making them. Consequence, said out loud: until they are made, the referenced build-thing artifact cannot answer tonight's question either — the ELF columns make it searchable from the binary's own receipt, which is the half that works today, but elf-recipe must report Unchecked for the toolchain image, by name, rather than pretend.

A caveat travels with (b), from dwarves/.nornir/build-thing-model-plan.md:79: image_sha256 is only reproducible for images whose recipe is itself pinned (edda's kernel-toolchain.Containerfile — digest-pinned base, Debian snapshot, pinned pip versions, SOURCE_DATE_EPOCH, OCI manifest format — four mints, one digest). nordisk-cargo-toolchain has had none of that work. Pinning it is a real check; reproducing it is not yet. And toolchain_archive.rs:61,720 warns the kernel-side value can be a per-box image id rather than a manifest digest — so the column must record what it found and never normalise the difference away.

6.4 Missing thing 3 — verification must look at the disk

The statement's subject[].digest must be compared against the bytes actually on the disk, recomputed at verification time. Statement::attests(&self, want: &ArtifactDigest) (attest.rs:410) is the comparison; supplying want from a fresh read of the file is the part that must not be skipped. "The payload is signed" proves a statement exists — it proves nothing about the file next to it. The verifier therefore:

1. read the file on disk → SHA-256 → ArtifactDigest
2. AttestStore::accept(envelope, certs, Some(&that_digest)) — subject binding before cryptography, which is already the order accept enforces (attest.rs:768+)
3. follow resolvedDependencies[build-thing-provenance], load it, digest_matches_record()
4. compare the ELF index row's recipe columns against the re-derived record → elf-recipe
5. anti-rollback floor

6.5 The anti-rollback floor

holger's floor (plugin/host/src/floor.rs) is keyed (supplier, product) over the module's own MAJOR.MINOR.PATCH, persisted at <dir>/<supplier>/<product>.floor, with Floor::{InForce, Absent} — absence is a value, not an Option, and an unreadable floor is a refusal (plugin-11), never Absent. That is the right shape and the right implementation and an ELF artifact does not get a second one.

But the floor's lane is (supplier, product) from ModuleIdentity, which a plain ELF binary does not have. Extending FloorStore to a second kind of lane is a change to holger's plugin host. This note records the need and stops there — it is the same class of decision as §6.3 and it is named here so it cannot be forgotten.

6.6 Where the envelope lives

One statement per artifact, as its own member of the archive:

provenance/<elf_build_id>.dsse.json

elf_build_id is the key because it is the one identifier that survives stripping and renaming (§4.1) — and it is the same key debuginfod uses (§8). One key, three uses.

The archive as a whole is still sealed by znippy's existing detached CMS over the archive_root merkle fold (.nornir/signing-design.md). Two layers, both already built: CMS answers "was this archive tampered with", DSSE answers "where did this one binary come from". Neither replaces the other; this is the same additive shape Attestation already uses, where dsse sits beside seal rather than replacing it (provenance.rs:450-456).

7. Airgap first, connected supported — one format, two modes

The requirement: nothing may need a network to be trusted. A connected box must use the same objects the ordinary way.

Airgap (the default). The DSSE envelope is a member of the archive (§6.6). The referenced build-thing attestation is a member too. The signer certificates ride in the envelope's signer_certs — DsseSeal carries them precisely because "a DSSE envelope itself carries only a keyid" and airgap verification must recover the public key with no directory to call (provenance.rs:394-410). Trust roots are local DER files (attest.rs:551). The store is a local directory. Every step of §6.4 runs with the NIC down.

There is also an existing on-disk convention for the sealed record and it is kept: <artifact>.build-provenance.znippy beside the binary (nornir/src/build/seal.rs:223 dump_beside_artifact), with ProvenanceMode { Cms (default), Dsse } at :104 selecting which seal is applied. The ELF statement rides in the archive (§6.6); the sibling file remains what a loose artifact carries. Same bytes, two places.

Connected. The same envelope bytes are published as an OCI 1.1 referrer, hanging off the binary's digest, with artifactType set from the payloadType. An SBOM hangs off the same subject the same way. A connected box does a plain GET /v2/<name>/referrers/<digest> and gets the identical envelope it would have found in the archive.

That door does not exist yet — say it plainly. referrers appears nowhere in holger, dwarves or gunnar. What exists is OCI manifest handling only: gunnar/crates/gunnar-server/src/oci.rs:350-353, gunnar/crates/gunnar-http/src/registry.rs:314, holger/server/repository/file-oci/, holger/server/repository/znippy-docker/. None implements /v2/<name>/referrers/<digest>. So the connected mode is a new door on holger, built alongside debuginfod in step 4 and following the same rules (§8): new door, existing hyper pattern, nothing else edited, no deploy without telling the owner. Until it exists, the connected case is served by the airgap objects over ordinary HTTP — which works, and is not an excuse to invent a second format for it.

The property that makes this one format and not two: the envelope is content-addressed by its subject, which is the binary's own digest. The archive and the registry are two transports for one object. Nothing is re-signed, re-encoded or re-shaped in transit; a byte comparison of the two must succeed, and the test says so.

8. holger speaks debuginfod

debuginfod is the standard HTTP protocol for fetching debug information by build-id. Three endpoints, and that is the whole protocol:

GET /buildid/<build-id-hex>/debuginfo
GET /buildid/<build-id-hex>/executable
GET /buildid/<build-id-hex>/source/<path>

200 with the bytes, 404 with nothing. gdb, perf, eu-*, coredumpctl and rustc's backtraces all speak it already, out of the box, by setting DEBUGINFOD_URLS.

Why it belongs to this lane. §4.4 already splits .debug_* out and measures it, and §4.1 already keys on elf_build_id. A lean appliance ships no debug info at all — debug_bytes of saving — and a developer still gets a symbolized backtrace, because holger can serve exactly the split this format already made. The three endpoints are a SELECT … WHERE elf_build_id = ? against the index this note designs.

Constraints, in order:

• Airgap first. It must work against a local holger with no internet, serving only what is in local archives. No upstream federation in v1.
• Do not touch holger-server's other doors. holger's HTTP is hyper-based in server/lib (server/lib/Cargo.toml:131). New door, existing pattern, nothing else edited.
• Back up what is changed. Do not deploy to holger.rs without telling the owner.
• It is read-only and unauthenticated by default — it serves debug info, which is derived from artifacts the requester already has. If that turns out to be wrong for this estate it becomes a gate, and the gate gets a name.

This is step 4. It is designed here so the columns in §4.4 are chosen for it, and built after §6.

9. Laws honoured

10. Deliberately left out — and why

11. Build order

1. this note ✔
2. plugins/native/znippy-plugin-elf — type_id 7, 22 columns, the skeleton deleted, the refusal enum, a test per column and a test per refusal, each shown red before green
3. the DSSE binding — the resolvedDependencies references to the build-thing attestation and recipe coordinate, and verification that recomputes the subject digest from the disk
4. holger speaks debuginfod, and the OCI 1.1 referrers door beside it

Three owner decisions block the authority half of step 3 — §6.3(a) build_thing on BuildProvenance, §6.3(b) the toolchain image in the record, §6.5 a second floor lane. None of them blocks step 2, step 4, or the searchable columns — which is the half that answers tonight's question, and it answers it from the binary's own receipt.