crates/korp-collectors-build-thing/src/recipe.rs
//! **The recipe** — korp-collectors' `impl BuildThing`, the ONE declaration of
//! what this repository builds. `.nornir/build-variants.toml` is emitted from
//! [`KorpCollectorsBuildThing::variants`] (see [`crate::emit`]); nothing here
//! reads it.
//!
//! # What korp-collectors declares, honestly
//!
//! **One package. One shipped binary. One library.** The 2026-07-30 suite
//! inventory listed two CLIs, `collectors_parse` and `tile_gatling`; measured
//! against the tree, those are `[[bench]]` targets (`harness = false`), not
//! binaries. The crate ships `src/bin/korp-collectors.rs` — the unified collector
//! driver — plus the object-safe parser/registry library a consumer such as
//! `facett-demo` links in-process. The stale names stay in this doc comment
//! rather than becoming phantom `[[bin]]` rows.
//!
//! | variant | kind | features | what it is |
//! |---|---|---|---|
//! | `korp-collectors` ⭐ | exe | `parsers` | the driver on the LIGHT surface: DCAT-AP-SE RDF, PxWeb/JSON-stat2, ENTSO-E XML, REST/JSON, Trafikverket → Arrow. No heavy geo dep, no skade, no Iceberg |
//! | `korp-collectors-osm` | exe | `osm` | the driver with the OSM geometry connectors (way→WKB, admin-relation read via osm-katana) and the `RegionSource` fetch seam |
//! | `korp-collectors-osm-daemon` | exe | `osm-daemon` | the driver with the tile DAEMON: planet → region catalog → tiles → ONLINE, plus the post-condition suite that decides whether a fresh pyramid may replace the one being served. MEASURED on oden 2026-08-22: `cargo test -p korp-collectors` runs 112 lib tests, the same command `--features osm-daemon` runs 241 |
//! | `korp-collectors-lib` ⭐ | lib | `parsers` | the parser/registry library itself |
//!
//! `linkage = "not-applicable"` on the library row is a declaration, not an
//! omission: a library is linked by whoever consumes it, and an omitted key reads
//! as the `fat` default — a self-containment claim nobody checked. The driver
//! rows are plain exes: korp-collectors reaches its data sources over HTTP, it
//! embeds no suite sibling, and it is not itself a suite service.
//!
//! # The feature axis is the interesting half
//!
//! Fifteen features; four rows build four of them; the other eleven are excused
//! BY NAME in [`allow_unbuilt`]. That is not bookkeeping. An offered-or-declared
//! feature that no variant builds and nothing excuses is ROT — it is the exact
//! shape that let korp's `spark41` accumulate three invisible compile breaks —
//! and the hand-written recipe this replaces offered four features and named
//! eleven others nowhere at all, so the gate had nothing to ask about them.
//!
//! Every excuse says what the feature IS and why nothing here compiles it. Where
//! a test suite compiles it, the excuse says so, because that is the honest
//! defence against rot: a surface a `cargo test --features …` covers is not
//! silently rotting even if no delivery names it.
use std::collections::BTreeMap;
use nornir_build_thing::cargo_container::CargoToolchain;
use nornir_build_thing::contract::Coordinate;
use nornir_build_thing::plugin::{BuildThing, Manifest, Role, ABI_VERSION};
use nornir_build_thing::variant::{
AllowEntry, ArtifactKind, BuildVariant, BuildVariantSet, Profile, ServiceShape,
};
/// The crate a receipt names as its recipe.
pub const KRATE: &str = "korp-collectors-build-thing";
/// The content hash over this crate's sources, computed by `build.rs`.
pub const CONTENT_HASH: &str = env!("KORP_COLLECTORS_BUILD_THING_CONTENT_HASH");
/// **The delivery this repo ships in.** `suites` is a LIST — one row may
/// ship in more than one flavour — but korp-collectors has exactly one, and writing it
/// down is the whole point: an EMPTY list is not a third flavour, it is a row
/// that `cargo xtask build --suite ORIGINAL` cannot select. All four rows here
/// were exactly that until 2026-09-20. The only other flavour anywhere today is
/// holger's MVP; every other product declares ORIGINAL alone and TAGS it, so
/// the build verb has ONE shape everywhere.
pub const SUITE: &str = "ORIGINAL";
/// The one package this repository builds.
pub const PACKAGE: &str = "korp-collectors";
/// The one shipped binary.
pub const DRIVER_BIN: &str = "korp-collectors";
/// korp-collectors' build thing.
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub struct KorpCollectorsBuildThing;
impl KorpCollectorsBuildThing {
/// The coordinate a receipt names this recipe by: `krate:version:hash`.
#[must_use]
pub fn coordinate() -> Coordinate {
Coordinate {
krate: KRATE.to_string(),
version: env!("CARGO_PKG_VERSION").to_string(),
hash: Some(CONTENT_HASH.to_string()),
}
}
}
fn s(x: &str) -> String {
x.to_string()
}
/// A driver build point: the one `[[bin]]`, with `features` compiled in.
fn driver(name: &str, features: &[&str], is_default: bool) -> BuildVariant {
BuildVariant {
name: s(name),
package: Some(s(PACKAGE)),
bin: Some(s(DRIVER_BIN)),
kind: ArtifactKind::Exe,
role: Some(Role::Tool),
suites: vec![s(SUITE)],
features: features.iter().map(|f| s(f)).collect(),
profile: Profile::Release,
is_default,
..BuildVariant::default()
}
}
/// The rows, in declaration order.
pub fn variants() -> Vec<BuildVariant> {
vec![
driver("korp-collectors", &["parsers"], true),
driver("korp-collectors-osm", &["osm"], false),
driver("korp-collectors-osm-daemon", &["osm-daemon"], false),
BuildVariant {
name: s("korp-collectors-lib"),
suites: vec![s(SUITE)],
package: Some(s(PACKAGE)),
kind: ArtifactKind::Lib,
role: Some(Role::Library),
shape: ServiceShape::NotApplicable,
features: vec![s("parsers")],
profile: Profile::Release,
is_default: true,
..BuildVariant::default()
},
]
}
/// Every feature this crate's `Cargo.toml` declares, `default` excluded (it is
/// an alias for `parsers`, not a surface of its own).
fn offers() -> Vec<&'static str> {
vec![
"parsers",
"osm",
"daemon",
"osm-daemon",
"osm-daemon-download",
"kvm-guest",
"service",
"systemd",
"skade-sink",
"csv",
"mainframe",
"gtfs",
"gribnc",
"znippy",
"testmatrix",
]
}
/// **The eleven features no row builds, each with a written reason.**
pub fn allow_unbuilt() -> BTreeMap<String, String> {
[
("daemon", "the collector DAEMON architecture (Config + State + Collector trio, the CollectorCtx seams, the Phase machine, the registry). Reached by `osm-daemon`, which a row DOES build; no row names it alone because a daemon surface with no daemon implementing it is not an artifact anybody ships."),
("osm-daemon-download", "the in-collector librqbit planet TORRENT (forwards osm-katana/planet-fixture). Deliberately off every build point: a lean build and every test rely on the on-disk seed, and a BitTorrent client in a delivery is a network dependency nobody asked for. Mirrors korp's own `planet-download` gate."),
("kvm-guest", "builds `korp-chain-init`, the PID 1 of the a–e chain drill's QEMU guest. Compiled by `tests/osm_chain_kvm.rs`, never by a delivery — it is a test instrument, and it is kept a NAMED feature precisely so the guest binary cannot be dragged into a substrate build by accident."),
("service", "the nursery: one process supervising many collectors, taking its two runtime decisions from `skidbladnir::readiness`. This is a RUNTIME SUBSTRATE choice made by whoever deploys a collector, not a shape this repository ships; korp wires it."),
("systemd", "the third runtime substrate — a collector as its own systemd unit, through Skidbladnir. Same reason as `service`, which it implies: the substrate is the deployer's decision."),
("skade-sink", "land the parsed Arrow batches in Iceberg through skade. A SINK is the consumer's choice — the same collector feeds a memory sink, a znippy archive or Iceberg — so no driver row picks one. Compiled by `cargo test --features testmatrix,osm,skade-sink`, which is what keeps it from rotting."),
("csv", "the RFC-4180 CSV collector (delimiter sniff, BOM strip, Latin-1 transcode, decimal-comma, typed two-pass schema sniff). A per-SOURCE collector surface: which sources a driver carries is a deployment decision, and pulling every parser into one binary is not one this repo has taken."),
("mainframe", "the EBCDIC + COBOL-copybook + COMP-3 fixed-width collector. Same per-source reason as `csv`; it is fully hand-rolled and pulls zero new deps, so it costs nothing to leave selectable."),
("gtfs", "the GTFS static transit collector (a pure-Rust ZIP reader over the feed, then the csv reader per known file). Same per-source reason as `csv`."),
("gribnc", "the NetCDF/GRIB weather-model collector (hand-rolled pure-Rust GRIB2 + NetCDF-classic decoders, Copernicus CDS + DWD ICON). Same per-source reason as `csv`."),
("znippy", "the collected-data CONVERTER: pack parsed batches into a znippy archive and read them back byte-identical. A SINK/transport choice like `skade-sink`, and it pulls the OpenZL C codec, which a light driver must not carry by default."),
("testmatrix", "turns the tests' `functional_status` self-reports into real nornir test-matrix rows. A TEST-HARNESS mode, never a delivery: a shipped binary that appends matrix rows would be reporting on a run nobody is measuring."),
]
.into_iter()
.map(|(k, v)| (s(k), s(v)))
.collect()
}
/// **The `[[bin]]` no delivery builds**, with its reason.
fn allow_unbuilt_bins() -> BTreeMap<String, AllowEntry> {
[
// 2026-10-16 build-thing sweep (loki): the gate counted these and no row or reason covered them.
("korp-collectors-build-thing", "this recipe crate's own emit door (`cargo run -p korp-collectors-build-thing -- emit` regenerates .nornir/build-variants.toml); the forge loads the crate as wasm, the bin is repository tooling, never shipped."),
("collector-fixtures", "fetches every provider's published sample payloads (with url + date) into committed test fixtures; run by hand to refresh them, never shipped."),
(
"korp-chain-init",
"the KVM guest's PID 1 for the a–e chain drill (`tests/osm_chain_kvm.rs`): mounts /proc /sys /dev and a real ext4 work disk, runs the plan off /proc/cmdline, writes its report to a mailbox block device and powers off. Built by that test behind `kvm-guest`, never by a delivery — a restart is not a `drop()` and `st_blocks` is a property of a real filesystem, which is why the drill needs a machine that can be cut off at the power.",
)]
.into_iter()
.map(|(k, v)| (s(k), AllowEntry::Reason(s(v))))
.collect()
}
/// **The pinned build image every cargo variant compiles in** — `[build.toolchain]`.
///
/// Until 2026-09-10 this recipe declared none, and the consequence was NOT that
/// korp-collectors built on the host. `dwarves buildthing` builds this very
/// crate's wasm32 module INSIDE this container
/// (`dwarves::build_thing::code::run_cargo`), so an absent table meant
/// `CargoToolchain::validate` refused — *"no `image_sha256` — a named tag is a
/// convenience, NOT a pin"* — the code road closed, and the forge fell back to
/// reading the emitted `.nornir/build-variants.toml` as if it were the source.
/// The code-is-source ruling was out of force here for want of one line; the
/// 2026-09-10 inventory render says `toml` for korp-collectors and six siblings.
///
/// The digest is the **MANIFEST** digest (64 lowercase hex, no `sha256:`), read
/// off the engine rather than typed from memory:
/// `cargo run -p nornir-build-thing --features image-build --example
/// image_digest -- nordisk-cargo-toolchain:live` — the bollard door added the
/// same day, because before it the only way to learn a digest was the
/// `podman image inspect` the no-shelling law forbids. The image recipe is
/// reproducible (edda `ac3852e`), so every box that mints it lands on this
/// digest. It moves only when the image RECIPE moves, in the same commit.
/// MOVED 2026-09-20 (loki). The value below was
/// `2cf9b012c9d2ffa42b447e5b0d127184e5a3f998dd2574d771477e9c8e366659` — a
/// second mint of this same recipe which exists on exactly one box, t14s, and
/// sat there UNTAGGED (`<none>:<none>`, 2026-07-13). On loki it does not exist
/// at all, so `CargoToolchain::validate` refused every container build of this
/// repo BY DIGEST — correctly, and that is the whole point of a pin: it told
/// the truth about an environment that was not there. The digest now names the
/// mint that is tagged `nordisk-cargo-toolchain:live` and `:live-uefi` on both
/// boxes, and which composed the live gunnar appliance ISO; `gunnar` and
/// `holger` have pinned it since 2026-09-15. Read back on loki through the
/// engine door, not from podman hearsay:
/// `cargo run -p nornir-build-thing --features image-build --example
/// image_digest -- nordisk-cargo-toolchain:live` ->
/// `3d5115baa07d03ca51eb3e44ebb86b4e64e679bd88f38e156ef9da12c35d2143`.
/// Same `rust_version` 1.97.1, and the target list below is a subset of what
/// that image carries (it adds `x86_64-unknown-uefi`), so the pin moves and
/// nothing else does.
fn toolchain() -> Option<CargoToolchain> {
Some(CargoToolchain {
image: s("nordisk-cargo-toolchain"),
// RE-PINNED 2026-10-01 (lane R, dwarf), `3d5115ba…` → `668468e7…`: edda 6b5f905 added
// clang + llvm to cargo-toolchain.Containerfile (same Debian snapshot, same
// SOURCE_DATE_EPOCH) and no longer mints `3d5115ba…`; this is the new manifest digest.
image_sha256: s("668468e775ad80f32d700cbc9709d41cac726f40d00649461a0932f30d1b2067"),
rust_version: s("1.97.1"),
// What the pinned image CARRIES std for (`rustup target list
// --installed` inside it), not what korp-collectors delivers. wasm32 is
// not optional here: it is the target this crate's own plugin module is
// built for, and a declaration that omitted it would refuse the code
// road by name before the container ever ran.
targets: vec![
s("x86_64-unknown-linux-gnu"),
s("x86_64-unknown-linux-musl"),
s("wasm32-unknown-unknown"),
],
timeout_secs: 3600,
mem_limit_mb: 8192,
cpus: 8,
network: false,
remap_paths: true,
// MEASURED 2026-09-10
// (`--example escaping_siblings /home/rickard/git/korp-collectors`),
// not guessed: `Skidbladnir` (the service support every nordisk service
// must use), `edda` (this crate's own `nornir-build-thing`), `tunnr`
// (`tunnr-vm`), `znippy` (`znippy-common`) and `znippy-zoomies`
// (`osm-katana`, `rotaryengine`). If a sibling path dep is added or
// removed, this list moves in the SAME commit.
//
// Two more that a first-order scan of this repo cannot report, both
// MEASURED on the code road on 2026-09-10 once the pin let the container
// actually run:
//
// * `draupnir` is SECOND ORDER. A sibling is a CHECKOUT the container
// mounts, not a file it copies, so cargo resolves it in its own
// right -- and the `Skidbladnir` above path-depends on draupnir from
// its root manifest.
//
// * `skade` is reached from THIS repo, through a table the scanner
// never visits: `[patch.crates-io] skade = { path = "../skade/skade" }`
// (Cargo.toml:317). A patch entry is not a dependency table, but its
// path is resolved just as hard, and it applies exactly here --
// when korp-collectors is the build ROOT, which is what building
// this recipe crate makes it.
//
// If a sibling path dep is added or removed -- here, in the patch table,
// OR in Skidbladnir's root manifest -- this list moves in the SAME commit.
// 2026-10-16 (build-thing sweep): `openzl-sys-rs` is reached one hop past znippy
// (znippy-common's DEFAULT `openzl` feature → `../../openzl-sys-rs`); cargo loads it
// (measured on loki: hiding the checkout fails `cargo metadata --locked`), so the
// container must carry it. `tests/siblings.rs` walks the transitive closure.
siblings: vec![
s("Skidbladnir"),
s("draupnir"),
s("edda"),
s("openzl-sys-rs"),
s("skade"),
s("tunnr"),
s("znippy"),
s("znippy-zoomies"),
],
})
}
impl BuildThing for KorpCollectorsBuildThing {
fn manifest(&self) -> Manifest {
Manifest {
abi: ABI_VERSION,
repo: s("korp-collectors"),
source_crate: s(KRATE),
summary: "korp-collectors — the object-safe collector trait/registry and the \
parsers behind it (DCAT-AP-SE RDF · PxWeb · ENTSO-E · REST/JSON · \
Trafikverket · OSM geometry · the tile daemon) → Arrow. One driver \
binary and the library a consumer links."
.to_string(),
}
}
fn variants(&self) -> BuildVariantSet {
BuildVariantSet {
// ONE delivery, named rather than left implicit, so --suite ORIGINAL
// resolves here instead of falling through to UNSUITED.
base_suite: Some(s(SUITE)),
offers: offers().into_iter().map(s).collect(),
allow_unbuilt: allow_unbuilt(),
allow_unbuilt_bins: allow_unbuilt_bins(),
variants: variants(),
toolchain: toolchain(),
..BuildVariantSet::default()
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use nornir_build_thing::feature_axis::FeatureAxis;
/// **The feature axis does not rot.** Every feature offered is built by a row
/// or excused by name, and no excuse is stale (a feature a row now builds
/// must lose its excuse, or the next real gap hides behind a line nobody
/// re-reads).
#[test]
fn the_feature_axis_gate_is_clean() {
let set = KorpCollectorsBuildThing.variants();
let axis = FeatureAxis::of("korp-collectors", &set);
if let Err(rot) = axis.gate() {
panic!("the feature axis is rotting: {rot:#?}");
}
}
/// The excuses are about THIS crate: every allow_unbuilt key is a feature the
/// recipe offers. An excuse for a feature that does not exist is a line that
/// can never come true and never be deleted.
#[test]
fn every_excuse_names_a_feature_this_repo_offers() {
let set = KorpCollectorsBuildThing.variants();
for f in set.allow_unbuilt.keys() {
assert!(
set.offers.contains(f),
"'{f}' is excused as unbuilt and is not a feature this repo offers"
);
}
for (f, why) in &set.allow_unbuilt {
assert!(!why.trim().is_empty(), "'{f}' is excused with no words");
}
}
/// **Two benches are not two CLIs.** The 2026-07-30 inventory said this crate
/// shipped `collectors_parse` and `tile_gatling`; they are `[[bench]]`
/// targets. No row may name a binary this crate does not have.
#[test]
fn the_only_binary_a_row_names_is_the_one_that_exists() {
for v in KorpCollectorsBuildThing.variants().variants {
if let Some(bin) = &v.bin {
assert_eq!(
bin, DRIVER_BIN,
"variant '{}' names a binary this crate does not ship",
v.name
);
}
}
}
}