nordisk/edda

build-info — the second half of a build (schema + emitter)

A build produces two things at once: the thing (artifact bytes, hashed by BuildProvenance and — a later slice — stored in holger's znippy CAS) and the build info (the columnar record ABOUT the build). This note wires the second half in nornir-build-thing: the §f3 Arrow schema, a flat dep-light row, and a pluggable BuildInfoSink with an in-process default. The full constellation design (Flight seam, at-rest seal, jera DAG) is workspace_nordisk/.nornir/ build-info-flight-stream-design.md; this is the build-thing-local slice 2.

The schema (§f3) — one row, six kinds

Build info is inherently record-shaped, so it IS columnar. One row shape (build_info::BuildInfoRow), discriminated by the kind column, carries the whole stream — the Artifactory-parity manifest rows (provenance / dep / artifact, derived from BuildProvenance) AND the live feed (log / metric / step). Columns, in declared order:

build_id (Utf8) · step_id (Utf8) · ts_micros (Int64) · kind (Utf8) · level (Utf8) · name (Utf8) · value (Utf8) · content_id (Utf8) · blob (Binary, the only nullable column).

Dep-light by construction

The rows + the sink are pure std + serde — a leaf repo's xtask gets the emitter for free. The Arrow RecordBatch binding (build_info_schema() / build_info_batch()) is gated behind the build-info-arrow feature, so the default crate keeps its no-heavy-deps promise. The feature reuses the arrow builder plumbing (arrow-array / arrow-schema, the crates + version znippy pins) rather than re-rolling arrow; znippy's own build_metadata_batch builds a different schema (blob-index), so only the plumbing is shared, not the shape.

The sink seam

BuildInfoSink is transport-agnostic (emit(&[BuildInfoRow])):

• InMemBuildInfoSink — the in-process default (a shared, cloneable Vec): single-box localhost, zero transport tax. Hand a clone to the build, read the rows back afterwards.
• NullBuildInfoSink — the no-op default so existing build_variant() callers pay nothing.
• Flight-backed sink — a later slice (the edda/nornir-flight seam is the transport); not wired here. Same schema + sink API both ways.

The emitter

build_variant_with_info(set, name, repo_root, sink) runs the same build as build_variant (which now just routes to a NullBuildInfoSink, so callers are unchanged) and emits: the live feed (step/log/metric for the cargo-build step) as it finishes, then the manifest rows from the captured BuildProvenance. Emission is best-effort — a sink error never fails the build.

Tested (RED-when-broken)

• arrow_schema_has_f3_columns_in_order — exact column order + types + the blob-only-nullable invariant + the schema-version metadata key.
• manifest_rows_cover_provenance_dep_and_artifact — a known captured build yields the expected provenance / dep / artifact rows; content_id == sha256.
• in_mem_sink_round_trips_rows — what is emitted comes back byte-identical.
• arrow_batch_matches_schema_and_values — the RecordBatch shape + cell values.
• emits_functional_rows_for_build_info (feature testmatrix) — honest matrix rows for the emit + round-trip.

Slice 3 — native warehouse landing + at-rest seal (shipped)

The build-info feed no longer rides test_results as an interim; it lands in its OWN warehouse table with its OWN submit path, and seals at rest:

• build_info table (warehouse::build_info, iceberg_schema::build_info()) — the §f3 columns 1:1 (build_id · step_id · ts_micros · kind · level · name · value · content_id · blobnullable), the exploded sibling of the one-row-per-build build_provenance ledger. append_build_info / query_build_info reuse nornir_build_thing::build_info::BuildInfoRow verbatim — the same struct the emitter and the seal use, so live wire, at-rest blob, and warehouse row are one shape (no second format).
• Submit RPCs (Telemetry, mirroring SubmitTestResults): SubmitBuildProvenance ships each BuildProvenance as JSON (record can gain fields with no proto bump) → append_build_provenance; SubmitBuildInfo ships typed BuildInfoRowPb rows (has_blob distinguishes a NULL blob from an empty one) → append_build_info. Client helpers: remote::submit_build_provenance / submit_build_info. A dwarves build on the dev's hardware ships both to the locked server warehouse, exactly like bakeoff/test/release rows.
• At-rest seal (build::seal::seal_build_info_beside_artifact, feature build-provenance) — the §f3 rows sealed as znippy over Arrow-IPC → <artifact>.build-info.znippy, the airgap-portable sibling of <artifact>.build-provenance.znippy.
• Viz — the generic warehouse deck lists tables via table_names(), so build_info + build_provenance show up natively with no bespoke pane.

Guarded: submit_build_info_and_provenance_land_natively (tests/data_submit_rpcs.rs) submits real rows to an ephemeral server and reads the exact values back over Warehouse.Scan; warehouse::build_info unit tests round-trip the manifest + the nullable blob; seal_build_info_writes_znippy_ beside_artifact proves the seal. The root build-info-arrow feature forwards to nornir-build-thing/build-info-arrow, and tests/cargo_features_guard.rs fails RED if any feature ever references an undeclared dep:.

Slice 4 — jera pipeline-DAG sub-jobs (§f5, shipped)

A build/pipeline is not one job — it is a root jera job whose body is a DAG, and every step is a sub-job. Modelled in nornir::build::pipeline:

• The DAG is data — reuse, don't re-roll. A PipelineSpec (a name + a list of PipelineStep{ id, name, needs, kind }) projects onto funnel-core's PlanDag verbatim (PipelineSpec::to_plan_dag): nodes = sub-jobs, edges = needs dependencies (EdgeKind::Dep, need → step). So topo_order() / critical_path() / depths() (the ready-wave layering) all work for free — no new graph type. A one-step spec is a one-node zero-edge "DAG", so the existing single-job path is unchanged (additive).
• The ledger is the sub-job hierarchy — reuse, don't re-roll. PipelineSpec::run_on_ledger starts a root nornir-jobs job (kind ci_build) that OWNS the DAG, then admits every step as a child sub-job (parent_id = root's job_id) queued up front, promotes each running in ready-wave order, and terminates it done/failed. A step whose needs did not all reach done is left blocked → failed without running. The needs edge set is stamped into each sub-job's detail_json, so the DAG is recoverable from the ledger, not only the projection. The viz Jobs tree already folds children under their parent by parent_id — no viz change needed.
• build_id / step_id contract. The root job_id is the §f3 build_id; each sub-job's job_id is the §f3 step_id. PipelineOutcome::build_info_for(rows, step) filters a slice-2 stream to one sub-job (build_id == root && step_id == sub-job job_id) — "each step's log/metric stream is the f3 feed filtered by step_id."
• One engine for nornir AND dwarves (LAW: reuse) — the same root job → pipeline DAG → sub-jobs model, not two schedulers.

Guarded (src/build/pipeline.rs, RED-when-broken): three_node_dag_makes_root_plus_three_subjobs_with_needs_and_status (a fetch → compile → test DAG creates 1 root + 3 sub-jobs on the ledger with the right needs edges + per-step done), failed_step_blocks_dependents_and_fails_root (compile fails → test blocked → root failed), single_step_is_a_one_node_dag_additive, build_info_stream_filters_per_step_id_to_each_subjob, cyclic_pipeline_is_rejected, diamond_dag_reuses_funnel_core_critical_path. Emits nornir_testmatrix::functional_status rows for the three headline surfaces.

Slice 5 — the Flight-backed sink + consumer (§f1/§f2/§f5, shipped)

The arc closes: the same build-info stream is now in-process (default) OR Flight (remote) behind the ONE BuildInfoSink trait, exactly as the design says.

• FlightBuildInfoSink (nornir::build::flight_sink, feature build-info-flight) — the remote counterpart of InMemBuildInfoSink. It serializes each BuildInfoRow batch to the §f3 Arrow RecordBatch and ships it via edda/nornir-flight's do_put. It lives in nornir because it is the one place that deps BOTH nornir-build-thing (the trait + rows) and edda/nornir-flight (the raw transport) — no cycle (edda knows nothing of build-info; build-thing knows nothing of Flight). emit is best-effort: rows are buffered to a worker that groups by route key and do_puts one batch per key; a transport error is surfaced via flush() -> FlightStats, never failing the build.

• Route key = per sub-job step_id. route_key(build_id, step_id) → build_id for root/manifest rows (empty step_id), else build_id/step_id. So a consumer subscribes by ticket per sub-job (§f5: "each step's log/metric stream is the f3 feed filtered by step_id").

• Consumer. subscribe(endpoint, token, key) does do_get(key) and decodes the batches back into BuildInfoRows byte-identical to what was emitted.

• Auth via nornir-service. The sink and consumer carry a bearer token; the FlightServer composes the shared nornir_service::auth_interceptor. An unauthenticated do_put/do_get is rejected — proven RED-when-broken.

• The arrow version boundary — ⚠ IT IS GONE (verified 2026-08-24). This used to read: "nornir-flight rides arrow-56 (its arrow-flight → tonic 0.13 matches nornir-service; arrow-flight 57+ needs tonic 0.14), while build-info-arrow is arrow-58. Those are semver-incompatible majors…"

Measured against the tree, both sides are arrow 59: nornir-flight/Cargo.toml:42-44 declares arrow-array/arrow-schema/ arrow-flight all "59" on tonic 0.14, nornir-build-thing:109-110 declares arrow-array/arrow-schema "59", and edda's Cargo.lock carries exactly one arrow family — 59.2.0, arrow-flight 59.2.0 included. The tonic 0.13/0.14 split that forced the gap resolved itself when nornir-service moved to 0.14.

rows_to_batch / batch_to_rows stay, and are not dead code: converting BuildInfoRow to a RecordBatch is the serialization layer, needed at any arrow version. What changed is the REASON — they are no longer a bridge across incompatible majors, so nobody should preserve them as one or go looking for the arrow-56 half. The test pinning their column shape against the canonical build_info_schema still earns its keep: it is now a same-major drift guard.

• Additive. The whole module is gated behind build-info-flight (OFF by default): the in-process InMemBuildInfoSink stays the default with zero transport tax and no second arrow in the default build.

Guarded (src/build/flight_sink.rs, RED-when-broken): flight_schema_matches_canonical_f3_columns (the §f3 shape + no-drift), batch_round_trips_rows_byte_identical (arrow-56 columnize/decode lossless incl. the NULL-vs-present blob), flight_sink_ships_and_consumer_receives_byte_identical (a mixed manifest+feed stream shipped over do_put, consumed per route key over do_get, byte-identical), auth_is_enforced_on_the_build_info_wire (no bearer ⇒ do_put/do_get rejected), and emits_functional_rows_for_flight_build_info (feature testmatrix — honest do_put/do_get matrix rows).

Slice 5 tail — the viz build-info LIVE-TAIL pane (shipped)

The arc's last open thread — surfacing the received stream in the viz — is now wired: a build-info live-tail pane in the 📡 Live Run area subscribes the §f3 stream per sub-job step_id and shows the rows arriving as they land.

• BuildInfoTail (src/viz/build_info_live.rs, feature viz) — a pure fold over BuildInfoRows that knows nothing about how the rows arrived. ingest(&[BuildInfoRow]) folds each row into the per-sub-job state keyed by step_id: a step row's status (carried in the §f3 level column — queued/running/done/failed) drives the sub-job's status + a recorded transition when it changes; log/metric rows append to that sub-job's tail; manifest rows (provenance/dep/artifact, empty step_id) are counted, not tailed (they are the static build record, not the live feed).
• The live seam is flight_sink::subscribe — BuildInfoTail::pull(endpoint, token, key) (gated build-info-flight) subscribes one sub-job's ticket (route_key = build_id/step_id) over do_get and folds the decoded rows through the SAME ingest, so live-wire and injected rows fold identically. The default viz build keeps the injectable ingest path and pays no Flight tax.
• Reuse (LAW #5) — the pane paints with the shared facett palette (Theme) and the vendored facett widgets (facett_ui::status_chip, zebra rows); nothing is re-rolled. It renders as data (state_json()["build_info_live"]), folded into the app just like bench.live, so it is robot-testable, not pixel-only.
• Additive — a new viz-gated module + a field on the app; existing viz unchanged. Drawn beneath the release-run feed in the 📡 Live Run tab.
• Continuous auto-refresh (the timer, shipped). The pane no longer needs a manual pull: a background poller (BuildInfoTail::start_poller, gated build-info-flight) re-pulls every tracked sub-job ticket (route_key(build_id, step_id)) on a fixed POLL_EVERY (1500 ms) cadence on its own OS-thread tokio runtime — the same pattern FlightBuildInfoSink uses, so a pull never blocks the egui frame nor needs an ambient runtime. Decoded rows come back over a plain std channel (the inbox); draw drains the inbox through the SAME synchronous ingest every frame and calls request_repaint_after(POLL_EVERY) so rows that land mid-build appear with no user input (mirrors bench_live's warehouse re-read). The poller forwards only rows past a per-key high-water mark, so re-pulling the full stream never double-folds — the append-only ingest (and every existing inject test) is unchanged (L2). The live-ticket set is tracked (subscribe_ticket) and auto-grown as the fold discovers each sub-job; the app starts the poller in remote mode. Both the tick count (poll_count) and the live tickets are exposed in state_json.

Guarded (RED-when-broken): ingest_grows_tail_and_tracks_status + manifest_rows_counted_not_tailed + auto_refresh_drains_inbox_and_advances_ticks (a simulated interval pull GROWS the tail + ticks poll_count, no re-inject) + empty_interval_pull_is_not_a_tick (src/viz/build_info_live.rs); build_info_pane_tails_feed_per_subjob + build_info_pane_auto_updates_on_poll_timer (the REAL app auto-folds delivered batches across egui frames) + poller_pulls_live_flight_rows_across_intervals (feature build-info-flight — the poller re-pulls a live edda/nornir-flight server across intervals, deduped by the high-water mark) + a wgpu snapshot (viz_snapshot_build_info_live) (tests/viz_build_info_live.rs). Emits nornir/viz/build-info-live functional_status rows (feature testmatrix).

Deferred (later slices)

dep rows currently carry the enabled features (the consumed inputs present on BuildProvenance); a resolved crate-graph dependency row is a later slice. (Continuous auto-refresh of the live-tail pane — the timer that re-pulls each ticket while the pane is open — is now shipped; see the "Slice 5 tail" section above.) run_on_ledger runs steps sequentially in topo order today (each wave's dependents wait on the prior); parallel intra-wave execution of independent sub-jobs is a later refinement (the PlanDag already exposes the waves via depths()).