nordisk/facett

The pipeline DAG subsystem — reasoning

Class: reasoning. This file holds the why, the trade-offs, and every design we rejected. The descriptive shape we actually run is in pipeline-design.md, which by rule mentions none of this. If you want to know "why not X?" — X is here.

f0 — why an engine-agnostic DAG subsystem at all

The payoff is the same one behind the whole Elm split (see elm-reasoning.md): a component whose whole input is a neutral, serializable model can be driven and asserted headless, and — for these two viewers specifically — can be reused across engines without a rewrite. "Show a pipeline" and "show a query plan" are useful independent of Spark, so neither viewer is allowed to import Spark/SDP types.

Why compose facett_graph, not paint the DAG from scratch, and not call DepGraphView::show either. A hand-rolled toposort would duplicate work facett_graph already does correctly; calling DepGraphView::show directly would hand away the paint surface, so neither facett-pipeline's animated flow/glow nor facett-plandag's heat-fill/metric-body would be possible without forking the graph crate. Composing DepGraphView for layout only (.columns()) and painting the nodes/edges independently gets correct DAG placement for free while keeping full control of the look.

f1 — why PipelineModel / PipelineAdapter (the road from the knut copy)

facett-pipeline used to be fed a Spark/SDP-shaped model directly: knut's ViewerState mapped by each consumer onto (name, Status) nodes + (from, to) index edges. That worked, but it coupled "show a pipeline" to "show a Spark pipeline" — every new engine would mean teaching the viewer (or every consumer) a new mapping ad hoc, and the mapping code lived wherever a given host happened to write it, not in one place.

The fix was to name the neutral shape once — [PipelineModel] — and move the knut-specific mapping into a single PipelineAdapter impl living in knut (knut_pipelines::FacettPipelineAdapter, feature facett-adapter) rather than in facett or scattered across consumers. Nothing about the viewer needed to change to make this split possible: PipelineView::from_model/from_adapter are additive, and the legacy new/from_nodes builders (which predate the model) remain as thin wrappers so existing callers (korp) did not need to migrate.

Rejected: keep the direct Spark-typed binding. The tidy alternative was to leave ViewerState → (name, Status) mapping as "the" pipeline interface and special-case future engines later. Rejected because every future engine (Flink, Beam, skade-streaming) would then need either a shim back to ViewerState's shape or a new viewer entry point — exactly the coupling the model exists to remove.

Why StageStatus gained Unknown — and why it is not grey

The four-state enum (Idle · Running · Done · Failed) is right for a pipeline: the engine owns the run, so a stage that has not started really is idle and the engine knows it. It is wrong the moment the nodes stop being steps of one run and become live environments — a promotion ladder, a fleet, anything observed over a wire. There, "we have never observed this environment" and "this environment is idle" are different facts, and there was no way to say the first.

Saying it as the second is the fake-green pattern. This codebase has found it repeatedly — five separate cases in the socket fixtures alone: a DHCP request read as a lease, a handshake initiation read as a completed handshake. Each one shipped a green light over something that was not working. A ladder rung rendered Idle because the probe timed out is the same bug with a calmer colour.

So Unknown(UnknownReason) is a value, not a convention:

• It carries a machine-readable reason (NotProbed/Unreachable/ ProbeFailed/NotAuthorised/Ambiguous/Stale/NotApplicable). "Unknown" alone is a shrug; the caller must say which kind of not-knowing it hit, and label() puts it in the text/JSON ("unknown:unreachable") so no reader has to go hunting.
• The reason is a Copy enum, not a String. That is what keeps StageStatus and Status Copy/Eq/Serialize, so every existing .copied() and == call site in knut, dwarves and Njord compiles untouched. A String payload would have been a breaking change wearing an additive costume.
• It is not grey. Idle is th.text_dim — a recessive neutral whose job is to let the eye skip the node. If Unknown shared it, an environment nobody has ever looked at would render identically to one we positively know is idle, and the variant would be decoration. It is UNKNOWN_AMBER (the same amber facett-releaseview uses for drift, so the constellation reads consistently), distinct from the failed red because unknown is not failure.
• Colour is never the only channel. A dashed border (a solid box asserts a known state; a broken one does not) and a ? corner marker carry the same meaning in greyscale and under colour-vision deficiency — the belt-and-braces the failed node's ⚠ already established. The glow is a steady non-zero 0.5: non-zero because an unobserved node is the one that must not be skipped, steady because pulsing would claim activity nobody observed.

Rejected: a separate observed: bool field on Stage. It would have left status free to say Idle anyway, so every renderer and every JSON consumer would have had to remember to consult two fields — and the one that forgets is exactly the fake green we are trying to make unrepresentable. Folding it into the status makes the wrong rendering impossible rather than merely discouraged.

Rejected: Option<StageStatus> at the call sites. Same problem one level up (unwrap_or_default() is Idle), plus it would have broken every signature.

#[non_exhaustive] was added in the same change, on both StageStatus and Status. Adding an enum variant is a breaking change for any downstream exhaustive match — the class of break that took nornir down the same day funnel-core added a variant. Every consumer was audited first (facett-demo, korp, dwarves' brokkr::facett_adapter + cicdwizard + demo, knut's facett_adapter, Njord's deployer-ui): all of them produce these enums and none matched one exhaustively, so nothing broke, and this is the last time the question has to be asked. In-crate matches are still exhaustiveness-checked, so label(), UnknownReason::token() and StatusTally::add deliberately carry no _ arm — a future variant must be given a treatment, not swept into a bucket.

The lossy legacy tuple was kept on purpose. status_counts() / PipelineView::counts() still return (idle, running, done, failed) with the same four numbers, so no caller changed — but they now need not sum to the node count. StatusTally (which does) is the additive replacement. Widening the tuple to five would have broken every destructuring caller for no gain.

Why the promotion ladder is an adapter, not a 71st crate

Per Njord/.nornir/deployer-layering-decision.md, Njord's deployer split: the generic engine to skidbladnir::promote, the view to facett. A ladder is a DAG with per-node status — the exact thing PipelineView already draws — and this crate's own header says "show another engine's pipeline is an adapter, not a fork." There are already 70 facett-* crates, three of which draw boxes-with-status (facett-releaseview, facett-fleet-view, facett-plandag); a fourth would be the duplication this constellation keeps auditing itself for.

Why the adapter takes a local PromotionLadder instead of skidbladnir's types. facett is a UI library; skidbladnir is a heavy headless provisioning lib path-depended by nornir, dwarves, korp and holger — four consumers that must never be handed an egui stack. So the edge is wrong in both directions, and the answer is the one RunaPipelineAdapter already uses for a leaf that must not learn about facett: a small local input type the caller populates. The Njord host maps skidbladnir::promote's serde topology onto it in a few lines and owns that mapping; facett owns neither the engine nor the customer's slot names.

Why it is not feature-gated like runa-adapter. That flag exists to keep an optional dependency (dwarves-runa) out of a default build. The ladder adapter pulls in nothing at all, so gating it would be copying the flag without its reason — and would make the honest default (Unknown) opt-in.

Why RungStatus is its own enum rather than StageStatus reused. A ladder's vocabulary differs (Promoted/Promoting/Pending), but the real reason is the default: RungStatus::default() is Unknown(NotProbed), the inverse of StageStatus's Idle. Keeping the type separate is what lets the cheapest thing a caller can write be the honest one, without changing what Idle means for every existing pipeline.

DEFERRED, not built (the seam exists for this, nothing more):

1. Render through facett-graphview with a status/metrics overlay. Today PipelineView paints its own themed DAG, composing facett_graph only for layout. The future path is to lay the PipelineModel out via facett-graphview (still spike-stage) and overlay StageStatus + StageMetrics on top. Because the model is renderer-independent, that swap would touch only the view, never the adapters. Not started.
2. More engine adapters — Flink, Beam, skade-streaming. Each is meant to be "just another impl PipelineAdapter"; streaming engines would populate StageMetrics::lag_s/throughput_rows_s (the fields already exist for this). Only the Spark/SDP adapter ships today.

f2 — why PipelineView's specific choices

Why impl_facet_via_elm! is invoked in form 3 (custom_state_json). The pipeline's state_json predates the Elm migration and is read verbatim by facett-demo's synthetic_sdp path and mega_matrix; it is richer than a plain serde(state()) (per-node/status/progress/nav keys plus the composed DAG's own block). Form 3 lets the bridge supply everything else (title, the view→update loop) while this one method stays hand-written and byte-for-byte identical to the pre-Elm surface.

Why the edge convention is (reads_idx, target_idx), painted source → consumer. This mirrors knut's Flow { reads, target } exactly, field for field, so the korp bridge from a live Spark run to a PipelineView is a direct copy rather than a translation — deliberate parallelism between knut's NodeStatus (idle/running/done/failed) and facett's Status for the same reason. Get the pair backwards and the arrows point upstream; this is the one mistake the for-idiots guide calls out explicitly.

Why Stage::id is a separate field from Stage::name. Most engines have a display name that is a stable id, so the common case (Stage::new) sets both to the same string. The field is kept distinct because an engine with opaque ids (e.g. Flink vertex ids) needs somewhere to carry them without colliding on the human-readable label the box actually shows.

Why throughput labels are host-supplied strings, not computed. The component has no rate model of its own — computing "12k rows/s" would mean guessing at a sampling window the viewer has no visibility into. The producer already knows its own rate; the viewer just places the text.

Accepted trade-off: fixed-grid column layout, no edge-crossing minimisation. facett_graph's toposort gives correct left-to-right column placement with naive per-column row stacking — good enough for medial pipelines, visually busy for very wide fan-outs. A real crossing-minimising layout was not built because the common case (a handful of ETL stages) never needed it; this is exactly the gap the deferred facett-graphview render path (above) would close.

Why Status is a fixed four-state enum. Richer states (queued, skipped, retrying) would need a wider enum, but the only shipped producer (knut) derives exactly these four states from a Spark run — the enum is sized to the one real producer, not speculatively widened.

f3 — why facett-plandag's specific choices

Why impl Facet for PlanDagView is hand-written instead of using impl_facet_via_elm!. This view publishes a richer state_json (the resolved graph + per-node cost-heat as data, plus version/collapsed) and additionally implements selection_json, portable_state/load_state, and a typed copy — the macro's generated impl only emits the plain serde(state()) default and has no seam to override it. facett-security made the same call for the same reason (see facett-core/src/elm.rs); plandag follows the established pattern rather than fighting the macro for a form it doesn't support.

Why resolve() is a free function over &PlanVersion, not a method that mutates view state. Collapse/metric/version are all read from PlanDagState each time resolved() is called, so the painter and every introspection path (state_json, copy_text, selection_json) always agree — there is exactly one place a "what does the picture show" bug could hide, and it's a pure function that a test can call directly with synthetic input, no view required.