nordisk/facett

The pipeline DAG subsystem — design

Class: design · Function axis: f0 (the DAG subsystem as one) · f1 PipelineModel/PipelineAdapter · f2 PipelineView · f3 facett-plandag. Strict: this file states only the shape we run. Every why, every alternative, lives in pipeline-reasoning.md.

f0 — the DAG subsystem (the whole)

Two crates paint dataflow/operator DAGs: facett-pipeline (a dataset/stage pipeline with run status + job progress) and facett-plandag (a metric-bearing Spark query-plan / Query-Profile operator DAG with cost-heat, WholeStageCodegen collapse, and an AQE initial-vs-final toggle). Both use facett_core::layered for producer-left column layering and persistent facett_flow::FlowScene geometry for shared GPU/CPU rendering. Their metric, status, selection and accessibility overlays remain component-owned.

Both are Elm components (facett_core::Elm) under the FC contract: a single serializable Model, a Msg enum as the only writable surface, a pure view that returns the Msgs a frame produced, and Effect as side-work-as-data (see elm-design.md for the trait itself). PipelineView carries a real Effect::PreviewRequest (facett-pipeline/src/lib.rs:307); facett-plandag::Effect is uninhabited (facett-plandag/src/lib.rs:83) — it does no I/O.

facett-pipeline is fed a normalized, engine-agnostic [PipelineModel] (f1) produced by a [PipelineAdapter] impl — the viewer owns no Spark/SDP types. facett-plandag is fed a [PlanModel] (facett-plandag/src/model.rs:196) built directly by the host (no adapter trait); PlanDagView::local() (facett-plandag/src/lib.rs:245) is a self-contained demo plan exercising every capability with no host at all.

Reuse map: both crates depend on facett_flow (persistent geometry and GPU/CPU painting) and facett_core (layered layout, Elm, Facet, theme, effects::glow_rect, a11y, nav::Navigable, clip::CopySource). facett-pipeline ships Runa and promotion-ladder adapters; the downstream Spark adapter is knut_pipelines::FacettPipelineAdapter (feature facett-adapter) — see knut-pipelines design. The bridge into a real app is korp's Spark pipeline example.

f1 — PipelineModel / PipelineAdapter (facett-pipeline/src/model.rs)

The normalized shape every engine adapter produces and [PipelineView::from_model] consumes:

pub struct PipelineModel {
    pub title: String,
    pub stages: Vec<Stage>,           // model.rs:262
    pub edges: Vec<PipelineEdge>,     // model.rs:264
    pub metrics: Option<PipelineMetrics>, // model.rs:266
}

PipelineModel::stage_index(id) (model.rs:493) resolves an edge id to a stage index; status_counts() (model.rs:505) gives the legacy (idle, running, done, failed) and status_tally() (model.rs:511) the five-way StatusTally.

StageStatus::Unknown — the state we could not determine

Idle asserts a fact: the stage has not started, and we know it. Unknown asserts the absence of a fact: we could not determine this node's state, for this reason. They are different values and they render differently. An adapter whose probe came back empty maps onto Unknown, never Idle and never Done.

• label() (model.rs:212) carries the reason: "unknown:not-probed", "unknown:unreachable", … family() (model.rs:231) is the bare "unknown" for legend chips. The four pre-existing labels are byte-identical.
• Idle is still #[default]; every pre-Unknown variant, label and From mapping is unchanged.
• status_counts() / PipelineView::counts() keep their four-tuple signature and are therefore lossy: an Unknown stage is in none of the four buckets, so they no longer have to sum to the stage count. StatusTally (total()) always does.
• Both StageStatus and the render-layer Status are #[non_exhaustive], so the next variant is not a breaking change either. In-crate matches are still exhaustiveness-checked, so label()/token()/StatusTally::add deliberately carry no _ arm.

Render treatment (lib.rs). Unknown is UNKNOWN_AMBER = rgb(240,180,70) (lib.rs:86) — the same amber facett-releaseview uses for drift — in theme_color (lib.rs:136) and color, in every theme. Never th.text_dim (that is Idle), never the failed red. Two non-colour channels back it up so the distinction survives greyscale/CVD: a dashed node border (dashed_rect, lib.rs:1097) instead of the solid rect, and a ? corner marker in the same slot the failed node's ⚠ uses. status_glow (lib.rs:305) returns a steady 0.5 — non-zero (unlike Idle's 0.0, which is how the eye skips a node) and non-pulsing (pulsing would claim activity nobody observed).

state_json. statuses gains an additive unknown count; each node gains unknown_reason (the token, or null); a new top-level unknowns array lists {node, reason, why} beside errors. The four legacy statuses keys and every pre-existing node key are unchanged.

The seam:

pub trait PipelineAdapter {
    fn to_pipeline_model(&self) -> PipelineModel;   // model.rs:321
}

facett-pipeline itself depends on nothing beyond crate::Status for the StageStatus conversion; every engine-specific mapping lives in the adapter impl, outside this crate.

Shipped adapters

The runa adapter is feature-gated because it pulls an optional dependency (dwarves-runa). The ladder adapter pulls none, so gating it would be cargo-culting the flag rather than its reason.

PromoteLadderAdapter — a promotion ladder is a DAG with per-node status

Per Njord/.nornir/deployer-layering-decision.md: the promotion engine goes to skidbladnir::promote, and the view is an adapter over this crate, not a 71st facett-* crate.

facett must not depend on the engine (facett is a UI library; skidbladnir is a headless provisioning lib path-depended by four consumers that must never be handed an egui stack) — and the engine must not depend on facett. So the adapter reads a small local input type the caller populates, exactly as RunaPipelineAdapter reads runa's model without facett owning runa's engine.

Mapping (to_pipeline_model, promote_adapter.rs:331):

The rungs are DATA. Njord's ladder is test2 → test1 → lab → g11.se; another customer's is dev → qa → staging → prod; a ladder may branch. No name, length or shape is baked into this module.

The one behavioural difference from RunaPipelineAdapter. There, a stage missing from the status map is Idle — correct, because the engine owns the run and knows it has not started. Here, a rung missing from with_observations keeps the rung's own status, which defaults to Unknown(NotProbed). A live environment nobody probed is not idle; it is unobserved. unobserved_reason(r) restates that for the whole ladder when the probe run failed (NotAuthorised, Unreachable) rather than never happening.

f2 — PipelineView (facett-pipeline/src/lib.rs)

Construction. Four entry points, all producing the same internal state:

State (PipelineState, lib.rs:241, the FC-1/3 Model): nodes: Vec<PipelineNode>, edge_ends: Vec<(usize, usize)>, progress: Option<JobProgress>, animated: bool, speed: f32, throughput: Vec<Option<String>>, selected: Option<String> (keyed on stable node name, FC-5), clock_s: Option<f64> (injected animation clock, FC-7), nav: Navigable (shared pan/zoom camera), fitted: bool.

Control surface (Msg, lib.rs:282): SelectNode(String), Pan(Vec2), Zoom { factor, anchor }, Fit(Rect), RequestRefit, SetClock(f64), Tick(f64). PipelineView::update (lib.rs:541) is the sole mutation path; SelectNode is the only variant that raises an Effect::PreviewRequest (lib.rs:307), drained via take_preview_request (lib.rs:737).

Layout. Construction uses core layered with Align::Sources and RowOrder::Input, and places node centres on the existing fixed grid (NODE_W/NODE_H/COL_GAP/ROW_GAP/LEFT_PAD/TOP_PAD, lib.rs:330-335) in scene coordinates; scene_bbox (lib.rs:602) is the fit-to-view bounding box. The camera (Navigable) maps scene → screen (lib.rs:718, to_screen).

Paint (paint_dag, lib.rs:748). Pure function of &self + time: builds the navigable canvas, deferred-fits on first paint, reads camera input via nav_input_msgs (lib.rs:619 — drag-pan, wheel/pinch zoom-to-cursor, arrow-key pan, +/-/PageUp/PageDown zoom, F/Ctrl-0 fit) and returns the resulting Msgs. Each frame: Falsett paints cached curves, throughput-driven packets and arrowheads at consumer ends; throughput labels sample FlowScene::edge_point. Zero rate emits no packets. The injected clock feeds PaintOpts.seconds; an active legacy flowsim overlay uses Animation::External to avoid double motion. Nodes retain local themed boxes with a status.theme_color stroke (lib.rs:136) — dashed when the status is Unknown — an effects::glow_rect underlay at status_glow(status, time, speed) intensity (lib.rs:305 — running pulses, done steady, failed blooms, unknown holds a steady 0.5), the name, a status/rows/throughput/lag sub-label (the status text carries the unknown reason), a selection ring, a ⚠ marker when failed and a ? marker when unknown. Each node also carries an a11y::Semantics::button node (lib.rs:823) keyed on the node name; a click emits Msg::SelectNode.

Top-level view (PipelineView::view): paints the status-count legend (● n done · running · failed · idle, plus an amber ? n unknown chip only when tally().unknown > 0, so a pipeline with no unknowns renders exactly as before) + ⤢ Fit button, the optional JobProgress bar and live-run readout (rows/s, rows, bytes, inflight, elapsed — only the parts the run reported, lib.rs:1003), paint_dag, then the selected-node detail panel. It is pure: it returns the accumulated Vec<Msg>; impl_facet_via_elm! applies them through update.

Introspection. state_json_value emits node_count, edge_count, per-node nodes[] (name/status/unknown_reason/detail/rows/error/throughput/lag), statuses counts (idle/running/done/failed/unknown), errors[], unknowns[] ({node, reason, why}), selected, progress, animated, speed, idle (is_idle, lib.rs:488 — no node Running and no task inflight), clock_s, nav (scale/offset/fitted), and dag (the preserved {nodes, edges, columns, selected: null} shape, derived from shared layout). impl Elm for PipelineView is at lib.rs:1172; impl_facet_via_elm!(PipelineView, custom_state_json, { .. }) (lib.rs:1197) supplies copy, caps (FacetCaps::NONE.themeable().resizable().selectable().navigable().copyable(), lib.rs:1207), state_json, and selection_json. CopySource (lib.rs:1160) returns the selected node's line, or the whole node list as a TSV rectangle (copy_text, lib.rs:943).

f3 — facett-plandag

Model (facett-plandag/src/model.rs):

pub struct PlanModel { pub initial: PlanVersion, pub final_plan: Option<PlanVersion> } // :196
pub struct PlanVersion { pub label: String, pub nodes: Vec<PlanNode>, pub edges: Vec<PlanEdge> } // :169
pub struct PlanNode { pub id, pub op, pub group: Option<String>, pub photon: bool, pub status: OpStatus, pub metrics: Metrics } // :121
pub struct PlanEdge { pub from: usize, pub to: usize }  // :161 — index-based, not id-based
pub struct Metrics  { pub time_ms: f64, pub peak_mem: u64, pub rows: u64, pub spill: u64 } // :88

CostMetric (model.rs:25, Time | Memory | Rows) selects which Metrics field drives the cost-heat fill; OpStatus (model.rs:58) is the neutral per-operator run status, with rank() (model.rs:76) used to aggregate a collapsed group's status (failed > running > done > idle).

resolve(version, &collapsed, metric) -> Resolved (model.rs:288) is the pure core the painter and state_json both read: it collapses every node whose group is in collapsed into one super-node (Metrics::add, model.rs:109, sums time_ms/peak_mem/spill, takes max of rows; aggregated photon = AND of members; aggregated status = highest rank), remaps edges to the resolved index space (dropping intra-group and duplicate edges), then computes each node's heat: f32 in [0,1] as value_for(metric) / max_across_nodes (model.rs:370). Resolved { nodes: Vec<ResolvedNode>, edges: Vec<(usize, usize)> } (model.rs:244) and ResolvedNode (model.rs:220, carrying id, title, status, metrics, members, photon, group, collapsed, heat) are the single source of truth: Resolved::state_json(metric) (model.rs:251) dumps exactly what the painter draws.

View (facett-plandag/src/lib.rs). Version (Initial | Final, lib.rs:44) is the AQE toggle. PlanDagState (lib.rs:90, the FC-3 Model): model: PlanModel, version: Version, metric: CostMetric, collapsed: Vec<String>, selected: Option<String>. Msg (lib.rs:63): SetMetric, SetVersion, ToggleGroup(String), CollapseAll, ExpandAll, Select(String), ClearSelection. Effect (lib.rs:83) is uninhabited. PlanDagView::update (lib.rs:144) is the sole mutation path — SetVersion(Final) is a no-op when model.has_aqe() is false (lib.rs:149); CollapseAll sets collapsed to active_version().groups() (lib.rs:160); Select only accepts an id present in the current resolved() (lib.rs:163).

PlanDagView::resolved() (lib.rs:138) calls model::resolve with the current (active_version, collapsed, metric) — the one call the painter and every introspection path share. render (lib.rs:325, pure, returns Vec<Msg>) paints: a cost-heat metric switch, an AQE version switch (only when has_aqe()), a WholeStageCodegen collapse-checkbox row + collapse/expand-all buttons; then, per resolved node, a box filled by heat_color(heat) (lib.rs:188 — a cool-grey→amber→hot-red lerp), bordered Photon-purple / status-colour / selected-accent (lib.rs:417), a multi-line metric body (time/mem/rows, spill in the corner), and an a11y node; a click selects (toggling off if already selected), a double-click on a collapsed group expands it. Falsett draws box-clipped edges and arrows; critical-path strokes sample the same edge_point geometry. Producers appear on the left. The scene rebuilds on collapse or version changes, while metric switches and navigation retain it. Pan, Zoom, Fit and RequestRefit messages use shared gestures; nav and fitted are included in observable and portable state.

impl Elm for PlanDagView is at lib.rs:509. impl Facet for PlanDagView (lib.rs:535) is hand-written — not the impl_facet_via_elm! bridge — and adds selection_json, kind() -> "plandag", and portable_state/load_state (lib.rs:582, :590 — round-trips metric/version/collapsed/selected). state_json (lib.rs:557) is Resolved::state_json(metric) with title, version, has_aqe, collapsed, groups, and selected merged in. CopySource (lib.rs:307) mirrors the pipeline pattern: selected node's title\tstatus\theat line, or the full resolved-node TSV.