nordisk/facett

The pipeline DAG subsystem — for idiots

Class: idiot. Every step, nothing assumed. Goal: draw an animated pipeline DAG and a query-plan DAG, and prove both work with a headless test, from a cold shell to green. If you already know Rust, read pipeline-guide.md instead — this one spells out every keystroke.

What you need first

• A terminal (the black window where you type commands).
• The facett repo on disk. In the terminal, type this and press Enter:

cd /run/media/rickard/T9/git/facett

f0 — the whole thing, start to finish

You have some stages (named boxes) with a status each, and some edges between them, and you want a nice animated DAG. That's facett-pipeline. It draws the picture — it has zero idea where the data came from (Spark, a CSV, your imagination — doesn't care). If instead you have a query plan (operators with time/memory/rows, maybe two versions because AQE re-planned it), that's facett-plandag — same idea, richer node bodies.

nodes  = [(name, Status), …]              ← boxes, coloured by status
edges  = [(reads_idx, target_idx), …]     ← arrows; index into `nodes`
        ↓
PipelineView::new(title, nodes, edges).ui(ui)   ← drop into any egui Ui

Status is one of Idle, Running, Done, Failed. That's it.

f1 — draw a pipeline DAG in 10 lines

Step 1. Open (or create) a small Rust file/test in a crate that already depends on facett-pipeline, and type:

use facett_pipeline::{PipelineView, Status, JobProgress};

let nodes = vec![
    ("bronze".to_string(), Status::Done),
    ("silver".to_string(), Status::Running),
    ("gold".to_string(),   Status::Idle),
];
// edge (a, b) means "a reads b" → flow is drawn b → a (source → consumer):
let edges = vec![(1, 0), (2, 1)];          // silver reads bronze; gold reads silver
PipelineView::new("my etl", nodes, edges)
    .with_progress(JobProgress { fraction: 0.5, tasks_done: 25, tasks_total: 50, ..Default::default() })
    .ui(ui);                                // ← that's the whole render

You get: themed boxes (green=done, pulsing amber=running, dimmed=idle, red bloom=failed), marching-ants edges flowing in the dataflow direction with an arrowhead, a status-count legend, and a job-progress bar.

Step 2 — the one thing beginners get wrong: edge direction. An edge (from, to) means from reads to. The animation then flows to → from — i.e. source → consumer. So if silver reads bronze, write (silver_idx, bronze_idx) = (1, 0), and the dashes correctly march bronze → silver. Flip it and your arrows point upstream.

Step 3 — tweaks you'll actually want:

PipelineView::new(title, nodes, edges)
    .with_animation(false)   // freeze it (screenshots, headless tests)
    .with_speed(1.2)         // faster marching ants (cycles/sec)
    .with_throughput(vec![Some("12k rows/s".into()), None])  // per-edge labels

counts() gives you (idle, running, done, failed) if you want your own summary line.

Step 4 — check it without opening a window. Type and press Enter in the crate that has this test:

cargo test -p facett-pipeline renders_headlessly_under_a_theme

Or write your own headless check:

let mut v = PipelineView::new("etl", nodes, edges);
let r = facett_core::harness::render_themed(&mut v, facett_core::Theme::cyberpunk_neon());
assert!(r.drew());
assert_eq!(r.state["dag"]["nodes"], 3);   // state_json mirrors the picture

Green test result: ok = done.

f2 — mapping a real engine onto it

If your statuses come from a live run (knut's ViewerState is the worked example), you write one small adapter instead of hand-building (name, Status) pairs each frame:

Step 1. Add the neutral model types:

use facett_pipeline::model::{PipelineAdapter, PipelineModel, Stage, StageStatus, PipelineEdge};

Step 2. Write the adapter — one function, to_pipeline_model:

struct MyAdapter<'a>(&'a MyRun);
impl PipelineAdapter for MyAdapter<'_> {
    fn to_pipeline_model(&self) -> PipelineModel {
        let mut m = PipelineModel::new(&self.0.name);
        for s in self.0.stages() {
            m = m.with_stage(Stage::new(&s.name, StageStatus::from(s.status)));
        }
        for e in self.0.edges() {
            m = m.with_edge(PipelineEdge::new(&e.from, &e.to));
        }
        m
    }
}

Step 3. Build the view straight from it:

let view = PipelineView::from_adapter(&MyAdapter(&run));

That's it — no (name, Status) bookkeeping, no index arithmetic; the model resolves ids to indices for you. knut's real, fuller adapter (rows, throughput, lag, errors) is in knut pipelines for dummies; the full, runnable end-to-end bridge is in korp's Spark pipeline example.

The one trap: don't say "idle" when you mean "I don't know"

Your adapter asks the engine what each stage is doing. Sometimes it can't find out — the call timed out, the credentials were rejected, you never asked.

The tempting thing is to write StageStatus::Idle and move on. Don't. Idle means "it has not started, and I checked." It paints a calm grey box. If you use it for "I couldn't check", you have put a quiet, fine-looking box over something nobody has looked at. That is a fake green, and it is a bug this codebase has found again and again.

Say this instead:

use facett_pipeline::{StageStatus, UnknownReason};

StageStatus::Unknown(UnknownReason::Unreachable)   // asked, no answer
StageStatus::Unknown(UnknownReason::NotProbed)     // never asked
StageStatus::Unknown(UnknownReason::NotAuthorised) // not allowed to look

It paints amber, with a dashed border and a ? — impossible to mistake for grey — and the reason rides along into the text and the JSON ("unknown:unreachable").

One gotcha when you read the numbers back: counts() still returns the old (idle, running, done, failed) and leaves the unknowns out entirely. If you need the total, use tally():

let t = view.tally();          // { idle, running, done, failed, unknown }
assert_eq!(t.total(), t.idle + t.running + t.done + t.failed + t.unknown);

f2b — draw a promotion ladder (dev → qa → staging → prod)

Showing "which environment has which release" is the same DAG with different labels, so there is no new component to learn — you fill in a ladder and hand it to the same view.

Step 1. Describe the rungs, in promotion order:

use facett_pipeline::{PromotionLadder, Rung, SlotRole};

let ladder = PromotionLadder::linear("acme", vec![
    Rung::new("dev",     SlotRole::Test),
    Rung::new("staging", SlotRole::Preprod),
    Rung::new("prod",    SlotRole::Prod).guarded(true),
]);

(The names are yours. Nothing in facett knows or cares what they are.)

Step 2. Say what you actually saw. Anything you did not check, just leave out:

use std::collections::HashMap;
use facett_pipeline::{PipelineView, PromoteLadderAdapter, RungStatus};

let seen = HashMap::from([
    ("dev".to_string(),     RungStatus::Promoted),   // green
    ("staging".to_string(), RungStatus::Promoting),  // running
    // `prod` omitted — the probe never got there.
]);
let view = PipelineView::from_adapter(&PromoteLadderAdapter::new(&ladder).with_observations(seen));

Step 3. Look at it. prod is amber with a ?, not grey — because you didn't check it, and the view refuses to pretend otherwise. That is deliberate: leaving a rung out cannot produce a reassuring box.

If you did check and it simply hasn't got the release yet, that's different — say RungStatus::Pending, and it paints the ordinary grey idle box.

f3 — draw a query-plan DAG

Step 1. The fastest way to see one on screen: there's a built-in demo, no data of your own required.

use facett_plandag::PlanDagView;
let mut view = PlanDagView::local();
view.ui(ui);

That single line draws a 5-operator plan with a WholeStageCodegen group, a Photon operator, an AQE re-plan (initial SortMergeJoin → final BroadcastHashJoin), and cost-heat colouring already wired up.

Step 2 — try the controls. In the running app, click:

• cost-heat: Time / Memory / Rows — repaints every node's fill colour by that metric's relative cost.
• plan: initial / final (AQE) — flips between the two plan versions.
• The WholeStageCodegen checkboxes — collapse two operators into one box; double-click a collapsed box to expand it again.

Step 3 — check it headlessly. Type and press Enter:

cargo test -p facett-plandag

or drive it yourself:

use facett_core::harness;
use facett_plandag::{Msg, CostMetric};

let mut v = PlanDagView::local();
let snap = harness::snapshot(&mut v, [Msg::SetMetric(CostMetric::Rows)]);
assert_eq!(snap.metric, CostMetric::Rows);

Green test result: ok = done — you just drove a query-plan DAG with no window and no GPU.

Step 4 — build your own plan (once the demo makes sense). Same recipe as f1, but with plan types:

use facett_plandag::model::{PlanModel, PlanVersion, PlanNode, PlanEdge, Metrics, OpStatus};

let nodes = vec![
    PlanNode::new("scan", "BatchScan").metrics(Metrics { time_ms: 40.0, peak_mem: 1 << 20, rows: 2_000_000, spill: 0 }),
    PlanNode::new("filter", "Filter").status(OpStatus::Done).metrics(Metrics { time_ms: 20.0, ..Default::default() }),
];
let edges = vec![PlanEdge { from: 0, to: 1 }];
let model = PlanModel::new(PlanVersion::new("initial", nodes, edges));
let view = facett_plandag::PlanDagView::new("my plan", model);

Step — save your work

Type these three, pressing Enter after each:

git add -A
git commit -m "wire up a pipeline/plandag view"
git push

That's the whole tour: a pipeline DAG in 10 lines, a real engine wired through one adapter fn, a query-plan DAG from a one-line demo, and both proved with a headless test — no window opened for any of it.