nordisk/knut

knut-pipelines — reasoning

Class: reasoning. This file holds the why, the trade-offs, and the gotchas. The descriptive shape we actually run is in pipelines-design.md, which by rule mentions none of this. If you want "why not X?" — X is here.

Why one model, many frontends

The goal was debugging and robot-testing a live pipeline viewer without a display. If the whole observable truth is one folded ViewerState, and the only way to change it is to apply a ViewerUpdate, then a test can drive it headless, dump state_json(), and assert — no window, no GPU, no Spark even (feed synthetic updates). Every frontend (CLI text, egui GUI, run API) becomes a pure render of that one state, so they cannot disagree. That single payoff is why the apply/state_json seam exists, and everything else serves keeping it true.

Why the CLI-parity law

The GUI is the seductive place to grow features, and a feature that lives only in eframe::run cannot be asserted headlessly — it becomes invisible to the test matrix. So the law is: every capability the GUI shows is reachable from the command line (dump_text/dump_json/dump_state) and appears in state_json(). This keeps the GUI a rendering of the model rather than a second model, and it keeps the whole viewer testable from a shell. "Any new visible panel MUST be added to state_json()" is the operational form of the same law.

Why Spark jobs, not only pipelines

The narrow design would stream only SDP run events. But ExecutionProgress snapshots ride off any ExecutePlan, including a plain spark.sql(...). Rather than throw that away, execute_sql exposes it: a plain SQL query streams stage/task counts with no pipeline involved, so the same viewer doubles as a live Spark job monitor for one extra code path. run_updates merging events + progress into one ViewerUpdate stream is what lets both feed the identical fold.

Why heavy stacks are off by default

gRPC (tonic + vendored proto + protoc codegen) and egui/winit/glow are large, slow-compiling trees that most consumers of the model don't need. Gating them behind grpc/gui (with live = gui + grpc) keeps the default build a fast, pure model + CLI show — importable anywhere, including as a library that only builds DataflowGraphs. egui/eframe are pinned to 0.28 not by preference but because egui-snarl 0.4 (the node-graph widget) requires it; bumping egui means waiting on snarl. The facett adapter deliberately sits on the other side of a feature wall because facett brings egui-0.34, which cannot coexist with the gui-0.28 tree in one build.

Why one adapter per engine (not per viewer)

The reusable visual component belongs in facett, engine-neutral, so a Flink or Beam or skade-streaming pipeline could reuse the exact same PipelineView. That forces the knowledge of "how Spark's ViewerState maps to neutral stages/edges" into one file — the Spark/SDP adapter — instead of a Spark-specific viewer. "Show a Spark pipeline" is one adapter; the next engine is another adapter, not a fork of the viewer. Spark ships now; Flink / Beam / skade-streaming are deferred (the PipelineAdapter trait is the only reserved seam).

Why the facett dep never leaks into knut's public API

If facett's PipelineModel types appeared in knut's signatures, every knut consumer would transitively pull facett (and its egui-0.34 tree), defeating the whole point of the feature gate. So the adapter is strictly a boundary: it takes knut types in and hands a PipelineModel out, under optional = true / dep:facett-pipeline. The default build stays byte-for-byte unchanged with the feature off — that back-compat is a hard law, pinned by the default build not referencing facett at all.

The gotchas (all pinned by live tests)

• pipelines.proto is on Spark master (4.2-era), not the v4.0.0 tag. The vendored copy uses ResolvedIdentifier flow ids. Re-vendor from whatever Spark tag is pinned when it moves.
• EventLevel is a heuristic. PipelineEvent on the wire carries only timestamp + message, so Info/Warn/Error is derived from the message until Spark surfaces structured severity.
• DefineFlow.once must be unset for normal flows — Spark errors otherwise.
• StartRun needs a storage URI (file:///s3a:///…) or outputs are never materialised; a run without it is only a dry validation. The path must be writable by the Spark workers, not just the caller.

The gaps

• Graph-id selection from the GUI (today you pass it on the CLI).
• Richer event taxonomy (blocked on Spark's structured severity, above).
• A web frontend — axum + SSE over the Serialize model — is sketched but deferred; the Serialize-able ViewerState is the seam it would use.
• Per-dataset metrics are now real, not stand-ins: the adapter fills each stage's StageMetrics.rows from ViewerState::rows_written (the counts parsed from the run's "wrote N rows" events), and the pipeline rows/elapsed_s from the real row sum + the measured first→last timestamped-event span. What's still deferred: per-node throughput/lag (Spark's ExecutionProgress is per-job-stage, so there's no per-dataset timer to derive rows/sec from).