nordisk/knut

knut-pipelines — design

Class: design · Function axis: f0 one-model-many-frontends · f1 DAG model · f2 run state + job progress · f3 Spark Connect connector · f4 renderers · f5 GUI · f6 introspection · f7 facett adapter. Strict: this file states only the shape we run. Every why, every alternative, lives in pipelines-reasoning.md.

Reuse: pipelines.uses = [] — standalone (default build is pure model + CLI). The facett adapter (f7) is an optional outbound edge to facett-pipeline.

f0 — one model, many frontends (the whole)

knut-pipelines is a viewer for Spark Declarative Pipelines (SDP, Spark 4.x) and plain Spark Connect jobs. It ships the knutpipe binary. The whole design is one invariant:

ViewerState is the single source of truth; the CLI text/JSON view, the egui GUI, and the headless run API are all pure functions of it. A Spark Connect server streams three things — the compiled dataflow DAG, pipeline run events, and per-stage job progress — and PipelinesClient::run_updates merges them into one ViewerUpdate stream that ViewerState::apply folds into that one state (progress.rs:240). Every frontend renders the same state; none holds state of its own.

Two laws hold the loop:

Heavy stacks are opt-in. The default build is pure model + CLI show. grpc (tonic 0.14 + vendored Spark Connect proto) and gui (egui/winit/glow) are cargo features; live = gui + grpc. egui/eframe are pinned to 0.28 for egui-snarl 0.4 compatibility.

f1 — the DAG model (src/model.rs)

A pipeline is a dataflow graph: datasets (nodes) + flows (edges).

f2 — run state + job progress (src/progress.rs)

ViewerState (progress.rs:188) is the folded truth: the DataflowGraph, a per-dataset NodeStatus, the event log, and the latest ExecutionProgress.

f3 — the Spark Connect connector (src/client.rs [feature grpc])

A thin hand-shaped driver over the Spark Connect gRPC service — not JDBC, not the REST API, not Arrow Flight SQL. The one RPC that matters is server-streaming SparkConnectService.ExecutePlan. See assets/spark_connect_flow.svg.

Everything is an ExecutePlan. There is no bespoke pipelines RPC: every pipeline operation is a Command in a Plan on an ExecutePlanRequest, every answer arrives on the returned server stream (pipeline_request, client.rs:234).

The four-command dance. define_graph (client.rs:348) drives, in order: CreateDataflowGraph → returns a dataflow_graph_id (create_dataflow_graph, client.rs:254); DefineOutput once per Dataset (define_output, client.rs:295); DefineFlow once per Flow — a flow with a query becomes a runnable WriteRelationFlowDetails (define_flow, client.rs:320); then start_run against the graph id opens the event stream (client.rs:367).

One merged stream. run_updates (client.rs:449) returns an UpdateStream (type UpdateStream = Pin<Box<dyn Stream<Item = Result<ViewerUpdate>> + Send>>, client.rs:156); response_to_update (client.rs:122) keeps only events + progress and drops arrow batches / sql results. The progress channel is orthogonal: execute_sql (client.rs:475) streams ExecutionProgress for any Spark SQL query (a Plan with a root relation, no Command), so the viewer is a live Spark job monitor as well as a pipeline-DAG viewer.

URL + session. connect (client.rs:205) accepts sc://host:port[/;params] (translated by sc_url_to_http, client.rs:519) or plain http(s)://, opens a tonic Channel, and mints a per-connection session_id (new_uuid, client.rs:539) that rides every request.

Feature isolation. The generated bindings (proto::, client.rs:32) carry their own tonic 0.14 tree via the crate build.rs (tonic-prost-build, gated), so it never collides with the workspace's tonic 0.13 — the same trick knut-bifrost uses for spark-connect.

f3b — the Spark runtime + Structured-Streaming VIEW (src/spark_view.rs)

The backend API korp's Spark pane calls to see a live runtime — its version, the Structured Streaming queries running on it, each query's status + recent progress, and (the one control verb Connect exposes) to STOP one. Same ExecutePlan transport; different Command variants:

The model types (StreamingQueryInfo / StreamingQueryStatus / SparkRuntimeInfo, spark_view.rs) are neutral and always compiled — korp depends on them without pulling gRPC in; the live methods that fill them live on PipelinesClient behind grpc. Honest scope: cluster executor/worker topology is not on the Connect wire (it lives in Spark's separate REST UI), so this view deliberately covers only what Connect can answer and does not invent fields the protocol cannot fill. The live methods need a reachable Spark Connect server (no way to fake a cluster); the neutral fold is unit-tested (spark_view::tests) and the gRPC wiring is compile-verified against the vendored proto.

f4 — renderers (src/viewer.rs)

Pure functions of the model/state; the CLI-parity surface.

f5 — the egui / snarl GUI (src/gui.rs [feature gui])

launch(state, connect) (gui.rs:493) / launch_live_pipeline (gui.rs:517) run an eframe app; DagViewer impls egui_snarl::SnarlViewer<NodeData> (gui.rs:59) to draw each dataset as a node coloured by NodeStatus, with a Spark job-progress panel and the event log. The background stream feeds through spawn_live (gui.rs:688) / spawn_pipeline (gui.rs:620), which consume the same UpdateStream as the CLI. KNUT_SCREENSHOT=<path> captures one frame and exits (gui.rs:496) — the headless GUI smoke path.

f6 — introspection (headless robot-testing)

The viewer's entire UI state is observable from outside — no display, no screenshot. Two always-on channels (mirrors nornir viz's urdr-threads):

1. State dump. ViewerState::state_json() (progress.rs:335) — datasets, node_status, status_counts, job_progress (fraction/tasks/stages), events+last_event, topo_order, connect_status/source/error, and placeholders (empty-state strings). The GUI writes it every ~300 ms to $KNUT_VIEWER_STATE (default <tmp>/knut_viewer_state.json, gui.rs:243), overwritten in place. It lives in the default build, so the data-layer matrix asserts on it with no eframe::run (progress.rs tests state_json_*).
2. Activity trail. one stderr line per real state change.

Rule: any new visible list/status/panel MUST be added to state_json().

f7 — the facett adapter (src/facett_adapter.rs [feature facett-adapter])

knut owns the data + connector side; the reusable visual component lives in facett as an engine-agnostic viewer. The seam is one feature-gated file (facett-adapter = ["dep:facett-pipeline"]) — the Spark/SDP adapter, the only place that knows both knut's Spark types and facett's neutral model. It maps knut's ViewerState onto facett's PipelineModel. "Show a Spark pipeline" is exactly one adapter; a future Flink / Beam / skade-streaming viewer is another adapter, not another viewer. See assets/facett_adapter_flow.svg.

The mapping (precise):

Three entry points, one impl (to_pipeline_model, lib.rs:98): the free to_pipeline_model(&ViewerState); SparkPipelineAdapter<'a>(&'a ViewerState) impls PipelineAdapter; and ViewerState itself impls PipelineAdapter, so the ergonomic form is PipelineView::from_adapter(&viewer_state). A test pins that all three agree.

Back-compat law. The default build is UNCHANGED. facett is pulled in only under facett-adapter (optional = true), its model types never leak into knut's public API (adapter takes knut types in, hands a PipelineModel out), and facett's egui-0.34 tree stays off the default and gui (egui-0.28) builds.

Architecture map