nordisk/knut

knut-pipelines — crate card

Reverse-engineered (from source — verify, don't trust the name)

• One-line pitch: A viewer for Spark Declarative Pipelines (SDP / spark.pipelines, SPARK-51727): fetch a pipeline's dataflow DAG and stream run/job events from a Spark Connect server, and render them (text, JSON, or a live egui graph).
• Does: Core is a dependency-light model mirroring spark/connect/pipelines.proto: DataflowGraph, Dataset, Flow, OutputType, PipelineRun/PipelineRunEvent, with validate()/topo_order(). progress adds ExecutionProgress/StageProgress and the unified ViewerState; viewer dumps text/JSON and defines PipelineEventSink. Feature grpc adds the Spark Connect client (own tonic 0.14 tree + vendored protos, protoc at build time); gui adds an egui/egui-snarl node-graph desktop viewer; live = both. The knutpipe binary is the front door (show/watch/gui). Feature facett-adapter maps ViewerState into facett's engine-agnostic PipelineModel.
• Key deps: serde/serde_json, thiserror, chrono, tracing, clap+tracing-subscriber (binary); feature-gated tonic/tonic-prost/prost/tokio/futures-* (grpc), eframe/egui/egui-snarl/image (gui, pinned egui 0.28), facett-pipeline (adapter). Build-dep tonic-prost-build + protoc-bin-vendored.
• Shape: lib + knutpipe bin; medium. Modules: model, progress, viewer, error, feature-gated client, gui, facett_adapter.
• Observed role in the workspace: The pipeline viewer half. Reused by knut-thund (which lowers its IR onto this crate's DataflowGraph and Spark Connect client as its Spark backend); adapter is the only knut→facett dependency.

Fill in (one line each — only the human knows these)

• Primary consumer(s):
• Why it exists (vs the obvious alternative):
• Hard constraint (airgap / no-C-deps / no-FFI / pure-Rust / etc.):
• Release cadence & independence (rides a parent? own repo someday?):
• Status (production / prototype / experimental / stubbed):
• Must-NOT-change invariant:
• Biggest known gap or TODO:
• Anything a newcomer would get wrong about it: