nordisk/knut

knut-thund — crate card

Reverse-engineered (from source — verify, don't trust the name)

• One-line pitch: Þund — a Rust-native, Arrow-centric batch+streaming dataflow engine with a pluggable execution backend (native Arrow/DataFusion, or lower-to-Spark-Declarative-Pipelines over Spark Connect); the "Airflow killer" authoring+runtime for knut.
• Does: Three layers around one IR: authoring (typed Rust builder + thin RON DSL, feature dsl), IR (one Arrow-typed dataflow graph — Pipeline, Dataset, Flow, OutputType, schemas, expectations, streaming/watermarks — with validate()/topo_order()/to_sdp()), and backends (ExecBackend trait + Capabilities/RunHandle/RunEvent/RunPhase). NativeBackend (feature native) lowers IR to DataFusion ExecutionPlans over RecordBatch streams; SparkBackend (feature spark) lowers IR to a knut-pipelines DataflowGraph and drives a Spark Connect server. Dagster-style asset model: declare artifacts, infer the DAG, recompute only stale.
• Key deps: serde/serde_json, thiserror, chrono, tracing, arrow-schema (IR is Arrow-typed); knut-pipelines (default dep for SDP model types, grpc for Spark backend); feature-gated ron (dsl), datafusion v54 (native), tokio+futures (runnable backends). Does NOT depend on skade today (only graphar-flight does) — the Iceberg write-path is worklist item 1 (see the gap doc).
• Shape: lib + wordcount_stream example; medium. Modules: authoring/ (dsl), ir/ (dataset, flow, pipeline, schema, streaming, expectation), backend/ (native, spark, checkpoint), error.
• Observed role in the workspace: The engine/runtime that produces and runs pipelines; reuses knut-pipelines as its viewer + Spark client half. Sibling to knut-bifrost (the materialisation engine).
• Native = a complete Spark alternative (in progress): the SDP-over-Connect drop list (watermark, window, ProcessingTime trigger, Update/Complete output modes, CDC apply-changes + apply_as_deletes + UPDATE_BEFORE, SCD2 history, expectations Warn/Drop/Fail) is fully executed by the native DataFusion backend with real row-level tests (tests/sdp_drop_coverage.rs + tests/iceberg_spark_gap.rs + tests/streaming_microbatch.rs). The SparkBackend::lowering_report strings now name the native counterpart ("RUN BY NATIVE: …"). Full scope + Databricks/Delta-vs-Iceberg catalogue + prioritized worklist: .nornir/thund-vs-spark-gap.md.

Fill in (one line each — only the human knows these)

• Primary consumer(s):
• Why it exists (vs the obvious alternative):
• Hard constraint (airgap / no-C-deps / no-FFI / pure-Rust / etc.):
• Release cadence & independence (rides a parent? own repo someday?):
• Status (production / prototype / experimental / stubbed): real prototype — the native backend genuinely executes batch + streaming (watermark/window/CDC/SCD2/expectations) on DataFusion, proven by tests; not yet landing to Iceberg (in-process/file materialisation only).
• Must-NOT-change invariant: the korp-consumed API stays ADDITIVE/stable (the ExecBackend/Capabilities/SdpLoweringReport/RunEvent surface korp's Spark view reads).
• Biggest known gap or TODO: Iceberg write-path via skade for the native backend (worklist item 1 in the gap doc) — turns "runs the computation" into "runs it AND lands it in the lakehouse". Blocked on the skade version collision (published 0.9.8 vs fork 0.9.1); knut-thund has no skade dep yet, so it is a clean add once pinned. Then partition-transform write + position-delete/MERGE onto Iceberg.
• Anything a newcomer would get wrong about it: the native-gated test files compile to ZERO tests under a default cargo test and report a green "ok" — the real coverage is the explicit cargo test -p knut-thund --features native CI lane. Also: Update output mode is an upsert delta, not a retraction stream (honest, not a bug).