nordisk/knut

Spark pipelines with knut — for idiots

Class: idiot. Every step, nothing assumed. Goal: from a cold shell, start a Spark Connect server, then define, run and watch a real Spark Declarative Pipeline with knutpipe. If you already know Rust and Spark, read pipelines-guide.md instead — this one spells out every keystroke.

The 30-second mental model

• A pipeline = a dataflow graph: some datasets (tables/views you want to produce) and some flows (the SELECTs that fill them).
• You hand that graph to Spark Connect (a gRPC server; address looks like sc://host:15002). Spark figures out the order and runs it.
• knut streams back events ("flow X is RUNNING") and job progress (task counts). The viewer shows both. You write the graph; Spark does the work; knut shows what's happening.

What you need first

• A terminal (the black window where you type commands).
• Rust installed. Type this and press Enter; if it prints a version you have it:

cargo --version

If it says "command not found", install Rust: type this, press Enter, and accept the default:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

Then close and reopen the terminal.

• podman (runs the Spark server in a container). Check it:

podman --version

If missing, install it from your OS package manager (sudo apt install podman on Ubuntu/Debian).

• The knut repo on disk. Type this and press Enter:

cd /run/media/rickard/T9/git/knut

Step 1 — see a pipeline with NO server (learn the shape, zero setup)

Type this and press Enter:

cargo run -p knut-pipelines --bin knutpipe -- show

The first run compiles for a minute, then prints a built-in bronze → silver → gold demo pipeline as text. Now see the same thing as JSON — type and Enter:

cargo run -p knut-pipelines --bin knutpipe -- show --json

That JSON is the spec format. A pipeline spec is a dataflow-graph JSON where each flow that should run carries a query. A ready-made one ships in the repo at crates/knut-pipelines/examples/pipeline.json — that's the file we'll run.

Step 2 — build the Spark Connect server container (once)

Type this and press Enter (it downloads Spark + Iceberg and bakes an image; takes a few minutes the first time):

podman build -t knut-spark-iceberg -f containers/spark-iceberg/Containerfile .

When the prompt comes back with no red errors, the image is built. You only ever do this once.

Step 3 — start the server

Type this and press Enter:

podman run -d --name knut-spark -p 15002:15002 knut-spark-iceberg

That starts Spark in the background and maps its port 15002 to your machine. Now wait — Spark's JVM needs 10–25 seconds before it accepts connections. Type this and press Enter:

sleep 25

Step 4 — define, run and WATCH the pipeline (one command)

Type this whole block and press Enter (the \ lets one command span lines):

cargo run -p knut-pipelines --features grpc --bin knutpipe -- \
  pipeline sc://127.0.0.1:15002 \
  --spec crates/knut-pipelines/examples/pipeline.json \
  --storage file:///tmp/knut_pipeline

--features grpc builds the live client (first time compiles a bit longer). That single command does the whole dance for you:

1. connects to Spark Connect,
2. defines the graph (CreateDataflowGraph → a DefineOutput per dataset → a DefineFlow per flow),
3. starts the run and streams the lifecycle to your terminal.

You should see something like:

# defining pipeline (2 datasets, 2 flows)
# graph 03ce891f-…; starting run
INFO   Flow spark_catalog.default.knut_bronze is QUEUED.
INFO   Flow spark_catalog.default.knut_bronze is RUNNING.
INFO   Flow spark_catalog.default.knut_bronze has COMPLETED.
INFO   Flow spark_catalog.default.knut_silver has COMPLETED.
INFO   Run is COMPLETED.

=== final state ===
Dataflow graph 03ce891f-… — 2 dataset(s), 2 flow(s)  [2 done · 0 running · 0 failed · 0 idle]
  ✓ [materialized_view] knut_bronze  (done)
  ✓ [materialized_view] knut_silver  (done)

Run is COMPLETED with both nodes done = you just ran a real Spark pipeline and watched it. That's the whole thing.

The two mistakes that trip everyone the first time:

• --storage is required. Spark needs a place for checkpoints/metadata to actually materialise outputs. Without it, the run only dry-validates.
• The storage path must be writable by the Spark workers, not just by you.

Step 5 — want the pretty node graph instead of text?

Add --gui (and --features gui). Same run, drawn as a live node graph: nodes go yellow while running, green when done, red on failure, with job-progress bars. Type and Enter:

cargo run -p knut-pipelines --features grpc,gui --bin knutpipe -- \
  pipeline sc://127.0.0.1:15002 \
  --spec crates/knut-pipelines/examples/pipeline.json \
  --storage file:///tmp/knut_pipeline --gui

Step 6 — just watch a plain SQL job (no pipeline at all)

Any Spark query streams progress, so knut doubles as a job monitor. Type and Enter:

cargo run -p knut-pipelines --features grpc --bin knutpipe -- \
  --json-logs watch sc://127.0.0.1:15002 \
  --sql "SELECT count(*) FROM range(0, 50000000, 1, 24)"

--json-logs prints machine-readable lines ("event":"connected", then climbing "event":"job_progress") — the same lines a test drives.

Step 7 — clean up

When you're done, stop and remove the server. Type these two, pressing Enter after each:

podman rm -f knut-spark
rm -rf /tmp/knut_pipeline

Cheat sheet

Where this goes next

• The engine-agnostic viewer (one adapter turns knut's Spark state into facett's neutral model) — the "facett adapter" section of pipelines-design.md.
• The reusable visual node-graph component lives in facett: facett-pipeline for dummies.
• A full app wiring knut + facett together (connect → submit → render): korp's worked example.