nordisk/knut

Watch a Spark pipeline turn green — for idiots

Class: idiot. Every step, nothing assumed. Goal: from a cold shell, open a web page and watch a pipeline's DAG animate from grey to green — with no Spark server at all — then run a dead-simple change-data-capture example. If you already know Rust, read pipeline-run-guide.md instead; this one spells out every keystroke.

The 10-second mental model

• A pipeline is boxes (datasets — tables you want) joined by arrows (flows — the queries that fill them).
• knut can draw those boxes in your web browser and colour them as the pipeline runs: grey = waiting, yellow = running, green = done.
• The demo needs no server. knut plays a pretend run so the boxes light up on their own. You just watch.

What you need first

• A terminal (the black window where you type).
• Rust. Type this, press Enter; a version number means you have it:

cargo --version

If it says "command not found", install it, then close and reopen the terminal:

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

• A web browser (Firefox, Chrome — whatever you already use).
• The knut repo on disk. Go into it (adjust the path to wherever you cloned it):

cd ~/git/knut

Step 1 — start the demo (one command)

Type this whole line and press Enter:

cargo run -p knut-pipelines --features web --bin knutpipe -- web

The first run compiles for a minute. Then it prints one line like:

# knutpipe web viewer on http://127.0.0.1:8787

That address is your pipeline page. Leave this terminal running — it is the server. (To stop it later, click the terminal and press Ctrl+C.)

Step 2 — open the browser and watch it turn green

Open your web browser and type that address into the address bar, then Enter:

http://127.0.0.1:8787

You'll see three boxes — bronze_events → silver_events → gold_daily — joined by arrows. Watch for a few seconds:

1. all boxes start grey (idle),
2. each turns yellow as it runs,
3. then green as it finishes,
4. the top line counts up: … — 3 done · 0 running · 0 failed · 0 idle,
5. the bottom line shows the last event, ending in COMPLETED.

When all three boxes are green, you just watched a pipeline run. That's the whole thing. No Spark, no setup — knut replayed a synthetic run over the page.

Refresh the page (F5) to watch it animate again from grey.

Step 3 (optional) — point it at a real Spark server

If you do have a Spark Connect server running (see pipelines-for-idiots.md for how to start one), stop the demo (Ctrl+C) and run the live version instead — add ,grpc and --connect:

cargo run -p knut-pipelines --features web,grpc --bin knutpipe -- \
  web --connect sc://127.0.0.1:15002

Same page, but now the colours follow a real run. If the server needs a login, put it right in the address, like --connect sc://127.0.0.1:15002/;user_id=alice;token=YOUR_TOKEN.

Step 4 (optional) — the tiniest change-data-capture example

"CDC" means: instead of rebuilding a table from scratch, you feed it a list of changes (add this row, change that one, delete another) and knut merges them in. Each change row carries a _change_type word: INSERT (add), UPDATE_AFTER (change), or DELETE (remove).

SCD type 2 means knut keeps the history: every version of a row is stored with a "valid from" and "valid to" time, so you can ask "what did this row look like last week?".

Here is the smallest possible one, in Rust. It says: keep a customers table, matched by id, ordered by seq, keeping full history (Type2):

use knut_thund::ir::{CdcSpec, Flow, FlowKind, ScdType};

let flow = Flow {
    name: "apply_customers".into(),
    target: "customers".into(),
    reads: vec!["customer_changes".into()],   // the stream of change rows
    kind: FlowKind::Cdc { cdc: CdcSpec {
        keys: vec!["id".into()],               // match rows by id
        sequence_by: "seq".into(),             // newest change wins
        apply_as_deletes: None,
        scd_type: ScdType::Type2,              // Type2 = keep history
    }},
    query: Some("SELECT id, name, seq, _change_type FROM customer_changes".into()),
    expectations: vec![],
};

Feed it changes like:

id=1 name="Ada"    seq=1  _change_type=INSERT
id=1 name="Ada L." seq=2  _change_type=UPDATE_AFTER
id=2 name="Bo"     seq=1  _change_type=INSERT
id=2 ...           seq=3  _change_type=DELETE

and the customers table ends up with Ada's two versions (the first closed off, the second still open) and Bo's version closed by the delete. Change Type2 to Type1 and you'd keep only the latest version of each row (no history).

You don't have to hand-build these change rows for a real table: knut's bifröst reads them straight out of an Iceberg table (it borrows skade's changelog reader, so the _change_type words always mean the same thing).

Run the built-in tests that prove both behaviours:

cargo test -p knut-thund --features native \
  cdc_scd_type1_upserts_and_deletes cdc_scd_type2_keeps_version_history

Two green tests = the merge works.

Step 5 — stop the server

Click the first terminal (the one still running the web viewer) and press Ctrl+C. Done.

Cheat sheet

Where this goes next

• Running a real Spark server and a real pipeline (not the demo): pipelines-for-idiots.md.
• The full workflow (auth, Airflow events, CDC in Rust): pipeline-run-guide.md.
• How it all fits together under the hood: pipeline-run-design.md.