nordisk/knut

knut — guide (f0, the whole workflow)

Class: guide · Function axis: f0 — the end-to-end session that spans several functions. Each function has its own *-guide.md for the detail; this file is the spine that runs them in order. The what is in design.md; the why in reasoning.md.

This is the 10-minute path from nothing to a live, queryable graph that stays fresh under the query. Every step links to the function chapter that goes deep.

The whole session, in order

1 — Start the backing services

First, work out how little you actually need. Not every knut path wants a service; the Spark-free Iceberg read wants none. This map is the answer to "what do I have to start before this works", per path:

The short version: an embedded box is a crate you link or a directory on disk — nothing to start. A service box is a process that must already be running. Reading Iceberg with the skade feature needs zero services; adding the FalkorDB graph sink needs exactly one; only the Spark Connect lane needs a JVM. Transfer methods into the graph are detailed in falkordb-guide.md, the Iceberg lanes in bifrost-guide.md.

knut talks to FalkorDB (on Valkey 9) for the graph and, for the lakehouse path, a Spark+Iceberg endpoint. The Containerfiles that back the live test matrix are in containers/ (falkordb-valkey9, spark-iceberg):

podman build -t falkordb-valkey9 containers/falkordb-valkey9
podman run -d --name falkordb -p 6379:6379 falkordb-valkey9

Or let a --features autostart build bring FalkorDB up from Rust (testcontainers → Podman).

2 — Get a graph in (bulk load)

Two easy on-ramps, both knut-native (no falkordb_bulk_loader):

# The synthetic 12-label Swedish-population graph, straight to FalkorDB:
cargo run -p graphar-falkordb --release --example population
# The same load with a live per-label progress GUI + headless introspection:
cargo run -p knut-popviz --features gui -- --target falkor --scale 1.0 --autostart

Under the hood this is Arrow batches → FalkorDbLoader::insert_vertices/_edges (one UNWIND … CREATE/MERGE per batch). Detail: falkordb-guide.md, popviz-guide.md. To persist the same batches as files first, write GraphAr chunks with graphar::io::write_chunk — graphar-guide.md.

3 — Serve the graph over Arrow Flight SQL

Front the graph with a Flight SQL server so any ADBC/ODBC client (including Power BI) can read it. Name the queries you want to expose as "tables":

let mut registry = SchemaRegistry::new();
registry.register_named("people", "MATCH (p:Person) RETURN p._gar_id, p.name", schema);
run_server(redis_url, "pop", registry, "0.0.0.0:50051").await?;

Add TLS / bearer / Basic / mTLS with run_server_with(.., ServerOptions { .. }). Full API + the bearer-rotation and per-query-authz workflow: flight-guide.md.

4 — Query it (any Arrow client)

# The thin Python client → pandas / NetworkX:
python -m knut.query "SELECT * FROM people"

A generic SELECT * FROM people is rewritten to the registered Cypher server-side; the rows stream back as Arrow. Power BI connects with the stock ADBC/Flight SQL driver.

To pull a graph straight into NetworkX for analysis in Python (shortest path, centrality, …), KnutClient.to_networkx(vertex_query, edge_query) walks FalkorDB → Arrow → pandas → NetworkX in one call — see networkx-export.md and python/examples/social_graph.py.

5 — Keep it fresh from the lakehouse (Bifröst CDC)

Point Bifröst at a file-drop directory and it materialises rows into the same graph on a poll loop, idempotently:

bifrost gen-sample --files ./drop --table lake.people --rows 100000   # try it with no lakehouse
bifrost run --mapping mapping.yml --files ./drop --falkor redis://localhost:6379 --graph pop --interval 30

The MERGE sink means the live query in step 4 keeps working while the graph updates. Mapping YAML, CDC, the log contract, and the state ledger: bifrost-guide.md.

The Iceberg sources are library-API today, not CLI. The shipped bifrost binary wires FileSource only — there is no --warehouse / --spark flag. To watch an Iceberg table you call knut_bifrost::listener::watch_iceberg_delta(...) (Spark-free, --features skade) or assemble Listener::new(spec, reg, SparkConnectSource::connect(..), sink, state) from your own main. Which of those needs a service running is lane A vs lane C on the map in step 1. Both lanes end to end, the transfer-method menu into FalkorDB, and the no-pushdown caveat: iceberg-falkordb-guide.md.

Alongside — author + watch the compute

The compute that fills the lake is a pipeline. Watch a Spark Declarative Pipeline or any Spark Connect job with knutpipe (pipelines-guide.md):

knutpipe watch sc://localhost:15002 --sql "SELECT count(*) FROM lake.people"

Or author the pipeline itself in Þund's one IR and run it on the native or Spark backend — thund-guide.md.

Where each capability lives

Never done this before? for-idiots.md spells out every keystroke.