nordisk/knut

knut-flight — an Arrow Flight server for the rel2graph output (N4)

Class: design. A plain Apache Arrow Flight server (not Flight SQL) that serves knut's rel2graph output — nodes, edges, and the source relational tables — as ticketed Arrow streams. Sibling of the Flight SQL server (graphar-flight, Cypher-in/Arrow-out over a live FalkorDB): this one has no database and no query language; it just streams the graph the mapping fabric built. Source: crate crates/knut-flight — lib.rs (Catalog, GraphFlightService), bin src/bin/knut-flight-server.rs; test tests/roundtrip.rs; Python client demo/flight_networkx_demo.py.

Why a new crate, not graphar-flight

graphar-flight implements the Flight SQL trait: it takes Cypher, runs it against FalkorDB, and returns Arrow. knut-flight implements the plain Flight trait ([FlightService]) over an in-memory [Catalog] built from a GraphMutation at startup. Different trait, no DB, no query language — so it stays a clean single-binary service. Ports are disjoint: FlightSQL uses :50051/:50054, this server defaults to :50055.

The catalog

Built once at startup from a rel2graph GraphMutation through the single seam Catalog::from_mutation(mutation, source_tables). There are two entrypoints onto that seam:

• Catalog::from_bench(scale) — the synthetic path: runs the engine over the seeded 12-table ERP schema (deterministic, DEFAULT_SEED).
• Catalog::from_parquet_dir(spec, dir) — the live-data path: reads <table>.parquet for every table in a mapping spec and runs the engine's canonical knut_bifrost::rel2graph::build_mutation over the collected ArrowTables. This serves a REAL graph read off disk — an Iceberg / object-store export — with no bench and no live DB. It is the shape korp's iceberg2graph step produces when it lands lakehouse tables as parquet and hands the directory to this server. See the engine seam.

Both funnel through from_mutation, so a bench catalog and a real-data catalog are identical in shape. At scale 1 the bench path materialises 35 datasets:

The ticket is the dataset name as UTF-8 bytes — no protobuf envelope — so any Flight client (Rust, pyarrow, Go) round-trips it verbatim. The nodes/edges aggregates carry a UNIFORM schema across labels: exactly what a networkx loader wants (see below). The node/* and edge/* flights carry the full native typed columns; table/* exposes the rel2graph inputs for provenance.

The Flight surface

Read-only. Implemented verbs:

• list_flights — one [FlightInfo] per dataset: its Arrow schema, a Cmd descriptor and a Ticket (both the name bytes), and total_records.
• get_flight_info — the FlightInfo for a descriptor (cmd bytes, or a /-joined path).
• get_schema — a dataset's schema without pulling data.
• do_get — streams the dataset's RecordBatches for a ticket, encoded with FlightDataEncoderBuilder.

The mutating verbs (do_put, do_exchange, do_action, handshake, poll_flight_info) return unimplemented — this is a read-only projection server; list_actions returns an empty list.

The binary

knut-flight-server — config via flags/env only, no interactive setup:

With --parquet <dir> the server serves a REAL graph from a parquet export (the live-data path above); --mapping supplies the rel→graph YAML for that data and defaults to the bench 12-table spec, so a rel2graph-export … --parquet <dir> dump serves out of the box. Without --parquet, --scale sizes the bench.

--version prints knut-flight-server v<version> (<git-hash>) — the git hash is stamped by build.rs (KNUT_GIT_HASH, pure-Rust file walk, no shelling), per the build-version law. --help lists the flags.

cargo run -p knut-flight --release --bin knut-flight-server -- --addr 0.0.0.0:50055 --scale 1

Proving it end-to-end

Two proofs, one per audience:

• Rust — tests/roundtrip.rs stands the server up on an ephemeral port, connects a real FlightClient, list_flights (asserts the 35 datasets incl. nodes, edges, edge/Employee_REPORTS_TO_Employee, table/erp.hr.employees), and do_gets nodes/edges/a source table, asserting 1106/2955 rows and that every REPORTS_TO edge has src != dst (no self-loop, a7208dc).
• Python — demo/flight_networkx_demo.py connects with pyarrow.flight, lists flights, pulls nodes+edges as Arrow, and rebuilds a MultiDiGraph with the same demo/knut_networkx.py loader the file-export demo (N3) uses — then asserts the Flight-path graph equals the file-path graph AND knut's oracle. Same graph, two delivery roads.
• Live data — tests/live_data.rs writes the scale-1 source tables to parquet (write_source_parquet, the export bin's --parquet), builds a catalog with from_parquet_dir over that directory, asserts it equals the bench catalog's shape (35 datasets, 1106 nodes, 2955 edges, 21 REPORTS_TO with no self-loop), and pulls nodes/edges/a source table with a real FlightClient — a client pull of REAL on-disk data, engine-driven end to end.

N5: a service, two ways (DONE)

This clean single-binary service — in-memory state built at startup, env-only config, no interactive setup, disjoint port — is now delivered as a service two ways, both driven from ONE ComponentSpec::knut_flight in Skidbladnir (so the Runtime targets never drift):

1. systemd — SystemdInstaller stages the binary into a TarZstd bundle, unpacks it under the airgap layout, and drops knut-flight.service (User=_knut, Environment=KNUT_FLIGHT_ADDR=0.0.0.0:50055 / KNUT_FLIGHT_SCALE=1, TCP health on 50055), enable+start over systemd D-Bus. Driver: skidbladnir --example install-knut-flight.
2. Kubernetes — containers/knut-flight/Containerfile images the version-stamped release binary onto a slim glibc base (knut-flight-server:0.1.3); Skidbladnir's KubernetesRenderer projects the same spec to a Deployment + Service in namespace knut (skidbladnir --example render-knut-flight-k8s). Manifests + apply steps: Skidbladnir/deploy/knut-flight/.

The image build copies the host-built binary (the knut workspace has path deps on sibling repos, so it does not cargo build in isolation inside a container); the binary needs glibc ≤ 2.39, so a trixie base runs it directly.

Future work: source the graph from a live sink or object store instead of the bench generator (the Catalog builder is the single seam to swap).

See also

• The file-export sibling that feeds the same networkx loader: rel2graph-networkx-design.md.
• The Flight SQL server (Cypher/FalkorDB): flight-design.md / flight-guide.md.
• The graph seam + correctness oracle: rel2graph-design.md, benching-benchmarks.md.