.nornir/graph-to-python-guide.md · nordisk/knut

Guide — getting knut's graph into Python (networkx)

Class: guide. How to run the two DB-free roads from a knut rel2graph result into a networkx.MultiDiGraph: the file export (N3) and the Arrow Flight server (N4). Design: rel2graph-networkx-design.md, flight-graph-server-design.md.

0. Python deps (once)

python3 -m venv .venv
.venv/bin/pip install -r demo/requirements.txt   # networkx (+ pyarrow for the Flight demo)

Road A — file export → networkx (N3)

# 1. build the graph and dump node-list + edge-list + oracle (deterministic):
cargo run -p knut-rel2graph-bench --release --bin rel2graph-export -- 1 target/rel2graph-export
#    → nodes.jsonl (1106), edges.jsonl (2955), oracle.json

# 2. load it into a MultiDiGraph, run analytics, assert against the oracle:
.venv/bin/python demo/rel2graph_networkx_demo.py target/rel2graph-export

The demo prints counts by label/type, a degree distribution, the weakly-connected component count, and a shortest path, then prints ALL ASSERTIONS PASSED once the networkx graph matches knut's oracle — including REPORTS_TO present with zero self-loops (the a7208dc self-ref proof).

SCALE (arg 1) grows the graph linearly (10, 100, …); the demo asserts at any scale because the oracle is recomputed each run.

Road B — Arrow Flight server → networkx (N4)

# 1. start the server (in-memory graph at startup; Ctrl-C to stop):
cargo run -p knut-flight --release --bin knut-flight-server -- --addr 0.0.0.0:50055 --scale 1
cargo run -p knut-flight --release --bin knut-flight-server -- --version   # v<version> (<git-hash>)

# 2. (optional) also produce the file export so the demo can cross-check:
cargo run -p knut-rel2graph-bench --release --bin rel2graph-export -- 1 target/rel2graph-export

# 3. connect, list flights, pull nodes+edges as Arrow, rebuild the graph:
.venv/bin/python demo/flight_networkx_demo.py grpc://localhost:50055 target/rel2graph-export

The client lists the 35 advertised datasets, do_gets nodes + edges (and one source table), rebuilds the MultiDiGraph with the same loader Road A uses, and asserts the Flight-path graph equals the file-path graph and the oracle.

What the server serves

list_flights advertises: nodes, edges, node/<Label> (×10), edge/<src>_<rel>_<dst> (×11), table/<erp.name> (×12). The ticket is just the dataset name as bytes, so do_get(Ticket(b"nodes")) works from any Flight client. Bind address and scale come from --addr/--scale or KNUT_FLIGHT_ADDR/KNUT_FLIGHT_SCALE.

Road B-live — serve a REAL graph from parquet (not the bench)

The same server serves a real graph read off disk — an Iceberg / object-store export — instead of the synthetic generator. Produce a parquet export, then point the server at it:

# 1. dump the source tables as parquet alongside the file export:
cargo run -p knut-rel2graph-bench --release --bin rel2graph-export -- 1 ./graph-export --parquet

# 2. serve the graph the engine builds FROM those parquet files (read → rel2graph → Flight):
cargo run -p knut-flight --release --bin knut-flight-server -- --parquet ./graph-export

# 3. same client, same graph — now sourced from real files:
.venv/bin/python demo/flight_networkx_demo.py grpc://localhost:50055 ./graph-export

--mapping <file> supplies the rel→graph YAML for other data (it defaults to the bench 12-table spec, so a rel2graph-export --parquet dump serves out of the box); env KNUT_FLIGHT_PARQUET / KNUT_FLIGHT_MAPPING mirror the flags. This is the seam korp's iceberg2graph step drives — see rel2graph-engine-seam.md.

Verifying without Python

cargo test -p knut-flight runs tests/roundtrip.rs — a full Rust FlightClient round-trip (list + do_get + self-ref assertion) on an ephemeral port, no Python, no external server — and tests/live_data.rs, the same round-trip over a real parquet export (write parquet → from_parquet_dir → engine → client pull), asserting the live graph equals the bench graph. cargo test -p knut-rel2graph-bench runs the correctness bench the file export is built on.

The one loader both roads share

demo/knut_networkx.py — build_multidigraph(nodes, edges) is fed by load_json_export(dir) (Road A) or nodes_from_arrow/edges_from_arrow (Road B). Node id is the (label, id) pair; the graph is always a MultiDiGraph so parallel edges and the self-referential REPORTS_TO survive.