nordisk/knut

Native GRAPH.BULK fast-lane — design

Class: design · Function axis: f6 native bulk encoder · f7 auto-select · f8 streamed/appendable sharding · f9 chunked multicore encode. Extends falkordb-design.md (the UNWIND load bridge). Reuse graph: falkordb.uses = [graphar] — endpoint identity is graphar's _gar_id. Strict: this file states only the shape we run. The why is in falkordb-reasoning.md.

f6 — the native GRAPH.BULK encoder (the fast lane)

Alongside the UNWIND … CREATE path, graphar-falkordb speaks FalkorDB's binary bulk-insert protocol natively (src/bulk.rs) — the same wire the falkordb_bulk_loader uses, but emitted in pure Rust straight from Arrow columns: no CSV, no Python, no shell (L9). GRAPH.BULK bypasses the Cypher parser/planner entirely, so it is the initial / empty-graph population fast lane; UNWIND stays the incremental/upsert path.

The wire (mirrors FalkorDB src/bulk_insert/bulk_insert.c, verified live):

GRAPH.BULK <key> [BEGIN] <node_count> <relation_count> <label_blob_count> <reltype_blob_count> <node_blob…> <reltype_blob…>

BEGIN creates the graph (first call to an absent key; errors if the key already exists); the tokenless form appends (errors if the key is absent). Node global ids are assigned in blob order — the first node of the first node blob is id 0 — which is exactly how a relation blob's uint64 endpoints reference nodes.

Typed value = 1 type byte + payload: NULL(0)→∅ · BOOL(1)→1 byte · DOUBLE(2)→f64-LE · STRING(3)→bytes+\0 · LONG(4)→i64-LE · ARRAY(5)→u64-LE len+that many values. Integers and floats are little-endian (the server reads them with a host-order pointer cast).

Non-finite floats and nulls both encode as NULL — exactly what the UNWIND path renders as Cypher null — so the two paths store the same value (B-2).

f7 — auto-select (initial → bulk, incremental → UNWIND)

choose_bulk(graph_is_empty) returns the fast path only for an empty graph (bulk CREATEs, it can't MERGE onto existing nodes). FalkorDbLoader::graph_is_empty() probes count(n); bulk_insert then reads EXISTS (a non-mutating check) to decide BEGIN (absent key) vs append (a key a probe already materialized).

Consumer — knut-popviz auto-selects bulk. The population demo's load_falkor (crates/knut-popviz) now takes the same fork: after the --drop DETACH DELETE, an empty graph auto-selects the GRAPH.BULK streamed lane (IngestMode::Auto → choose_bulk(graph_is_empty)), a non-empty one keeps the parallel-UNWIND path. Two reconciliations make popviz's graph byte-identical either way: its vertex batches carry a global, offset-based _gar_id column (reused verbatim by B-2's supplied-column mode — no duplicate, no 0-based flip), and the bulk lane regenerates edges with zero offsets so the encoder's base + position (B-3) reproduces the same global endpoints the UNWIND path MATCHes. Node groups are passed in LABELS order, so each group's base offset equals popviz's offsets[li]. Proven live by knut-popviz's popviz_bulk_matches_unwind_byte_identical (byte-identical across all 12 labels

• 11 rels) and pinned DB-free by bulk_groups_reconcile_gar_id_and_edges.

f8 — streamed / appendable sharding

bulk_insert_streamed sends each node group, then each edge group, as its own appendable GRAPH.BULK call (BEGIN only on the first to an absent key) instead of one giant command — bounding per-command size/memory for very large loads. It is ordered on purpose: the server assigns global node ids in call order, and every node must exist before its edges, so the streamed result is byte-identical to the single-shot bulk_insert. This is the seam for future multi-graph-key sharding (split disjoint subgraphs across keys, then alias/merge) — deferred until a live-DB GRAPH.RESTORE/merge story is proven (see reasoning).

f9 — chunked, multicore encode (no single-core stall, no broken pipe)

At demo-zone max scale the dominant label (a 60M-row TaxRecord) used to encode its whole blob on one core and then ship it as a single GRAPH.BULK command whose payload blew past FalkorDB's proto-max-bulk-len (512 MB default) — the server closed the socket mid-write and the fill died with Redis error: Broken pipe. Fixed in build_bulk_payload: each node/edge group is split into ≤ MAX_ROWS_PER_BLOB (2M) blobs and the blobs are encoded in parallel across all cores via gatling (LAW #0: gatling, never rayon), laid down in order. Because blobs stay ordered — every chunk of label li before li+1, chunks in row order — the server assigns the same contiguous global ids, so the graph is identical byte-for-byte; only the wire is split (and progress reporting gets finer for free). encode_node_blob gained a gar_base so a split label keeps assigning contiguous synthesized _gar_ids (byte-identical to the one-blob path; popviz's supplied _gar_id column is still passed verbatim). Every command now stays well under the wire limit.

Guarded by chunk-window splitting, gar_base continuity across a split, and chunked-vs-single-blob graph equivalence (edges resolve across the boundary). Measured live (e0c0378, scale 0.3, oden): 46 GRAPH.BULK blobs encoded across up to 32 cores in 3.65 s (36,360,000 nodes + 33,060,000 relations; each blob < proto-max-bulk-len) → fill OK in 31 s, no broken pipe — the exact scenario that crashed before now saturates the box.

Parity + numbers

The container-gated bulk_matches_unwind_byte_identical proves a GRAPH.BULK load (single-shot and streamed) yields the byte-identical graph — same nodes, _gar_id, properties, and relationships (incl. edge props) — the UNWIND path produces from the same Arrow batches. DB-free byte-exact tests in src/bulk.rs guard the encoder (header layout, type bytes, endianness, _gar_id scheme, endpoint translation, out-of-range errors) with no server.

Live bench (bench_bulk_vs_unwind, 200k nodes, debug, Loki, one FalkorDB container):

End-to-end consumer number — knut-popviz's whole 12-label population (nodes + edges + _gar_id indexes) via bench_popviz_bulk_vs_unwind (60,600 vertices, debug, Loki 12c, one container): parallel-UNWIND 16.1k nodes/s → streamed GRAPH.BULK 356k nodes/s = ≈22× (lower than the raw ingest ratio because it wraps edge load + index creation, the honest demo-fill wall).