nordisk/knut

Arrow → FalkorDB bulk load — reasoning

Class: reasoning. This file holds the why, the trade-offs, and the gotchas. The descriptive shape we run is in falkordb-design.md, which by rule mentions none of this. Reuse graph: falkordb.uses = [graphar] — the _gar_id identity is defined in graphar-reasoning.md.

Why UNWIND batches, not row-by-row

The naïve loader issues one CREATE per row. That's one Redis round trip per node — for a 120M-node graph, 120M round trips, each dominated by network latency, not by the work. So we render a whole batch of rows as a single UNWIND [ {..}, {..} ] AS r CREATE …: one round trip amortises DEFAULT_BATCH_SIZE (512) rows, and FalkorDB plans the CREATE once for the whole list. The batch size is the tuning knob (with_batch_size); the shape — one statement per batch — is the law (F-1).

Why the hot path is column-resolved (H1)

The obvious way to build a Cypher map is per cell: match on the Arrow DataType, downcast the array, format the value into a fresh String, join. For a batch of 512 rows × M columns that's 512·M downcasts and as many temporary Strings — the downcast and the allocation, not the formatting, dominate. ColRenderer::resolve downcasts each column once (M downcasts per batch, not 512·M), and write_row_map renders every cell straight into one pre-sized buffer — no per-cell downcast, no intermediate String. This is the §H1 win the knut.cypher_render bench measures; it's why the renderer is column-shaped, not row-shaped.

Why Arrow over CSV

FalkorDB ships a falkordb_bulk_loader that eats CSV. We don't use it and we don't emit CSV:

• Typed, columnar, no text parse. Arrow batches are already the typed columnar form the renderer wants; CSV would mean serialize-to-text then parse-back, throwing away the types on the way out and re-guessing them on the way in.
• The same batches persist elsewhere. An Arrow batch loaded into FalkorDB is byte-for-byte the same batch that persists as GraphAr/parquet or an Iceberg table, read back zero-copy. A CSV intermediate is a dead-end artifact — strictly worse than the Arrow we already hold.

Why knut never shells out to falkordb_bulk_loader

Shelling out to the external bulk loader would mean: materialize CSV to disk, spawn a Python process, and hand it a second connection with its own auth, batching, and error surface that knut can't observe. Instead every knut bulk path — FalkorDbLoader, parallel loaders, Flight SQL DoPut, Bifröst's FalkorDbSink — goes over the same Redis/RESP wire through the same insert_* methods. One code path, one error type (FalkorError), one place to tune, and the loads compose (Flight and Bifröst call the loader rather than reinventing it).

Why explicit-and-total value mapping (loud, not silent)

arrow_value_to_cypher maps a fixed set of Arrow types and returns FalkorError::UnsupportedType on anything else. The tempting alternative — fall back to format!("{:?}") for unknown types — is silent data corruption: a type we never validated gets a debug-string rendering that looks like it loaded and is wrong. Making the map total means an unmapped type is a visible gap that fails the load, not corrupt data sitting in the graph. Adding a type is a deliberate act (extend the match), which is exactly the friction we want on the correctness-critical seam. Date/Timestamp/List rendering was added this way (done 2026-06-11).

Why edges MERGE on _gar_id, index first

Edges reference endpoints by graphar's _gar_id, and both endpoints are matched before the relationship is created. Without an index, each endpoint match is a label scan — O(nodes) per edge, quadratic over a load. So the workflow is create_index(label, "_gar_id") before any edge MERGE: build the range index while the graph is still just vertices, and every endpoint lookup becomes a point read. This ordering is a workflow rule, not something the loader can enforce — hence it lives in the guide and here, loudly.

Gotcha: FalkorDB on Valkey 9

The FalkorDB module gates on redis-compat ≥ 8.0, but Valkey 9 reports redis_version as 7.2.x — so the module refuses to load on an otherwise-fine Valkey 9. The containers/falkordb-valkey9 Containerfile lowers that gate to 7.0 at build time so the module loads. Live tests are container-gated (--ignored) and run against both stock FalkorDB and FalkorDB-on-Valkey 9 (KNUT_FALKORDB_IMAGE), asserting node/edge counts and properties — so this compatibility path is actually exercised, not assumed.

Why a native GRAPH.BULK fast lane (and why it's not the default path)

UNWIND … CREATE makes the server parse + plan + execute Cypher per batch. For an initial population that planning is pure overhead — the shape is always the same CREATE. FalkorDB's GRAPH.BULK protocol skips the planner and ingests a typed, columnar node/relation payload directly. knut already holds the typed Arrow columns, so a native Rust encoder (src/bulk.rs) emits that payload with zero CSV, zero Python, zero shell (L9) over the same RESP wire and the same FalkorError — the reasoning doc's "one code path over the wire" law holds, we just add a faster verb on it. Measured ≈48× the parallel-UNWIND path and ≈108× the single-connection baseline at 200k nodes (debug).

It is not the default because GRAPH.BULK CREATEs and cannot MERGE — it is safe only on a fresh/empty graph. So choose_bulk gates it on graph_is_empty(), and the incremental/upsert loads (bifrost sink, re-populate) stay on UNWIND. The result is provably identical: the container-gated parity test reads both graphs back and asserts byte-equal node/edge rows.

Gotcha discovered live (the BEGIN/EXISTS dance). BEGIN creates the graph and errors if the key already exists; the tokenless append errors if the key is absent. But the auto-select probe (graph_is_empty, a GRAPH.QUERY) materializes an empty key — so after probing, BEGIN would fail "key already exists". The loader therefore reads a non-mutating EXISTS and sends BEGIN iff the key is absent, appending otherwise. Both branches are exercised by the live test (single-shot on a probed key = append; streamed on a fresh key = BEGIN-then-append).

Gotcha: the two blob-count tokens. The wire is GRAPH.BULK <key> [BEGIN] <nodes> <rels> <label_blob_count> <reltype_blob_count> <blobs…> — the two entity counts are followed by the count of node blobs and count of relation blobs (FalkorDB's "number of node descriptor tokens"). Omitting them makes the server read a blob as that count and fail; this is only discoverable against a live server, hence the container gate.

Deferred: true multi-graph-key sharding + offline RDB

bulk_insert_streamed chunks one load into ordered appendable calls (bounded memory) but still targets one graph key. Splitting a load across several graph keys for parallel ingest needs a correct cross-shard-edge story (edges whose endpoints land in different shards) plus a merge/alias step — undocumented and module-version-specific in FalkorDB — so it is deferred to a version-pinned spike. Likewise the genuine "server-off" RDB-assembly path: the serialized graph form is module-version-specific and not appendable; GRAPH.BULK (fast + appendable + safe) is the shipped answer.

Note: composite keys + multi-label, done

Composite MERGE keys and multi-label nodes landed 2026-06-11 (plan §8). The design point worth remembering: the single-key / single-label path is kept byte-identical to the historical one — insert_vertices_multi with a one-label slice renders exactly Label , and insert_edges_keyed with ["_gar_id"]/["_gar_id"] still stages the legacy src/dst carriers. New capability, zero regression on the old path.