nordisk/knut

GraphAr gar/v1 substrate — reasoning

Class: reasoning. This file holds the why, the trade-offs, and the gaps. The descriptive shape we run is in graphar-design.md, which by rule mentions none of this. If you want to know "why not X?" or "what doesn't it model yet?" — it's here.

Why five interchangeable formats

The gar/v1 layout is format-agnostic on purpose: a property group is a chunk, and the chunk's encoding is a storage decision, not a data one. We support all five — parquet, csv, orc, json, arrow-ipc — because each earns its place downstream: parquet for compact analytic storage (the default), orc for Spark/Hive interop, arrow-ipc for zero-copy hot paths, csv/json for human-readable fixtures and debugging. The law that makes this safe is G-1: every codec round-trips the same Arrow schema, so the format is swappable per property group without touching a single consumer. If two formats disagreed on schema, this whole property would collapse — which is what the round-trip conformance test guards.

Why arrow 58 pinned workspace-wide (one tree)

Arrow is not just a dependency here, it's the lingua franca between graphar, orc-rust, spark-connect, Flight SQL, and FalkorDB load. If two of those linked different arrow major versions, an arrow::RecordBatch produced by one would be a different, incompatible type to the other — no zero-copy hand-off, silent version-mismatch errors at the seam. So the whole workspace pins one arrow tree (58): orc-rust and spark-connect are held to the same version graphar reads, and a batch flows edge-to-edge without a re-encode. The cost is that bumping arrow is a workspace-wide event, not a per-crate one — a deliberate trade for one coherent columnar type across every crate.

Why _gar_id is the identity

A property graph needs a stable internal handle for endpoints. We use GraphAr's _gar_id — a vertex's absolute 0-based row index across its chunks — rather than any user property, because it is dense, contiguous, and assigned by the writer, so it's always present and always unique even before the user declares a primary key. Edges carry src/dst as _gar_ids (Edge::new(i64, i64)), which is exactly the key FalkorDB MERGE and the Flight SQL vertex/edge schemas build on. A user-property identity would be optional, possibly non-unique, and typed — three ways to be missing when an edge needs to resolve an endpoint.

Why object storage is first-class, not bolted on

The ChunkStore is generic over object_store from the start rather than being a local-filesystem type with an S3 adapter grafted on later. A graph the same size as the bench fixtures lives happily on disk; a 120M-node population graph does not, and the moment you need it on S3/GCS/Azure the entire read/write path — not just a copy step — has to speak the remote dialect. Making ChunkStore Arc<dyn ObjectStore>-generic means local and remote are the same code path (G-3); the only backend-specific thing is the constructor.

Why retry and multipart are first-class too. Remote object stores fail transiently (throttling, connection resets) and reject single-put uploads above a size limit — so a substrate that's "remote-capable" but retries nothing and can't upload a big chunk is remote-capable in name only. We front object_store's native retry with the typed RetryOptions (so callers tune backoff without importing object_store's config type) and route large writes through put_multipart. The multipart part size is kept ≥ 5 MiB because S3/GCS/R2 all reject smaller non-final parts; any error abort()s the upload so those stores don't leak orphaned parts. Default local / in-memory traffic stays under the 16 MiB threshold and never goes multipart, so the common case pays nothing.

Gotcha: from_url_with_retry can't use object_store::parse_url_opts — that entry point won't thread a typed RetryConfig through — so it dispatches on ObjectStoreScheme and applies with_retry on the per-scheme builder instead. Failure-injection against real S3 stays container-gated; the unit tests only cover the config plumbing.

Corpus model-gaps (validator skips, not failures)

Validating against the real apache/incubator-graphar-testing corpus surfaced two gaps that are skips, not bugs:

• list<float> parameterized type — knut's DataType enum has no parameterized list type, so the one corpus graph using list<float> is skipped rather than mis-modelled. Adding it means teaching DataType about parameterized element types.
• The java/ subtree — uses a different, non-gar/v1 info-file schema (label instead of type). It isn't a gar/v1 graph, so the corpus test skips it by design rather than reporting spurious errors.

To read the upstream info files at all, GraphInfo.prefix had to become #[serde(default)] — the corpus *.graph.yml omit it. During development the corpus test validated 12 of the corpus's gar/v1 graphs cleanly with those two skips.

Why round-trip conformance always, corpus opt-in

Two tiers, deliberately asymmetric:

• round_trip_conformance runs always: it builds a graph with VerticesBuilder/EdgesBuilder, dumps it, asserts the exact spec paths exist, and checks validate() is error-free. This is the gate that keeps G-1 honest and catches a codec or layout regression on every CI run, with zero external dependencies.
• corpus_validates is #[ignore]d and gated on GRAPHAR_TESTDATA pointing at a local checkout of the upstream corpus (validated with ValidateOptions::corpus() for extensionless chunks). It never fetches over the network. It's opt-in because it needs a multi-hundred-MB checkout no CI runner should clone on every build — it's the conformance proof you run deliberately, not the regression guard you run constantly.

Notes / measured

• Bench targets knut.format_{parquet,csv,json,orc,arrow_ipc}, knut.builder_dump, knut.merge_property_groups — see benching-benchmarks.md. Fixtures are synthetic and on /tmp, never the T9 USB drive (external IO skews throughput).
• Object-store round-trips (SeaweedFS / RustFS) are container-gated --ignored tests, not in the release gate. Validation is read-only and never touches the network.