nordisk/knut

GraphAr gar/v1 substrate — design

Class: design · Function axis: f0 (the chunked-file substrate) + f1 formats/io · f2 metadata/info-files · f3 readers/builders · f4 ChunkStore/object-store · f5 validation. Strict: this file states only the shape we run. Every why, every rejected alternative, lives in graphar-reasoning.md; the workflow is in graphar-guide.md.

f0 — the chunked-file substrate (the whole)

A graph is a directory of chunks: column-oriented files of a fixed row count, described by YAML info-files, storing one Apache GraphAr gar/v1 property graph. graphar reads and writes those chunks as Arrow RecordBatches, in five interchangeable file formats, over local disk or object storage (crates/graphar/src, re-exported from src/lib.rs).

Three laws hold across the whole substrate:

Reuse map (what the graphar subsystem uses): it depends only on arrow (58, one tree workspace-wide), object_store, and the format codecs (parquet, orc-rust, arrow-ipc). It is used by graphar-falkordb (bulk load), graphar-flight (Flight SQL), knut-bifrost, and the bench suite — graphar is the columnar floor they all stand on.

f1 — formats & io

io::write_chunk(path, &[RecordBatch], &FileType) (src/io/mod.rs:22) and io::read_chunk(path, &FileType) (src/io/mod.rs:12) are the file-level seam. FileType (src/types.rs:50) is the five-way format enum, each with a fixed extension via FileType::extension() (src/types.rs:60):

write_chunk creates parent dirs then matches the FileType to the codec's write; each codec has a mirror read. Value types the schema may carry are DataType (src/types.rs:8) and edge topology is AdjListType (src/types.rs:73).

f2 — metadata / info-files

The gar/v1 graph is described by YAML info-files, modelled in src/metadata/:

A vertex label's columns are partitioned into property groups, each group stored as its own vertex/<label>/<group>/chunk0..chunkN run; edge groups are partitioned per source/dest chunk. This is the layout f5 validates and f3 reads.

f3 — readers & builders

Builders (write side, src/writer/) accumulate rows then dump chunks:

• VerticesBuilder::new(info, base_dir) (src/writer/vertex_writer.rs:66) → add_vertex(Vertex) (:74) → dump() -> u64 (:79, returns vertex count). A Vertex (:24) carries PropertyValues (src/writer/vertex_writer.rs:30) built with add_property (:48).
• EdgesBuilder::new(...) (src/writer/edge_writer.rs:56) → add_edge(Edge) (:71) → dump() (:76). Edge::new(src, dst) (:27) keys endpoints by _gar_id.

Readers (read side, src/reader/) stream a graph's chunks back to Arrow:

• VertexReader::new(info, base_dir, vertex_count) (src/reader/vertex_reader.rs:19); read_property_group(group) (:36), read_merged() (:64) joins all groups row-aligned, read_merged_groups(groups) (:70), read_by_property(name) (:77).
• EdgeReader::new(...) (src/reader/edge_reader.rs:23); read_adj_list(...) (:50), read_offset(...) (:78), read_property_group(...) (:98), iter_vertex_chunks(...) (:124).

Row-aligned joining of property groups is exposed standalone as merge_property_groups(&[Vec<RecordBatch>]) -> RecordBatch (src/merge.rs:21).

f4 — ChunkStore over object_store

ChunkStore (src/storage.rs:246) is the storage seam: one API over any object_store backend. Constructors: new(store, prefix) (:254), local(root) (:372), from_url(url) (:284), s3_compatible(...) (:387). Async chunk ops: read_chunk(rel, &FileType) (:442), write_chunk(rel, &[RecordBatch], &FileType) (:452), get_bytes (:463), put_bytes (:469). Free-standing codecs read_chunk_bytes/write_chunk_bytes (:166,:199) do the in-memory encode/decode.

Retry is object_store's native backoff, fronted by the typed RetryOptions (src/storage.rs:51: max_retries, init_backoff, max_backoff, retry_timeout; Default matches the library's 10 retries / 100 ms→15 s / 3-min budget; RetryOptions::none() (:78) opts out; From<RetryOptions> for RetryConfig at :97). from_url / s3_compatible use the default; from_url_with_retry (:297) and s3_compatible_with_retry (:407) override it — from_url_with_retry dispatches on ObjectStoreScheme and applies with_retry per-scheme (it can't thread RetryConfig through parse_url_opts).

Multipart upload is fronted by MultipartOptions (src/storage.rs:136: threshold default 16 MiB, part_size default 8 MiB). Attached builder-style via with_multipart(opts) (:267), read back by multipart_options() (:273). A write whose encoded length exceeds threshold is sliced into part_size parts and uploaded with put_multipart (sequential, bounded memory, abort() on error); at or under it a single put runs unchanged.

f5 — validation

validate::validate_graph_dir(graph_yml) (src/validate.rs:127) walks a written graph and returns Vec<ValidationIssue> (src/validate.rs:41; empty ⇒ conformant). Each issue carries a Severity::{Error,Warning} (src/validate.rs:32), a message, and the offending path. It is read-only and never touches the network. validate_graph_dir_with(.., ValidateOptions) (:132) relaxes checks via ValidateOptions (:91: extensionless_chunks, require_chunks; ::corpus() preset for the upstream extensionless corpus). Info-struct entry points: GraphInfo::validate(base) (src/validate.rs:571) and EdgeInfo::validate_dir(base) / VertexInfo::validate_dir(base) (:578,:585); the split checkers are validate_graph_info (:152), validate_vertex_info (:197), validate_edge_info (:247).

Rules checked: info-file schema — names/types/triples non-empty, version == gar/v1, chunk_size/src_chunk_size/dst_chunk_size > 0, ≥ 1 property group (vertex) / adjacency list (edge), valid adj-list type, each property in exactly one group, ≤ 1 primary property. Chunk layout — every declared group has its vertex/<label>/<group>/ dir with a gapless chunk0..chunkN run; each edge adjacency list has <adj>/adj_list/partK/chunkN, ordered lists have an <adj>/offset/ dir (unordered must not), edge property groups partition the same way. Partitions may be sparse (a vertex chunk with no incident edges has no partK); numbering within a partition must be gapless. Stray/extra files are warnings, not errors.