nordisk/knut

GraphAr gar/v1 substrate — guide

Class: guide · audience: a techie reading, writing, or validating a GraphAr graph. The what is in graphar-design.md; the why in graphar-reasoning.md. This file is the API and worked examples.

f0 — a graph in one round trip (the whole workflow)

Four moves take Arrow batches to a validated gar/v1 directory and back:

1. Write chunks. io::write_chunk(path, &batches, &FileType::Parquet) — pick any of the five formats; the Arrow schema is preserved either way.
2. Read them back. io::read_chunk(path, &FileType::Parquet) → the same Vec<RecordBatch>.
3. Build a real graph. VerticesBuilder/EdgesBuilder accumulate typed rows and dump() the whole chunk layout; VertexReader/EdgeReader stream it back.
4. Validate. validate::validate_graph_dir(graph_yml) — empty Vec means gar/v1-conformant.

Put a graph on S3/GCS/Azure by swapping the file calls for a ChunkStore; everything else is identical.

f1 — chunk io (the raw seam)

use graphar::{io, types::FileType};

// Write one chunk file (parent dirs created for you):
io::write_chunk("vertex/person/id/chunk0.parquet", &batches, &FileType::Parquet)?;
// Read it back to the same Arrow batches:
let batches = io::read_chunk("vertex/person/id/chunk0.parquet", &FileType::Parquet)?;

FileType is Parquet | Orc | Csv | Json | ArrowIpc; FileType::extension() gives the file extension. Same schema in, same schema out — format is a storage decision, not a data one.

f2 — builders & readers

use graphar::{VerticesBuilder, EdgesBuilder, Vertex, Edge, PropertyValue, VertexReader};

// Vertices: accumulate, then dump the chunk layout described by `info`.
let mut vb = VerticesBuilder::new(vertex_info, "graph_dir");
vb.add_vertex(Vertex::new().add_property("name", PropertyValue::from("Ada")));
let n = vb.dump()?;                 // -> u64 vertex count; each row's index is its _gar_id

// Edges reference endpoints by _gar_id:
let mut eb = EdgesBuilder::new(edge_info, "graph_dir", /* ... */);
eb.add_edge(Edge::new(0, 1).add_property("since", PropertyValue::from(2020)));
eb.dump()?;

// Read back — read_merged() row-aligns every property group into one batch:
let vr = VertexReader::new(vertex_info, "graph_dir", n);
let batch = vr.read_merged()?;

Standalone row-aligned join (when you already hold group batches): merge_property_groups(&[group_a_batches, group_b_batches]) -> RecordBatch.

f3 — ChunkStore over object storage

use graphar::storage::{ChunkStore, RetryOptions, MultipartOptions};
use graphar::types::FileType;
use std::time::Duration;

// Local, S3/GCS/Azure by URL, or S3-compatible (SeaweedFS/RustFS/MinIO):
let store = ChunkStore::from_url("s3://my-bucket/graphs/social")?;      // default retry policy

// Explicit retry/backoff, and a tuned multipart policy:
let store = ChunkStore::from_url_with_retry(
        "s3://my-bucket/graphs/social",
        RetryOptions::new(5, Duration::from_millis(200), Duration::from_secs(10)),
    )?
    .with_multipart(MultipartOptions::new(8 << 20, 5 << 20)); // 8 MiB threshold, 5 MiB parts

// Async chunk ops, keyed by a relative path under the store prefix:
store.write_chunk("vertex/person/id/chunk0.parquet", &batches, &FileType::Parquet).await?;
let batches = store.read_chunk("vertex/person/id/chunk0.parquet", &FileType::Parquet).await?;

RetryOptions::none() disables retries; RetryOptions::default() reproduces the library's policy (10 retries, 100 ms→15 s, 3-min budget). Writes larger than MultipartOptions::threshold (default 16 MiB) automatically stream via put_multipart; smaller ones use a single put.

f4 — validation

use graphar::validate::{validate_graph_dir, validate_graph_dir_with, ValidateOptions, Severity};

let issues = validate_graph_dir("graph_dir/social.graph.yml");
assert!(issues.iter().all(|i| i.severity != Severity::Error));  // errors = non-conformant

// Upstream corpus (extensionless chunks, optional chunk dirs):
let issues = validate_graph_dir_with("graph_dir/social.graph.yml", ValidateOptions::corpus());

Or validate from already-loaded info structs: GraphInfo::validate(base), VertexInfo::validate_dir(base), EdgeInfo::validate_dir(base). Validation is read-only and offline — it walks the directory and reports every deviation from gar/v1; an empty Vec<ValidationIssue> (or a list with only warnings) means the graph is conformant.

Surfaces (Rust API)