nordisk/knut

graphar chunk codec — roundtrip coverage + light baseline

The through-line hot path of the graph warehouse is the chunk codec: write_chunk_bytes(&[RecordBatch], FileType) -> Vec<u8> and its inverse read_chunk_bytes(Bytes, FileType) -> Vec<RecordBatch> (crates/graphar/src/storage.rs) — every GraphAr vertex/edge chunk goes through it, for all five FileTypes (Parquet · ArrowIpc · Csv · Json · Orc).

What landed

crates/graphar/tests/chunk_roundtrip.rs — pure, in-memory, no network/container:

• Correctness (all 5 formats): write → read preserves row + column count. Drives the codec IN CODE (AI-free at runtime) and emits a chunk_roundtrip functional-status cell per format via graphar's nornir-testmatrix wrapper (--features testmatrix).
• Light timing smoke (NOT a heavy bench — heavy runs stay on the quiet bench box): a 100-iter write+read loop over the binary through-line formats.

Baseline (this box, 1000-row batch, write+read/iter — light, not the record)

Reading it

ArrowIpc is ~17× cheaper than Parquet for a small-chunk roundtrip (Parquet pays column-encode + footer/statistics cost). Implication for the "loop till times down" work: for hot, small, ephemeral chunks (cache/shuffle/intermediate), prefer ArrowIpc; keep Parquet for the durable columnar store where its compression/pushdown earns back the encode cost. Next bench step (bench box): sweep batch size + column mix, and measure the parquet write props (compression/dictionary) knobs against read pushdown.