nordisk/znippy

znippy index — design

Class: design · Function axis: f0 (the seekable footer) + f1 whole-index open · f2 locate_file resolver · f3 external SQL · f4 metadata browse. Strict: this file states only the shape we run. Every why, every alternative, lives in index-reasoning.md; the how-to is in index-guide.md. The engine that consumes these rows is in decompress-design.md.

f0 — the seekable footer (the whole)

znippy is a content-addressed, seekable store: O(key) per-file seeks, no full scan. That property lives in a self-describing footer written at finalize. Layout, in order:

[ blobs ]
[ data sub-index per (pkg_type,repo) ]
[ lookup sub-index  (base cols, re-sorted by relative_path,chunk_seq) ]
[ fst trie blob ]
[ manifest  (Arrow IPC) ]
[ 8-byte MAGIC "ZNPYMIDX" ]
[ u64 manifest_offset ]

The 16-byte footer (MAGIC + manifest_offset) is unchanged from the pre-seekable format, so old readers still open new archives. Base columns (schema znippy-common/src/index.rs):

Two derived structures ride in the ordinary manifest under reserved discriminants so old readers skip them: pkg_type = i8::MIN (RESERVED_PKG_TYPE, index.rs:312) with module_name = "__znippy_lookup__" (LOOKUP_MODULE, index.rs:293) and "__znippy_trie__" (TRIE_MODULE, index.rs:298). index.rs:594 read_znippy_manifest filters both out, so the data view is unaffected.

• LOOKUP sub-index — the base columns re-sorted by (relative_path, chunk_seq), plugin columns stripped: a second sorted copy. The main index keeps its original chunk_seq write order (Law 3 untouched). Sorting makes a file's chunks contiguous and lands paths byte-lexicographic (what fst needs).
• TRIE — fst::Map<relative_path → first lookup row>. Exact lookup is O(key length); prefix/range scans fall out for free.

Every reserved module_name is listed once in RESERVED_MODULES (index.rs), which is_reserved_module is the only reader of — so adding a reserved section is one edit, not two that can disagree. Beyond the lookup and the trie the catalog carries the two sign sections, __znippy_meta__, and the git package format's three: __gunnar_oid__ (raw stree over the first 8 bytes of each oid → its first lookup row), __gunnar_graph__ (the commit graph as Arrow) and __gunnar_reach__ (roaring reachability bitmaps). A writer adds its own through ArrowIpcSink::with_reserved_builder, which is handed the final sorted lookup at seal time — so an extra index addresses the rows the archive actually has — and which refuses a module name is_reserved_module does not know, because such a section would be merged into the data index and corrupt list, decompress and the iceberg sink.

Both are built once by meta_sink.rs:148 write_lookup_and_trie during finalize (append path: meta_sink_append.rs:245).

f1 — whole-index open (ZnippyArchive)

znippy-common/src/archive.rs:73 ZnippyArchive::open reads the whole index up front into a HashMap<String, FileEntry> — O(n) parse + memory per open, then O(1) per lookup. archive.rs:239 extract_file(relative_path) pulls a file from that map. This is the shape for one archive, many files: pay the parse once, amortize over many extractions.

f2 — the locate_file resolver

znippy-common/src/index.rs:736 locate_file(path, target) never materializes the whole index. It returns the target's contiguous ChunkLoc run (index.rs:342) via a three-tier fallback, fastest first:

Once the first row is found, it expands to the contiguous chunk run; get_file (decompress.rs:318) then preads each blob, decompress_into (or raw), blake3-verifies, places at fdata_offset. Use for one/few files ("Holger asks for a path").

Cold-read cost the design pays: a cold locate_file re-decodes the whole lookup, so per-call latency grows ~linearly with N. Acceptable for occasional seeks; a cached reader is future work (see index-reasoning.md).

f3 — external SQL (no znippy code)

The lookup sub-index is a valid Arrow IPC stream sorted by relative_path. Slice it out of the archive by the manifest's index_offset / index_len and hand the bytes to DuckDB / Polars / DataFusion:

SELECT blob_offset, blob_size, fdata_offset, compressed, uncompressed_size, checksum
FROM lookup
WHERE relative_path = 'org/foo/Bar.class'
ORDER BY chunk_seq;

Zero znippy code runs: the on-disk index is the query surface.

f4 — metadata browse (no bytes)

Two calls list files without reading any blob, returning ArtifactMeta { relative_path, uncompressed_size (summed across chunks), chunk_count, compressed } (index.rs:774):