nordisk/znippy

znippy index — guide

Class: guide · audience: a techie querying a .znippy index for locations or metadata. The what is in index-design.md; the why in index-reasoning.md. This file is the workflow.

f0 — pick your access shape (the whole)

There are three ways to get a row and two ways to browse metadata. Choose by how many files you touch and whether you want to run znippy code at all:

Every path is O(key)-seekable because the footer carries a sorted lookup sub-index plus an fst trie — no full scan needed for a single file.

f1 — many files from one archive

use std::path::Path;
use znippy_common::archive::{ZnippyArchive, ArchiveReader};

let arc = ZnippyArchive::open(Path::new("archive.znippy"))?;  // reads whole index once
let a = arc.extract_file("org/foo/Bar.class")?;               // O(1) from the HashMap
let b = arc.extract_file("org/foo/Baz.class")?;               // amortized — no re-parse

Pay the index parse once, then every extract_file is a hash hit. Use this when you'll pull a batch of files from the same archive.

f2 — one file, fastest resolver

use std::path::Path;
use znippy_common::{get_file, index::locate_file};

// full read (locate + pread + verify + reassemble):
let bytes = get_file(Path::new("archive.znippy"), "org/foo/Bar.class")?;

// just the locations (a contiguous ChunkLoc run), no bytes read:
let locs = locate_file(Path::new("archive.znippy"), "org/foo/Bar.class")?;

locate_file tries the trie first (O(key)), falls back to binary search of the sorted lookup (O(log n)), then a linear index scan for legacy archives. Never materializes the whole index. Note: a cold locate_file re-decodes the lookup, so back-to-back single seeks on a huge archive pay ~O(N) each — batch through f1 if you're pulling many.

f3 — query the index with DuckDB (no znippy code)

The lookup sub-index is a plain Arrow IPC stream sorted by relative_path. Slice it by the manifest's index_offset/index_len, then point any Arrow engine at it:

-- lookup.arrow = the bytes you sliced out of the archive footer
SELECT blob_offset, blob_size, fdata_offset, uncompressed_size, compressed, checksum
FROM read_ipc('lookup.arrow')
WHERE relative_path = 'org/foo/Bar.class'
ORDER BY chunk_seq;

You now have every chunk's location without linking znippy — pread those blob_offsets yourself, or just inspect the layout. Works in DuckDB, Polars, and DataFusion.

f4 — browse metadata without reading bytes

use std::path::Path;
use znippy_common::index::{get_all_files_meta, get_files_meta_with_prefix};

// every file, one slim sorted row each:
for m in get_all_files_meta(Path::new("archive.znippy"))? {
    println!("{}  {} bytes  {} chunks", m.relative_path, m.uncompressed_size, m.chunk_count);
}

// just one subtree (trie range-seek, O(prefix len)):
let sub = get_files_meta_with_prefix(Path::new("archive.znippy"), "org/foo/")?;

ArtifactMeta carries relative_path, uncompressed_size (summed across the file's chunks), chunk_count, and compressed. No blob is read — this is the cheap way to list or size an archive's contents.