nordisk/znippy

Searchable archive metadata — the __znippy_meta__ sub-index

Status: implemented (2026-07-22, oden) Question it answers: which entries in this archive carry key X, and what are their values — read from an index, never by streaming the payload. First consumer: dwarves' build thing (a wasm module describing how to build an app, embedded in artifacts and pulled from a registry). Finding one inside an archive without extracting the archive is the same problem as reading it from an ELF section without executing the binary.

Why this is a sub-index and not a new format

A sealed v0.7 .znippy already ends:

[ blob_0 … blob_N ][ data sub-idx(es) ][ lookup ][ trie ][ manifest ][ ZNPYMIDX ][off]

The 16-byte footer gives the manifest offset with one seek; the manifest frames every section by (offset, len). A metadata region that is independently readable at O(index) cost is therefore already a solved problem in this format — it only needed a module. META_MODULE = "__znippy_meta__" is registered exactly like LOOKUP_MODULE and TRIE_MODULE, written through the existing ArchiveMetaSink::push_subindex, and read back through the existing reserved- section reader.

There is deliberately no second directory. The metadata rows key on relative_path — the same identity the data sub-indexes and the lookup already use — and a hit is resolved to bytes through the existing locate_file / ArchiveReader. Nothing here duplicates the index, so nothing here can drift out of agreement with it. (This was an explicit constraint: twinned sources of truth bit this constellation three times in one session.)

On-disk shape

Arrow IPC, one RecordBatch, schema metadata carrying znippy_format_version (so a section from a newer znippy is refused rather than mis-parsed — the same guard every other sub-index gets).

Columnar rather than a serialized blob so the section stays directly queryable by DuckDB/Polars — SELECT relative_path FROM … WHERE key = 'build-thing' — which is the same property the rest of znippy's index layer sells.

Rows are stored sorted by (key, relative_path). Key-major, so find_by_key and find_by_prefix are partition_point ranges (O(log n) + O(hits)), not scans, and archive-level rows for a key sort ahead of its entry-level rows (NULL first).

value_type and the value columns are cross-checked on read: a row that declares a string with a NULL string column, or an unknown tag, is an error. Nothing defaults. The section is attacker-reachable input.

API

// write
let mut t = MetaTable::new();
t.insert("app/main.wasm", "build-thing", MetaValue::Bytes(wasm))
 .insert("app/main.wasm", "build-thing.abi", "wasi-p2")
 .insert_archive("producer", "znippy");

create_archive_with_meta(path, &files, 3, Some(t))?;      // fresh
append_files_with_meta(path, &more, 3, Some(extra))?;     // merge into existing
ArrowIpcSink::new(f, b).with_meta(t)                      // compress path, via MetaSinkFactory

// read
match read_archive_meta(path)? {
    ArchiveMeta::NoMetadata => …,                 // nothing was searched
    ArchiveMeta::Index(ix) => {
        ix.find_by_key("build-thing");            // &[MetaEntry] — entry refs
        ix.find_by_prefix("build-thing");         // sweeps build-thing.abi too
        ix.archive_value("producer");
        ix.keys(); ix.iter(); ix.len();
    }
}
// or, keeping the distinction at the call site:
meta.find_by_key("build-thing").hits()            // Option<&[MetaEntry]>

Queries return entry refs, so the caller reads the index, decides, and only then extracts the one entry it wants:

for hit in meta.find_by_key("build-thing").hits().unwrap_or(&[]) {
    let bytes = ArchiveReader::open(path)?.read_file(hit.path().unwrap())?;
}

CLI

znippy append -i a.znippy -a dir \
      --meta 'app/main.wasm=build-thing=@build.wasm' \
      --meta 'app/main.wasm=build-thing.abi=wasi-p2' \
      --meta-archive 'producer=znippy'

znippy meta -i a.znippy -k build-thing          # exit 0 — index exists, hits
znippy meta -i a.znippy -p build-thing --paths-only
znippy meta -i a.znippy -k absent               # exit 1 — searched, no match
znippy meta -i old.znippy -k anything           # exit 2 — NO index; nothing searched

Values are typed by shape, narrowly and predictably: @file → raw bytes, true/false → bool, integer → i64, decimal → f64, anything else → string (so 1.0.2 stays a string rather than half-parsing).

NoMetadata is not an empty index

This is the part with a scar on it. An archive written before this module has no __znippy_meta__ entry; a reader that turns that into an empty result set will answer "searched, found nothing" about an archive it never searched.

So:

• read_archive_meta returns ArchiveMeta::{NoMetadata, Index(_)} — not Option<MetaIndex>, and with no Default, so there is no unwrap_or_default() that quietly collapses them.
• ArchiveMeta::find_by_key returns MetaSearch::{NoMetadata, Hits(&[..])}, which carries the distinction all the way to the call site. Hits(&[]) means searched, found nothing; NoMetadata means there was nothing to search. MetaSearch::hits() is Option<&[MetaEntry]> — naming the absent case is the point.
• On the write side the same distinction is a real on-disk difference: create_archive_with_meta(.., None) seals no section (byte-identical to create_archive, and what every pre-existing archive looks like), while Some(MetaTable::new()) seals a present, empty section.
• The CLI carries it into exit codes: 2 no index · 1 searched, no match · 0 hits.

The re-seal trap, closed

ArrowIpcSinkAppend::open_existing truncates the entire metadata tail and rebuilds it. Without carrying the metadata section forward, every append would silently erase it — and the reader would then honestly report NoMetadata for an archive that used to have some, which is the worst possible version of this bug because it looks correct. open_existing therefore recovers the section into the sink, and a plain append_files (which says nothing about metadata) preserves it. Asserted in meta_sink_append::tests::metadata_is_searchable_survives_a_reseal_and_absence_is_reported_as_absence.

One more trap, closed

push_subindex accumulates every pushed batch into the random-access lookup. Its guard was != LOOKUP && != TRIE; the metadata sub-index is Arrow IPC and does go through push_subindex, so its key/value rows would have been folded into the lookup as bogus chunk locations. The guard is now !is_reserved_module(..). The test asserts the lookup and the data index still contain exactly the file rows.

Cost — measured, not asserted

Two independent forms of evidence, because a timing bench on a shared box can be contaminated and a structural fact cannot.

Structural (always-on test). The bytes a search reads are the footer, the manifest and the metadata section. a_search_reads_only_the_index_and_its_size_is_independent_of_the_payload builds two archives with identical metadata and payloads 128× apart, and asserts those byte counts are equal: 129 952 B on a 0.6 MB archive and 129 952 B on a 13.7 MB one — 0.95 % of the larger archive. This runs in the ordinary suite, so a regression that made the search touch the payload fails CI.

Timing (--ignored benches). Metadata held constant at 512 keyed entries (1538 rows); only the payload grows. The verify all column is the control — it reads and hashes every chunk, so it is O(payload) by construction, and the bench refuses to call a flat search number evidence unless the control actually moved. (An earlier draft used single-entry extraction as the control; the numbers showed it stays flat too, because znippy's random access is also O(1) in payload. It is now reported but not used as the control.)

oden · 32 cores — non-heavy

payload 82.4× · search 0.81× · control 34.9×

oden · 32 cores — HEAVY

payload 57.1× · search 0.92× · control 44.9×

Reported figure is the minimum of 25 repeats — the least contaminated estimator on a box with other agents on it. New host row (oden, 32 physical cores, SMT off, 499 GB). Never merge with the loki · 12 cores tables.

Run them:

cargo test -p znippy-tests --release --test meta_search_bench -- --ignored --nocapture
ZNIPPY_BENCH_HEAVY=1 cargo test -p znippy-tests --release --test meta_search_bench \
    -- --ignored --nocapture meta_search_heavy

Not done

• The compress path does not seal metadata by itself. compress_dir takes a MetaSinkFactory, and ArrowIpcSink::with_meta is the injection point, so a caller can already do it with no change to compress_dir's signature — but no CLI flag on compress wires it, because there is no per-file metadata source for a directory walk yet. The obvious next step is a plugin hook: a ZnippyPlugin that returns MetaEntrys for the files it claims, which is how a maven/python/rust handler would publish an artifact's build thing without the caller enumerating it.
• No index over value. Search is by key (exact or prefix). "Which entries have build-thing.abi == wasi-p2" is a filter over the key's hits, which is fine at these row counts and would need a second sort order to be better.
• The section is not deduplicated. Two entries carrying the same 4 MB wasm store it twice. If build things get large, the value should become a relative_path reference into the archive's own blobs rather than inline bytes — the format already supports that, and the row model would carry a MetaValue::Str(path) under a *.ref key. Not needed yet; noted so it is a decision rather than an oversight.