nordisk/znippy

⚠ SUPERSEDED (2026-06-09). Frozen history (v0.4–v0.8 cycles). The live backlog is now in /plan.md. Do not add to this file.

Znippy Backlog

Done (v0.4.0)

• Streaming API (compress_stream, StreamCompressor, ArchiveEntry)
• Wire up stream_packer in lib.rs
• Integration tests (13 tests, full round-trip)
• OpenZL backend (replaced zstd)
• Single-file format (no .zdata, inline LargeBinary zdata column)
• Multi-repo support (repo column, nullable)
• Streaming batch writes (64MB flush threshold)
• Zero-copy batch builder (build_batch_zero_copy)
• Footer custom_metadata for checksums (reader merges transparently)
• Empty file handling (emit one empty compressed chunk)

Done (v0.5.0)

• Valid Arrow IPC Stream format (DuckDB/Polars/pyarrow can query directly)
• Direct buffer construction (bypass LargeBinaryBuilder copy)
• Removed custom hybrid format (trailer, magic, raw section)

Done (v0.7.0)

• Module-owned schema: ArchiveTypePlugin::schema_fields(), compose_index_schema, build_metadata_batch generic over ext columns (Step 1)
• WASM plugin: plugin_schema export (name:type pairs), plugin_extensions — fully documented in design.md §5
• Multi-index container format: manifest Arrow IPC + ZNPYMIDX magic footer (Step 2)
• ArchiveEntry gains pkg_type: Option<i8> + repo: Option<String> for per-entry grouping
• v0.7 writer (stream_packer + packer): always emits manifest; groups blobs by (pkg_type, repo)
• v0.7 reader: read_znippy_index detects magic, reads manifest, merges sub-indexes into one batch
• read_znippy_manifest for direct manifest inspection
• Dropped v0.6 read support (archives must be re-compressed)
• 20 integration tests (5 new v0.7: manifest roundtrip, footer detection, multi-index write/read, single-group v0.7)

Done (v0.8.0) — Iceberg support

• ArchiveMetaSink trait (znippy-common): sync abstraction over the archive metadata layer (sub-indexes + manifest + footer), keeping the blob/compression pipeline untouched
• ArrowIpcSink: default backend reproducing the v0.7 container format byte-identically (verified via structural fingerprint; slot_packer + stream_packer now write their metadata tail through the trait)
• znippy-iceberg crate: IcebergSink backend using the official iceberg crate (0.9.1) — writes a real on-disk Iceberg table (Parquet + metadata.json + Avro manifests) to a local-fs warehouse (MemoryCatalog + LocalFsStorageFactory)
• One namespace per archive, one table per (pkg_type, repo, module) sub-index
• Arrow 58 → IPC bytes → Arrow 57 bridge across the version gap; unsigned columns widened (UInt32→Int32, UInt64→Int64) since Iceberg has no unsigned types
• tokio/iceberg/Arrow-57/Parquet confined to znippy-iceberg — all existing crates stay tokio-free (sync trait bridged to async API via private runtime + block_on in finish())
• Round-trip test (write base index → read rows back via StaticTable)
• Follow-up: CLI --format arrow-ipc|iceberg + --warehouse flags — requires injecting Box<dyn ArchiveMetaSink> into compress_dir/compress_stream (can be done without adding tokio to znippy-compress); blobs stay in the .znippy sidecar, Iceberg table holds the index incl. blob_offset/blob_size
• Follow-up: IcebergZnippyReader implementing ZnippyReader for read-back

Arrow IPC Serialization Bottleneck — Technical Analysis

The numbers

Ratio: 3.6x slower due to Arrow IPC serialization copy.

Root cause

arrow-ipc-57.3.0/src/writer.rs line 2083:

fn write_buffer(arrow_data: &mut Vec<u8>, buffer: &Buffer, ...) {
    arrow_data.extend_from_slice(buffer);  // ← mandatory memcpy
}

Every buffer in every RecordBatch is copied into a contiguous Vec<u8> before being written to disk. No scatter-gather / write_vectored support exists.

What we tried (eliminated, made no difference)

1. LargeBinaryBuilder::append_value() → replaced with direct Buffer::from(vec) construction
2. Direct OffsetBuffer + Buffer assembly (avoids builder's internal realloc)
3. Result: 799 → 777 MB/s (noise). The IPC writer copy dominates.

Why direct NVMe write is 3.6x faster

Direct:  ring_buffer → write_all(data) → NVMe    (1 syscall, 0 userspace copies)
Arrow:   ring_buffer → Vec values → Buffer → extend_from_slice → write() → NVMe
                        copy #1              copy #2 (IPC writer)

Path to fix (upstream)

• Issue: apache/arrow-rs#9835 — "Support write_vectored / scatter-gather in IPC writer"
• We commented on the issue with our benchmark data (2025-05-21): 3.6x throughput loss, 777 vs 2775 MB/s
• Fix: IPC writer writes buffer slices directly instead of concatenating into Vec
• Our code is already structured for this — if upstream ships the fix, cargo update recovers full speed. Zero code changes needed on our side.

Decision: we keep Arrow IPC (v0.5)

• 777 MB/s is still fast (saturates 6 Gbit/s network, exceeds SATA SSD)
• DuckDB/Polars queryability is worth the tradeoff
• Upstream fix will eventually land — we get the speed back for free

TODO

Performance

P4: Small-file coalescing — pack multiple small files into one chunk before compression

Problem (observed in benchmark):

• rust_deps_real: 41k .rs/.toml files averaging ~24 KB → 18 MB/s, 54 seconds
• Root cause: each file is ~1 chunk, so the full reader→compressor→writer→tx_ret round-trip is paid per file. Channel overhead and thread synchronisation dominate useful work.
• 100k_small_files_10kb decompresses at only 627 MB/s vs 2000+ MB/s for large files — same per-chunk overhead on the read side.

Idea: Reader accumulates small files (below a threshold, e.g. file_split_block_size / 4) into a single revolver chunk, recording individual fdata_offset boundaries in the chunk metadata. A single chunk is sent to the compressor for the whole group. Decompressor splits it back using the stored offsets.

Trade-offs:

• Complicates chunk metadata (needs a sub-file offset table or sentinel records in the index)
• Compression ratio may improve (cross-file dictionary opportunities)
• Need to decide threshold — too large means latency spike waiting to fill a chunk

Relevant commit: e77d9a4 (zero-copy Blob pipeline — shows current per-chunk cost baseline)

P1: mixed_repo compress regression (-68% vs v0.3) ✅ FIXED

Solved by hybrid format (v2.1): raw data section + Arrow IPC Stream for metadata. Write path now does zero userspace copies — chunk bytes go directly to file. mixed_repo recovered from 901 → 2775 MB/s (v0.3 was 2850).

P2: decompress regression for incompressible data (-34%) ✅ FIXED

Solved by hybrid format: reader now seeks directly in raw section (sequential I/O). No Arrow IPC parsing overhead for data bytes. random decompress: 3125 MB/s (was 2110, +48% improvement).

P3: Investigate Arrow IPC compression (body-level)

Arrow IPC supports LZ4 Frame and ZSTD body compression (per-batch). This could reduce file size for the metadata columns without affecting zdata (which is already compressed).

Features

F1: Selective decompression by repo

znippy extract archive.znippy --repo libs-release --out ./libs/ Filter rows by repo column during decompression.

F2: Multi-batch streaming for very large archives

Current writer still buffers up to 64MB before flush. For extreme archives (>100GB), could reduce to smaller batches or adaptive threshold.

F3: Refactor compress_dir to use compress_stream internally

pub fn compress_dir(input: &Path, output: &Path, no_skip: bool) -> Result<CompressionReport> {
    let handle = compress_stream(output, no_skip)?;
    for entry in WalkDir::new(input)... {
        handle.sender().send(ArchiveEntry { relative_path, data: fs::read(path)? })?;
    }
    handle.finish()
}

F4: Plugin metadata on chunk_seq=0

Extension DenseUnion column currently always null. Wire up plugin metadata injection for the first chunk of each file.

F5: Self-contained package-handler modules (znippy as a thin host)

Vision: znippy core knows nothing about any specific package type. A "module" (= package handler, native or WASM) is fully self-contained:

• It declares which files it handles (plugin_extensions) — already implemented.
• It declares its own CLI commands and parses its own parameters.
• znippy's only responsibility is to enumerate the registered handlers and delegate.

znippy must never hardcode a plugin's parameters. Adding a new package type (maven, cargo, npm, docker, …) requires zero changes to znippy core — you register the module and it brings its full CLI surface with it.

Why: otherwise znippy would have to know every present and future plugin's flags (--plugin-type-id, resolver depth, etc.). That doesn't scale and couples the host to every handler. Self-description inverts it: the module owns its parameters.

Design sketch (extends the ArchiveTypePlugin interface):

• Interface gains a delegated CLI entrypoint, e.g. fn cli(&self, args: &[String]) -> Result<()> (default no-op). Native modules implement it directly; WASM modules export a matching plugin_cli(args_ptr, args_len) and parse the opaque arg blob themselves.
• znippy CLI: znippy <module> <args...> looks up the module by name in the registry and forwards everything after the module name verbatim. The registry's job is discovery + dispatch, not argument parsing.
• znippy modules (or list-handlers) enumerates registered package handlers — the only plugin-aware command znippy itself owns.
• Replaces the current --plugin <path> --plugin-type-id <n> flags, which leak plugin knowledge into the host. type_id and name become module self-description (exports), not CLI args.

Module owns its Arrow schema (currently violated): Today pkg_type + the 5 maven_* columns are hardcoded in ZNIPPY_INDEX_SCHEMA (znippy-common/src/index.rs) and filled by maven-specific code in build_metadata_batch. That's the host-knows-about-maven coupling to remove. Target:

• Trait gains fn schema_fields(&self) -> Vec<Field> (WASM: plugin_schema export).
• Writer composes the final schema = base(9) + ⋃ registered modules' fields.
• Each module fills only its own columns; build_metadata_batch has no per-type code.
• Column-name prefix convention is a hard rule (maven_*, cargo_*, python_*) so modules never collide. Schema is defined by the registered module set, so it's stable/predictable even though some columns are null for a given row.

Archive model: one .znippy is a multi-type container. A single archive holds multiple package types, each with multiple repos: { maven: [repo1, repo2], cargo: [repo1, repo2, repo3], pip: [...] }.

• pkg_type (Int8) discriminates which module owns a row → which *_ columns are populated.
• repo (Utf8) is the repo within that type. A "logical sub-archive" = rows sharing (pkg_type, repo).
• Selective extract by (type, repo) — ties into F1: znippy extract dump.znippy --type maven --repo repo1.

create/decompress are FORMAT-level (shared), not per-module — important. The container format (blobs + Arrow IPC index + footer) is universal, so modules must NOT each reimplement the compressor/decompressor. A module plugs into:

• write/create: matches_path + extract_metadata + schema_fields hooks the core pipeline calls.
• read/query: package-specific subcommands (znippy maven lookup --gav …) that translate to a DuckDB/DataFusion query over the Arrow-IPC file — NOT a hand-rolled index walk. The engine does execution; the module owns only the schema + the ergonomic command surface. Generic file extraction (decompress) stays type-agnostic in core.

CLI surface:

• znippy create -i … -o … / znippy extract … -o … — core, type-agnostic.
• znippy <module> <subop> … — module-provided query/ops, forwarded verbatim (module parses args).
• znippy modules — enumerate registered handlers + their subcommands (the only plugin-aware core cmd).

Open questions:

• WASM transport for an argv array (length-prefixed UTF-8 blob vs JSON).
• How a WASM module surfaces its sub-command help text to znippy <module> --help.
• Whether modules can register more than one command each.
• Schema = union of all registered modules vs only modules used in this archive (Arrow writes schema first, so "all registered" is simpler; nulls are cheap via the validity bitmap).

F6: Spatial archive — znippy as a geo-indexed coordinate store

Motivation: katana-osm needs to store ~6.8 billion node coordinates (id, lat, lon) and look them up by ID during way geometry resolution. Current approach is a 100+ GB flat file sorted by node ID, with Ragnar's STree for tree-routed mmap access. Problem: nodes referenced by the same way are spatially close but may be far apart in ID-sorted order → random 4K page faults across the entire file.

Idea: Use znippy as a spatial archive where each "file" is a geographic tile (S2 cell, geohash bucket, or Hilbert curve segment) stored as a compressed Arrow IPC chunk (columns: id, lat, lon). Ragnar's STree indexes node_id → tile, then znippy's per-file random decompression gives you just that tile's data.

spatial_nodes.znippy:
  ├── tile_0x3a1f.arrow   (compressed Arrow IPC — ~1M nodes)
  ├── tile_0x3a20.arrow
  ├── ...  (~4096 tiles for planet)
  └── [znippy Arrow IPC index with STree overlay]

Why this wins:

• Ways reference nodes in 1-3 adjacent tiles → decompress only those tiles
• Each tile (~1M nodes × 16B = 16 MB uncompressed) fits in L3 cache
• Delta-encoding of spatially-local coords compresses extremely well (maybe 4-8x)
• Znippy already supports per-file random decompression — no new format work
• Arrow IPC gives zero-copy columnar access within each tile once decompressed
• Total archive size maybe ~15-20 GB instead of 100 GB (with compression)

Two-level index:

1. STree (Ragnar's tree): node_id → tile_index (in-RAM, ~2 GB for planet)
2. Within tile: binary search or secondary STree on the id column

Simpler alternative — Hilbert-sorted flat file: Sort nodes by Hilbert curve of (lat, lon) instead of by ID. Then nodes that are spatially close ARE file-adjacent. Same flat file + STree architecture, but the mmap access pattern becomes nearly sequential for way lookups. No znippy compression, but massively better page-cache locality. Could be a stepping stone.

Open questions:

• Tile granularity: too few tiles = large decompressions; too many = index overhead
• Hot tiles (Western Europe, Japan) vs cold tiles (ocean) — adaptive sizing?
• Can we build the spatial index incrementally during pass 1, or need a second pass?
• Hilbert vs S2 vs geohash for the partitioning scheme
• Whether to cache recently-decompressed tiles in an LRU (likely yes, ~8-16 tiles)

F7: Transfer/resurrect VTD into znippy-zoomies as generic parallel XML parser

The VTD scanner (xml_vtd in katana-osm) finds XML element boundaries fast using memchr. Combined with gatling's worker pool, it gives parallel XML parsing with zero allocation for the index itself (streaming callback pattern). This is useful as a generic parallel XML parser module — not OSM-specific. The element-boundary finding + safe-split logic (find_safe_slot_end) is reusable for any large XML.

Transfer the core scanning primitives (build_elem_index_slice, find_safe_slot_end, attribute parsing) into znippy-zoomies as a vtd module. The OSM-specific record building stays in katana-osm; the generic XML scanner moves to znippy-zoomies alongside gatling (which already orchestrates the parallel pipeline around it).

Performance Baseline (v0.5.0 Arrow IPC, 8-core T14s laptop, release)

Note: mixed_repo limited by Arrow IPC serialization copy (apache/arrow-rs#9835). Direct NVMe write achieves 2775 MB/s for same workload.

Performance Baseline (v0.6.0, zero-copy Blob pipeline, 8-core T14s laptop, release)

Key changes vs v0.5.0:

• v0.6 streaming format: blobs first, Arrow index last, 8-byte footer
• Zero-copy Blob pipeline: eliminated Arc::from memcpy on both skip and compress paths
• random_500mb regressed vs v0.5 (was 152 MB/s) — incompressibility detection not yet implemented (P3/backlog)