nordisk/znippy

znippy-zoomies — design

Class: design · Function axis: f0 (zoomies as one system) + the parts: linflate core · the decoder family · the candidate-offset output model (B6) · the gatling engine · stree. Strict: this file states only the shape we run. Every why, every alternative, lives in zoomies-reasoning.md; the how-to in zoomies-guide.md; the measured numbers in benchmarks.md.

f0 — zoomies as one system (the whole)

znippy-zoomies is a sibling repo consumed as path deps: pure-Rust parallel decoders and fast primitives that znippy leans on. One invariant runs through everything: a decoder takes a borrowed &[u8] and never copies its input. That single rule is what lets a 164 GB planet file stream through on one read-copy — bytes are read into a slot once, sliced, decoded in place, and stitched, with no intermediate owning buffer.

The workspace is tokio-free except the iceberg crate (and the async gatling::io sibling). The pieces, and what each uses:

Consumed by: znippy proper (decompress path), and osm-katana (OSM→GeoParquet/PBF), whose xml→pbf path rides gatling and whose node store uses stree (re-exported as stree64).

linflate — the shared inflate core

linflate/ is a pure-Rust, full-buffer, SIMD DEFLATE decoder: the libdeflate full-buffer model (no streaming window) fused with zlib-ng branchless tricks. Hot working set ≈ 13 KB — it fits in L1-D. Files:

API (linflate/src/lib.rs):

inflate_segment decoding a slice that does not require BFINAL=1 is the linchpin of parallel decode: mid-stream segments — split at Z_FULL_FLUSH boundaries by deflate_scan — are each independently decodable.

the decoder family

Four format front-ends, one shared core. Each takes a borrowed &[u8]:

All four inflate through linflate; none owns a second DEFLATE implementation.

the candidate-offset output model (B6)

The zero-copy parallel decode layout. One pre-allocated buffer; the flow:

1. Candidate offset per segment = prefix sum of generous caps (not true sizes).
2. Decode all segments in parallel, in place, each at its candidate offset — zero per-segment allocation (ljar/src/chunk.rs:220, output.resize(acc, 0) pre-allocates the whole candidate region once).
3. Forward-seek compaction (chunk.rs:267): walk segments in order, moving each down to its true cumulative offset. Because actual ≤ cap, every move is leftward, so nothing not-yet-moved is ever clobbered. Then set_len(total).
4. Spill path (chunk.rs:258): a segment that outgrows its cap spills to an owned Vec (rare) → sequential rebuild. It never corrupts; output is byte-identical.

Two implementations of this model:

the gatling engine

src/gatling.rs — a generic, no-barrier streaming engine. A Read source is split into segments in a bounded zero-copy slot ring; N worker threads transform each segment with no cross-worker barrier; an ordered collector reassembles in stream order and hands to a Sink. Two modes:

Config (:156): chunk_size, carry_headroom, ring_slots, initial_carry, slot_fill (slot size = carry_headroom + chunk_size). SlotFill (:141): Incremental grows a slot to the bytes read (small/partial inputs); Big sizes once and reuses without re-zeroing (planet-scale).

Async sibling src/gatling_io.rs (gatling::io) — same invariants on tokio: run / run_ordered, bound by max_in_flight; payloads are bytes::Bytes moved, never cloned. For I/O-bound work (e.g. S3 multi-PUT).

The 1→32→1 principle: 1 reader only reads (a cheap O(n_workers) candidate-offset split, no CPU work); N workers carry a unit through all heavy stages — boundary-find, decode, parse, encode, sort — with no barrier and self-dispatch; 1 writer only stitches in order. Heavy work on a 1-end is the serial ceiling.

stree

src/stree.rs — a static search tree for id→position lookup under the OSM node store, re-exported into osm-katana as stree64. A cache-line S⁺-tree with branchless SIMD compare and prefetched batch traversal.

Derived from Ragnar Groot Koerkamp's static-search-tree (MIT, © 2025) — acknowledged in stree.rs:3. Our contribution at i64 width: B=16 (two cache lines) internal nodes.