nordisk/znippy

znippy-zoomies — guide

Class: guide · audience: a techie decoding through, or building on, zoomies. The what is in zoomies-design.md; the why in zoomies-reasoning.md; the numbers in benchmarks.md. This file is the workflow and the worked code.

gatling flow

f0 — decode a stream, end to end (the whole workflow)

The zoomies rule you never break: hand the decoder a borrowed &[u8]; never copy the input. The path from bytes to output:

1. Read once into a slot (or mmap the file). Do not clone.
2. Split at safe boundaries — Z_FULL_FLUSH marks for DEFLATE, BWT blocks for bzip2.
3. Decode segments in parallel through linflate, each into its region of one buffer.
4. Stitch in order and set_len(total).

For a single file you call linflate directly; for a Read stream at scale you let gatling do steps 1–4 and you only supply a Codec + Sink.

linflate — decode into your own buffer

// Simple: one shot into a fresh Vec.
let out: Vec<u8> = linflate::inflate_to_vec(&compressed, expected_size)?;

// Zero-alloc: decode into a caller buffer. It MUST carry the headroom.
let mut buf = vec![0u8; expected_size + linflate::OVERWRITE_HEADROOM];
let n = linflate::inflate_into(&compressed, &mut buf)?;   // n == bytes written
buf.truncate(n);

OVERWRITE_HEADROOM is not optional slack — SIMD match-copy writes past the last real byte, so an output region that omits it returns InflateError::OutputOverflow (a clean error, never memory corruption). Every region you decode into — including each segment's slot — must include it.

Mid-stream segment (does not need to end on BFINAL=1):

// prefix = the already-decoded tail that back-references may reach into.
let n = linflate::inflate_segment_with_prefix(&segment_bytes, &prefix, &mut region)?;

the candidate-offset output model (B6) — parallel in one buffer

Use ljar::chunk::decode_chunk_into when you have many segments to inflate into a single output:

// One buffer, generous per-segment caps → candidate offsets (prefix sum of caps).
// All segments decode IN PLACE in parallel (rayon fork-join), then a forward-seek
// compaction slides each down to its true offset. actual <= cap ⇒ every move is
// leftward ⇒ nothing un-moved is clobbered. A segment that overflows its cap spills
// to an owned Vec and is rebuilt sequentially — output stays byte-identical.
ljar::chunk::decode_chunk_into(&segments, &mut output)?;   // ljar/src/chunk.rs:173

Rule of thumb: make caps generous (over-estimate). The cost of a too-small cap is a rare spill + sequential rebuild; the cost of a too-large cap is only transient address space in the one buffer.

gatling — stream a Read source through N workers

Byte mode: workers emit Vec<u8>, the collector concatenates:

use zoomies::gatling::{self, Config, SlotFill, Codec, Sink};

struct Inflate;                 // your transform
impl Codec for Inflate { /* fn transform(&self, seg: &[u8]) -> Vec<u8> { … } */ }

let cfg = Config {
    chunk_size:     64 << 20,
    carry_headroom: 1 << 20,    // room for a prepended carry across a split
    ring_slots:     n_workers + 2,
    initial_carry:  Vec::new(),
    slot_fill:      SlotFill::Big,   // planet-scale: size once, reuse, no re-zero
};
gatling::run(reader, Inflate, my_sink, cfg)?;

Typed mode when workers produce records, not bytes (no reassembly — the split pre-cuts logical segments and the collector just forwards each in order):

gatling::run_typed(reader, my_typed_codec, my_typed_sink, cfg)?;   // src/gatling.rs:99

SlotFill choice: Incremental for small/partial inputs (grows a slot to bytes read); Big for planet-scale (sizes each slot once to carry_headroom + chunk_size, reuses, never re-zeroes). Keep heavy work inside the worker — boundary-find, decode, parse, encode, sort all ride one unit with no barrier (the 1→32→1 principle). A reader or writer that does CPU work is your ceiling.

Async / I/O-bound (S3 multi-PUT and friends):

use zoomies::gatling::io;         // src/gatling_io.rs — tokio, bytes::Bytes moved never cloned
io::run_ordered(jobs, /* max_in_flight = */ 32).await?;

stree — id → position lookup

use zoomies::stree::STree64;

let tree = STree64::build(&sorted_keys);         // keys: sorted i64
let pos  = tree.find_exact(id);                  // serial, Option<usize>

// Batched + software-pipelined + prefetched — this is where the speedup lives:
let hits = tree.batch_stream::<16>(&query_ids);  // src/stree.rs:773

// mmap-backed store (leaf layer is the mmap):
let hits = mmap_tree.lookup_batch_pipeline::<16>(&query_ids, &mmap);   // :387

Build with RUSTFLAGS="-C target-cpu=native" — the AVX2 count_lt path roughly doubles batch throughput over the portable default. Call advise_hugepage on large trees for the RAM-bound regime.

running the benchmark suite

One suite, via nornir's Bencher trait — not cargo bench / criterion:

cargo run -p xtask --release --example nornir-bench      # prints one BenchRun JSON line
NORNIR_BENCH_SKIP_HEAVY=1  cargo run -p xtask --release --example nornir-bench   # skip heavy scales
NORNIR_BENCH_SKIP_PLANET=1 cargo run -p xtask --release --example nornir-bench   # skip planet scale

Results are historized in the Iceberg warehouse by git SHA, gated at >10% regression, and rendered into the README. See benchmarks.md for the current numbers.