nordisk/znippy

plan — znippy

The planning spine: the single source of truth for what's next, the locked decisions behind what is, and the working order. The docs describe what exists (.nornir/design.md = the map; .nornir/{benching,random-access}.md = the territory); this file is the live worklist. Supersedes the frozen legacy/TODO_NOW.md + legacy/backlog.md (kept only as history — do not add to them).

Convention: this is the nornir plan.md "spine" (see nornir/.nornir/docs-generation.md §Doc taxonomy). One source per fact — decisions live here; the docs link to them.

Laws (locked)

1. Zero-copy channel communication. Inter-thread messages are slot indices / ChunkMeta, never payloads. (Design law 1.)
2. No allocation in the hot path. Reused compress/decompress buffers, long-lived codec context. (Design law 2.)
3. Deterministic checksums independent of core count. blake3 bound to chunk_seq order, over the original (pre-compression) bytes. (Design law 3.)
4. Writer is concurrent; compressors never wait on the index. Workers pwrite their own payload at an AtomicU64 cursor; the finalizer only builds the Arrow index from ChunkMeta. (Design law 4.)
5. No-barrier Gatling pipeline — ever. One global slice queue, no slot-completion barrier (the lbzip2-v1 bug). Decided 2026-05-24.
6. Benchmark-first for any performance change. Baseline the OLD code, change it, re-run the SAME bench, compare on the same machine; proceed only if not slower (>10% same-machine = release-gate fail). The bench suite is nornir's (xtask/examples/nornir-bench.rs), not criterion — see .nornir/benching.md. Applies squarely to all of #7.
7. Public/private hygiene. znippy is public; nornir is private. Never copy nornir's internal docs into znippy/.nornir. znippy's own docs follow nornir's doc conventions but carry only znippy content.

Done (0.9.x)

• #7 B6 — ljar zero-copy chunk output (ljar::chunk::decode_chunk_into). The lbzip2 pattern done right: one pre-allocated buffer, candidate offsets per segment, all segments decoded in parallel in place (no per-segment Vec), then forward-seek compaction to the true cumulative offsets — caller reuses the buffer across chunks. Single-segment hits a true zero-copy fast path; a slot overflow (rare) falls back to owned buffers, never corrupts. Test decode_chunk_into_matches_decode_chunk; A/B bench benches/chunk_into.rs (decode_chunk_flatten vs decode_chunk_into_reused).

• Compress uses the gatling no-barrier ordered-sink (gatling::ordered::run_ordered_sink, off the old per-thread ChunkRevolver; int_ring deleted). Decompress is a different design: positioned pread (FileExt::read_exact_at) driven by a shared atomic row cursor over thread::spawn — no gatling, no slots, no channel. Each worker grabs the next row via the lock-free cursor, preads its blob, decompresses/verifies, and pwrites to fdata_offset (znippy-common/src/decompress.rs).

• Random-access reads — sorted lookup sub-index + fst trie (.nornir/random-access.md).

• Iceberg metadata sink — ArchiveMetaSink + ArrowIpcSink (byte-identical default) + znippy-iceberg::IcebergSink.

• Store-if-incompressible (commit 8bca9aa) — write path stores raw when the codec frame ≥ input; never inflates a blob, skips the decode cost. Test test_incompressible_data_stored_raw; bench metrics raw_chunks / compressed_chunks.

• nornir bench suite + warehouse + workspace_znippy/ control plane (registered, monitored). Bench cores/file-count metrics; region geo benches (geo_liechtenstein always-on, geo_europe skip-until-present).

• osm-katana region-clip + synthetic generator (item B / J-perf) — see .nornir/region-clip.md. bbox-at-decode, None-free, vtd_bench-gated: clip is one branchless point-in-box test in reader::parse_nodes before tag-JSON, behind the default-OFF region-clip feature (lean build byte-identical). Presets europe/nordics, CLI --clip, lib extract_region. write_synthetic_geoparquet drives the real NodeWriter. Inject-assert tests (synthetic round-trip + liechtenstein None/europe/tight/empty

  • KATANA_FIXTURE opt-in real-data arm). vtd_bench decode arm proves the None-path matches the no-branch baseline (≈62.7 ms vs 64.36 ms, 353 167 nodes, 3975WX/32c) — no regression; tight box is faster (skips 91 % tag-JSON).

• gatling::io async-sibling circuit SVG (J2-old) — assets/gatling_io_flow.svg in the house style (dark gradient, cyan/ice, monospace, left→right), embedded in gatling.md alongside the sync gatling_flow.svg.

Open work

Priority order. Each perf item is gated by Law 6 (baseline first).

Bounded / ready

• #5b — IcebergZnippyReader: ZnippyReader (the remaining half of #5). ✅ UNBLOCKED — skade shipped 2026-06-09 (skade/skade, 0.1.0) and verified against znippy's wishlist (nornir/znippy_agents_whishes.md §4): every item delivered — skade::open(dir) one-call warehouse, Table::{append,ingest,read,count,sql}, bridges incl. unwiden with bit-reinterpret for u64 > i64::MAX (exact blob_offset round-trip), windowed parquet reads, SQL via datafusion 52.2 (shares arrow 57 — no bridging). 8/8 skade tests green locally. Implementation now bounded:

  1. znippy-iceberg: swap MemoryCatalogBuilder → skade::open(warehouse) in IcebergSink (durable catalog.redb, fixes the ephemeral-catalog flaw); write sub-indexes via Table::append.
  2. IcebergZnippyReader: skade::open → table(name).read() → skade::unwiden(batches, znippy_schema) → file→chunks map → pread + codec::decompress_into from the .znippy sidecar.
  3. Roundtrip test: compress --meta-format iceberg → read back → byte-compare. Gate CLEARED 2026-06-09: skade 0.1.0 published to crates.io (by us, user-authorized; dry-run verified against stock deps). znippy-iceberg can depend on skade = { version = "0.1.0", path = "../../skade/skade" } — wait, skade lives OUTSIDE the znippy workspace (sibling repo), so use version-only skade = "0.1.0" in znippy-iceberg. Implementation may start. Original analysis (kept for context):

  • Root problem with the current sink: IcebergSink::finish uses MemoryCatalogBuilder — the catalog (table pointers) is in-process and dies with the process; only warehouse files survive. Read-back would mean crawling metadata/*.metadata.json + StaticTable and guessing which metadata file is current. Fragile by construction.

  • The fix: swap MemoryCatalog → skade_katalog::RedbCatalog (crates.io 0.1.x, pure-Rust, redb-backed, implements iceberg::Catalog). A catalog.redb beside the warehouse becomes the durable catalog. Versions align exactly (iceberg 0.9.1, arrow 57.x both sides); dependency direction is acyclic (znippy-iceberg → skade-katalog → znippy-zoomies).

  • Write side bonus: RedbCatalog::atomic_release = multi-table atomic commit — znippy writes one table per (pkg_type, repo) sub-index, so a whole archive's index can flip in ONE transaction (today each table commits separately).

  • Read side becomes trivial (the skade-katalog pattern): RedbCatalogBuilder::default().db_path(…).warehouse_location(…).load(…) → catalog.load_table(&ident) → table.scan().select_all().build()?.to_arrow() → RecordBatch stream → un-widen the sink's signed cast (Int32→UInt32, Int64→UInt64; don't reuse ZnippyArchive::build_file_index, it expects UInt64) → file→chunks map → pread + codec::decompress_into from the .znippy blob sidecar. Roundtrip test: compress --meta-format iceberg → read back → byte-compare.

  • SQL for free: iceberg_datafusion::IcebergCatalogProvider::try_new(catalog)

    • DataFusion SessionContext → SELECT … FROM nornir.<ns>.<subindex> — proven in skade-katalog's datafusion_sql.rs (they vendor a patched iceberg-datafusion 0.9.1 with a parallel per-file partitioned scan). Gives znippy SQL over an iceberg-format archive index without DuckDB. Optional feature, after the reader.

  • F6 cross-pollination: skade-katalog's parquet_layout + rowgroup_windows + read_row_groups (bounded-memory windowed row-group ingest, parallel across threads) is exactly the shape the spatial archive (F6) needs for billion-row coordinate ingest.

Large / standalone (still benchmark-first)

• #7 — ljar decode perf.

  • B6 — ✅ done, library and binary (see Done). Library: decode_chunk_into. Binary (wired 2026-06-09): the candidate-offset pattern woven into the pread + shared atomic row cursor decompress workers (thread::spawn, no gatling/slots/channel) — per chunk ONE pooled output buffer with per-segment candidate offsets (exact cap when the entry's uncompressed size is known; ratio-hint cap seg_cap() otherwise), each worker inflate_segments straight into its region (overflow → owned spill, rare), forward-seek compacts in place, then pwrites straight to fdata_offset and reuses the buffer for its next cursor row. Zero per-segment allocation in steady state. A/B (law 6), 163 840-entry 10 GB corpus, best-of-5: 7.12 s → 6.70 s (~6 %), variance 7.1–11.3 → 6.7–7.4 s. Output byte-identical. The library decompress_jar path was already pre-sized via linflate::inflate_to_vec(uncompressed_size).
  • B7 — ✅ was already shipped as linflate (discovered 2026-06-09; this backlog entry was stale). The custom SIMD DEFLATE decoder exists as the linflate crate — all five design modules (bitreader/tables/fixed/ copy/fastloop per linflate/DESIGN.md), shared by ljar+lgz+lzip-parallel. The leftover pre-extraction fork ljar/src/inflate/ (diverged; still carried the refill_slow corruption bug) was deleted; ljar::inflate is now a re-export of linflate, so the criterion inflate benches measure the live core. New nornir scorecard: linflate_guava/linflate_scala (vs miniz_oxide). Chapter: .nornir/inflate.md. #7 is closed.
  • BUG — FIXED 2026-06-09 — linflate refill_slow looped on bits <= 56, so entering with exactly 56 bits (what a fast refill leaves) pushed bits to 64: invariant breach (debug assert), and in release the next fast refill's << bits masked to << 0 on x86 → 8 raw bytes OR'd over the whole register, silent corruption. Fix: loop bound < 56 (max exit 63, ≥56 post-condition kept). Regression test refill_slow_at_56_bits_keeps_invariant; ljar batch_timing green again; lgz + lzip-parallel suites pass.

• F6 — spatial archive. znippy as a geo-indexed coordinate store for osm-katana (tiled coords, per-tile random decompress). Roadmap / research.

• F7 — VTD → zoomies. Move the generic parallel-XML scanner out of osm-katana into znippy-zoomies. Roadmap.

Blocked on upstream

• Arrow IPC scatter-gather write (apache/arrow-rs#9835). The metadata-write copy costs ~3.6× on metadata-heavy archives. Our code is already structured for it; cargo update recovers the speed when it lands. No action.

Working order / release gate

Publish bottom-up per CLAUDE.md §Release procedure; gates per workspace_znippy/release/SPEC.md (bench → no >10% same-machine regression → history append → docs fresh → tag → publish). The README's <!-- nornir:gen --> tables regenerate from the warehouse — never hand-edited.

Session checkpoint 2026-06-10 — in-flight work LANDED + RELEASED

RELEASED 2026-06-10 (master 5c017bb): znippy-zoomies 0.1.9 (xml+json scanners, additive) and znippy-iceberg 0.1.1 (#5b reader + durable skade catalog, additive — znippy-cli's ^0.1 dep unaffected) published to crates.io; tags zoomies-v0.1.9 + iceberg-v0.1.1 pushed to codeberg. Other workspace crates unchanged at 0.9.1 (not republished). Sibling releases the same day: skade-katalog 0.1.4, nornir 0.4.4. Full bench-gate/docs-render (SPEC gates 2–5) NOT run — these are additive feature/scanner releases with no perf change; the bench harness (xtask→nornir dev-dep) is unaffected. Original pre-release checkpoint below.

Session checkpoint 2026-06-10 — in-flight work LANDED, release pending

All three in-flight streams from the 2026-06-09 checkpoint are finished, green, and on master (5983058 merge + fix(xtask) doctest on top):

• F7a — generic XML scanner. znippy_zoomies::xml extracted from vtd; vtd is now the OSM specialization on top (public API preserved; osm-katana builds+tests green). pub mod xml + benches/xml_scan.rs wired. Fixed the doc example (TagSet::new needed &[u8] slice coercion). Commit 5a3fe82.
• F7b — parallel NDJSON scanner. znippy_zoomies::json (the worktree commit 24db989) integrated by copying json.rs + json_scan.rs and wiring pub mod json + serde_json dev-dep + bench. 14 json tests green. Commit 66fa30e. Agent worktree retired.
• #5b — IcebergZnippyReader. Built on the feat/iceberg-reader branch (was znippy-5b/ worktree), merged no-ff into master. skade 0.1.0 version-only dep, default-features = false. Roundtrip test green. Commit c538252, merge 5983058. Worktree retired.

Cargo.lock note (resolved): the big crypto/server tree (russh/tonic/axum/ quinn/aws-lc) is legit — it is nornir's transitive tree, pulled as a dev dep of xtask/ljar/lzip-parallel (the bench harness). Not in any published crate's runtime closure. The 2026-06-09 "investigate origin" caution is closed.

Full cargo test --workspace is GREEN (lib + integration + doctests). Only remaining work: the release itself (bench-gate → version bump → tag → cargo publish bottom-up → push to codeberg). Release order per CLAUDE.md §Release procedure / WORKSPACE.md.

Open question for the human (auto-authorized "YES on all" this session): versions chosen by the agent — see the release checkpoint appended at run time.

Session checkpoint 2026-06-09 (late) — in-flight work

Two background agents were launched on F7 and may have left UNCOMMITTED work (they were instructed not to commit; review before committing):

• F7a — generic XML scanner. Working in the MAIN tree (/run/media/rickard/T9/git/znippy). Goal: extract tag-generic znippy-zoomies/src/xml.rs out of vtd.rs (vtd = OSM specialization on top; osm-katana's xml_vtd::* paths must keep working), + criterion bench benches/xml_scan.rs, benchmark-first A/B (no regression vs current vtd). Check git status in the main tree for its output; verify cargo test -p znippy-zoomies -p osm-katana before committing.
• F7b — parallel NDJSON scanner. Working in an ISOLATED WORKTREE (check git worktree list). Goal: znippy-zoomies/src/json.rs mirroring the vtd pattern — RFC 8259 raw-\n-illegal-in-strings ⇒ memchr newline = safe split; zero-copy RecordSpan callback API; string-aware find_key (find_attr analog); criterion bench benches/json_scan.rs vs serde_json per-line baseline (serde_json dev-dep only). Single-array JSON = documented v2. Merge the worktree into master after review. After both land: write the .nornir chapter for the scanners (xml+json), pointer in design.md, close F7 here.

#5b is READY TO IMPLEMENT (deliberately not started — agent F7a owns the main tree's build): skade 0.1.0 is on crates.io. Steps are in the #5b entry above. Use skade = "0.1.0" (version-only; sibling repo) in znippy-iceberg.

Done this session (all pushed, master 7158b85): store-if-incompressible; .nornir/ doc model (design map + plan spine + inflate chapter); iceberg write backend (--meta-format/--warehouse) + compress_stream_with_sink; selective extract (--type/--repo); B6 (library decode_chunk_into + binary candidate-offset pipeline, A/B 7.12→6.70 s); linflate refill_slow corruption fix; B7 reality (linflate IS the SIMD DEFLATE; deleted ljar's diverged fork; linflate_* nornir scorecard 1.33–1.39× vs miniz); skade analyzed, verified, published to crates.io by us; workspace_znippy created+registered+pushed; nornir wishes/mcp_todo coordination files.

Still open elsewhere: _st bench refresh + docs-render-all (nornir agent); F6 spatial archive (roadmap); pre-existing dirty Cargo.lock at session start (left uncommitted on purpose — investigate origin before committing).

2026-06-11 — stree (static search tree) benched + perf-fixed

Worried about the Ragnar-derived id-lookup tree (znippy-zoomies/src/stree.rs). Found the article ("Static search trees: 40× faster than binary search", curiouscoding.nl) and reproduced his comparison.

• Benches added to nornir bencher (NOT criterion): znippy.stree_8mb / _128mb / _1gb in xtask/examples/nornir-bench.rs — single find_exact + batched batch_stream vs binary_search baseline, swept cache→RAM. stree_1gb gated behind NORNIR_BENCH_SKIP_HEAVY.

• Gap found: original ~110 ns/q @ 1 GB, ~55 ns @ 8 MB batched — 2–4× off Ragnar even in cache. binary_search baseline matched his (~1050 vs ~1150 ns), so harness sound.

• Two fixes (both keep API + all 32 stree tests green):

  1. AVX2 multiversioning — count_lt was dispatching is_x86_feature_detected per node + calling a non-inlinable #[target_feature] fn. Hot traversals now generic over the counter, AVX2 check hoisted once per query/batch into a #[target_feature] wrapper so the intrinsic inlines.
  2. advise_hugepage() (madvise(MADV_HUGEPAGE)) on the tree Vec in both constructors — host is transparent_hugepage=madvise, so the tree got 4 KB pages → dTLB thrash at 1 GB.

• Result: default build 1 GB batch 110→96 ns (11.2×); target-cpu=native 1 GB 66 ns (16×) and 8 MB 19 ns — in Ragnar's single-thread range.

• Doc: .nornir/stree.md (full write-up + remaining levers: B=16 two-line node, native planet build, interleave/left-max, multi-thread → his ~7 ns), pointer added in benching.md.

• Not committed — left for review. Remaining levers need format/layout sign-off (B=16 changes internal-node on-disk layout).

Follow-up (same day): tried const B=16 (two-cache-line node) per request. All tests green but it regresses the batched path (best ns/q B=8→B=16: 8 MB 40→47, 128 MB 68→82, 1 GB 104→120 default) — P-way batching already hides the miss latency fewer levels would save, leaving only B=16's doubled node cost. Helps serial-only (1 GB serial 580→457) but production is batched. Reverted to B=8; count_lt_avx2 left B-agnostic so it's a one-line flip. Ragnar credit strengthened in the module header + stree.md. NOTE: couldn't re-run the full nornir suite — the sibling ../../nornir working tree is mid-edit by another agent and currently fails to compile (viz/model.rs gix mismatch); benched via a throwaway znippy-zoomies example instead (no nornir dep). Cargo.lock shows only a skade 0.4.6→0.4.8 path-dep bump, unrelated.