znippy-zoomies — crate card
Reverse-engineered (from source — verify, don't trust the name)
- One-line pitch: A grab-bag of "fast-as-hell" building blocks (parallel XML/JSON scanning, static search trees, a no-barrier streaming engine, a parallel sort, and a vector-ANN index) extracted from katana-osm so multiple downstream repos can share them.
- Does: Provides
vtd/xml(VTD-style parallel OSM-XML element scanners producing compact byte-offsetElemIndex),stree/stree32(Ragnar Groot Koerkamp's AVX2 static search tree over sorted i64/u32 keys, incl. mmap-backed leaf layer),gatling(generic no-barrier worker-pool engine: split → N decode → in-order collect → sink, with an asyncgatling::iosibling on tokio),chunk_revolver(zero-alloc slot pool for 1-reader/N-worker streaming),psort(parallel sample sort over 16-byte AoS records),json(zero-copy parallel NDJSON scanner), andvann(exact brute-force f32 nearest-neighbour index with SIMD/int8-VNNI cosine kernels). - Key deps: rayon (data parallelism), memchr (delimiter scan), memmap2 (mmap stree/vann blobs), bytes+futures (async gatling::io payloads/in-flight set), libc on Linux; dev: criterion, tokio, tempfile, rand, serde_json.
- Shape: lib only (~9k LoC across 15 modules); biggest are stree (1653), vann (1469), vtd (1132), stree32 (942), gatling (866), xml (691). Multiple criterion benches (xml_scan, json_scan, gatling_roundtrip). (The
vann-mmapfeature named here gated zerocfgsites and was deleted 2026-08-22; the row blob is read, not mapped.) - Observed role in the workspace: The workspace root crate and shared foundation.
osm-katanadepends on it and re-exportsvtd as xml_vtdandstree as stree64. Doc comments say both skade (stree/gatling) and znippy (codecs/gatling) depend on it; extracted into its own leaf repo to avoid a release-cascade cycle.
Fill in (one line each — only the human knows these)
- Primary consumer(s):
- Why it exists (vs the obvious alternative):
- Hard constraint (airgap / no-C-deps / no-FFI / pure-Rust / etc.):
- Release cadence & independence (rides a parent? own repo someday?):
- Status (production / prototype / experimental / stubbed):
- Must-NOT-change invariant:
- Biggest known gap or TODO:
- Anything a newcomer would get wrong about it: