nordisk/znippy-zoomies

Scanners (json / xml / vtd) — guide

Class: guide · Function: scanners. How to call the three parallel record scanners. Shape in scanners-design.md; why oversplit in scanners-reasoning.md.

Build

CARGO_TARGET_DIR=/mnt/work4t/cargo-target/zoomies cargo build --release

All three live in the root znippy-zoomies crate (src/json.rs, src/xml.rs, src/vtd.rs).

Call them

// NDJSON: count records, get a zero-copy span per line.
let n = znippy_zoomies::json::scan_records_parallel(&bytes, 0, |span| {
    let rec = &bytes[span.offset..span.offset + span.len];   // borrowed, no copy
});                                                           // json.rs:127

// XML: scan top-level elements, map each span to your own T on the worker.
let tags = /* TagSet of element names */;
znippy_zoomies::xml::scan_parallel(&bytes, 0, &tags, |kind, span| my_map(kind, span),
                                   |t| collect(t))?;          // xml.rs:387

// OSM: build a file_offset-sorted ElemIndex (pick the collector by scale).
znippy_zoomies::vtd::build_elem_index_parallel(&bytes, 0, |ix| push(ix))?;   // vtd.rs:284
znippy_zoomies::vtd::build_elem_index_to_mmap(&bytes, 0, idx_path)?;         // planet, mmap
znippy_zoomies::vtd::build_elem_index_pipelined(path, 0, idx_path)?;         // I/O-ring, streaming

n_workers = 0 ⇒ one per core (the gatling default). Below 4 MiB, xml/vtd run serially on their own (no pool spin-up).

Which vtd entry

• build_elem_index_parallel — in-memory buffer, callback per element.
• build_elem_index_to_mmap — planet-scale, writes the index straight to an mmap at prefix-sum offsets.
• build_elem_index_revolver — mpsc + BTreeMap committer, over-provisioned chunks.
• build_elem_index_pipelined — reads from a path via an I/O-ring, 100 MB slots.

Bench

cargo bench runs benches/json_scan.rs and benches/xml_scan.rs; the scanners also ride the osm-katana convert bench (they are its XML front-end).