nordisk/znippy-zoomies

Scanners (json / xml / vtd) — design

Class: design · Function: scanners — the three zero-copy parallel record scanners that fan through the gatling fork-join engine. Strict: shipped shape, every claim a real file:symbol. Run them via scanners-guide.md; the oversplit rationale in scanners-reasoning.md. Engine: gatling-design.md.

What they are

Three scanners in the root znippy-zoomies crate that turn a byte buffer into a zero-copy span index — byte (offset, len) pointers into the original input, never a copy of the records. They are not decompressors; they are the reading half that a decompressed stream feeds. All three share one shape: split at legal record boundaries, oversplit ~8× the core count, self-dispatch the per-unit parse through gatling_forkjoin::gatling_for_each (gatling/src/gatling_forkjoin.rs:60), and merge partial results in unit order — deterministic and byte-identical to a serial scan for any worker count.

json — NDJSON / JSON-Lines (src/json.rs)

scan_records_parallel(bytes, n_workers, on_record) -> u64 (src/json.rs:127). Boundaries are raw \n: per RFC 8259 §7 a literal newline is illegal inside a JSON string, so every raw \n is a record boundary — found with memchr, no parsing needed. It oversplits into chunks = (n * SPLIT) units (SPLIT = 8, json.rs:148), each candidate i * chunk_size forward-seeked past its next \n by find_safe_split (json.rs:55); borders are monotonic so no record straddles a cut (a candidate landing inside a prior record yields an empty unit). The fan-out is gatling_for_each at json.rs:165, returning Vec<Vec<RecordSpan>> in slice order, merged to on_record on the calling thread. Output span: RecordSpan { offset, len } (json.rs:36), the newline and trailing \r excluded. No size threshold — the parallel path runs whenever n > 1.

xml — generic VTD-style element scanner (src/xml.rs)

scan_parallel(bytes, n_workers, tags, map, on_item) -> Result<u64> (src/xml.rs:387), with map: Fn(u8, ElemSpan) -> T + Sync. Boundaries are top-level element starts: find_top_level_start (xml.rs:191) does memchr(b'<') then tags.starts_element (name followed by whitespace). It oversplits into units = (n * SPLIT) (SPLIT = 8, xml.rs:409); border[i]..border[i+1] bound each unit so no element straddles a cut. Fan-out is gatling_for_each at xml.rs:421; map runs on the worker, partials merged in file order. The parallel path is gated behind a 4 MiB threshold (xml.rs:402) — below that it runs the serial scan_slice. Output span: ElemSpan { offset, len, tag_len, self_closing } (xml.rs:101).

vtd — OSM specialisation + planet-scale writers (src/vtd.rs)

Layers OSM decoding (<node>/<way>/<relation>) on xml, exposing four entry points across three collection engines, all producing a file_offset-sorted ElemIndex (32-byte #[repr(C)], vtd.rs:104):

• build_elem_index_parallel (vtd.rs:284) — delegates straight to xml::scan_parallel (vtd.rs:292), mapping each span to an ElemIndex.
• build_elem_index_to_mmap (vtd.rs:347) — std::thread::scope two-scope rendezvous: a parallel count, then each thread writes its elements to its prefix-sum offset in a shared mmap (file order by construction).
• build_elem_index_revolver (vtd.rs:468) — std::thread::scope + mpsc + a BTreeMap committer draining (chunk_index, Vec) in file order; n_chunks = n * 5 / 4 (vtd.rs:479, "+25% over-provision keeps cores busy during commits").
• build_elem_index_pipelined (vtd.rs:599) — an I/O-ring reader thread emits 100 MB slots (SLOT_BYTES, RING_DEPTH = 4) trimmed at find_safe_slot_end; each slot is re-split into units = n * SPLIT sub-chunks and fanned through gatling_for_each (vtd.rs:695), slots processed sequentially so output stays in file order with no sort.

Whereabouts

Saturation state (just-landed)

All three oversplit by SPLIT = 8 so a dense region cannot strand one worker — the self-dispatch engine lets siblings steal the tail. json takes the parallel path unconditionally (n > 1); xml/vtd gate behind 4 MiB and carry the elaborate oversplit-under-skew machinery (rendezvous / revolver / pipelined committers) for planet-scale inputs. No cores_busy is fabricated here; the honest figures ride the osm-katana bench wave (the scanners are that converter's XML front-end).