nordisk/znippy-zoomies

Single-pass VTD — discard-until-boundary-then-store

Rickard's idea. The planet is bz2 (non-seekable) — the changesets + whole node prologue (~86 GB) MUST be decompressed sequentially anyway. The two-pass VTD ([crate::vtd], and osm-katana's seek_past_nodes) assumes a seekable mmap and re-seeks past the nodes, wasting that mandatory leading decompression. The single-pass makes it do double-duty: one stream-order pass that scans the node prologue (index-only, no store) and, at the node→way boundary, FLIPS to storing the VTD ElemIndex for ways + relations — no seek, no re-decompress.

The flip

 bz2 decompressor ──chunks(stream order)──▶  StreamingBoundaryDetector
                                                    │  monotonic <way / <relation  (= WAYREL_NEEDLES)
                                                    │  cross-chunk straddle carried (MAX_NEEDLE-1)
     ┌──────────────── before flip ────────────────┼──────────── after flip ───────────────┐
     ▼                                              ▼                                        ▼
  scan node prologue                        node→way boundary                     store ElemIndex
  (count, DISCARD — no store)                (first <way offset)                   (ways + relations)
                                                                                   offsets absolute in-stream

• Seek-free: boundary found by streaming the same monotonic <way /<relation predicate as seek_past_nodes, chunk-by-chunk, with a MAX_NEEDLE-1 straddle carry → O(1) memory, no mmap.
• Byte-identical: the stored way/relation entries match a full two-pass scan of the same bytes (offset · length · id · kind · tag_flags) — proven in vtd_single_pass::tests.

Status

• StreamingBoundaryDetector (the seek-free flip) — REAL + tested (4 tests, incl. cross-chunk edges).
• single_pass_index(chunks, store) wires detect → discard-prologue → store-way/rel. Store side currently buffers the post-boundary region before scanning (correct/byte-identical). Remaining refinement: incremental post-boundary store (per-element carry) so the full planet never buffers the way/relation tail. The detector already proves the seek-free flip — the rest is adding incremental scan, not a rewrite.
• Opt-in (store flag) — changes no default path, so the WORLD_RECORD_DECOMPRESSORS discipline holds.

Build + compare protocol (HEAVY — deferred to the quiet bench box)

Baseline = the current two-pass full-planet convert. Then, on a QUIET box (Loki/Odin): run the SAME planet with the single-pass, quiet → compare wall-time + phase-log. Keep only if FASTER AND byte-identical (roundtrip-verify the parquet/vtd matches the baseline). If not faster, keep behind the flag + record the honest result. Bench data → /home; never a heavy run on a busy box. See also [[WORLD_RECORD_DECOMPRESSORS]].