osm-katana
Fast OpenStreetMap format converter — OSM XML / BZ2 / GZ / PBF → GeoParquet / PBF with parallel decompression. A sub-crate of the znippy-zoomies workspace.
CLI
osm-katana convert <input> # OSM XML/BZ2/GZ/PBF → GeoParquet
osm-katana convert <input> --clip "min_lon,min_lat,max_lon,max_lat" # bbox region clip
osm-katana convert <input> --buildings # keep building=* footprints + heights/levels
osm-katana convert <input> --vtd-workers 1 # single-worker (single-core lens)
osm-katana xml2pbf <input> [output] # OSM XML/BZ2 → PBF
osm-katana pbf2geo <input> # PBF → GeoParquet
osm-katana geo2arrow <input> <output> # GeoParquet → Arrow IPC
osm-katana verify [dir] # inspect / verify GeoParquet output
osm-katana admin-poly <input> --name sweden --admin-level 2 -o sweden.poly
# cut a region's administrative boundary
# out of a planet (.osm.bz2 or .osm.pbf)
# as an Osmosis .poly — a clip polygon
# comes from OUR data, not a downloadLibrary
The osm_katana crate exposes the same conversion as a library API (convert()), run in-process on a worker pool — no subprocess, no shell-out. It reads compressed input directly (the parallel lbzip2 / lgz decompressors), so a .osm.bz2 or .osm.pbf is converted in one pass. The node prologue — mandatory sequential work on a non-seekable bz2 stream — is skipped with a streaming, seek-free VTD (see Streaming VTD below).
Convert benchmarks
osm-katana convert (OSM XML → GeoParquet) head-to-head with the legacy single-stream C++ reference tools osmium / osmconvert for the same input. One convert pass, min-of-3. Columns: MB/s · cores busy · wall seconds · peak RSS (MB). The convert rows share the workspace bench harness (../examples/nornir-bench.rs) and warehouse under repo=znippy-zoomies; the head-to-head also renders in that repo's README under OSM converter (../README.md).
Multi core — all cores
osm-katana fans the convert across every core on a no-barrier gatling pipeline; osmium / osmconvert are the single-stream C++ references (benched only when installed).
(no bench history yet — run a benchmark to populate this section)
Single core — one core each
osm-katana_st pins a single VTD worker (convert … --vtd-workers 1), racing the inherently single-stream references at one core each — the pure-Rust per-core advantage before parallelism is added on top.
(no bench history yet — run a benchmark to populate this section)
convert_mbs = input MB / wall; cores_busy = average CPU cores busy during the convert (full saturation = the host core count). By default a small synthetic OSM-XML corpus exercises the emit path on any box — the HEADLINE number comes from the off-box planet run (NORNIR_OSM_KATANA_INPUT=<planet.osm> nornir bench run znippy-zoomies). Region clip headline (cut Sweden out of the planet with our tool): NORNIR_OSM_KATANA_CLIP=10.9,55.3,24.2,69.1.
Staged pipeline
The classic planet chain — each stage eats the previous stage's output file: xml2pbf (OSM XML/bz2 → PBF) → pbf2geo (PBF → GeoParquet) → geo2arrow (GeoParquet → Arrow IPC). NORNIR_OSM_KATANA_CHAIN=1 nornir bench run znippy-zoomies warehouses one row per stage; point NORNIR_OSM_KATANA_INPUT at a planet for the heavy chain.
Streaming VTD
The planet is bz2 — a non-seekable stream — so the changesets + the whole node prologue MUST be decompressed sequentially no matter what. osm-katana's two-pass VTD (seek_past_nodes) assumes a seekable mmap and re-seeks past the nodes, wasting that mandatory leading decompression. The single-pass streaming VTD (znippy_zoomies::vtd_single_pass) makes the mandatory pass do double-duty: fed the decompressor's byte chunks in stream order (never a seek), it scans and discards the node prologue and, the instant it crosses the node→way boundary, flips to storing the ElemIndex for ways + relations — seek-free, O(1)-memory boundary detection with a cross-chunk straddle carry. Byte-identical to the two-pass way/relation subset (proven by unit tests). Opt-in; the full-planet FASTER-and-byte-identical compare is deferred to the quiet bench box.
License
MIT