nordisk/znippy-zoomies

region-clip — clip-at-decode for osm-katana (bbox + polygon)

Design + guide for the region-clip feature on the current fast convert (coarse PBF split + streaming sort + worker-side encode). Peer of the gatling saturation work. Flow diagram: .nornir/assets/region-clip-flow.svg.

The pitch in one line: clip Sweden's REAL polygon shape during the already 32-core parallel convert — the clip is a per-node test run inside the pass-1 Gatling workers, the moment a node's lon/lat are decoded and before any tag-JSON work. There is no extra pass and no serial scan; every core that decodes a blob also clips it. Crushes osmium, whose planet-bz2 polygon clip is single-threaded (~8 h).

What it is

Keep only the OSM data inside a geographic region:

1. bbox / preset — --clip europe | nordics | <min_lon,min_lat,max_lon,max_lat>. A branchless four-compare test.
2. polygon — --poly <file> where <file> is an Osmosis .poly (e.g. Geofabrik's sweden.poly) or a GeoJSON Polygon / MultiPolygon. Clips to the region's true shape with an even-odd ray-cast, bbox-pre-filtered (the polygon's own bbox is a free reject for the common far-outside node).

--clip and --poly are mutually exclusive. Both require the default --geometry resolved mode (raw keeps node-ID lists, not resolved coords, so node-membership clipping does not apply — it is rejected with a clear error).

Design

Types (lib.rs)

• Bounds { min_lon, min_lat, max_lon, max_lat } — inclusive box, branchless contains (&-combined compares). Bounds::parse handles presets + raw bbox.
• Polygon { rings: Vec<Vec<(f64,f64)>>, bbox: Bounds } — contains runs the bbox pre-filter first, then an even-odd ray-cast across all rings combined (so .poly holes/islands compose). from_poly_str / from_geojson_str / from_file load it.
• Clip { Bbox(Bounds), Poly(Polygon) } — the worker-shared region. contains dispatches; bbox is the cheap path, poly does its own bbox pre-filter.
• ConvertOptions.clip: Option<Arc<Clip>> — None (default) is the lean path. active_clip(opts) hands a cheap Arc clone to each pass-1 reader.

Where the clip lives — clip-at-decode, in the worker

The current convert decodes via the coarse split (workers own candidate ranges, align_and_collect, the typed pass-1/pass-2 codecs). The clip is inserted at the node-parse point of every pass-1 path so it filters DURING the parallel decode (zero extra pass, all cores):

In parse_nodes/parse_node_coords (reader.rs) each node decodes its (lon,lat) to degrees, then if !clip_keeps(clip, lon, lat) { continue } — before collect_dense_tags/serialize_tags. For dense nodes a clipped-out node still advances the shared key/value cursor via skip_dense_tags (a pointer walk, no alloc, no JSON) so the next node stays in sync. clip == None is a never-taken branch the optimiser folds away.

Way / relation semantics (documented, intentional)

Clipping is by node membership. Pass-1's node store now holds only in-region nodes, so in pass-2 a way's refs resolve only against in-region coords. A way with zero resolvable coords lies entirely outside the region → it is dropped (reader::parse_ways_rels / xml_reader::parse_chunk / the codec inline assembly, gated on clip_active). A way that crosses the boundary keeps its in-region endpoints — it is truncated, not edge-clipped (the simplest correct choice: no second pass, no separate way-membership test). In the Sweden bench only 544 of 9.27 M kept ways had <2 in-region coords (geometry null).

Relations are not node-clipped — they are kept by their member-JSON (geometry is already null for relations). A relation referencing only out-of-region members survives as a (geometry-less) record. This matches the original design (clip is node/way membership) and is acceptable because rel rows are tiny; tightening it would need a member-resolution pass.

Tests (inject-assert — real input, asserted output) — lib.rs mod clip_tests

All synth PBFs go through the production encoder (pbf_enc) and decode through the exact parse_nodes / convert path the converter runs — green ⟺ the production clip works (the fail-on-bug: remove the clip branch and bbox_clip… reads 400/400, polygon… reads poly==bbox, way_clip… reads 3 ways).

Bench — Sweden polygon clip from a 32 GB europe PBF

Threadripper PRO 3975WX · 32 cores · --poly sweden.poly (Geofabrik, 1 ring, 146 verts) · output on /home, source read-only:

Result: 103.5 M nodes, 9.27 M ways, 8.32 M rels — a coherent Sweden subset (matches the known sweden_pbf_resolved reference of 104 M / 9.29 M). The clip stays Gatling at ~30/32 cores throughout. osmium's planet-bz2 polygon clip is single-threaded (~8 h). Appended to bench_history.json.

Quick reference

cargo build --release -p osm-katana

# bbox / preset
osm-katana convert europe.osm.pbf -o out --clip nordics
osm-katana convert planet.osm.pbf -o out --clip 9.47,47.05,9.64,47.27

# real polygon shape (Geofabrik .poly or GeoJSON)
curl -L -o sweden.poly https://download.geofabrik.de/europe/sweden.poly
osm-katana convert europe.osm.pbf -o out --poly sweden.poly

use std::sync::Arc;
use osm_katana::{Bounds, Clip, Polygon, ConvertOptions, convert, decode_pbf_node_count};

let poly = Polygon::from_file("sweden.poly".as_ref())?;
let opts = ConvertOptions { clip: Some(Arc::new(Clip::Poly(poly))), ..Default::default() };
convert("europe.osm.pbf".as_ref(), &opts)?;

let kept = decode_pbf_node_count(&pbf_bytes, Some(&Clip::Bbox(Bounds::nordics())));
osm_katana::extract_region("europe.osm.pbf".as_ref(), "out".as_ref(), "nordics")?; // bbox helper