nordisk/znippy-zoomies

optimize — shrink a GeoParquet dir into an embed-ready GeoParquet

The first-class home for the "make OSM data small enough to include_bytes! into a wasm demo" job the facett team used to hand-roll. Flow diagram: .nornir/assets/optimize-flow.svg. Peer of region-clip (clip-at-decode) and the gatling saturation work.

The pitch in one line: turn a 20 MB clip into a few-hundred-KB GeoParquet — trim to geometry + a slimmed tag set, keep named POIs first, cap the feature count, zstd — the gatling (1 reader → N workers → 1 writer) way, not a read-all-then-process script.

What it is

osm-katana optimize <input-geoparquet-dir> -o <out.parquet> [flags]

Reads nodes.parquet (always) and — unless --points-only — ways.parquet (if present). Relations are skipped: their geometry is null, so there is nothing to embed. Output is a single GeoParquet file with exactly two columns, the facett-osm read contract:

facett-osm::read_points reads the WKB geometry; read_ways reads geometry + tags. The output is schema-compatible with both, on disk or include_bytes!'d (read_points_bytes / read_ways_bytes).

Flags

A feature is named if its tags JSON has any of NAMED_KEYS (name, place, amenity, shop, tourism, historic, leisure, natural, aeroway, railway, public_transport, office, man_made) with a non-empty value. slim_tags keeps that set plus the keys facett's theme classifier reads for ways (highway, waterway, landuse, building, barrier, boundary, bridge, tunnel) and drops everything else.

The gatling shape (1 → N → 1) — src/optimize.rs

Mirrors geo2arrow and convert's pass2 exactly:

• 1 reader — emits only row-group indices (usize) down a bounded sync_channel. It does no decode (a 1-reader-decodes design was the serial smell: it pinned ~2 cores). Backpressure keeps it a few slots ahead, never flooding RAM.
• N workers (= available_parallelism(), capped at n_rg) — each grabs the next row-group index (self-dispatch via Mutex<Receiver>), opens its own projected reader over that one row group, and carries it through all stages with no barrier: zstd-decompress + arrow-decode (projected to geometry + tags, so version/changeset/node_refs are never touched) → is_named → slim_tags → prune to two columns → split into named / unnamed.
• 1 writer (the caller, after the scope) — saves only. Named-first is a partition + truncate, NOT a sort: emit named (capped), top up with unnamed to the cap. Build one arrow batch, write one zstd GeoParquet.

Verify as data

The optimize.process phase emits cpu_cores / busy_pct via phase_log, like convert. On a real 104 M-node file the refactor (reader emits indices, workers decode) lifted the process phase from 1.93 → ~4 cores and halved its wall, and replacing the 100 M-row global sort with the named/unnamed partition cut the rank tail 9.4 s → 2 s. Per-row work for points is light (21-byte Point WKB + a substring pre-checked tag scan), so points-only is decode/scan-bound at ~4 cores — the gatling LAW's one legitimate low-core case — while folding ways in (larger LineString WKB) climbs to ~5.5 cores.

Bench — real clip

The 20 MB Sigtuna clip → a 691 KB embed (named POIs first), the exact knob the facett team needed. Output validated: two-column WKB GeoParquet, geometry[0] byte = 1 (little-endian Point), first row a named village.

Tests

• src/optimize.rs mod tests (unit): cap + named-first + tag-prune; named-only; output-smaller + facett-readable; is_named / slim_tags.
• tests/optimize_embed.rs (integration): end-to-end on a multi-row-group nodes+ways dir — asserts smaller, two-column WKB, named-first, tag-prune, the --log JSONL carries optimize.process cpu_cores/busy_pct/workers (the cores-busy perf check), and --points-only --named-only.

Quick reference

cargo build --release -p osm-katana

# embed knob: ~40 k named-first features from a clip → one zstd GeoParquet
osm-katana optimize /tmp/sigtuna_geo -o embed.parquet --max-features 40000

# points-only, named POIs only (tiny)
osm-katana optimize /tmp/sigtuna_geo -o pois.parquet --points-only --named-only

use osm_katana::{optimize, OptimizeOptions};
let opts = OptimizeOptions {
    output: "embed.parquet".into(),
    points_only: true,
    max_features: 40_000,
    ..Default::default()
};
optimize("out".as_ref(), &opts)?; // reads out/nodes.parquet (+ ways)