nordisk/znippy-zoomies

znippy-zoomies — Årsstädning audit (2026-06-28)

Reconciliation (2026-07-17): the dated snapshot below is preserved as authored on 2026-06-28. Findings that no longer reflect the tree have since been closed in place — marked [x] with a dated RESOLVED note — rather than deleted, so the original audit stays honest. The tallies under Counts are the original 2026-06-28 figures and are left unchanged.

Overall health is good: the low-level core (gatling sync/forkjoin/io engines, psort sample sort, stree/stree32 SIMD search, linflate fast loop, chunk_revolver slot pool, vtd/xml/json scanners) is genuinely state-of-the-art and the resolved-mode pass-2 Parquet path is clean and parallel. The recurring themes are (1) the four codec sub-crates (lgz, ljar, lzip-parallel, lbzip2) hand-roll the gatling + chunk_revolver pipeline four times and duplicate chunk-split/deflate-scan/zip-core code instead of driving the in-repo helpers — with ljar/lzip-parallel being near-verbatim twins (JAR == ZIP); (2) the osm-katana way/node hot paths force per-way coord clones and throwaway HashMaps and carry two superseded node-store implementations (~700 dead lines); (3) the flagship sync gatling engine and chunk_revolver (unsafe pointer code) have zero tests and no perf bench; (4) capabilities (verify, streaming decompress) live inside binaries with no library/Serialize twin; and (5) docs/README drift around stree numbers and an absent xtask harness. Two test suites also shell out to curl/unzip, the only no-shelling violations in the repo.

Tier 1 — Hygiene & principles

zero-copy-alloc

• [HIGH] encode_wkb forces a per-way coords Vec clone at every call site — osm-katana/src/shared/gpu_backend.rs:9-26 (callers osm-katana/src/reader.rs:106, :169, osm-katana/src/xml_reader.rs:2439) — GpuBackend::encode_wkb takes &[RawWay] where RawWay{coords: Vec<(f64,f64)>} owns coords it only reads, so every way is deep-cloned (coords: collected[i].coords.clone()) into a throwaway RawWay; on planet that is hundreds of millions of coord-Vec allocs+memcpys across all three way pipelines for zero benefit. Fix: make WKB encode borrow — change RawWay.coords to &'a [(f64,f64)] (or add encode_wkb_borrowed/take &[CollectedWay]), build the list with coords: &collected[i].coords; CpuBackend already only reads the slice. (effort: M)
• [HIGH] Per-way throwaway HashMap<String,String> built just for is_closed_area when a zero-alloc slice helper exists — osm-katana/src/xml_reader.rs:680-684, osm-katana/src/reader.rs:374-395 — parse_way_meta does tag_pairs.iter().cloned().collect() then is_closed_area(&refs,&tags_map); reader.rs is worse, building the map with (*key).to_string(),(*val).to_string() (two heap allocs/tag) only to area-test and serialize — all discarded per way. geometry::is_closed_area_slice already takes the tag-pair slice directly. Fix: call is_closed_area_slice(&refs,&tag_pairs), delete the HashMap, and serialize tags straight off the borrowed &str pairs. (effort: S)
• [MEDIUM] Raw bz2/gz path serializes way/rel Parquet zstd encode on a single writer thread while nodes are parallel-encoded — osm-katana/src/xml_reader.rs:1003-1083 (RawParallelParquetSink), wiring :1136-1145 — in geometry=raw single-pass, node row groups encode on workers but way/rel batches ship to one writer::ParquetSink thread doing serial zstd for ways AND rels; the resolved pass-2 path already solved this (parallel_way_writer/parallel_rel_writer, :2034-2051,:1989-2012). Raw mode is the asymmetric laggard with N-1 cores idle. Fix: switch RawParallelParquetSink to the Pass2TypedSink pattern — workers call enc.encode(&batch), collector only w.append(rg); drop the single ParquetSink. (effort: M)
• [MEDIUM] Arrow IPC writer re-copies the entire sorted column arrays with per-batch to_vec() — osm-katana/src/node_store_arrow.rs:223-234 — after sorting, the writer loops batches doing Int64Array::from(sorted_ids[start..end].to_vec()) (+lats/lons), copying every element of the already-materialized sorted Vec a second time into fresh per-batch Vecs — tens of GB of needless memcpy on the single finalize thread at planet scale. Fix: build each full column array once (consume the sorted Vec), then array.slice(start,len) per batch (zero-copy, shares Buffer); removes all three to_vec()s. (effort: S)
• [MEDIUM] optimize() copies every WKB geometry out of the zero-copy Arrow BinaryArray into an owned Feature.geometry Vec — osm-katana/src/optimize.rs:293-297 — geom.value(i) is a borrowed &[u8] into the decoded column but the worker stores geometry: g.to_vec() into Feature, deep-copying every kept feature's WKB (100M+ features = a full second copy of the geometry column). Fix: carry a zero-copy slice handle — keep the source Arc<BinaryArray> alive and store (array,row_index), or store bytes::Bytes sliced from the Arrow Buffer. (effort: M)
• [MEDIUM] Decode workers materialize owned-String NodeRecords (incl. String::from("{}") per untagged node) as an intermediate before Arrow build — osm-katana/src/reader.rs:295-309, consumed osm-katana/src/writer.rs:471-479 — parse_nodes builds Vec<NodeRecord> with an owned tags_json String per node (and a fresh 2-byte heap alloc for every untagged node — the vast majority), then NodeAccumulator copies each into the Arrow StringBuilder: every tag allocated once, copied again. Fix: append directly into the Arrow builders (serialized tag bytes; append_value("{}") as a borrowed literal for untagged), eliminating the Vec+owned-String hop; at minimum kill the per-untagged-node String::from. (effort: L)
• [LOW] Gatling collector reallocates the carry buffer every chunk with buf[safe..].to_vec() — src/gatling.rs:412-426 — the no-barrier collector flush allocates a fresh Vec for the boundary remainder each flushed chunk, a steady per-chunk allocation in the core helper used by every consumer. Fix: keep a reusable carry Vec — carry.clear(); carry.extend_from_slice(&buf[safe..]); (or copy_within+truncate+swap). (effort: S)

no-shelling

• [HIGH] Tests shell out to curl to download Maven JARs instead of a pure-Rust HTTP client — ljar/tests/maven_artifacts.rs:81 and lzip-parallel/tests/maven_artifacts.rs:81 (fn ensure_jar) — spawns Command::new("curl").args(["-sSfL","-o"]) (with .expect("curl must be available")) to fetch from repo1.maven.org: forbidden shelling, opaque failures, non-portable, duplicated across both files. Fix: replace with a pure-Rust blocking GET (ureq dev-dep, or reqwest blocking) writing the body to path, factored into one shared test-support helper. (effort: S)
• [HIGH] Benchmark baseline shells out to system unzip instead of the in-process zip crate (already a dev-dependency) — ljar/tests/maven_artifacts.rs:400 & :599; same lines in lzip-parallel/tests/maven_artifacts.rs — bench_unzip + the sequential compare spawn Command::new("unzip") (.expect("unzip must be available")); redundant since both crates already use the zip crate next door in reference_entries. Fix: replace with an in-process extraction loop over zip::ZipArchive writing each entry to the tempdir and timing that (apples-to-apples Rust vs Rust), shared between both files. (effort: S)
• [LOW] CLI end-to-end test spawns its own built binary via Command instead of calling the library entry point (symmetry) — ljar/tests/maven_artifacts.rs:259 (maven_cli_extract_guava) and lzip-parallel/tests/maven_artifacts.rs:259 — runs Command::new(env!("CARGO_BIN_EXE_ljar"))...status(), asserting only an exit code + on-disk files, not structured output. Fix: move extraction into a public lib fn (e.g. ljar::extract_jar_to_dir), have the bin and the test both call it; keep the subprocess test as a thin smoke check only. (effort: M)

helper-reuse

• [HIGH] Four codec CLIs hand-roll the gatling streaming pipeline + chunk_revolver slot pool — lgz/src/bin/lgz.rs, ljar/src/bin/ljar.rs, lzip-parallel/src/bin/lzip.rs, lbzip2/src/bin/lbunzip2.rs — each builds the identical Reader→Main(carry+split)→N Workers→ordered Collector→Writer engine by hand with raw mpsc::sync_channel wiring plus a manual Vec<u8> ring-slot recycle (FilledSlot/WorkItem/CollectorMsg/WriteMsg, is_ring_slot, done_count lifetime) — ~500-700 lines each — i.e. gatling::run + chunk_revolver reimplemented four times in the same repo. osm-katana already proves these codecs plug into gatling via TypedCodec/TypedSink. Fix: implement gatling::Codec/Sink per format and drive each CLI through gatling::run/run_typed, allocating ring slots from chunk_revolver::ChunkRevolver; delete the bespoke mpsc plumbing and Vec slot recycling in all four bins. (effort: L)
• [HIGH] lbzip2/src/par.rs is a local no-barrier work-stealing pool replacing gatling/rayon — lbzip2/src/par.rs — a hand-rolled scoped-thread work-stealing parallel-map (AtomicUsize claim of 0..n, UnsafeCell/MaybeUninit result slots, LBZIP2_THREADS sizing) whose own doc says "Replaces lbzip2's old dedicated rayon::ThreadPool" — exactly the role gatling/gatling_forkjoin own, a private copy one directory below the helper. Fix: drive lbzip2 block decode through gatling/gatling_forkjoin like osm-katana; if the dynamic claim-balancing is a genuinely new variant, move par_map INTO znippy-zoomies as a shared helper rather than a private copy. (effort: M) — RESOLVED 2026-07-17: lbzip2/src/par.rs no longer exists (deleted); lbzip2 block decode now fans out through the shared gatling fork-join engine. Finding no longer reflects the tree.
• [HIGH] ljar and lzip-parallel are near-identical twin crates (JAR == ZIP container) — ljar/src/ vs lzip-parallel/src/ — central_dir.rs differs by 96/350 lines; batch.rs/parallel.rs/reader.rs are parallel copies; the CLI bins share identical type layouts (FilledSlot, WorkItem, CollectorMsg, WriteMsg, is_ring_slot) — whole-crate duplication of central-dir parsing, entry decode, batching, and the streaming pipeline. Fix: collapse into one zip-core crate (one set of central_dir/entry/chunk/reader/batch) parameterized for the JAR↔ZIP deltas with thin front-ends, or make ljar depend on lzip-parallel for the shared machinery. (effort: L) — TODO / deferred (evaluated 2026-07-17): still open, intentionally NOT consolidated. Twin pair is ljar/src/{central_dir,deflate_scan,reader,batch,chunk,entry}.rs ↔ lzip-parallel/src/{central_dir,deflate_scan,reader,batch,chunk,entry}.rs. Full extraction into a shared zip-core leaf is an L-effort cross-crate refactor: the two are separately-publishable crates with distinct error types (JarError vs ZipError), CLIs, and bench suites, and lzip-parallel currently carries a pre-existing red unit test (chunk::tests::decode_chunk_two_segments, fails on a clean tree independent of this consolidation). No low-risk shared-helper slice lands green in isolation, so this stays a tracked follow-up rather than being shipped red.
• [MEDIUM] Sibling codecs diverge into four different parallel-map primitives instead of one helper — lgz/src/lib.rs (thread::scope par_map), ljar/src/lib.rs:39-56 (rayon ThreadPool singleton), lzip-parallel/src/lib.rs:32-49 (rayon ThreadPool singleton), lbzip2/src/par.rs (scoped-thread pool) — the same job (ordered parallel decode of N segments) done four ways; the two OnceLock<rayon::ThreadPool> singletons risk oversubscription when composed (osm-katana pulls in lgz + lbzip2 alongside gatling). Fix: expose one shared ordered par_map/fork-join helper from znippy-zoomies (extend gatling_forkjoin) and have all four crates call it; delete the per-crate thread_pool() singletons and lgz's local par_map. (effort: M) — RESOLVED 2026-07-17: the two OnceLock<rayon::ThreadPool> singletons (ljar, lzip-parallel) and lbzip2/src/par.rs are gone; all four codecs now fan out through the one shared gatling::gatling_forkjoin engine and rayon is no longer a dependency of any of them. Finding no longer reflects the tree.
• [MEDIUM] Per-codec chunk split/decode logic duplicates the gatling Split/Codec contract — lgz/src/chunk.rs, ljar/src/chunk.rs, lzip-parallel/src/chunk.rs, lbzip2/src/chunk.rs — each defines its own ChunkSplit{segment_starts,decode_segments,consumed} with split_chunk/decode_chunk + manual carry; lgz's header even says "Mirrors ljar-rs/lbzip2-rs chunk.rs" — exactly gatling::Split + Codec::split/decode reinvented per codec. Fix: re-express each split_chunk as Codec::split → gatling::Split and decode_chunk as Codec::decode; remove the bespoke ChunkSplit structs once the CLIs run on gatling. (effort: M)
• [MEDIUM] deflate_scan.rs SIMD flush-boundary scanner copied across lgz/ljar/lzip — lgz/src/deflate_scan.rs, ljar/src/deflate_scan.rs, lzip-parallel/src/deflate_scan.rs — the SIMD DEFLATE full-flush boundary scanner (00 00 FF FF) is duplicated three times (ljar/lzip differ by ~20 lines, lgz a third variant); all three already depend on the shared DEFLATE crate linflate, its natural home. Fix: move deflate_scan into linflate, reconcile the lgz variant's extra logic, delete the three local copies. (effort: M)

Cross-ref: the helper-reuse audit also flagged the curl/unzip test shelling (ljar/lzip-parallel tests/maven_artifacts.rs:81,400,599) — covered above under no-shelling.

cli-ui-symmetry

• [HIGH] verify capability has no library twin and exists as two divergent CLI impls — osm-katana/src/bin/main.rs (Cmd::Verify + verify_one/count_node_refs/count_geometry/count_refs_for_row_groups, ~119-end) and osm-katana/src/bin/verify_parquet.rs (full 61-line standalone reimpl) — GeoParquet verification lives entirely in binaries, never in osm_katana lib, so no programmatic/gRPC caller can verify output; worse, it is implemented twice with DIFFERENT behavior (main.rs: footer-stats fast-path + row-group-parallel projected decode; verify_parquet.rs: single-threaded full-column decode) that can report different numbers, and results are only println!'d (no Serialize/state_json twin). Fix: add pub fn verify(dir:&Path)->Result<VerifyReport> (+ #[derive(Serialize)] VerifyReport exposing state_json) in a new osm-katana/src/verify.rs (the parallel footer-stats impl); make both Cmd::Verify and the bin thin wrappers (or delete verify_parquet). (effort: M)
• [HIGH] Codec streaming file→file decompress pipelines exist only inside the CLI binaries, no library API twin — lgz/src/bin/lgz.rs (~830), ljar/src/bin/ljar.rs (~830), lzip-parallel/src/bin/lzip.rs (~674), lbzip2/src/bin/lbunzip2.rs (~473); libs expose only in-memory decompress (lgz/src/lib.rs:73, lbzip2/src/parallel.rs:15, lzip-parallel/src/lib.rs:63, ljar/src/lib.rs:70) — each CLI inlines the whole 1-reader→N-worker→collector→writer ring-slot streaming pipeline in main(), while the libraries offer only decompress_*(&[u8])->Vec<u8>; the bounded-memory streaming capability the CLI provides has no programmatic entry point (and duplicates gatling/chunk_revolver four times). Fix: lift the pipeline into each crate's stream module as e.g. pub fn decompress_stream<R:Read,W:Write>(r,w,workers)->Result<u64>, built on gatling + chunk_revolver; reduce each bin to arg-parse + one call. (effort: L)
• [MEDIUM] parquet2arrow bin reimplements Parquet→Arrow inline, diverging from osm_katana::geo2arrow — osm-katana/src/bin/parquet2arrow.rs vs osm-katana/src/geo2arrow.rs:36 + osm-katana/src/bin/main.rs Cmd::Geo2Arrow — two CLI ways into the same capability: the geo2arrow subcommand calls the lib fn, while the auto-discovered parquet2arrow bin hand-rolls its own ParquetRecordBatchReader→ArrowFileWriter loop, bypassing the library and able to drift (schema/compression). Fix: delete parquet2arrow.rs and keep the subcommand, or generalize the lib fn (pub fn parquet_to_arrow(src,dst)) and have both call it. (effort: S)
• [MEDIUM] Redundant standalone auto-bins duplicate unified osm-katana subcommands — osm-katana/src/bin/{xml2pbf.rs, pbf_to_parquet.rs, parquet2arrow.rs, verify_parquet.rs} duplicate main.rs subcommands {Xml2Pbf, Pbf2Geo, Geo2Arrow, Verify} — Cargo auto-discovers every src/bin/*.rs, shipping four extra CLIs as second entries into capabilities that already have subcommands; a flag/fix added to one path silently misses the other. Fix: collapse the benign duplicates into the subcommands (delete xml2pbf.rs/pbf_to_parquet.rs, or mark required-features/examples) so each capability has one CLI surface backed by the lib fn. (effort: S)
• [LOW] Phase-log telemetry is emit-only: no CLI/programmatic read or facett viz twin, no queryable state_json — osm-katana/src/phase_log.rs (PhaseSink writes JSONL to stderr/--log); no log-view subcommand in osm-katana/src/bin/main.rs — observability (wall_s, cpu_cores, busy_pct, per-phase counters) is one-way JSONL with no CLI to read/summarize a prior run, no PhaseEvent parse twin, no facett/state_json. Fix: add osm-katana log-view <path> + pub fn parse_phase_log(path)->Vec<PhaseEvent> (Serialize) and render the summary through a facett component whose state_json is the parsed events. (effort: M)

test-bench-coverage

• [HIGH] Sync gatling core engine (run/run_typed) has zero tests — src/gatling.rs:278 (run), src/gatling.rs:559 (run_typed) — the flagship no-barrier worker-pool the whole standard centers on (788 loc, Codec/Sink traits, slot ring, ordered collector, backpressure) has ZERO #[test] while its async (gatling_io) and fork-join (gatling_forkjoin) siblings both ship inject-and-assert tests, and its only caller (osm-katana raw_gatling_run_typed) is itself untested. Fix: add #[cfg(test)] with a fake Read + test Codec/Sink asserting REAL output: chunk_id-ordered emission, exactly-once decode, slot recycling (ring_slots < n_chunks completes without deadlock), backpressure, Err short-circuit propagation, observed peak concurrency reaching Config workers; plus a run_typed typed-output test. (effort: M) — RESOLVED 2026-07-17: the sync engine (now gatling/src/gatling.rs, re-exported as znippy_zoomies::gatling) is covered by tests/gatling.rs, which drives both gatling::run (byte mode) and gatling::run_typed (typed mode) over multi-chunk / multi-segment inputs and asserts the assembled output equals a serial baseline byte-for-byte (ordered emission), plus a carry-exceeds-headroom case. Finding no longer reflects the tree.
• [HIGH] chunk_revolver helper (ring + unsafe slot pool) has zero tests — src/chunk_revolver/ring.rs, src/chunk_revolver/revolver.rs — a named shared helper and the zero-copy slot pool gatling relies on, full of unsafe pointer work (SendPtr unsafe Send+Sync, unsafe get_chunk_slice, const-N RingInner variants behind dispatch!, try_get_chunk/return_chunk recycling) with ZERO tests — the highest-risk gap on this dimension. Fix: unit-test RingBuffer FIFO order/wraparound/is_full/is_empty/capacity at boundaries across each const-N size; ChunkRevolver acquire-until-exhausted (None), return_chunk→re-acquire (recycle), base_ptrs disjoint, and get_chunk_slice round-trips a known injected pattern. (effort: M)
• [HIGH] osm-katana XML reader and gatling pipeline driver entirely untested — osm-katana/src/xml_reader.rs (2468 loc, 0 tests), osm-katana/src/xml_to_pbf.rs:413 (raw_gatling_run_typed, 1034 loc, 0 tests); also reader.rs(509), coords_ipc.rs(301), phase_log.rs(305), pbf_enc.rs(267), pbf_decomp.rs(169) all 0 tests — the two largest modules of the flagship gatling consumer (the OSM-XML parse engine and the only place the SYNC gatling engine is wired up), plus the zero-copy coords IPC and the gatling-mandated phase telemetry, are unverified. Fix: add an integration test injecting a small known OSM-XML buffer through raw_gatling_run_typed asserting exact decoded node/way records (ids/coords/tags); coords_ipc round-trip (write→mmap-read identical i64/f32); phase_log assertion of recorded phase/cores-busy values. (effort: L)
• [MEDIUM] gatling::io tested only with fake sleep jobs, never a live/remote sink — src/gatling_io.rs:196 (mod tests) — gatling_io exists to be the SAVE-stage pool for network/S3 PUT/GET fan-out, but all three tests drive in-memory tokio-sleep+arithmetic jobs; no test moves a Bytes payload over a socket, so the real S3/HTTP path is untested. Fix: add a test spinning up a localhost HTTP server (or in-process mock object store) driving run/run_ordered with real async PUT/GET, asserting bytes received server-side and returned descriptors match. (effort: M)
• [MEDIUM] psort parallel sample sort has no bench despite explicit perf claims — src/psort.rs:62 (samplesort_aos_by_i64_key), src/psort.rs:257 (radix_sort_aos_by_i64_key) — correctness is well tested but the doc's "sample sort beats fixed-radix LSD" / "below the floor single-threaded sort_unstable wins" claims have NO bench guarding them. Fix: add a criterion bench over realistic OSM-id distributions and several N (below/above the parallel floor) comparing samplesort_aos vs radix_sort_aos vs slice::sort_unstable_by_key, plus a release-only perf-floor assertion. (effort: M)
• [MEDIUM] Sync gatling engine has no throughput/overlap bench — src/gatling.rs (run/run_typed), src/gatling_forkjoin.rs — the only root benches are xml_scan/json_scan; nothing measures the engine's whole value proposition (no-barrier read/decode/sink overlap). Fix: add a gatling bench (synthetic Read + cheap CPU codec + counting sink) measuring gatling::run vs single-threaded baseline across worker counts (release-asserting it scales past 1 thread), plus a gatling_forkjoin map/scanline bench. (effort: M)
• [MEDIUM] lgz parallel gzip decompressor ships no bench while every sibling codec does — lgz/ (no benches/ dir); cf. lbzip2/benches(3), linflate/benches(1), ljar/benches(2), lzip-parallel/benches(1) — lgz is a "pure Rust parallel gzip decompressor" whose entire point is parallel speed, and the only codec crate with zero benches. Fix: add lgz/benches/inflate_bench.rs mirroring siblings, with a pigz/Z_FULL_FLUSH corpus (parallel path) and a standard gzip corpus (flate2 fallback), measuring throughput vs single-threaded flate2 across worker counts. (effort: M)
• [MEDIUM] osm-katana (12k loc parallel pipeline) has no criterion benches — osm-katana/ (no benches/ dir); ad-hoc bin/planet_bench.rs, bin/vtd_bench.rs — the heavy gatling consumer (XML scan, PBF decode/encode, parallel coord sort, Parquet write, optimize) has zero tracked benches; the two "bench" files are hand-rolled bins producing no comparable numbers. Fix: add osm-katana/benches with criterion per-stage hot paths (xml_reader scan rate, pbf_decomp/pbf_enc, optimize gatling end-to-end on a fixture) reusing the fixture builder; keep the bins as manual planet-scale drivers. (effort: M)
• [MEDIUM] .nornir docs reference a non-existent xtask bench harness — .nornir/stree32.md:105-135, .nornir/README.md:13 — documents xtask/examples/nornir-bench.rs registering Stree32Bench and cargo run --release -p xtask --example nornir-bench, but there is NO xtask crate or nornir-bench example in the repo (members: lbzip2, lgz, linflate, ljar, lzip-parallel, osm-katana, zstd-sys-rs). Fix: either add the xtask crate with the documented example, or correct stree32.md/README to point only at the real in-test commands (cargo test --release stree32_bench -- --nocapture). (effort: S)
• [LOW] stree (i64) perf benches are print-only #[ignore] tests with no perf assertion — src/stree.rs:1517 (bench_compare_all_methods), :1574 (bench_stree64_batch_variants), :1609 (bench_compare_1m_records_osm_ids) — they time variants but only eprintln throughput, so they cannot fail on regression; the u32 sibling (stree32.rs:919/937) already release-asserts speedup > 1.0/10.0, but the i64 OSM workload has no such guard. Fix: have the i64 bench tests return the measured speedup and add a release-only assert!(speedup > floor), or port to a tracked criterion bench. (effort: S)

docs-vs-code

• [MEDIUM] Dangling .nornir/stree.md reference in three places — the file does not exist — .nornir/README.md:12, README.md:14 (generated), src/stree.rs:42 — all three point readers to .nornir/stree.md "for the full benchmark and the comparison to Ragnar's published numbers", but only stree32.md exists; src/stree.rs:42 even attributes its B=8-vs-B=16 measurement to the missing file. Fix: recreate .nornir/stree.md capturing the STree64 benchmark, or remove/redirect the dead reference at all three sites (point at the in-code bench comment src/stree.rs:41-47); edit the .nornir/README.md source and regenerate README.md. (effort: S, easy-fix)
• [MEDIUM] README stree benchmark numbers contradict the figures in src/stree.rs — .nornir/README.md:11-12 (and generated README.md:13-14) vs src/stree.rs:41-44 — README claims batched find_exact ~19 ns/query in cache and ~66-96 ns/query at 1 GB (vs ~1050 ns binary search), but the committed B=8 numbers are ~40 ns/q at 8 MB and ~104 ns/q at 1 GB — roughly 2× the in-cache figure and outside the 1 GB range, and now unsourced since stree.md is gone. Fix: reconcile to one canonical set — update .nornir/README.md to the actual src/stree.rs figures (or record a newer measurement in code/test and cite it), then regenerate README.md. (effort: S)
• [MEDIUM] README omits the flagship gatling family and 6 other public modules — .nornir/README.md:5-16 (source of generated README.md) — README lists only vtd, stree, chunk_revolver, but lib.rs exports 10 public modules and Cargo.toml's description headlines "generic no-barrier streaming engine (gatling) + parallel sample sort (psort)"; gatling/gatling_forkjoin/gatling_io/psort/json/xml/stree32 are all absent — the marquee feature is undocumented at the front door. Fix: add README bullets for gatling (+forkjoin/io), psort, json, xml, stree32 matching src/lib.rs:3-22 and the Cargo.toml description; edit .nornir/README.md and regenerate. (effort: S)
• [MEDIUM] chunk_revolver documented as the slot pool "behind the streaming engine" but gatling.rs never uses it — .nornir/design.md:162-163 (and :49-56) — design.md describes chunk_revolver as "the zero-allocation slot pool behind the streaming engine … slot indices cross threads, never Vec clones", but gatling.rs implements its own slot management (InFlight.slot: Vec<u8>, raw data_ptr, src/gatling.rs:163-189, fill_slot :211) and chunk_revolver has zero in-crate uses — it is a standalone primitive consumed by external crates (znippy). Fix: reword design.md to state chunk_revolver is a standalone slot-pool primitive used by external consumers while gatling manages its own Vec slots — OR, if intended, make gatling.rs actually consume chunk_revolver. (effort: S)
• [LOW] stree32.md "warehouse harness" section references an xtask that does not exist in this repo — .nornir/stree32.md:107-110 & :135 — says "xtask/examples/nornir-bench.rs registers Stree32Bench" with NORNIR_RUN_HEAVY=1 cargo run --release -p xtask --example nornir-bench -- --only znippy.stree32_512mb, but this leaf repo has no xtask crate; the harness lives in the consumer (nornir) repo. Fix: mark the warehouse-harness paragraph as cross-repo (consumer/nornir), consistent with how design.md flags sibling-repo docs; the in-repo cargo test stree32_bench commands are correct and can stay. (effort: S, easy-fix)
• [LOW] gatling-guide TypedSink trait block omits the finish() method — .nornir/gatling-guide.md:208-211 vs src/gatling.rs:119-126 — the guide's TypedSink<T> shows only fn process(&mut self, out: T, is_last: bool), but the real trait also defines fn finish(&mut self) -> Result<()> { Ok(()) } (called once after all segments), so a reader won't know the post-stream flush hook exists. Fix: add the fn finish(&mut self) -> Result<()> { Ok(()) } line (with its comment) to the TypedSink block to match src/gatling.rs:125. (effort: S, easy-fix)

dead-lost-code

• [HIGH] Entire node_store_arrow module is orphaned (superseded Arrow-IPC node store) — osm-katana/src/node_store_arrow.rs (539 lines, declared osm-katana/src/lib.rs:6 pub mod node_store_arrow;) — a complete Arrow-IPC node store (ArrowNodeStore, ArrowCoordWriter, convert_soa_to_arrow/convert_flat_to_arrow, open/open_mmap/lookup/lookup_batch + its own 6 tests) that NOTHING in production references (only the pub mod decl + its own test block); the live path is node_store.rs CoordFileWriter::finalize → external_merge_sort_inner → coords_ipc ArrowMmap, which hand-rolls the same single-RecordBatch IPC layout. dead_code missed it because the fns are pub and reachable from its own tests. Fix: confirm coords_ipc fully replaces it (it does), then delete the module and remove the pub mod from lib.rs; if its pyarrow/DuckDB-readable + per-batch LZ4/ZSTD advantages are wanted instead, wire ArrowCoordWriter into CoordFileWriter and delete the coords_ipc path — but do not keep both. (effort: M)
• [HIGH] Dead disk-merge node-store path (MmapTree / from_mmap / merge_sorted_chunks / CoordFileWriter::push) — osm-katana/src/node_store.rs:31 (Inner::MmapTree), :55 (from_mmap), :200 (push), :745-905 (merge_sorted_chunks) — a whole pre-ArrowMmap K-way disk-merge path is dead: Inner::MmapTree never constructed, from_mmap only called from inside merge_sorted_chunks (:771/:901) which itself has no external caller, and CoordFileWriter::push superseded by the bulk write_raw sink; the live finalize uses external_merge_sort_inner (Inner::ArrowMmap). ~170 lines kept alive only by mutual reference. Fix: delete merge_sorted_chunks + its private helpers, NodeStore::from_mmap, the Inner::MmapTree{tree,mmap} variant, and CoordFileWriter::push; re-run cargo to clean up newly-dead STree64Mmap/merge helpers. (effort: M)
• [MEDIUM] reader.rs blob_coords and blob_node_records superseded by fused blob_nodes_and_coords — osm-katana/src/reader.rs:59 (blob_coords), :71 (blob_node_records) — both pub fn flagged never-used; they were the decoupled per-blob decoders now replaced by blob_nodes_and_coords (~:82), which fuses coord extraction + node-record encoding into a single protobuf parse for the Gatling PBF pass-1 codec. Fix: delete blob_coords and blob_node_records; confirm parse_node_coords/parse_nodes still have other callers and remove parse_node_coords if it was only used by blob_coords. (effort: S)
• [LOW] geometry.rs encode_point and is_closed_area_slice are dead (live variants are *_inline / non-slice) — osm-katana/src/geometry.rs:21 (encode_point), :43 (is_closed_area_slice) — cargo flags both never-used; the pipeline uses encode_point_inline ([u8;21], called from writer.rs ×3) and is_closed_area (the tag-map variant). Fix: delete encode_point and is_closed_area_slice. (effort: S, easy-fix) (note: conflicts with the zero-copy-alloc fix that wants to adopt is_closed_area_slice — resolve by either keeping+using the slice variant or porting its logic; do not blindly delete if adopting it.)
• [LOW] Never-read struct fields: chunk is_last, coords_ipc count, Pass2TypedSink include_rels — lbzip2/src/chunk.rs:39 (is_last), osm-katana/src/coords_ipc.rs:84 (count), osm-katana/src/xml_reader.rs:1951 (include_rels) — three fields constructed but never read (logic uses the local is_last param / passes count separately / gates via rel_writer.is_some()), misleading readers about live state. Fix: remove each unused field + its assignment; if intended for future read, add a doc note + #[allow(dead_code, reason=...)] instead of a silent warning. (effort: S, easy-fix)
• [LOW] Unused imports left in source (Write, NamedTempFile, lbzip2::block) — osm-katana/src/node_store.rs:949 (std::io::Write), osm-katana/src/node_store_arrow.rs:396 (tempfile::NamedTempFile), lbzip2/benches/stage_breakdown.rs:6 & lbzip2/benches/profile_stages.rs:7 (use lbzip2::block) — pure compiler-warning clutter that hides real warnings. Fix: delete the unused imports (or cargo fix); node_store_arrow.rs:396 disappears anyway if that module is deleted. (effort: S, easy-fix)
• [LOW] lbzip2 bench files registered as multiple build targets — lbzip2/Cargo.toml (decompress_bench.rs, profile_stages.rs, stage_breakdown.rs) — each is both explicitly listed under [[bench]] AND auto-discovered from benches/, so each compiles twice with warnings on every build. Fix: set autobenches = false (or remove the redundant explicit [[bench]] entries) so each bench is a single target. (effort: S, easy-fix)

Cross-ref: the dead-code audit independently flagged ljar as a near-verbatim copy of lzip-parallel (ljar/src/* vs lzip-parallel/src/*) — covered above under helper-reuse (JAR == ZIP twin crates).

Tier 2 — Excellence (algorithms/datatypes)

• [MEDIUM] Concatenated-gzip parallel decode uses foreign flate2 instead of the in-house linflate fast loop — lgz/src/speculative.rs:407-432 (line 426) — decode_concatenated_members is the parallel fast path for multi-member .gz, yet decodes each member with flate2::read::GzDecoder (zlib-ng/miniz) — but the crate exists for linflate's SIMD preload/burst loop, which its own bench (ljar/src/bin/bench_inflate_cmp.rs) reports beats zlib-ng; parallel-but-foreign leaves per-core speedup on the table. Fix: decode each member through linflate (strip the 10-byte gzip header + optional extras, feed raw DEFLATE to linflate's segment decoder, as lgz/src/chunk.rs already does), keeping flate2 only as the genuine error fallback. (effort: M)
• [MEDIUM] Inverse-BWT test asserts only output length, never the reconstructed bytes — lbzip2/src/bwt.rs:48-68 — inverse_bwt is the single most correctness-critical bzip2 step, but its only test feeds a contrived array and asserts output.len()==6 ("we just verify it doesn't panic") — the forbidden "didn't panic" anti-pattern; a wrong T-vector permutation still yields 6 bytes and passes. Fix: inject a real BWT output + orig_ptr for a known string (e.g. 'banana' → 'annb$aa' with its documented orig_ptr) and assert next_byte() reproduces the exact original bytes; add a property test that bwt-encode→inverse round-trips random inputs. (effort: S, easy-fix)
• [LOW] Streaming gatling fans work through Arc<Mutexmpsc::Receiver> instead of a lock-free MPMC channel — src/gatling.rs:342-357 (and run_typed ~:627-640) — units are distributed to N workers via mpsc::sync_channel whose single Receiver is shared as Arc::new(Mutex::new(work_rx)), so every worker takes a global lock to recv(); std mpsc is the slowest channel and the Mutex serializes dispatch, contradicting design.md's (:28,:35) "no central scheduler / self-dispatch" story that the forkjoin/psort variant honors with an atomic fetch_add. Impact is muted because decode units are coarse. Fix: replace the Arc<Mutex> work queue with a lock-free MPMC channel (crossbeam-channel or flume) — Sync multi-consumer receivers with no mutex. (effort: M)
• [LOW] Byte-mode gatling allocates a fresh Vec per decode unit, bypassing the slot-pool philosophy — src/gatling.rs:70 (Codec::decode → Vec), consumed :364-369 — every worker returns a newly heap-allocated Vec<u8> per segment and the collector concatenates them, so the output side allocates+frees one Vec per unit in the steady-state hot loop of an engine that ships chunk_revolver specifically to avoid this (codecs already use thread-local reusable buffers internally, then copy into the returned Vec). Fix: offer a decode_into(&mut Vec<u8>) / pooled-output Codec variant so workers write into a recycled buffer leased from a small per-worker pool (or chunk_revolver ring) returned after the collector consumes it. (effort: M)
• [LOW] Speculative decode materializes every segment as its own Vec then copies all into a combined buffer — lgz/src/speculative.rs:286-322 — decodes each segment into a separate Vec<u8> (pass 1), rewrites prefixes in place (pass 2), then allocates a fresh output and extend_from_slices every segment (:316-321) — a second full pass over the entire decompressed output plus peak RAM of (all segments + combined copy), ~2× peak memory for multi-GB decompressions. Fix: after pass 1 the per-segment lengths are known — allocate the final buffer once and have each worker decode/fix-up directly into its computed offset range via inflate_segment_with_prefix into a shared output slice, eliminating the concatenation copy. (effort: M)
• [LOW] AVX2 node compare uses movemask_epi8 + popcount/8 instead of the 4-bit movemask_pd — src/stree.rs:612-626 — count_lt_avx2 reduces each 256-bit _mm256_cmpgt_epi64 with _mm256_movemask_epi8 (32 bits, 8 duplicated per lane) then count_ones()/8, wasting the redundant-byte expansion and a divide on the hottest inner primitive of the search tree. Fix: cast with _mm256_castsi256_pd and use _mm256_movemask_pd (exactly 1 bit per i64 lane); OR the (B/4) masks and a single count_ones() gives the count directly — no /8, same latency. (effort: S, easy-fix)
• [LOW] run_ordered reorder buffer is a heap-allocating BTreeMap when a fixed ring would be O(1) alloc-free — src/gatling_io.rs:164-187 — out-of-order completions buffer in BTreeMap<usize,R> (node alloc + O(log n) per insert/remove), even though the buffer is provably bounded by max_in_flight and indices are contiguous from next_emit. Fix: replace with a Vec<Option<R>>/VecDeque of length cap indexed idx % cap; insert at idx % cap, drain while slot[next_emit % cap] is Some — O(1), zero per-item alloc. (effort: S, easy-fix)
• [LOW] OSM integer/coord parsing is a scalar per-byte digit loop, missing SWAR fast parse — src/vtd.rs:153-177 (parse_coord_e7) and src/xml.rs:169-181 (parse_i64) — id/lat/lon parsing walks one ASCII byte at a time with a per-byte is_ascii_digit branch + mul-add; at planet scale over hundreds of millions of matched elements this is the per-element decode cost, while simdjson/fast_float parse 8 digits at once with a couple of SWAR multiplies. Fix: add a SWAR 8-digit parser (load 8 bytes, subtract '0', masked-compare validate, fold with the 1+10+100… trick) for the common fixed-width fast path, falling back to scalar for short/odd lengths. (effort: M)
• [LOW] bzip2 Huffman long-code path bit-walks a binary tree instead of a second-level table — lbzip2/src/huffman.rs:131-157 — codes longer than FAST_BITS (10) fall to decode_slow, a bit-by-bit pointer-chase down nodes[node_idx][bit] (one dependent load + one bit read per bit, up to 20 levels) — the least cache-friendly option, though a rare path. Fix: replace the node-array fallback with a second-level lookup table (offset+sub-bits, same shape linflate already uses via HUFFDEC_SUBTABLE) so long codes resolve in one extra table read. (effort: M)
• [LOW] Library decode paths print progress with eprintln! instead of a log facade — lgz/src/speculative.rs:413; lgz/src/parallel.rs and lgz/src/bin/lgz.rs (multiple) — reusable library decode fns (not just the bin) emit eprintln! diagnostics (e.g. "lgz: found N concatenated gzip members"), giving embedding consumers unsolicited stderr spam they cannot route/silence, plus a lock+syscall on the decode path. Fix: route diagnostics through the log/tracing facade (or return structured stats), keeping raw eprintln! confined to the CLI binary. (effort: S, easy-fix)

Easy fixes (safe to apply now)

• .nornir/README.md:12, README.md:14, src/stree.rs:42 — remove/redirect the dead .nornir/stree.md reference (edit the .nornir source, regenerate README). (docs-vs-code)
• .nornir/stree32.md:107-110,135 — mark the xtask "warehouse harness" paragraph as cross-repo (consumer/nornir). (docs-vs-code)
• .nornir/gatling-guide.md:208-211 — add the fn finish(&mut self) -> Result<()> { Ok(()) } line to the TypedSink block. (docs-vs-code)
• osm-katana/src/geometry.rs:21,:43 — delete dead encode_point and is_closed_area_slice (but reconcile with the zero-copy fix that wants to adopt is_closed_area_slice). (dead-lost-code)
• lbzip2/src/chunk.rs:39, osm-katana/src/coords_ipc.rs:84, osm-katana/src/xml_reader.rs:1951 — remove the never-read struct fields (is_last / count / include_rels). (dead-lost-code)
• osm-katana/src/node_store.rs:949, osm-katana/src/node_store_arrow.rs:396, lbzip2/benches/stage_breakdown.rs:6, lbzip2/benches/profile_stages.rs:7 — delete unused imports. (dead-lost-code)
• lbzip2/Cargo.toml — set autobenches = false (or drop the redundant [[bench]] entries) so benches build once. (dead-lost-code)
• lbzip2/src/bwt.rs:48-68 — make the inverse-BWT test assert exact reconstructed bytes for a known string + add a round-trip property test. (tier2)
• src/stree.rs:612-626 — switch count_lt_avx2 to _mm256_movemask_pd (no /8). (tier2)
• src/gatling_io.rs:164-187 — replace the run_ordered BTreeMap reorder buffer with a fixed-capacity ring. (tier2)

Counts

• High: 14
• Medium: 22
• Low: 18
• Total: 54 (after de-duping the curl/unzip test-shelling finding and the ljar==lzip twin-crate finding across dimensions)