nordisk/znippy-zoomies

Gatling — reasoning

Class: reasoning. The why, the trade-offs, and the road here. The descriptive shape we ship is in gatling-design.md, which by rule mentions none of this. If you want to know "why not X?" — X is here.

The cardinal sin — the "1-ends cook" bug

The disaster mode is doing heavy work on the reader or the collector instead of in the workers. There is only one of each, so the moment you sort/parse/assemble there, one core does everything and the pool starves. The kitchen picture (see gatling-for-idiots.md): the waiter and the plater never cook.

This is not hypothetical. It once dropped osm-katana from "all 32 cores" to ~3: the ends had been made to cook — read the whole 8 GB then sort it one-at-a-time, walk the whole file to find boundaries, assemble the final result at the pass. The cruel twist: while broken, the old rayon path looked faster, and people almost blamed gatling. Gatling was fine; we'd told the waiter to cook. Moving the cooking back into the workers restored a flat 100 % on all cores — even on the entire planet's map (10.58 billion points). The whole engine exists to make that failure mode structurally hard.

Why "no rayon" (the constellation law)

rayon is mature and ubiquitous, so the ban needs a reason. Its global thread pool is exactly what hid the osm-katana serialization: shared, implicit, and invisible when work lands on the wrong side of a fork-join barrier. std::thread::scope + one atomic cursor has no hidden global state, no barrier, and lets worker closures borrow caller stack data directly (no 'static/Arc). The crate carries no rayon dependency in its Cargo.toml and none in its shipped tree; the last two holdouts (src/json.rs NDJSON line-scan, src/vtd.rs sub-chunk parse) were ported off rayon::into_par_iter onto gatling_forkjoin::gatling_for_each, and on the json_scan bench the ported scan is if anything faster than the rayon version it replaced. The law is enforced by tests/rayon_free_law.rs. (The one honest exception — ljar/lzip-parallel's dedicated non-global pools, and criterion's dev-only transitive rayon — is documented in reasoning.md.)

Why balanced scheduling is a separate body, not the default

Plain gatling_for_each claims 0..n in order, one index at a time — optimal when units cost about the same, and cheaper (no weight computation, no batch bookkeeping). It is not optimal when a handful of units dwarf the rest: a monster claimed last leaves one worker grinding alone while its siblings drained the cheap tail and went idle (the classic long-pole). Rather than tax every caller with LPT machinery, the skew fix is a distinct body (gatling_for_each_balanced / gatling_map_balanced) you opt into with a real weight. Keeping the common case lean and the skew case explicit beats one always-balanced engine that pays for scheduling nobody needed.

Why gatling_map_balanced exists next to gatling_for_each_balanced

gatling_for_each_balanced schedules a bare 0..n. nornir's flattened knowledge scan doesn't hold a count — it holds a Vec of (member_idx, file, crate, section, size) tags unioned across every repo, and wants to map each to an owned per-file scan then group back by member_idx. Forcing that through …_balanced(tags.len(), …, |i| f(&tags[i])) works but pushes an i → tags[i] re-index into the hot closure and makes the weight close over the slice. The item-ref body (f(i, &item), size_of(&item)) reads one borrow, keeps distribution zero-copy, and let nornir delete the private gatling_map_balanced mirror it had carried and call the canonical primitive.

Why sync threads and an async sibling

The streaming engine is CPU-bound (one thread per core) — right for decode/parse. For I/O-bound work (S3 PUT/GET, fsync) the cost is per-op round-trip latency, not CPU, so a blocked OS thread is pure waste. gatling::io keeps every gatling invariant but swaps threads for ≤ max_in_flight tokio futures. Same soul, different substrate — chosen by whether the bottleneck is a core or a socket, not by taste.

The rayon regression + the fix (the real history)

The "no rayon" law was, for a while, asserted crate-wide but enforced narrowly — and the codecs slipped through the gap. The forensic record (crates.io tarballs + full git pickaxe):

• The codecs were rayon from first publish, never gatling. crates.io is immutable, so it settles it: lgz 0.2.0 (2026-05-21), lbzip2 0.1.0 (2026-05-15), ljar 0.1.0 and lzip-parallel 0.1.0 (2026-05-21) all shipped rayon = "1" at their earliest version. There is no gatling version of any codec to "restore" — the shared engine (znippy-zoomies::gatling, extracted from katana 2026-05-27) was only ever wired into json/vtd/the streaming compressor and osm-katana, never the codecs or skade.
• The partial port declared complete. 43c571a (2026-07-03, "Port json/vtd scanners off rayon; make the 'rayon-free' law true") touched only src/json.rs + src/vtd.rs and dropped the root Cargo.toml rayon dep — zero codec files. lgz/lbzip2 had independently moved to a private scoped-thread fork-join upstream (a hand-rolled clone of gatling_for_each, not the shared engine); ljar/lzip-parallel kept live OnceLock<rayon::ThreadPool> singletons — the exact global-pool shape that bit osm-katana.
• The law narrowed by decree, the guard baked to match. 769363f rewrote the law's prose from "no rayon … the whole crate" to "no rayon in the shipped, non-dev tree (only criterion's dev-dep left)", quietly reframing it around the root crate; c3d5512 then created tests/rayon_free_law.rs born scoped to src/ with an explicit written carve-out excusing ljar + lzip-parallel. So the guard was written to not notice the two shipped crates that still carried rayon.

The fix (what shipped here). The law is now real for the whole workspace, not the root alone. ljar and lzip-parallel were ported off their rayon pools onto gatling::gatling_forkjoin::gatling_for_each (their rayon deps dropped; lgz/lbzip2 folded their private clones onto the shared engine too — every codec now routes through the one pool). And tests/rayon_free_law.rs was widened from the src/-only scan to whole_workspace_is_rayon_free() (workspace_src_roots() walks every member's src/ + Cargo.toml), and its forbidden-token set now also bans private pools — rayon::ThreadPool and the pool builders — so a codec cannot quietly stand up its own pool without tripping red. The carve-out is deleted. This is why the law now holds crate-wide and not by footnote.

The writer story — ring-slots refuted, parallel writers win

The headline of this session. The codecs saturated cores very unevenly — lgz ~3.1, ljar ~3.5, lzip ~4.0 of 12, while lbz2 held ~10.4 — and the obvious first guess was slot starvation: widen ring_slots so more chunks are in flight. That hypothesis is REFUTED. Widening the ring moved nothing. The bottleneck is not the pipeline width; it is the single ordered writer/collector at the tail. Both the streaming (gatling::run) and fork-join engines funnel every decoded unit through one collector draining an mpsc and one writer thread — a serial drain that copies and writes gigabytes while N-1 cores idle. lbz2 hides it only because its per-block BWT/MTF/Huffman decode is heavy enough to dominate the serial tail; the DEFLATE codecs, whose inflate is cheap (see decompressors-design.md), are left writer-bound.

The real fix is to remove the single writer: let each worker write its own output. That is the parwrite layer — the same no-barrier, atomic-cursor, std::thread::scope pool, but with no collector and no writer thread on the critical path. Two disjoint-output shapes make it safe without ordering:

• Independent files (extract_entries_unordered) — ZIP/JAR members are independent, so each worker preads its entry from the shared archive and writes its own output file. No two workers touch the same path; no ordering needed.
• Independent regions of one stream (write_segments_positional / write_slices_positional) — the caller pre-computes each segment's byte offset (a prefix-sum of known output sizes), pre-sizes the file, and each worker pwrites its segment at its offset. Positional writes to disjoint regions never conflict, so exact stream byte-order is preserved with zero ordering on the hot path — the offset-pwrite breakthrough.

Measured cores_busy on the writer-lift branch:

The honest limits. lgz reaches only 7.8, not 11.5, and only on -o file output: a multi-member gzip stream still has a single-stream boundary-scan tail (find the next flush boundary before the offsets are known), and stdout cannot be offset-written — pwrite needs a seekable file, so a piped lgz input.gz | … falls back to the ordered writer. parwrite also assumes the output sizes are known up front (ZIP/JAR central directory, or a prior decode pass), which is why it is a sibling layer the codec opts into, not a silent replacement for gatling::run.

Why parwrite is a separate layer, not folded into gatling::run

gatling::run/run_typed guarantee ordered output through one collector — that ordering is the contract callers rely on (bz2→XML reassembly, deep_scan record order). parwrite trades that away: it only works when the output units are provably disjoint (independent files, or pre-offset regions of a pre-sized stream) so no ordering is needed at all. Baking disjoint-output assumptions into the ordered engine would either weaken its guarantee or bolt a mode-flag onto the hot loop. Keeping it a distinct body — the parallel-writer sibling of gatling_for_each, exactly as gatling_for_each_balanced is the skew sibling — keeps each engine's contract clean and lets a codec pick the writer that matches its output shape.

The road, briefly

Extracted from katana-osm into this leaf crate (see reasoning.md for the release-cascade rationale). The engines gained bodies as consumers needed them: the streaming byte/typed pair for OSM decompress→parse, the fork-join family for fixed work sets, the balanced bodies for nornir's skewed member-scan, the async sibling for znippy's I/O fan-out, gatling::ordered for znippy's io_uring batched small-file reader (produces (path, bytes), wants strict producer-order archive writes through a bounded reorder buffer), and finally the parwrite layer (landed on master) for the codec extraction/writer tail — the last un-parallelised stage, removed by letting every worker write its own disjoint output.