nordisk/znippy-zoomies

Gatling — design

Class: design · Function: gatling (the no-barrier engine). Strict: this file states only the shape we ship. The why (the rayon ban, the "1-ends cook" bug, the road here) lives in gatling-reasoning.md; the how-to is gatling-guide.md; the plain-words intro is gatling-for-idiots.md.

Source (leaf gatling crate, re-exported by znippy-zoomies): gatling/src/gatling.rs (streaming), gatling/src/gatling_forkjoin.rs (fixed set), gatling/src/gatling_io.rs (async I/O), gatling/src/gatling_ordered.rs (ordered item sink), gatling/src/parwrite.rs (parallel writers).

The shape every variant shares

One idea in several bodies:

1. No barrier. There is never a "submit all, then await all" stall. The instant a worker finishes a unit it claims the next — it does not wait for a slow sibling at a chunk boundary. An N-core box stays N-busy even when units cost wildly different amounts.
2. Self-dispatch via one atomic. A single shared AtomicUsize hands out the next index with fetch_add — no central scheduler, no per-unit channel send, the cheapest 1→N hand-off.
3. std::thread::scope substrate. The sync engines are pure scoped threads; the async sibling is tokio tasks. Scoped threads let worker closures borrow caller stack data directly — no 'static/Arc ceremony.
4. Zero-copy, zero-contention output. Each unit owns a disjoint slice of the output (a pixel row, an index slot, a byte range in a slot ring). Workers write in place — never sharing a cache line, never locking.

The reader only reads; the collector only stitches in order; all heavy work is in the workers in the middle.

Two families

A. Streaming — work arrives a chunk at a time (gatling.rs)

Reader → split → N workers → collector → sink. The caller plugs in a codec (how to split + transform) and a sink (how to consume); the engine carries no format knowledge.

• Byte mode — run + Codec/Sink: workers emit raw bytes; the collector concatenates and cuts at a Sink::safe_end boundary (e.g. end of an XML element).
• Typed mode — run_typed + TypedCodec/TypedSink: split guarantees logical boundaries so workers emit finished records; the collector forwards each in order, no reassembly.

Tuned by Config { chunk_size, carry_headroom, ring_slots, initial_carry, slot_fill }. ring_slots is the steady-state width; SlotFill::Incremental (default) faults only the bytes read, SlotFill::Big pays the full slot fault once for planet-scale inputs.

B. Fork-join — the work set is known upfront (gatling_forkjoin.rs)

No reader, no carry: a fixed 0..n range (or a row grid) and N workers draining a shared counter. The family you reach for when you already hold a Vec/grid/file list.

C. Async I/O sibling — latency-bound fan-out (gatling_io.rs)

For I/O-bound work (PUT/GET/fsync) a blocked thread is wasted. gatling::io keeps every invariant (no barrier, bounded N-in-flight, ordered output) but swaps the substrate to ≤ max_in_flight tokio futures. Payloads are bytes::Bytes, moved in and out, never deep-copied.

D. Parallel-writer — the writer itself fans out (parwrite.rs)

The families above (and the codecs built on them) funnel decoded output through one collector + one writer thread — the last un-parallelised stage. parwrite removes it: the same no-barrier, atomic-cursor, std::thread::scope pool, but each worker writes its own output — no collector, no writer thread on the critical path. It is safe without any ordering because the output units are provably disjoint:

• Independent files — extract_entries_unordered: each worker preads its entry's compressed bytes from the shared input (thread-safe positional reads, no shared file offset), decodes, and writes its own output file (ZIP/JAR members are independent).
• Independent regions of one stream — write_segments_positional (decode then pwrite) / write_slices_positional (already-decoded slices, pwrite): the caller pre-computes each segment's offset (prefix-sum of known output sizes), pre-sizes the file, and each worker pwrites its region at its offset. Positional writes to disjoint regions never conflict, so exact stream byte-order holds with no ordered collector.

Each worker reuses its scratch buffers across the units it claims (zero-alloc hot loop), never copies input into the pool, and is rayon-free. This is the family a codec reaches for when output sizes are known up front (a central directory, or a prior decode pass) and ordering can be dropped — see the writer story in gatling-reasoning.md.

The variants

The balanced bodies (skew killers)

gatling_for_each_balanced(n, n_workers, batch, weight, f) adds two refinements while keeping 0..n output order: a heaviest-first (LPT) schedule from a caller-supplied weight(i) (e.g. file byte size), and batched self-dispatch (claim batch indices per atomic fetch_add; batch == 0 ⇒ 1). gatling_map_balanced(items, n_workers, batch, size_of, f) is its sibling for an owned Vec of heterogeneous tags: size_of(& item) and f(i, &item) both read one borrow (no external re-index), results land in disjoint output slots in original order, ready to group-by-key. First real consumer: nornir's member-scan parsing every .rs across all cores scheduled by byte size.

Whereabouts map — where every piece lives

Every engine variant, codec, and the law test, with a file:line you can jump to. The engine lives in the leaf gatling crate; the codecs are workspace members; all are re-exported / path-depended by the root znippy-zoomies crate.

parwrite (the writer-lift) is landed on master alongside every other variant.

Gatling-adjacent primitives (same crate, same no-rayon law)

• chunk_revolver — the zero-allocation slot pool behind the streaming engine (chunk_revolver-design.md).
• psort — parallel sample sort over 16-byte AoS records keyed by a leading i64.
• stree/stree32 — the AVX2 static search tree (stree-design.md).
• xml/vtd/json — zero-copy parallel record scanners; all three parallelize through gatling_forkjoin::gatling_for_each.