.nornir/gatling-tricks.md

gatling — the reusable list of tricks

Written 2026-08-03, from reading the whole engine (gatling/src/*.rs, published as rotaryengine 0.1.10) and from building ldeflate on top of it. Deliberately written independently of that compressor: this is the pattern list, not a design doc for one consumer.

Everything here is either read from the engine source or measured on oden. Where a commonly-repeated claim turned out not to hold, it says so — three of them did not.

The soul, in one line each

No barrier. A worker that finishes unit i does not wait for a sibling, a scheduler, or a stage boundary. It takes the next unit itself.

Zero copy. Bytes stay where the caller put them. Work items carry indices, ranges and pointers — never payloads.

Zero wait. Nothing on the critical path blocks on something that is not the work itself: not a lock hand-off, not a scheduler round, not an allocator.

Trick 1 — Self-dispatch off one atomic, not a work queue

The cheapest possible hand-off. gatling_forkjoin is, at its core, four lines:

loop {
    let i = next.fetch_add(1, Ordering::Relaxed);
    if i >= n { break; }
    f(i);
}

No channel send per unit, no central scheduler, no queue to drain. Ordering::Relaxed is correct here because the counter carries no data — the data hand-off is the disjoint index, and the scope join is the synchronisation edge.

Why it beats a static carve-up. Give each of 32 workers a fixed 1/32 of the range and the whole run is as slow as the unluckiest slice. With a shared cursor, a worker that draws cheap units simply draws more of them. On skewed unit costs — which is nearly all real work — this is the entire difference.

Where it lives: gatling_forkjoin::gatling_for_each, gatling_run, gatling_for_each_balanced, gatling_map_balanced, gatling_map_owned, gatling_scanlines, and parwrite's two entry points. Same four lines every time.

MEASURED (oden, 2026-08-03, 32 workers): one empty unit costs 62 ns at claim 1 and 28 ns at claim 16, over 1 M units. At 100 k units it is 67/32 ns; at 10 k units it is ~290 ns, because pool spin-up (~340 µs for 20 workers, per skade's own note in read.rs) has not yet amortised. The pool has a floor: below ~50 k units of trivial work, you are timing thread creation.

Trick 2 — Disjoint output slots, so no worker ever locks

Every fan-out in the engine pre-allocates one output slot per unit and lets each worker write only the slot it uniquely claimed:

let mut out: Vec<MaybeUninit<T>> = Vec::with_capacity(n);
out.resize_with(n, MaybeUninit::uninit);
// ... worker writes out[i] for its own i only ...
unsafe { Vec::from_raw_parts(out.as_mut_ptr() as *mut T, len, cap) }

No Mutex, no Arc<Mutex<Vec>>, no per-worker Vec + final concat + sort. And because slot i holds unit i's result, the output comes back in input order for free — no reduction pass, no re-sort, and a caller can index straight into it.

The soundness argument is always the same and always stated at the site: each index is claimed exactly once by the atomic, so the pointer writes never alias; the scope joins before anyone reads. The OutPtr<T> / RowBase<P> wrappers exist only to carry a raw pointer across the scoped-thread boundary, and both are Copy with a hand-rolled impl so they do not require T: Copy.

The edition-2024 gotcha, and it is a real one. Disjoint closure capture will grab the raw pointer field rather than the Send wrapper struct, and the code stops compiling — or worse, stops being sound. The engine works around it by rebinding the whole struct inside the closure:

s.spawn(move || {
    let out_ptr = &out_ptr;   // whole-struct capture, so the Send impl applies
    ...
});

If you copy this pattern anywhere, copy that line too.

Trick 3 — LPT: claim the heaviest units first

gatling_for_each_balanced / gatling_map_balanced sort a schedule of indices by descending caller-supplied weight, then hand out schedule positions. Results are still written to out[original_index], so the reordering is invisible.

This is longest-processing-time-first, and it fixes the specific failure where the last unit claimed is the most expensive one: a monster drawn at the end runs alone while 31 workers sit idle. Drawn first, its cost is hidden under everything else.

Two constraints that are not obvious. weight runs serially on the calling thread for all n units before any work starts, so it must be O(1) and side-effect-free. And a panic inside a worker leaks the Vec<MaybeUninit<T>> — there is no catch_unwind anywhere in the engine. Return Result, never panic.

Trick 4 — Batched claims, when units are cheap

claim (the batch parameter) takes k indices per fetch_add instead of one. It is the knob for when the atomic itself starts to cost more than a unit.

MEASURED, and the received wisdom is workload-specific. The house note says claim 1 is a 2× regression (from znippy-zoomies/src/stree.rs, on very cheap units). On empty units that reproduces: 62 ns → 28 ns, a 2.2× difference. But on a DEFLATE per object — where a unit costs microseconds — claim 1 / 4 / 16 measured within 0.3% of each other across 45 877 real git objects. The knob is real; its importance is entirely a function of how cheap your unit is. Do not carry the 2× across workloads.

Trick 5 — The two-queue split: coalesce the small, split the large

The shape ljar gestures at and that ldeflate implements: classify each item by size and put it in a different queue.

Items below a threshold are coalesced — one task drains many, so per-unit fixed costs are paid once per batch rather than once per item.
Items above a threshold are split — one item across many tasks, so a single monster cannot serialise a worker.
Everything between is one task, one item.

Be precise about what ljar actually does, because it is less than it looks. Its 10 MB constant (ljar/src/lib.rs:48) picks between two entry points for a whole archive, not a per-item size class. Its batch of 64 (ljar/src/reader.rs:20) is a RAM bound — "only one batch of decompressed output is live at a time" — not dispatch amortisation. And it dispatches one gatling unit per entry with claim 1 (parallel.rs:32, batch.rs:109); it never coalesces. So take the silhouette from ljar and build the small queue yourself.

The measurement that decides the threshold is not the one you expect. The obvious model is "coalesce until dispatch cost stops mattering". On a real workload the dispatch was never what mattered: 62 ns of gatling against ~4400 ns of per-object cost. The win came from reusing the expensive per-item object (a zlib compressor: reset 1567 ns vs new 4082 ns) inside the batch. Measure what the per-item fixed cost actually is before assuming it is the scheduler.

Trick 6 — 31+1: give the device its own core

N−1 workers compute; one thread does nothing but stream results to the device. Compute never blocks on the device, and the device never idles waiting for a compute round to end.

In this engine the shape falls out of gatling::ordered::run_ordered_sink, whose collector runs on the calling thread. So the caller's own thread is the writer, and asking for available_parallelism() - 1 workers gives 31+1 on a 32-core box literally.

Why one writer and not 32, on local storage. A write to NVMe is a memcpy-class syscall with no round-trip latency to hide; one uncontended writer already saturates the device and extra writers only buy lock, seek and allocator contention. This is documented reasoning, not folklore — skade/.nornir/s3-ingest-concurrency.md, "Why local stays sequential (and must)". The inverse holds for object storage, where the cost is per-op latency that only concurrency hides; that case wants gatling::io (the async sibling, bounded and ordered over tokio tasks), not this.

⚠ Correction: skade is NOT the reference implementation of 31+1

This trick is often attributed to skade "doing 200 M rows/s on 31 cores of Parquet while one core streams to NVMe". That is not in skade. Searched the whole repo including git log --all --grep: no "200 M", no "31+1", no "31 workers", and no cores - 1 arithmetic anywhere. Every call site asks for available_parallelism(), all cores.

What skade actually implements is N workers → barrier → 1 sequential writer: gatling_map_owned is a fork-join, std::thread::scope joins every worker before the results vec is read, and write.rs:944 says outright that "the encode phase holds all files in memory before the write phase". Encode and write are not overlapped in ingest_parallel or ingest_pipelined.

The "31" almost certainly comes from a cores-busy measurement — s3-ingest-concurrency.md:143 reports zstd:9 NVMe ingest at "31-32 / 32 cores busy, 97% CPU". That is 32 workers of which 31-32 were observed busy, not a 31-worker topology. skade's own highest rows/s figures are 92.7 M (uncompressed NVMe, and memory-bandwidth-bound at ~4/32 cores) and 103 M.

The 31+1 pattern is still right, and the engine supports it directly. It just is not copied from skade, and the number should not be quoted.

Trick 7 — Slot pool + carry: streaming without a copy

For data that arrives rather than already existing, gatling::run / run_typed fill pre-allocated fixed-size slots on a reader thread and hand workers raw pointers into them. Two details make it zero-copy in the common case:

Carry headroom. Each slot reserves space at its front, so an unconsumed tail from the previous chunk is written into the headroom and the worker sees one contiguous carry ++ read slice with no concatenation. When a single unit outgrows the headroom there is a correct fallback to a dedicated buffer — the headroom is a fast path, not a cap on unit size.
SlotFill::Incremental vs Big. Incremental grows a slot to exactly the bytes read and never faults the unused tail (a 5 MB file used to zero 6 × 232 MB before reading a byte); Big sizes once and reuses with no re-zeroing, which amortises to nothing when every chunk is full. Small inputs want the first, planet-scale streams the second.

The slot is held alive in the collector's InFlight until every segment of that chunk is done, which is what makes the workers' raw pointers sound.

Trick 8 — Recover order at the edge, never in the middle

Ordering is a sink property in this engine, never a barrier.

Fork-join: order is free (Trick 2 — disjoint slots are already indexed by input).
Streaming: a collector thread holds out-of-order results and flushes strictly by chunk_id, so workers never wait for their turn.
gatling::ordered::run_ordered_sink: a bounded reorder buffer plus a counting-permit pool primed with cap tokens. The producer must hold a permit to pull; the collector releases one after each in-order emit. That single counter bounds the sum over { work channel, workers, result channel, reorder buffer } — memory is O(cap), never O(stream length).
parwrite: order recovered by positional pwrite to pre-computed offsets. No collector at all, no serial concatenation — N workers write disjoint regions of one pre-sized file and the byte order is exact.

The choice to make: if output sizes are known in advance, use parwrite and delete the writer from the critical path entirely. If they are not, use the ordered sink and accept one writer (which Trick 6 says you wanted anyway on NVMe).

Trick 9 — Per-worker scratch, reused across every unit it claims

parwrite gives each worker a comp and an out Vec outside the claim loop, so after the first (largest) unit the hot loop allocates nothing. Cheap, and it composes with Trick 5: a coalesced batch is exactly the place to hold reusable per-worker state.

The general form: anything expensive to construct should be constructed per worker or per batch, never per unit. Measured instance: a zlib compressor is 4082 ns to build and 1567 ns to reset, against ~95 ns of actual coding for a small object.

Trick 10 — `background::Job` is the one sanctioned raw thread

The rayon-free law forbids thread::spawn outside the engine, which leaves producer / consumer overlap (as opposed to fan-out) with nowhere to go. background::Job is that home: one closure, one thread, join for the result. Depth-1 pipeline, not a pool. Use it for a sampler, a spill-writer, a double-buffer — and nothing that wants work stealing.

How the law is actually enforced (and it works)

tests/rayon_free_law.rs scans the root crate's src/ plus every member's src/, read from the members = [...] line of the root Cargo.toml — so a newly added crate is covered automatically, with no carve-out to remember. Four checks: no rayon tokens, no private pool (thread::spawn / thread::Builder / crossbeam / ThreadPoolBuilder), no reinvented thread::scope outside the engine, and no manifest declaring rayon.

Correction, verified 2026-08-03 on main at 0e2fec4: this guard is often described as broken — "it include!s a missing file and cannot compile". It compiles and all four tests pass. There is no include! in the file. Verified further by injecting rayon::current_num_threads() into a new member crate: the guard went red and named the exact file and line. It is live and it does see new members.

The three things this list corrects

skade is not the 31+1 reference (Trick 6). The pattern is sound; the attribution and the 200 M number are not.
ljar does not coalesce (Trick 5). It has the silhouette and none of the small queue.
The claim-size 2× is workload-specific (Trick 4), and the rayon guard is not broken.

If a fourth is needed: the dispatch cost is usually not your problem. Measure the per-item fixed cost first. Twice now the answer has been "the expensive object you rebuild per item", not the scheduler.