chunk_revolver — design
Class:
design· Function:chunk_revolver(the zero-allocation slot pool). Strict: chosen shape only.
Source: gatling/src/revolver/ — revolver.rs (the fixed-slot pool), coalesce.rs (the coalescing / ref-counted mode), ring.rs (a per-worker ring), mod.rs. Reachable as gatling::revolver, with the historical chunk_revolver name kept alive as a crate-root re-export alias.
What it is
The zero-allocation buffer pool behind the streaming gatling engine. A 1-reader / N-worker pipeline needs somewhere to put each chunk of input while a worker decodes it, then reuse that memory for the next chunk — without allocating a fresh Vec<u8> per chunk. ChunkRevolver owns a fixed set of equal-size slots arranged as per-worker rings; the reader borrows a free slot, fills it, a worker processes it as &[u8], and the slot is recycled. Steady-state allocations: zero.
It has two modes, chosen at construction and mutually exclusive by construction.
Fixed-slot mode (revolver.rs) — one payload, one whole slot
| Item | Role |
ChunkRevolver::new(chunk_size, total_chunks, num_workers) (:87) |
build the pool: total_chunks slots of chunk_size bytes, split across num_workers rings |
try_get_chunk() -> Option<Chunk<'_>> (:170) |
non-blocking claim of a free slot; None when all are in flight (backpressure) |
return_chunk(ring_nr: u8, index: u64) (:189) |
recycle a slot back to its ring once the worker is done |
Chunk<'a> (:23) |
a borrowed handle to one slot — the &[u8] a worker reads, no copy |
SendPtr (:8) |
a Send wrapper over *const u8 so a slot's base pointer can cross the thread boundary safely |
get_chunk_slice(base, chunk_size, index, used) (:216) |
the worker-side zero-copy view of used bytes of a slot |
base_ptr(ring_nr) / base_ptrs() / chunk_size() / num_rings() |
pool geometry the engine uses to place workers on rings |
Invariants:
- Fixed-size slots. Every slot is exactly
chunk_sizebytes; the reader fills a slot incrementally and hands the worker aChunkover the bytes actually read. - Disjoint ownership. A slot is claimed by exactly one reader fill and read by exactly one worker at a time; workers never share a slot, so no lock and no false sharing — the gatling "disjoint output" soul, applied to the input buffers.
- Ring recycling. Slots belong to per-worker rings;
return_chunkpushes the slot back to its ring, so the reader's nexttry_get_chunkreuses warm memory. - Non-blocking claim.
try_get_chunkreturnsNonerather than blocking when the pool is saturated — the caller (the gatling collector) uses that as the backpressure signal.
Coalescing mode (coalesce.rs) — many payloads, one shared slot
Fixed-slot mode is ruinous for a corpus of small files: a 4 KiB file burns a whole slot, so a memory budget buys budget / max_file_size files in flight. Coalescing mode packs many variable-width payloads into ONE slot and reference-counts the slot, so the same budget buys an order of magnitude more.
| Item | Role |
ChunkRevolver::with_coalescing(num_slots, slot_size, slice_size) (coalesce.rs:255) |
build the pool in coalescing mode; one Box<[u8]> per slot |
claim_slot() -> Option<Filler<'_>> (:314) |
blocking claim of a free slot — the pipeline's only backpressure |
try_claim_slot() (:324) |
the non-blocking twin |
Filler::writable(max) -> &mut [u8] (:381) |
the reader reads payload bytes directly into the slot at the cursor |
Filler::commit_slice(len, skip, file_index, fdata_offset, chunk_seq) (:393) |
close the bytes at the cursor as one slice and advance |
Filler::publish_into(&mut Vec<PackedSlice>) (:431) |
arm the slot's reference count, drain the slices into a caller-owned buffer, recycle the internal one — the hot-path publish, zero allocation |
Filler::publish() -> Vec<PackedSlice> (:448) |
the same, handing the buffer to the caller instead of recycling it |
PackedSlice (:66) + unsafe as_slice() (:94) |
one unit of work: a Send view into a published slot plus the packing metadata |
Returner::release_one(slot_id) (:153) |
a worker is done with one slice; the last release frees the slot |
returner() (:290) / release_one() (:299) / slice_size() / num_slots() / is_coalescing() |
the shared handle and the geometry |
Slot lifecycle: FREE → (reader claims) FILLING → (reader publishes) DRAINING → (workers release_one; the last one frees it) FREE.
Invariants:
- A slice ⊆ one payload AND ⊆ one slot, contiguous.
slice_sizeis the granularity at which the caller is expected to cut a payload bigger than one slice; the pool only records it, because for an archiver it is output-affecting (it lands in chunk boundaries and checksums) and so must be planned by the caller. - Reference-counted release.
publishstores the slice count withRelease;release_onedoesfetch_sub(1, AcqRel)and pushes the slot back on the 1→0 edge. The slot is not handed to a reader again until then, so the reader's&mutwindow and the workers'&window never overlap. - No double-issue.
with_coalescingdrains every ring free list into the coalescing free list, sotry_get_chunkon a coalescing pool can only ever returnNone. The two modes cannot hand out the same bytes. - Un-published drop returns the slot. A panic or an early
?in the reader loop loses the packed bytes but never retires a slot. - Flat free list. A
Mutex<Vec<u32>>+Condvar, touched once per slot — never per slice, never per byte. A coalescing slot has no ring affinity worth preserving: its consumers are N workers pulling off one global queue, not one worker pinned to one ring.
History
Coalescing mode exists because znippy forked this pool into znippy-common/src/slotpool.rs ("Magazine") to get it. Phase 1 of the consolidation (chunk_revolver-unsorted.md) brought the capability back here; Phase 2 — flipping znippy-compress/src/slot_packer.rs onto it and deleting the fork — is znippy-side work and still open.
Correctness is pinned by tests/chunk_revolver.rs (claim/return lifecycle, no double-issue, ring geometry) and tests/chunk_revolver_coalescing.rs (coalescing lifecycle, concurrent worker reads, un-published drop, no cross-mode double-issue). Throughput is pinned by benches/revolver_coalescing.rs.