nordisk/znippy

PLAN — bound the compress slot-pool memory (Njord backup, issue #23)

Status: §1 IMPLEMENTED (2026-07-22, oden) · §2 (disk-backed slots) still proposed

2026-08-25 — the pool moved out from under this document. Everything below that names Magazine / slotpool.rs describes a fork znippy no longer owns: the slot pool is now gatling::revolver in coalescing mode (znippy_zoomies::chunk_revolver), and only the CUT POLICY stayed, as znippy-common/src/poolplan.rs. PoolPlan::plan and every bound §1 describes are unchanged and still hold — read Magazine::new as ChunkRevolver::with_coalescing and slotpool.rs as poolplan.rs throughout.

§2 is UNAFFECTED and still open. Disk-backed slots change where a slot's bytes live, not how slots are claimed, packed or released, so the collapse neither blocks nor advances it — but it must now be built behind the upstream Filler/PackedSlice surface, and therefore belongs upstream too, so no consumer changes.

What landed

The 1.6 GiB unconditional reservation is gone. slot_packer.rs no longer holds SLOT_SIZE/NUM_SLOTS consts; each pass plans its pool with slotpool::PoolPlan::plan(input_bytes, num_workers, budget), which reserves the smallest of three bounds — the old default (never more than 8 × 200 MiB), the input (ceil(input / slice_size) slices; a 5 MiB tree reserves 7 MiB, not 1.6 GiB), and the budget (common_config::slot_pool_budget_bytes(): ZNIPPY_SLOT_POOL_BYTES, else 50 % of the cgroup headroom, else 50 % of host available memory). Floor: one slot of one slice, so the pipeline always makes progress.

Deviation from the plan below, and the reason: the plan proposed shrinking slot_size, from which Magazine::new derives slice_size. slice_size is output-affecting — it is the big/small partition threshold and the big-file cut length, so it lands in chunk_seq, fdata_offset and every chunk checksum. Shrinking it under memory pressure would have made the archive depend on how much RAM the pod had. So Magazine gained with_slice_size/from_plan, slice_size stays pinned at 200 MiB / num_workers regardless of the plan, and only the reservation moves. slot_size × num_slots affects nothing but how many slices coalesce per buffer and how many buffers are in flight; the ordered sink writes in producer order either way.

Measured on oden (32 cores, 29 workers → slice_size 7 231 558 B), debug build:

No fat-path regression by construction: when neither the input nor the budget binds, plan early-returns PoolPlan::default_for(num_workers) — the same 8 × 200 MiB the consts gave — as the 2.4 GB row shows.

Still open after this. The slot pool is no longer the RAM floor, but it was not the only one: the 765 MB run still peaks at ~1.13 GB with a 14 MB pool. The residual is the per-worker codec state — 29 gatling workers × a zstd level-19 context (CONFIG.compression_level = 19) plus zstd_output_buffer_size scratch. On a small pod max_core_in_flight scales down with the core count so this shrinks with it, but if a backup pod is still tight, that is the next term to bound, not the slot pool.

Original plan follows.

Status (original): proposed (not yet implemented) Driver: Njord issue #23 "Bygg en stabil IT-arkitektur 2.0". The self-built backup/restore (tillsynia-admin) seals its staging tree with znippy as a lib (znippy_compress::compress_dir). Ops flagged that the backup "has previously needed a lot of memory because the snapshot is handled in memory". The Njord side is now fully streaming (S3 blobs → disk, DB COPY chunked, uploads multipart), so the remaining RAM floor is znippy's slot pool.

The problem

znippy-compress/src/slot_packer.rs hardcodes:

const SLOT_SIZE: usize = 200 * 1024 * 1024; // 200 MiB
const NUM_SLOTS: usize = 8;

Magazine::new (znippy-common/src/slotpool.rs) allocates the whole pool eagerly, up front, regardless of input size:

.map(|_| vec![0u8; slot_size].into_boxed_slice())  // × num_slots

So every compress_dir call reserves 8 × 200 MiB = 1.6 GiB of heap before touching a single file, plus per-worker zstd/OpenZL scratch buffers. On a backup pod with a modest memory limit (e.g. 512 MiB–1 GiB request) this OOM-kills or forces a large, wasteful reservation. The pool size is independent of the data volume — a 5 MiB staging dir still reserves 1.6 GiB.

Crucially, StrategicConfig (znippy-common/src/common_config.rs) already computes max_mem_allowed and min_free_memory_ratio from detected RAM — but SLOT_SIZE/NUM_SLOTS ignore it. The knobs exist; the slot pool just doesn't read them.

Goals

1. Cap slot-pool RAM to a caller/config-chosen budget (target: a backup can run comfortably under ~256 MiB total).
2. Offer a disk-backed slot mode ("use tmp disk rather than RAM") for memory-constrained hosts, trading throughput for a tiny RSS.
3. No throughput regression on the fat/bench path (Loki, Odin) when memory is plentiful — the current 1.6 GiB behaviour stays the default there.
4. Preserve the zero-copy skip/incompressible path and the two-pass (big/small) design.

Proposed changes (in priority order)

1. Make the pool config-driven (cheap, high value)

• Add slot_size: usize and num_slots: usize to StrategicConfig, derived from max_mem_allowed with a hard floor/ceiling. Default keeps today's 200 MiB × 8 when RAM is plentiful; under a cgroup limit it shrinks so slot_size × num_slots ≤ max_mem_allowed × (1 − min_free_memory_ratio).
• slot_packer.rs reads CONFIG.slot_size / CONFIG.num_slots instead of the two consts. slice_size math is unchanged (slot_size / num_workers).
• Env overrides for ops: ZNIPPY_SLOT_SIZE (bytes/MiB suffix) and ZNIPPY_NUM_SLOTS, parsed in common_config. This lets the Njord backup pod set e.g. ZNIPPY_SLOT_SIZE=32MiB ZNIPPY_NUM_SLOTS=2 → 64 MiB pool.
• Note: the big/small partition threshold is slice_size = slot_size / num_workers. Shrinking slot_size pushes more files into the big pass (sequential chunked reads — already bounded memory), which is exactly the behaviour we want when RAM is tight. Verify the big pass's per-worker chunk buffer is itself ≤ slice_size.

2. Disk-backed slots (tmpfs/tmp-disk mode)

• New SlotBacking enum in slotpool.rs: Heap (today) | MmapFile(dir).
• MmapFile mode mmaps each slot from a file under a caller-supplied dir (ZNIPPY_SLOT_DIR, default $TMPDIR). Pages are backed by the page cache and can be evicted under pressure → RSS stays low, spillover goes to disk instead of OOM. Use memmap2 (already in the constellation? confirm; else add).
• compress_dir gains an optional SlotBacking param (or reads ZNIPPY_SLOT_DIR); Njord's backup points it at the pod's ephemeral /tmp (or a mounted scratch PVC), matching the constellation build-target doctrine (heavy work on fast scratch disk, not RAM).
• Keep zero-copy: pwrite-from-slot still works from an mmap'd region.

3. Report the reservation

• CompressionReport (or a log::info!) states the chosen slot_size × num_slots and backing mode, so a backup run records its actual memory envelope — feeds the issue #23 requirement to "document actual RPO/RTO and resource use after each drill".

Tests (constellation testable-code law)

• slotpool: a unit test asserting Magazine::new(n, s, w) allocates exactly n slots of s bytes and that a config-shrunk pool reports the smaller size (red-when-broken: hardcode reintroduced → size wrong).
• common_config: env-override parse test (ZNIPPY_SLOT_SIZE=32MiB → slot_size == 32 * 1024 * 1024), and the max_mem_allowed clamp (pool ≤ budget).
• slot_packer: seal a small dir with a tiny pool (num_slots=2, slot_size=1MiB) and assert the archive round-trips (decompress → identical tree) — proves correctness is independent of pool size. Extend to MmapFile backing once (2) lands.
• Bench guard: a HEAVY bench row before/after on Loki confirming no throughput regression at the default (plentiful-RAM) pool size.

Out of scope

• Changing the Arrow-IPC index format or the on-disk archive layout — this is purely about the transient compress-time buffers.

Njord side — already done (no znippy dependency)

tillsynia-admin now streams everywhere it previously buffered: ObjStore::get_to_file / put_file / get_stream (multipart up, chunked down), restore DB load via chunked COPY FROM STDIN, and the admin-server snapshot up/download endpoints stream instead of read-into-Vec. That removes the per-object and per-bundle RAM spikes; this plan removes the last floor.