ldeflate
Pure-Rust parallel DEFLATE/zlib compressor — the write side of linflate.
linflate reads DEFLATE fast. This writes it fast. It exists because serving or repacking a git packfile means DEFLATE-compressing thousands of objects, and before this crate there was no compression code anywhere in the znippy-zoomies workspace.
use ldeflate::{compress_objects, Config};
let objects: Vec<&[u8]> = vec![b"tree 42\0...", b"blob..."];
let streams = compress_objects(&objects, &Config::default())?;
// Each element is a complete RFC-1950 zlib stream. `git` reads them verbatim.
# Ok::<(), ldeflate::DeflateError>(())What is parallel
Not the entropy coder — that is zlib-rs, reused unchanged. The scheduling is new, and it has two axes chosen per object by size, on one no-barrier gatling pass:
- small objects are coalesced into batches, so one task drains many and the expensive per-stream compressor setup is paid once per batch rather than once per object;
- large objects are split at
Z_FULL_FLUSHboundaries, so a single big blob cannot serialise a worker.
Git objects are 93.8% under 64 KiB (measured, 48 917 real objects), so they take the first path. LFS objects are megabytes, so they take the second. The same Config serves both.
Guarantees
- Stock
gitaccepts the output.tests/stock_git.rsbuilds a repository whose entire object store this crate wrote and runsgit fsck --full --strictover it, then assembles a real pack-v2 and runsgit index-pack --strict. Both have negative controls proving the checks can fail. - Pure Rust.
flate2is pinned todefault-features = false, features = ["zlib-rs"]. No C, no cmake, no bindgen, nobuild.rs, no build-time network — asserted mechanically bytests/no_c_toolchain.rsagainst the resolved dependency graph. - Deterministic. Byte-identical output at any worker count: the work plan is a pure function of the object sizes and the
Config, and never reads the core count. - rayon-free (constellation LAW 3). All threading routes through
gatling.
Measured, oden (Threadripper PRO 3975WX, 32 cores), 2026-08-03
| parallel cost, ~49 k real git objects, ~1.68 GiB, level 6 | 1.025×–1.054× the CPU of serial at 32 threads — n=7 runs, reproducible to 2.8% |
| efficiency vs cores actually held | 94.9%–97.6% at 32 threads, over occupancies from 10.9 to 24.3 of 32 cores held |
| sustained | 2136 MB/s across 39 consecutive passes, median 100 ms window fully saturated |
| ratio cost of coalescing | zero — not one output byte changes |
| ratio cost of splitting | 2.2% at the default 1 MiB segment, only on objects ≥ 4 MiB |
| skipping already-compressed input | ≈20× less CPU (94.8%–95.6% saved) for +0.41% size, on 24 real packfiles |
Why no headline “×N faster” figure. There was one — 28.30× at 32 threads — and a 7-run re-measurement on 2026-08-03 retired it as a crate claim: over those runs the 32-thread speedup ranged 10.38× to 23.20× purely with how many cores the box would give us, while the CPU cost of the schedule held to 1.025–1.054×. 28.30× was never contradicted (it is consistent with, and slightly conservative against, the re-runs), but it is unreproducible on a shared machine, so it is not quoted here as a property of the code. On a genuinely idle 32-core box the re-measured overhead predicts 30.4×–31.2×; that prediction is untested — oden has not been idle. See Finding 4b in the design doc.
Full tables, the two hypotheses these sweeps falsified, and the honest caveats: ../.nornir/ldeflate-design.md. The engine patterns it builds on, written down independently: ../.nornir/gatling-tricks.md.
When NOT to use it
- Serving a clone. The bytes are already deflated in the pack the client pushed; copying them is a
memcpyand no compressor beats that. - Large already-compressed content (most LFS objects: video, images, archives). Level-6 DEFLATE measured a compression ratio of 1.000 on high-entropy data — literally nothing — while running 42× slower than a SHA-1 pass over the same bytes. Route that content to store-raw before it reaches a compressor.
License
MIT OR Apache-2.0