Bench plan — one table, every crate on its own file types
Every crate is benched on real files of the types it reads, against a reference tool, by this repository's own bencher. README speed numbers come only from that bencher, on real data, dated, with wall time, MB/s, cores busy and peak RSS beside each row. Look up your file type in the first column.
| file type | crate | corpus | reference | state |
| .tar.gz, .tgz, .gz, OCI tar+gzip | lgz | real OCI layers by digest (nginx, postgres, python, clickhouse — benches/corpora/oci-layers.json), pigz -i files |
gzip -d, pigz -d, rapidgzip, flate2 | wired: benches/oci_layers.sh → benches/oci_table.py |
| raw DEFLATE (inside .gz, .zip, zlib, .png) | linflate | streams pulled out of the corpora below | zlib-rs (flate2), libdeflate | follow-up |
| writes raw DEFLATE, zlib | ldeflate | git objects, OCI layer tars | zlib-rs, libdeflate, pigz | follow-up |
| .jar, .war, .ear | ljar | real Maven jars | unzip, the JDK's jar | follow-up |
| .zip | lzip-parallel | real zips (e.g. GitHub release archives) | unzip, 7z | follow-up |
| .zip, .jar | znippy-zip-core | covered by ljar and lzip-parallel | — | — |
| .tar.bz2, .tbz2, .bz2 | lbzip2 | real .tar.bz2 (kernel or source tarballs) | bzip2 -d, lbzip2 (C), pbzip2 | follow-up |
| .tar.zst, .zst, OCI tar+zstd | zstd-sys-rs | real zstd layers or tarballs | zstd -d | follow-up |
| .osm.pbf, .osm → .parquet | osm-katana | a Geofabrik region extract | osmium | follow-up |
Each row of a result table: date, host, wall time, MB/s, cores busy, peak RSS. Corpora are fetched by digest at bench time and never committed.
Running the lgz rows
python3 benches/fetch_oci_layers.py benches/corpora/oci-layers.json /path/layers
podman build -t localhost/lgz-bench:1 benches/bench-image # rapidgzip, pigz, gzip
cargo build -p lgz --release --examples
LAYERS=/path/layers OUT=results.jsonl BIN=target/release/examples/layerbench sh benches/oci_layers.sh
python3 benches/oci_table.py results.jsonl # the README rowsThreads 1, 4 and all cores; lgz also at chunk sizes 50, 200 (the pipeline's slot) and 800 MB.
Read windows: one shared plan
Every parallel decoder sizes its reads from gatling::window_plan (segment = clamp(S / (N × k), min_seg, max_seg), read window = N × segment, ramp from N × min_seg doubling, small inputs read whole, unknown size ramps to N × max_seg). lgz uses it (1–50 MB segments, k = 4). Follow-up: lbzip2 (BZ2_CHUNK_SIZE, lower min_seg for ≤ 900 KB blocks), ljar, lzip-parallel and osm-katana (RAW_XML_CHUNK_SIZE, RAW_CHUNK_SIZE, PBF_CHUNK_SIZE) move to it. The bench rows carry k, min_seg and the time to the first output byte.