.nornir/compress-guide.md · nordisk/znippy

The compress engine — guide

Class: guide · audience: a techie compressing a directory or a stream with znippy. The what is in compress-design.md; the why in compress-reasoning.md. This file is the workflow.

The Gatling flow

f0 — zero to an archive (the whole workflow)

Three moves take a directory to a seekable .znippy:

Build. cargo xtask build --suite ORIGINAL --atom znippy — the ONE build door, which reads .nornir/build-variants.toml. (znippy-cli is the LIBRARY that holds the code; the binary is znippy, and since 2026-08-03 it is the only one.)
Compress. target/release/znippy compress -i ./mydir -o mydir.znippy — this drives Commands::Compress (znippy-cli/src/lib.rs:299), which calls znippy_compress::compress_dir (znippy-compress/src/slot_packer.rs:88).
Verify. znippy list mydir.znippy — every original file appears, big ones as several chunk_seq-ordered rows sharing one relative_path.

That is the Gatling directory path end-to-end: io_uring reader → 200 MB slots → N workers → one finalizer → Arrow index.

f1 — the compress CLI

znippy compress -i <dir> -o <out.znippy> \
  [--no-skip] \
  [--format rust|python|maven] \
  [--meta-format arrow-ipc|iceberg] [--warehouse DIR] \
  [--sign KEY --sign-cert CERT --sign-alg p256|ed25519]

flag	effect
`-i <dir>`	input directory to pack
`-o <out.znippy>`	output archive
`--no-skip`	compress every slice — disables the "store-raw for `.jar`/`.gz`/`.crate`/`.png`/`.parquet`" skip rule
`--format rust\|python\|maven`	ecosystem-aware layout/ignore profile for the input tree
`--meta-format arrow-ipc\|iceberg`	container tail backend; `iceberg` needs `--warehouse DIR`
`--sign KEY --sign-cert CERT --sign-alg p256\|ed25519`	sign the archive

f2 — driving the engine from Rust

Directory in, archive out:

use std::path::PathBuf;
znippy_compress::compress_dir(
    &PathBuf::from("mydir"),
    &PathBuf::from("mydir.znippy"),
    /* no_skip = */ false,
    /* plugin  = */ None,
)?;

In-memory / streaming (HTTP, holger) over Arc<Vec<u8>> rounds — same worker/writer/finalizer shape as the directory path:

// default inline ArrowIpcSink:
let mut c = znippy_compress::compress_stream(&PathBuf::from("out.znippy"), /* no_skip = */ false)?;
// …or choose the metadata backend (e.g. an Iceberg sink) via a factory:
let mut c = znippy_compress::compress_stream_with_sink(
    &PathBuf::from("out.znippy"), false, Some(sink_factory))?;

f3 — codec and store-raw behaviour

Compression is Codec::compress_into(&mut self, input, out) (znippy-common/src/codec.rs:42) — OpenZL zstd/lz4 into a reused out buffer. You get store-raw automatically in two cases (index compressed flag = false): an already-compressed extension on the skip path, and any slice whose frame ends up ≥ its input. Pass --no-skip when you want the extension skip disabled and every slice pushed through the codec anyway.

f4 — picking the metadata backend

Default is inline Arrow IPC (ArrowIpcSink, znippy-common/src/meta_sink.rs:73): sub-indexes + lookup + fst trie + manifest + ZNPYMIDX footer, all appended after the blobs in the .znippy. --meta-format iceberg --warehouse DIR swaps in a warehouse-backed sink via the MetaSinkFactory (meta_sink.rs:43) without changing the blob layout. Both share the compress engine unchanged.