mikronova/openzl-sys-rs

openzl-sys-rs

Low-level Rust bindings to OpenZL, linked statically.

The C sources are vendored

Building this crate needs a C compiler and nothing else:

• no network access,
• no curl, no tar, no git, no cmake — build.rs spawns no process at all,
• no C++ toolchain (the openzl target is pure C; 0.2.0's -lstdc++ was never needed).

OpenZL's C sources, plus the exact zstd 1.5.7 and lz4 1.10.0 releases OpenZL pins, live under openzl/. build.rs compiles them with the cc crate, reproducing the flags OpenZL's own CMake used. See openzl/VENDOR.md for provenance, upstream tarball hashes, licences and the exact build configuration.

Up to and including 0.2.0 this crate downloaded a pinned facebook/openzl tarball with curl at build time and drove cmake, which in turn fetched zstd and lz4. That made the crate unbuildable offline, in a sandbox, or anywhere a build is not allowed to reach the network. 0.3.0 removes all of it. The compiled library is unchanged: same OpenZL commit, same zstd, same lz4, same compile flags, and compressed output is byte-for-byte identical to 0.2.0's.

Which OpenZL is inside

The wrapper's version is not OpenZL's version. From 0.4.0 the crate version names the bundled release as build metadata: 0.4.0+openzl.0.3.0.

OpenZL 0.3.0 writes format 27 unless told otherwise, and a 0.3.x-wrapper reader cannot decode that. A writer whose output must stay readable by older readers sets ZL_CParam_formatVersion to 25 explicitly. 0.4.0's bindings follow OpenZL 0.3.0's API renames (ZL_MaterializerDesc2 → ZL_MaterializerDesc, registerBruteForceSelectorGraph → buildBruteForceSelectorGraph); see openzl/VENDOR.md.

An allocation budget for untrusted frames (0.4.2)

OpenZL sizes its internal stream buffers from the frame's graph, not from the decompressed size in the frame header: a 36-byte frame can make it allocate and fill a gigabyte before it fails. with_alloc_budget(limit, || ...) runs a closure with a per-thread budget that every allocation of OpenZL's allocation layer (ZL_malloc, ZL_calloc, ZL_realloc, the arenas' raw allocator) is charged against; one that does not fit fails as an OpenZL allocation error. The charge is cumulative (frees give nothing back). A thread without a budget pays one thread-local read per allocation. The vendored allocation.c is unchanged: csrc/zlsys_allocation.c includes it with malloc/calloc/realloc renamed.

Using a pre-built library instead

Set OPENZL_SYS_LIB_DIR to a directory containing libopenzl.a, libzstd.a and liblz4.a to link those and skip compiling the vendored tree. The build script says so on stderr, and fails loudly if any of the three is missing. The allocation budget is then NOT enforced (alloc_budget_enforced() is false).

Licence

The crate is BSD-3-Clause. It redistributes vendored C sources under BSD-3-Clause (OpenZL; zstd, whose BSD option is elected) and BSD-2-Clause (lz4's lib/). Package licence expression: BSD-3-Clause AND BSD-2-Clause.