nordisk/heimdall

The split verify-bundle guest: isolated decoders, driven by gatling on the host

Heimdall is a defensive sandbox. This page describes how a Holger bundle is verified with OpenZL's C decoder kept OUT of the verifier's memory, and how the host spreads that decoding over several cores (lane gatwasm, 2026-10-31). It follows the recommendation in OPENZL-WASM.md: "decoder in a separate instance with its own memory". wasm32 has no threads, so all parallelism stays on the host (the znippy host-decompressors pattern), through the constellation's one worker engine, gatling (crates.io rotaryengine; no rayon, no private pool).

The two components

The single-instance guest (world job, default features) is unchanged and gives the same verdicts. tests/split_wasm.rs proves the import lists of both modules ($root::decode-chunks, $root::digest-members, $root::emit-file for the verifier; none for the decoder), and the host refuses a verifier loaded without its decoder (heimdall-wasm-27) and a decoder that imports anything (heimdall-wasm-27).

Host API (heimdall-wasm): Host::load_decoder(path) (signed and pinned like every guest), Host::load_split(verifier, decoder), then Guest::run as before. RunLimits::decoders (from the new heimdall_core::Limits::decoders, default min(cores, 8)) is the number of decoder instances that run at once.

One run

1. The verifier parses the index and, per phase (the control members, then the content), hands EVERY compressed chunk to decode-chunks in one call, after the checks a member-by-member read makes. It then walks the answers in order, so the first member that fails decides exactly as in the single-instance guest.
2. The host cuts the chunks into units from their sizes alone (a unit closes at 8 MiB of frame + declared size, or 64 chunks), never from N, so verdicts, codes and fuel are the same for every N.
3. Every unit runs in a FRESH decoder instance: its own store, linear memory, fuel and deadline (the job's limits, per instance; the deadline is what is left of the job's wall clock). It is handed only its own frames, lent straight from the verifier's memory (no copy on the host side).
4. gatling drives the units over N workers: gatling::ordered when bytes come back (in order, at most 2N units in flight, stop at the first failing unit), gatling_forkjoin when only digests do.
5. The host checks every answer natively before the verifier sees it: one answer per chunk, a 32-byte digest, exactly the declared size, BLAKE3(bytes) = digest. A decoder that lies ends the job with heimdall-wasm-13; one that traps or hits its limits ends it with that limit's code (-2, -3, -4, -5), the reason naming the decoder instance. The first failing unit in chunk order decides.
6. The verifier holds every digest to the index checksum (Holger's container code on a mismatch, as before) and the members to the signed manifest. It asks digest-members for every member's SHA-256 and BLAKE3: the host hashes the lent members natively, in parallel (gatling_forkjoin), and the verifier compares them with the signed manifest, core.json and the report exactly as if it had hashed them (each member is now hashed ONCE; before, Holger's order hashed each member three times, in wasm).

The verifier never shares memory with a decoder: a C bug in OpenZL can at worst make one decoder instance return wrong bytes, which the host's BLAKE3 check, the index checksum and the signed manifest catch (tests: tests/split.rs, lying decoders in the library; tests/split_wasm.rs, a test-only lying decoder build whose mode is patched into the module: wrong bytes, wrong digest, a short chunk, a dropped answer, a consistent lie, a trap, a spin, a memory hog).

Stable codes added

Results

Measured with cargo test --release --test split_wasm -- --ignored --nocapture measure on dwarf (6 physical cores, 12 threads, other work running; oden was unreachable), mean of 3 runs (2 for 192 MB), WHOLE verification in the host: index, signatures, digests, emit. Payload: word-salad text in OpenZL 1 MiB chunks (about 2.6:1). CPU = this process's user + system time.

The single-instance guest also got faster: every member is now hashed once instead of three times (8 MB: 175-197 ms before this lane on oden, 125 ms now on dwarf). On dwarf N=16 is no faster than N=8 (12 hardware threads). What is still serial: the verifier's own wasm work (the index, copying each decoded chunk into its memory, assembling members, emit-file), so the speed-up grows with the bundle and is bounded by that part (Amdahl).

Before digest-members (oden, 32 cores, loaded): 8 MB 197 / 202 ms, 64 MB 1,551 / 1,381 ms, 192 MB 4,671 / 4,122 ms (single / split N=8): decoding alone was never the cost, the verifier's hashing in wasm was.

Fuel

Fuel is counted per instance: the decoders of one job may together burn far more than one budget (each_decoder_instance_has_its_own_fuel...). fuel_used in the outcome is the verifier's plus every counted unit's, and is the same for every N. For bigger bundles the job policy sets fuel to at least 128 x the bundle's content bytes (the split verifier), with wall as the real bound; the defaults are not changed here (a policy decision).

Fuzz

New target guests/verify-bundle/fuzz decoder_instance: ONE decoder instance (the 32-bit wasm build of OpenZL, componentized, signed, loaded with Host::load_decoder) driven through Decoder::decode exactly as a verifier's call is served; the declared size is the frame header's (at most 16 MiB). A trap of the C code, a stack overflow, a host fault or a lying answer is a finding; an answer, a per-chunk refusal or the instance's own memory/fuel/clock limit is not. libFuzzer sees only the host side's coverage (the decoder runs as JIT code), so the corpus was first grown with the native, coverage-guided openzl_frame target (C under ASan).

The run reported 49 "slow units"; replayed one by one (replay_frames_through_a_decoder_instance) every one takes under 60 ms (the time went into the forks' start-up, which compiles the component), and two of them are real finds kept as seeds and as a test: 45- and 50-byte frames whose OpenZL graph grows the instance's memory to ~274 MB and ~2 GB. Both hit the decoder instance's own memory limit (heimdall-wasm-5, named), never the host.

What is missing

• (Done in 0.2.0: a job names its decoder with JobKind::Wasm { module, decoder: Some(GuestRef) }, WasmJobRunner and heimdall run --decoder load it with Host::load_split; tests/cli_split.rs runs good, tampered, OpenZL and garbage bundles through the heimdall binary and WasmJobRunner, split = single.)
• The verifier's own serial work bounds the speed-up (see Results). Next steps if wanted: let the host emit decoded payload members directly (they would not have to cross into the verifier's memory at all, with gatling::ordered keeping the files in order), and blake3's wasm SIMD build for what the verifier still hashes.
• The decoder fuzz has no coverage from inside wasm; it leans on the native corpus.
• Measured on dwarf only after digest-members (oden went offline mid-lane).