nordisk/holger

Deb/RPM native metadata — design

Class: design · Function: serve (the Deb + RPM repository backends). Sibling how-tos: deb-rpm-native-metadata-guide (serve a repo) and deb-rpm-native-metadata-for-idiots (point-and-go). Whole-system view: design.md; serve path: serve-design.md.

The problem — a mirror of bare packages

Holger serves a Debian (APT) repo and an RPM (yum/dnf) repo straight out of an immutable .znippy archive. When the archive was built by the znippy deb_native / rpm_native plugin it carries a typed sub-index — a pre-parsed view of every package's control/header fields — and holger just reads that. But an archive is often a plain mirror of package bytes: someone dropped a folder of .deb / .rpm files in, no plugin, no sub-index. The client (apt, dnf) still demands a full index — APT Packages/Release, rpm-md repomd.xml/primary.xml — with real fields it can resolve dependencies against.

Previously holger filled the gap by guessing from the filename: it split {name}_{ver}_{arch}.deb / {name}-{ver}-{rel}.{arch}.rpm for coordinates and emitted nothing else — no Depends, no Maintainer, RPM epoch=0, no <format>. That is enough to list packages, not enough to resolve them. Both backends carried an honest RED maturity-disclosure row saying so.

This change closes the gap: holger now parses the real control stanza / RPM header out of the served package bytes, with no reliance on the znippy plugin. Filename-derivation drops to last-resort fallback and both maturity rows flip green.

Fallback order — three tiers, most-authoritative first

Both backends resolve each package's metadata in the same precedence:

There is a fourth, outer tier that wins over all of these for the index files themselves: authoritative path-passthrough. If the archive already carries a real Release/Packages or repomd.xml/primary.xml (a byte-exact upstream mirror), holger serves that verbatim and never synthesizes. Only when the index path misses does synthesis run — and synthesis is where the three tiers above pick each package's fields.

• Deb passthrough-vs-synthesize dispatch: server/repository/znippy-deb/src/lib.rs:347 (handle_http2_request) → passthrough get_file at :364, then synthesize_index at :369/:275.
• Deb per-package precedence (tiers 1→2→3): server/repository/znippy-deb/src/lib.rs:194, the controls.get(...).or_else(|| control::control_from_deb(&bytes)) chain.
• RPM per-package precedence: server/repository/znippy-rpm/src/lib.rs:169, the meta.get(...) → header::parse_rpm_header(...) match.

Deb — the ar + control.tar.* parser

A .deb is an ar(5) archive with exactly three members in order: debian-binary, control.tar.{,gz,xz,zst}, data.tar.{,gz,xz,zst}. Only the control tarball is needed, and it holds a single ./control file — the Debian control stanza (Depends / Maintainer / Description / …). Module: server/repository/znippy-deb/src/control.rs.

Pipeline (control_from_deb, control.rs:32):

1. Hand-parse the ar container (control_tar_member, :41). The ar format is trivial — an 8-byte global magic (!<arch>\n, AR_MAGIC :20) then fixed 60-byte member headers (AR_HEADER :22): name in bytes 0..16 (space-padded, GNU may append /), decimal size in 48..58, members padded to an even boundary. No ar crate — walking it is a dozen lines. Every offset is bounds-checked; a truncated or lying header returns None, never panics.
2. Pick the codec from the member extension (decompress_control_tar, :72) — a pure extension match, not content sniffing: control.tar → verbatim, .gz → flate2::read::GzDecoder, .xz → liblzma::read::XzDecoder, .zst → zstd::stream::read::Decoder. Anything else → None.
3. Untar ./control (control_from_tar, :89) via the tar crate; return its UTF-8 body.

The stanza is then normalized into Packages field lines by clean_control_stanza (lib.rs:308), and holger appends its own repo-level Filename / Size / SHA256 (computed over the real served bytes) on top — the three fields APT verifies a [trusted=yes] download against.

RPM — the binary header tag-table parser

An .rpm is a 96-byte lead, a signature header, then the main header. Both headers share the RPMv3 structure: magic 8e ad e8 01, 4 reserved bytes, a big-endian nindex (entry count) and hsize (data-store length), then nindex 16-byte index entries (tag, type, offset, count), then the hsize-byte store the offsets point into. Module: server/repository/znippy-rpm/src/header.rs.

Pipeline (parse_rpm_header, header.rs:174):

1. Skip the lead — validate the 4-byte lead magic (LEAD_MAGIC :22), step over 96 bytes (LEAD_LEN :21).
2. Step over the signature header without fully decoding it (header_end, :64 — reads its nindex/hsize, returns the byte just past its store), then align the main header to an 8-byte boundary: (sig_end + 7) & !7 (lib.rs/:186).
3. Decode the main header's tag table (parse_header, :84) into Entry records, each validated so its offset lands inside the store.
4. Read the wanted tags by number into an RpmMetaRow (znippy_common::views): EPOCH (1003, int32), SUMMARY (1004), VENDOR (1011), LICENSE (1014), URL (1020), SOURCERPM (1044), PROVIDENAME (1047, string-array), REQUIRENAME (1049, string-array). Tag constants: header.rs:27. Strings via read_string (:118), arrays via read_string_array (:125).

primary.xml synthesis then emits epoch="…", <summary>, <url>, and the <format> block (<rpm:license>, <rpm:provides>/<rpm:requires> as <rpm:entry> rows) from that row — server/repository/znippy-rpm/src/lib.rs:169 onward. repomd.xml references primary.xml by SHA256+size; the index is served uncompressed so dnf verifies it directly.

Bounds, bomb-caps, and hostile input

Both parsers treat the package as adversarial — a mirrored archive may contain anything. Rules held throughout:

• No unsafe, no panics. Every slice is get(..)-checked or width-validated; malformed bytes yield None, and the caller silently drops to the next fallback tier. Proven by the non_*_bytes_yield_none / truncated_store_yields_none unit tests in each module.
• Decompression-bomb cap (Deb): the control tarball is read through take(MAX_CONTROL_TAR) — 16 MiB (control.rs:25). A real control is a few KiB; a 4-byte .zst that inflates to gigabytes is truncated, not honoured.
• Resource caps (RPM): MAX_NINDEX = 65 536 tags, MAX_STORE = 64 MiB (header.rs:45). A header claiming billions of entries is rejected before any allocation.
• Every arithmetic on offsets is checked_* so a crafted size can't wrap and read out of bounds.

Both modules ship a synthetic-fixture encoder in a tests_support submodule (a real minimal .deb / byte-exact .rpm) so the crate's lib.rs integration tests drive the whole synthesis path — native_control_parse_enriches_packages and native_header_parse_enriches_primary prove the emitted index carries the parsed fields with no typed view present.

Codec dep note — one workspace-wide links = "lzma"

Rule for future contributors: use liblzma, never xz2. Both crates wrap the same native lzma library and both declare links = "lzma" in their -sys crate (xz2 → lzma-sys; liblzma → liblzma-sys). Cargo permits only one crate with a given links value in a dependency graph, so a workspace that pulls both fails to resolve — a hard error, not a warning.

DataFusion (reached via skade / knut / nornir) already pulls liblzma-sys. holger-znippy-deb therefore decodes control.tar.xz with liblzma (control.rs:81), and the two zip dev-deps that defaulted to an xz feature now set default-features = false, features = ["deflate", "time"] (they only ever write Deflated entries):

• server/repository/znippy-deb/Cargo.toml — liblzma = "0.4" (was xz2), with the rationale in a comment beside it.
• bench-scenarios/Cargo.toml, server/repository/znippy-rust/Cargo.toml — zip with default-features = false.

Reintroducing xz2 (or any crate that transitively adds lzma-sys) re-breaks workspace resolution for every consumer that also sees DataFusion. If you need xz, reach for liblzma.