Your browser sucks, JavaScript disabled! Gunnar's console needs JavaScript and WebAssembly. Git over SSH on port 2222 does not.

mikronova/modgunn

.nornir/cve-warehouse-ingest.md

CVE warehouse ingest — all major vuln DBs → Iceberg → modgunn scan

Plan for getting every major vulnerability database into the shared nornir Iceberg warehouse (cve_advisories), airgap-compatible, so modgunn Engine A can render a Pass/Warn/Block verdict over the full dependency closure of any repo in any ecosystem — fully offline. Extends modgunn-master-plan.md §1 (Engine A), §7.3 (adopt-list) and §8 (open decision: advisory-DB sourcing in airgap).

0. Goal & the one decision that shapes everything

Goal: one warehouse-resident advisory corpus, refreshed by sealed-znippy generations, that covers all the big feeds and all ecosystems holger serves (cargo · pip · npm · maven · go · nuget · gem · deb · rpm · apk · conda · composer · helm · docker base images), matched precisely against real versions.

The decision: store ranges, match at scan time — not expand-at-import.

Today cve_advisories stores affected_versions as exact versions and scan::Advisory::matches (scan.rs:127) does exact equality (empty list ⇒ all versions). To import a range-only OSV/NVD/GHSA entry you must expand the range to exact versions (versions::expand_range), which needs the full published version universe per package. Over all DBs × all packages that:

Recommendation: evolve to range-native matching — persist the OSV range events (introduced / fixed / last_affected) and match with versions::affected_by(eco, events, version) against the artifact's actual version at scan time. Expansion becomes lazy (only against the versions you actually scan) or unnecessary. This is the single highest-leverage change and P0 below; everything else layers on it.

1. The sources (the "big CVE databases"), by format

Source Format Ecosystems modgunn status
OSV.dev aggregate (per-ecosystem all.zip exports) — federates GHSA, PyPA/PYSEC, RUSTSEC, Go vulndb, npm, Maven, NuGet, Hex, Pub, … OSV (lingua franca) nearly all package ecosystems ✅ parser (osv.rs)
GitHub Advisory DB (GHSA) (github/advisory-database git, OSV files) OSV npm/pip/maven/nuget/rubygems/go/cargo/composer ✅ reuse osv.rs (just another dir)
CISA KEV JSON (CVE-keyed, cross-eco) ✅ feeds::parse_kev_json
FIRST EPSS CSV (CVE-keyed) ✅ feeds::parse_epss_csv
OpenSSF Malicious Packages OSV-ish JSON npm/pip/… ✅ feeds::parse_malicious
NVD (NIST) — CVE JSON 2.0, CVSS vectors + CPE NVD JSON 2.0 product-centric (CPE), weak pkg map ❌ new nvd.rs
Distro trackers — Debian Security Tracker, Ubuntu USN/OVAL, Red Hat CSAF/OVAL, Alpine secdb, SUSE CSAF, GitLab Advisory DB mixed (JSON/YAML/OVAL/CSAF) deb · rpm · apk ❌ prefer OSV.dev distro exports; native adapters for gaps
VEX / CSAF (not_affected suppression) CSAF cross-eco 🚧 partial (provenance.rs)

Coverage strategy: OSV.dev is the workhorse — its per-ecosystem exports already fold GHSA/RUSTSEC/PYSEC/Go/npm/Maven/NuGet and several distro feeds (Debian, Alpine, Ubuntu, Rocky, AlmaLinux, SUSE). Lead with it (P1) and only write native adapters where OSV.dev's coverage is thin (NVD CVSS backfill = P3, distro gaps = P4).

2. Schema v2 — cve_advisories (additive, db_rev-versioned)

Iceberg appends are additive and str_col reads are drift-tolerant (warehouse.rs:98), so we add columns without breaking the existing reader. The new source-of-truth is ranges; affected_versions stays as an optional pre-expanded cache for tiny/explicit-version advisories.

Column Type Note
id Utf8 primary advisory id (prefer CVE, else GHSA/RUSTSEC)
aliases Utf8 (CSV) dedup key — all ids this advisory is known by (CVE↔GHSA↔RUSTSEC)
ecosystem Utf8 canonical modgunn label (security::ecosystem_for)
package Utf8
ranges Utf8 (JSON) new source of truth — [{introduced, introduced_exclusive, fixed, last_affected}] per versions::RangeEvent (IntroducedExclusive = NVD versionStartExcluding, a strictly greater than lower bound)
affected_versions Utf8 (CSV) optional pre-expanded cache (""=all) — kept for back-compat
severity Utf8 info/low/medium/high/critical
cvss_vector Utf8 new — from NVD/OSV severity[]
cwe Utf8 (CSV) new
epss Float64 (null) exploit probability (FIRST)
kev Boolean confirmed exploited (CISA)
withdrawn Boolean new — OSV withdrawn ⇒ never matches
source Utf8 new — `osv
published_micros / modified_micros Int64 new — cadence/anomaly (A20, A33) + audit
db_rev Utf8 import generation; latest rev wins on read

Code touch points:

3. Pipeline — internet-side build → airgap carry → secure-side import

Mirrors holger's airgap model (build a sealed .znippy, carry it, import it).

 INTERNET SIDE (agent / world connector)            — LUFTGAP —     SECURE SIDE (modgunn + skade)
 ┌───────────────────────────────────────────┐                   ┌────────────────────────────────┐
 │ 1 fetch  OSV.dev all.zip · GHSA git · NVD  │   2 seal          │ 4 verify sig (blake3)          │
 │          JSON · KEV · EPSS · distro feeds  │ ─► .znippy bundle ─►  5 append_advisories(db_rev,…) │
 │ 3 normalize → Vec<Advisory> (ranges) +     │   db_rev =        │    → cve_advisories generation │
 │   enrich(EPSS,KEV) + dedup-by-alias        │   date+srchash    │    load_advisory_db() = latest │
 └───────────────────────────────────────────┘                   └────────────────────────────────┘
  1. Fetch (internet side): pull each feed snapshot to staging, pinned by date. Reuse holger-agent's world connector pattern; no live access on the secure side.
  2. Normalize: each source → Vec<Advisory> with ranges. Reuse osv.rs/feeds.rs; new nvd.rs + distro adapters. Dedup/merge by alias — collapse the same vuln across CVE/GHSA/RUSTSEC into one row (alias union, range union, max severity, best CVSS, OR of kev). feeds::enrich folds EPSS/KEV by CVE id.
  3. Seal: write the corpus as a signed .znippy stamped db_rev = <ISO-date>-<sha of source set> (master-plan §8). The carriable unit.
  4. Carry across the gap (USB/diode), no return channel — file self-verifies.
  5. Import (secure side): modgunn warehouse import <bundle.znippy> → verify → Warehouse::append_advisories(db_rev, &advisories) (warehouse.rs:156). Append = new generation; load_advisory_db already returns latest db_rev. Idempotent re-import is safe (a re-imported rev just re-wins identically).

4. The scan path — "scan all dependencies"

  1. Dep closure. nornir captures the resolved transitive closure per repo into sbom_components; modgunn reads it via read_sbom_components(repo) (warehouse.rs:263). Multi-ecosystem resolvers are the master-plan §6 cutover (cargo-metadata exists; add pip/npm/maven/go/…). Holger's per-archive listing (list_archive_files → security::parse_coords) is the lighter path already wired in Tool #3.
  2. Join. For a single repo / promotion gate: load_advisory_db(Some(eco)) per ecosystem (cheap, partition-scoped). For a fleet / whole-warehouse scan: push the sbom_components × cve_advisories join down to skade/DataFusion (skade sql feature) — match (ecosystem, package) then affected_by(version) — instead of loading the whole DB into RAM.
  3. Verdict. OsvScanner (range-aware) → per-component Verdict → ScanReport (Pass/Warn/Block) with KEV→Critical, EPSS≥τ→High, VEX not_affected suppression. Cache on (blob_sha256, policy_version, db_rev).
  4. Surface. holger Tool #3 / holger-server scan; nornir viz Security pane. holger-server scan swaps demo_scanner() for a warehouse-backed scanner when a catalog root is configured.

5. Phases (each independently shippable)

Phase Deliverable Repos
P0 ✅ Schema v2 + range-native matching — DONE. Advisory gained ranges/aliases/withdrawn/source; matches is range-native via affected_by; merge_by_alias dedup; cve_advisories v2 columns (ranges JSON, aliases CSV, withdrawn, source) round-trip drift-tolerantly. 4 new tests (range match, withdrawn, alias dedup, warehouse v2 round-trip) + holger's 11 security tests green. modgunn
P1 ✅ OSV.dev full ingest — osv.rs parses range-native (carries OSV ranges + aliases + withdrawn); parse_and_merge normalizes+dedups across feeds; modgunn warehouse import/stats CLI; internet-side fetcher tools/fetch-advisories.sh (OSV.dev per-ecosystem all.zip + NVD + KEV + EPSS → staging dir → the documented import invocation). Remaining: only the agent's sealed-znippy wrapping of the staging dir (carry step). modgunn (+ agent fetch)
P2 ✅ Exploit + malicious enrich — DONE (offline). EPSS + KEV wired into import (--epss/--kev → feeds::enrich); OpenSSF Malicious-Packages wired via --malicious <dir> (feeds::parse_malicious → Critical, source = openssf, folded into the corpus with merge_by_alias). Smoke-tested (OSV + malicious in one import). modgunn
P3 🔄 NVD — enrichment DONE. nvd.rs parses CVE JSON 2.0 (CVSS v3.1/v3.0/v2 → severity + vector, CWE ids, description); enrich_with_nvd folds them onto advisories by id OR alias (sets CVSS vector + CWE, lifts severity), wired into import as --nvd <dir>. Advisory/warehouse gained cvss_vector + cwe (schema). 3 enrichment tests. CPE backfill DONE: parse_nvd_advisories emits standalone advisories from vulnerable application CPEs (product → package, versionStart*/versionEnd* → ranges — versionStartExcluding maps to the EXCLUSIVE RangeEvent::IntroducedExclusive, never the inclusive Introduced, or the boundary version false-positives; an upper bound with no lower bound is anchored Introduced("0")), tagged ecosystem = "cpe" (separate namespace, won't false-match OSV) + source = "nvd"; CLI import --nvd-backfill <dir>. CVEs absent from OSV now land with coords. modgunn
P4 🔄 Distro trackers (deb/rpm/apk) — version grammars DONE: versions.rs gained dpkg deb_cmp (epoch/upstream/revision, ~ ordering) + rpm_cmp (rpmvercmp: ~/^, numeric-vs-alpha, leading-zeros), wired into compare/affected_by for deb/rpm/apk; normalize_ecosystem maps OSV distro labels (Debian/Ubuntu→deb, Alpine→apk, Red Hat/Rocky/Alma/SUSE→rpm) so OSV.dev's distro exports ingest with the right comparator. So deb/rpm advisories now range-match correctly via the existing OSV path — no native adapter needed for the OSV-covered distros. apkvercmp DONE: apk_cmp implements Alpine's grammar (numeric dot-parts, trailing letter, _alpha/_beta/_pre/_rc<release<_p suffix ranks, -rN revision) — apk no longer borrows dpkg. 4 tests. Remaining: native adapters only for the rare feeds OSV.dev doesn't carry. modgunn
P5 🔄 Whole-warehouse scan — in-memory variant DONE (dep-neutral): scan_fleet joins ALL sbom_components (every repo) × the latest cve_advisories in one sweep; scan_sbom_delta(from) is the incremental CDC path — only the components added since a snapshot, off skade's table_delta, returning the next watermark (poll commit_seq). CLI warehouse scan [--since <snapshot>]. Tolerant of an absent SBOM table. 2 tests + CLI smoke. Components assume cargo until nornir adds an ecosystem column (§6). Variant B (pushdown) DONE: scan_fleet_sql behind the opt-in warehouse-sql feature (skade/sql) runs the name=package equi-join in DataFusion, then range-matches + latest-rev in Rust on the narrowed candidates; CLI warehouse scan --sql. A test proves it returns exactly the in-memory result. The DataFusion tree is isolated behind the feature — the lean gate/build never pulls it. modgunn (+ skade)
P6 🔄 VEX/CSAF suppression DONE: vex.rs parses CSAF known_not_affected → suppressed CVE ids; warehouse scan --vex <dir> drops those findings (reports the suppressed count). 2 VEX tests. Scorecard storage DONE: scorecard_signals table + append_scorecard/load_scorecard (latest-per-subject). SBOM-drift A30 DONE: sbom_drift(snap_a, snap_b) reports (entered, left) (name, version) sets via the snapshot_id column. 2 more tests. modgunn

6. Cross-repo dependencies & risks

Needs from skade: confirm/enable the sql (DataFusion) feature for the join; verify large-append throughput (OSV.dev all-ecosystems is hundreds of thousands of records); consider partitioning cve_advisories by ecosystem so the gate read load_advisory_db(eco) stays a single-partition scan. skade is not checked out here (../skade/skade, path dep) — clone from codeberg.org/nordisk before P0.

Needs from nornir (master-plan §6 cutover): multi-ecosystem components resolvers feeding sbom_components; retire the rust-only rustsec mirror (NORNIR_ADVISORY_DB, security.rs:483) in favour of the warehouse table.

modgunn surface to add: nvd.rs + distro adapters; alias-merge in normalize; CLI verbs modgunn warehouse build-bundle | import | scan (modgunn is lib-mostly today — add the bin surface or expose via holger/nornir CLIs).

Risks