Your browser sucks, JavaScript disabled! Gunnar's console needs JavaScript and WebAssembly. Git over SSH on port 2222 does not.

mikronova/modgunn

.nornir/feeds-guide.md

feeds — guide

Class: guide · audience: someone importing advisory feeds or building an AdvisoryDb from disk. The what is in feeds-design.md; the why in feeds-reasoning.md.

f0 — import a generation from the CLI

The secure-side operator path (needs --features warehouse,scan):

modgunn warehouse import --warehouse <root> --rev 2026-07-04-a \
        --osv               advisories/osv/ \
        --malicious         advisories/openssf/ \
        --nvd               advisories/nvd/ \
        --epss              advisories/epss.csv \
        --kev               advisories/kev.json \
        --exploit-poc       advisories/poc-in-github/ \
        --metasploit        advisories/metasploit_modules.json \
        --nuclei            advisories/nuclei-cves.json \
        --exploitdb         advisories/files_exploits.csv \
        --crates-yanks      advisories/crates/ \
        --npm-deprecations  advisories/npm/ \
        --pypi-yanks        advisories/pypi/ \
        --go-retract        advisories/gomod/

It parses every *.json under each dir, merges + dedups by alias, enriches with EPSS/KEV, and appends the corpus as a new db_rev generation. The summary prints counts of KEV / EPSS-scored / public-exploit / CVSS-vectored advisories, the registry yanks + npm deprecations, and any skipped source files. The network fetch that fills those dirs runs internet-side (tools/fetch-advisories.sh) and hands a sealed znippy across the airgap — this verb is offline.

The sources (each optional; at least one required):

Anti-silent-empty. A per-doc parse failure (a truncated OSV/registry doc) is named on stderr + counted, never silently dropped; a well-formed doc that yields a real 0 (nothing yanked) is fine; but an enrichment feed (--epss/--kev) or the public-exploit union (--exploit-poc/--metasploit/--nuclei/--exploitdb) that was provided yet parsed to 0 rows is a truncated catalog that would silently disable its escalation — so the import bails rather than write a fake pass. --exploitdb adds one more edge: a CSV whose header carries no codes column is not an Exploit-DB index at all (a wrong file, or a fetched HTML error page), so it is rejected outright rather than read as “no exploits”.

f0 — build an AdvisoryDb in code (no warehouse)

use modgunn::{osv, feeds};
use std::collections::{HashMap, HashSet};

// 1 · parse + dedup an OSV directory's worth of docs
let mut advisories = osv::parse_and_merge(osv_json_strings);

// 2 · enrich with exploit intelligence
let epss: HashMap<String, f64> = feeds::parse_epss_csv(&epss_csv);
let kev:  HashSet<String>      = feeds::parse_kev_json(&kev_json)?;
feeds::enrich(&mut advisories, &epss, &kev);

// 3 · hand to Engine A
let db = modgunn::scan::AdvisoryDb::new("2026-07-04-a", advisories);

f1 — per-source recipes

f2 — version matching

To test whether a concrete version is hit by an advisory's ranges:

use modgunn::versions::affected_by;
let hit = affected_by("cargo", &advisory.ranges, "1.2.3");

Advisory::ranges is a disjunction — one event list per OSV range object — so the scan-time call is affected_by_any, which ORs them. affected_by evaluates ONE range; never concatenate two ranges' events into a single list, because the walk is a monotone on/off sweep and the narrower range's Fixed would cancel the wider one's window.

use modgunn::versions::affected_by_any;
let hit = affected_by_any("cargo", &advisory.ranges, "1.2.3");

The comparator is picked from the ecosystem automatically — semver for cargo/npm/go/nuget/composer, PEP440 for pip, RubyGems segments for gem, dpkg/rpm/apk for distro packages. Anything else (notably NVD's cpe) falls back to a loose dotted-numeric compare that is NOT prerelease-aware. Feed OSV distro labels through versions::normalize_ecosystem first so Debian/Alpine/Red Hat map to deb/apk/rpm.

f4 — score a package's install behaviour (A23, no CVE needed)

To flag a malicious package that has no advisory yet, fold its package.json text into the A23 behaviour-feature vector (needs --features scan; offline, no code run):

use modgunn::behavior::{parse_npm_package, parse_pypi_setup, parse_pyproject_toml, parse_setup_cfg};

// npm — reads scripts{}, no exec (Result: the JSON can be malformed)
let b = parse_npm_package(package_json_text)?;
if b.network_in_install {
    // dropper pattern: an install hook fetches stage-2 from the network
}

// PyPI setup.py — scans the whole file text, no Python run (infallible text fold)
let p = parse_pypi_setup(setup_py_text);

// PyPI pyproject.toml — SECTION-AWARE: only build/install-active sections scanned
let q = parse_pyproject_toml(pyproject_toml_text);

// setuptools setup.cfg — SECTION-AWARE INI: only build/install-active keys scanned
let c = parse_setup_cfg(setup_cfg_text);

for f in b.findings().iter().chain(p.findings()).chain(q.findings()).chain(c.findings()) {  // Vec<core::Finding>
    println!("{} · {} — {}", f.severity, f.id, f.summary);
}

npm scores only the auto-run hooks (preinstall/install/postinstall/prepare) — a test/build script the user invokes on purpose is not the malware vector and is ignored. PyPI's setup.py runs in full at pip install from an sdist, so parse_pypi_setup scans the whole file; its install-hook signal is a cmdclass install/develop override, and it extracts the same features (a whole-file urllib.urlopen + os.system still trips the network finding with no cmdclass). A pyproject.toml is declarative TOML, so parse_pyproject_toml is section-aware (NOT a whole-file scan): it folds only the build/install-active sections ([build-system] / a PEP 517 in-tree backend-path / a cmdclass table / a [tool.*.build.hooks…] build-hook) and skips the declarative [project] deps & URLs — so an honest project's urllib3 dependency or https:// homepage does not false-positive, while a build-hook that curl … | bashes a payload IS flagged High. It uses a tiny line-scanner (no toml dep) so it stays in the lean scan gate. A setup.cfg is declarative INI with the same hazard, so parse_setup_cfg is likewise section-aware (no ini dep): it folds only the [options] cmdclass / dependency_links keys and a build-active [options.entry_points] group, and skips [metadata] (incl. url) and every declarative dep key (install_requires / extras_require / …) — so a benign install_requires urllib3 or a [metadata] url = https://… does not false-positive, while a dependency_links payload fetch IS flagged High. All four land in the SAME PackageBehavior. The strongest finding is High MODGUNN-BEHAVIOR-NETWORK-IN-INSTALL (a curl/wget/fetch(/urllib/URL fetch); the sensitive_apis set (npm child_process/eval(/atob(; Python os.system/marshal.loads/…) and a high-entropy base64 blob are Medium; merely declaring an install hook is Low.

f4 — from the CLI (no code)

Vet a single package on a laptop, offline, without writing Rust:

# ecosystem inferred from the basename
#   package.json → npm ·  setup.py → pypi ·  pyproject.toml → pyproject ·  setup.cfg → setupcfg
modgunn behavior --path ./suspect/package.json
modgunn behavior --path ./suspect/setup.py
modgunn behavior --path ./suspect/pyproject.toml
modgunn behavior --path ./suspect/setup.cfg

# force the parser for an oddly-named file, and rename the subject
modgunn behavior --path ./blob.json --ecosystem npm --subject suspect-pkg

# gate a promotion: exit 3 if ANY finding fires
modgunn behavior --path ./package.json --fail-on-finding

# PERSIST the analysis into the warehouse (verdict ledger), then read it back
modgunn behavior --path ./package.json --warehouse ./wh      # append vector + verdict
modgunn warehouse behavior-log --warehouse ./wh              # latest per (ecosystem, subject)

It prints the feature vector (hooks=… network_in_install=… urls=… entropy=… apis=[…]) then one line per finding (High MODGUNN-BEHAVIOR-NETWORK-IN-INSTALL — …) and a worst <sev> footer. It folds through the SAME parse_npm_package/parse_pypi_setup/parse_pyproject_toml/parse_setup_cfg the library exposes — no network, no code executed. Without --fail-on-finding it always exits 0 (a report); an unknown basename with no --ecosystem, or a missing/malformed file, is a usage error (non-zero).

Add --warehouse <root> to persist the ecosystem-tagged feature vector + its derived verdict (worst severity + finding ids) into the behavior_features table via the SAME warehouse-persist path scorecard --warehouse uses (no second seam). warehouse behavior-log --warehouse <root> [--ecosystem <eco>] [--subject <name>] reads the latest analysis per (ecosystem, subject) back (worst-severity first) — the append→load round-trip. The persist leg needs a --features warehouse binary; a plain behavior (no --warehouse) touches no warehouse code.

Gotchas