mikronova/modgunn

feeds — design

Class: design · Function axis: f0 the ingest fold · f1 per-source parsers · f2 version grammars · f3 enrichment & suppression · f4 package behaviour features (A23). Strict: only the shape we run. The why is in feeds-reasoning.md; the system view in design.md.

f0 — the ingest fold

feeds turns a set of offline feed documents into a deduplicated Vec< scan::Advisory> carrying ranges, ready for the warehouse or a direct scan. The pipeline (as wired in main.rs:cmd_import): parse each source → merge_by_alias (scan.rs:224, dedup CVE↔GHSA↔RUSTSEC into one row via alias union / range union / max severity / OR-of-kev) → optional NVD enrichment → feeds::enrich (EPSS/KEV) → append.

An Advisory is range-native: ranges: Vec<RangeEvent> is the source of truth, affected_versions an optional pre-expanded cache, and Advisory::matches (scan.rs:183) is withdrawn ⇒ false else versions::affected_by(ecosystem, &ranges, version) OR the exact-list fallback.

Reuse: feeds is leaf-of-leaf — pure parse, no warehouse/network. It produces the Advisorys that verdict matches and warehouse round-trips.

f1 — per-source parsers

AffectedRange is the intermediate { id, ecosystem, package, severity, summary, events: Vec<RangeEvent> } — one per OSV range object, not per affected entry; expand_advisories (osv.rs:274) expands ranges to exact Advisorys against a known version list when a cache is wanted.

f2 — version grammars (versions.rs)

RangeEvent (versions.rs:26) is the OSV range vocabulary (Introduced/Fixed/LastAffected) plus one non-OSV event, IntroducedExclusive — a strictly greater than lower bound. OSV cannot express it, but NVD's versionStartExcluding means exactly that, and folding it onto the inclusive Introduced would flag the boundary version itself as vulnerable (a false positive on the version the vendor called clean). These functions drive matching:

• compare(ecosystem, a, b) — per-ecosystem version ordering: semver (cargo/npm/go/nuget/composer) / PEP440-lite / maven-lite / gem (RubyGems segments, 1.0.0.rc1 < 1.0.0) / loose, plus distro grammars dpkg (deb_cmp: epoch/upstream/revision, ~ ordering), rpm (rpm_cmp: rpmvercmp) and apk (apk_cmp: Alpine suffix ranks). The loose fallback (only cpe + unknown ecosystems now) is NOT prerelease-aware — it ranks 1.5.0-rc.1 above 1.5.0.
• affected_by(ecosystem, events, version) — the scan-time predicate for one range: is version inside its [introduced, fixed) / [introduced, last_affected] window under the ecosystem's comparator — or (introduced, … when the lower bound is an IntroducedExclusive.
• affected_by_any(ecosystem, ranges, version) — the OR over an advisory's range objects. Advisory::ranges is Vec<Vec<RangeEvent>> (one inner list per OSV range object) precisely so ranges union instead of cancelling; GIT ranges are dropped at import (commit hashes are not versions).
• decode_range_sets — warehouse ranges column decode; a pre-v3 flat event array reads back as a single range object.
• expand_range (versions.rs:87) — the lazy exact-list expander.

normalize_ecosystem maps OSV distro labels (Debian/Ubuntu→deb, Alpine→apk, RedHat/Rocky/Alma/SUSE→rpm) so OSV.dev's distro exports match with the right comparator.

f3 — enrichment & suppression

• enrich(advisories, epss, kev) (feeds.rs:70) folds EPSS scores and the KEV flag onto advisories matched by id OR any alias (same as enrich_with_nvd): the EPSS/KEV feeds are keyed by CVE, but merge_by_alias keeps the first-seen record as primary, so an actively-exploited CVE whose merged advisory is GHSA/RUSTSEC-primary carries the CVE only in its aliases — matching on aliases keeps the KEV→Critical / EPSS→High escalation firing.
• enrich_exploited(advisories, exploited) lifts an advisory whose id OR alias has a public PoC (PoC-in-GitHub) to at least High and prepends the stable [public-exploit] marker to its summary (idempotent; never lowers Critical). Schema-free — the lift + marker ride the existing severity/summary warehouse columns, so the signal round-trips and the baddie-detection join ranks exploit-backed findings above version-only ones. A public PoC floors at High (weaker evidence than KEV's confirmed-in-the-wild → Critical).
• enrich_with_nvd(advisories, entries) (nvd.rs:255) sets CVSS vector + CWE and lifts severity, matched by id OR alias.
• collect_suppressions(docs) (vex.rs:51) unions CSAF known_not_affected ids — the set a scan drops (warehouse-scan --vex, or verdict's vex_not_affected).
• Offline CVSS v3 scoring (osv::cvss_base_score): an OSV/RUSTSEC advisory whose only severity signal is a CVSS v3.0/3.1 vector (no numeric score, no database_specific label — very common) is scored by the spec's deterministic base-score formula and bucketed via the shared severity_from_base, instead of silently under-rating to Medium (OSV) / High (rustsec). Both the CVSS v3.0/3.1 vector (cvss_v3_base) and the older CVSS v2 vector (cvss_v2_base, detected by its Au: metric) are computed; validated against FIRST's published reference scores. Only a v4 vector (a far more involved formula) still falls back to the label/Medium default.

f4 — package behaviour features (A23)

Where f1–f3 import advisory feeds keyed by a known vulnerability, behavior scores a package that has no CVE yet — the malicious-package detection axis. It is the read-side feature extractor, the same parse_* shape: offline, no code executed — it reads a package's published metadata text and folds the install-time code into the A23 feature vector academic detectors use (Socket H1-2025; arXiv 2412.05259 / 2309.02637).

Four ecosystem front-ends, one folder. parse_npm_package reads an npm package.json's auto-run scripts{}; parse_pypi_setup reads a PyPI setup.py; parse_pyproject_toml reads a PEP 621/517 pyproject.toml; parse_setup_cfg reads a setuptools setup.cfg (Python is the #2 malicious-package ecosystem). All funnel their install-time text through the SAME pure extract_from_install_text folder and land in the SAME PackageBehavior — never a per-ecosystem twin (L5); a new ecosystem is one more thin front-end. They differ only in what counts as the install text:

• npm — only the auto-run hooks INSTALL_HOOKS (preinstall / install / postinstall / prepare) are scored: those are the scripts npm install executes without the user asking, so a test / build script is not the vector and is ignored.
• PyPI setup.py — a setup.py runs in full at pip install from an sdist, so the whole file is the install text. The install_hooks signal is a cmdclass install/develop override (the documented persistence trick, has_cmdclass_install_override); a declarative setup(name=…, packages=…, install_requires=…) with no dangerous call is clean, while a setup.py that urllib.urlopens + os.systems a payload trips the network / sensitive-API findings from the whole-file scan even with no cmdclass.
• PyPI pyproject.toml — declarative TOML that does not itself run code, so a whole-file scan would be WRONG: a [project] dependencies = ["urllib3"] trips the urllib needle and a [project.urls] Homepage = "https://…" trips https:// → a spurious network_in_install on nearly every honest project. So parse_pyproject_toml is TOML-section- aware: it scans ONLY the sections/keys that run at build/install time — the [build-system] backend spec, a PEP 517 in-tree backend backend-path (an install-hook signal in itself), a cmdclass table, a [tool.*.build.hooks…] build-hook table — and SKIPS the declarative [project] block, the [tool.poetry] / linter config, and the dependency-list keys requires / dependencies / optional-dependencies even inside an active section (a build requires = ["urllib3"] is a declared dep, not a network call). A benign declarative pyproject (deps incl. urllib3, a https:// homepage) is clean; a build-hook that curl … | bashes a stage-2 payload IS flagged High. A tiny line-scanner — no toml dependency, so the pyproject front-end stays inside the lean scan gate (the toml crate never enters the default binary — holger / static-holger purity, matching the rustsec / manifests gating).
• setuptools setup.cfg — declarative INI, the same false-positive hazard as pyproject and the same cure: parse_setup_cfg is a section-aware INI line-scanner (tracks [section] + indent-continuation, no ini / configparser crate, so it too stays inside the lean scan gate). It scans ONLY the build/install-active keys — the [options] cmdclass mapping (a custom install/build command), the [options] dependency_links list (URLs setuptools fetches at build → a genuine build-time network source), and a build-active [options.entry_points] group (BUILD_ENTRY_POINT_GROUPS — a distutils.commands / setup-keyword hook that runs at build) — and SKIPS the entire [metadata] block (incl. url / project_urls), every declarative dependency-list key (SETUP_CFG_DECLARATIVE_KEYS — install_requires / extras_require / python_requires / …) even inside [options], and a runtime console_scripts / gui_scripts entry-point group. So a benign declarative setup.cfg (install_requires incl. urllib3, a [metadata] url = https://…) is clean, while a dependency_links payload fetch + a cmdclass override IS flagged High + Low install-hook.

Either way PackageBehavior carries: install_hooks, network_in_install (a curl/wget/fetch(/urllib/raw-URL fetch — the dropper stage-2 download), url_count, max_token_entropy (max Shannon entropy over long install-script tokens — the base64/packed-payload signal, reusing the SAME core::shannon_entropy the SCIP A17 obfuscation tripwire keys on; one metric, no twin, L5) and the sensitive_apis set (npm: child_process / eval( / atob(; Python: os.system / subprocess / marshal.loads / __import__(; shared: /dev/tcp / base64 / …).

PackageBehavior::findings ranks by academic signal strength:

Same core::Finding model the verdict surface consumes — the scoring model (the isolation-forest / behaviour-sequence classifier over the vector) is the dynamic-plane consumer; the feature extraction lives here so every consumer extracts it the same way.

Durable + queryable (the VERDICT ledger). A parsed PackageBehavior is not only printed: behavior --warehouse <root> PERSISTS it into the behavior_features warehouse table via Warehouse::append_package_behavior (ecosystem-tagged coords + the full feature vector + the derived verdict = worst finding severity + finding ids), through the SAME warehouse-persist plumbing scorecard --warehouse uses — no second warehouse seam (L5). The read back is load_package_behaviors (latest per (ecosystem, subject)), whose inner PackageBehavior round-trips field-identical (mutation-verified); the CLI read verb is warehouse behavior-log. All behind the warehouse feature — the lean scan/default gate stays free of the skade/arrow tree, so a plain behavior still vets a package with no warehouse build.

Surface

The modgunn warehouse import CLI (main.rs:cmd_import) is the operator face: --osv, --malicious, --nvd, --nvd-backfill, --epss, --kev, --crates-yanks, --npm-deprecations, --pypi-yanks, --ossindex, and the four public-exploit sources --exploit-poc / --metasploit / --nuclei / --exploitdb (which union into one exploit-CVE set before the single enrich_exploited lift) each feed one source into this pipeline; a malformed doc is named on stderr and counted, never silently dropped.

Anti-silent-empty (two edges): a per-doc parse failure (a truncated OSV/registry doc missing its versions/releases array) is counted in n_fail and named on stderr; a well-formed source doc that legitimately yields 0 rows (nothing yanked) is a real zero. But an enrichment feed (--epss / --kev) that was provided yet parsed to 0 rows is a truncated catalog that would silently disable the KEV→Critical / EPSS→High escalation — a fake pass — so cmd_import bails rather than import it (a provided-but-empty enrichment feed is an error, not a shrug). The same law covers the exploit union: if ANY of --exploit-poc / --metasploit / --nuclei / --exploitdb was provided yet the union is 0 CVEs, the import bails rather than silently skip the [public-exploit] lift.

f5 — superplan §D security tail (2026-07-29)

The nordisk super-plan (workspace_nordisk/.nornir/nordisk-super-plan-2026-07-29.md) §D closed its collector gap list — "Remaining: VulnCheck, socket.dev, OSSF Scorecard ingest, Libraries.io, GUAC" — as follows. All parsers are OFFLINE, zero-shell, file-in (obtaining the file — token/key/ running-instance — is the dynamic plane's job, out of modgunn's scope); every one holds empty-is-RED (a well-formed-but-empty doc is a real zero; a malformed doc is an Err, never a fake pass) and ships a real red-when-reverted test.

Why socket.dev / GUAC are PENDING, not fake-green: each has a real parser + real unit test, so --socket / --guac are honestly usable on a correctly-shaped file. The PENDING mark is about source fidelity — I could not obtain a live sample (paid key / running instance) to confirm the real envelope matches the documented one. That is a truthful "scaffolded, unverified against the live source", never an empty-that-passes-as-green.