nordisk/holger

12. Watched applications

Since 3.2 the thing Holger watches is the customer's application, not only the store. A customer declares an application, its CI sends Holger the signed graph of each commit, and Holger keeps a lamp on it: red when a package it ships pulls in a reachable CVE or malware, yellow when the CVE is present but never called, none when it is clean. The lamp is judged again whenever the advisory data changes, so a new advisory lights it without a new build, and a red lamp tells the application's owners. The code is server/lib/src/apps.rs (AppsHub), the board band is in server/lib/src/flow.rs, and the console pages are holger-ui's /apps and /apps/<name>. It works the same online and airgapped: the only difference is where the advisory data comes from.

Declare an application

An application is data: its name, the repository it is built from, the products (the workspace members that are shipped), the owners to tell, and the Ed25519 keys whose signature on a graph Holger believes.

apps: [
    (
        name: "shop",
        repo: "https://git.example.org/acme/shop.git",
        products: ["shop-server", "shop-cli"],   // empty: every workspace member
        owners: ["shop-team@example.org"],
        graph_keys: ["<ed25519 public key, hex>"],
    ),
],

The same declaration goes in from the console (Security → Watched applications → "Declare an application") or as POST /-/apps (a Repository Admin); those are kept in <data_root>/holger-apps/declared.json. A name is 1 to 64 of a-z 0-9 . _ -; a declaration that names no repo, an empty product, or a key that is not 64 hex characters is refused (holger-apps-2). An application needs a Holger with the intake chain and its scanners: a build without them answers holger-apps-7 on every apps door, and the downstream build is such a build.

The graph

The graph is the one cargo lens graph-package writes (dwarves; chapter 8): graph-package.znippy, Arrow IPC tables dep_nodes, dep_edges, symbols and call_edges, every table's metadata saying dwarves.graph_package.schema = 2. Holger reads schema 2 and nothing else (holger-apps-5); a graph may be at most 512 MiB.

The CI sends it per commit:

PUT /-/apps/shop/graph?commit=<sha>
X-Holger-Graph-Envelope: <base64 of a DSSE envelope>
<the znippy bytes>

The envelope is a DSSE envelope over an in-toto v1 statement whose one subject is the sha256 of the bytes and whose predicate type is https://holger.rs/watched-app-graph/v1, with the application's name and the commit in the predicate, signed with one of the declared graph_keys. An envelope that cannot be read is holger-apps-3; one that is signed by another key, over other bytes, or for another application or commit is holger-apps-4. With sign-in configured the sender must be signed in; the signature is what Holger believes. Holger keeps every commit's graph and judges the latest one.

Holger does not yet build the graph itself (the in-Holger Heimdall job that runs cargo lens graph-package on the declared repository is not started; GraphRecord.signer = holger is reserved for it). Today the customer's CI sends the signed graph.

The lamp

Per product, Holger takes the dependency closure out of dep_edges and matches every package against the local intel database by its purl (chapter 5). Then, per hit, it asks whether the product calls the vulnerable code: RustSec advisories name their vulnerable functions (affected.functions), and modgunn's reachability (reachability_gated_vulns, A8) walks the graph's call_edges from the product to them, with a breadth-first call path as the evidence (server/lib/src/apps.rs, judge_app).

An application's lamp is the worst of its products'. The advisory data is the same database the intake chain reads: there is no second source, and no data means unknown (the fail-closed rule of 3.2, chapter 5). The scope analyzer (a dev- or build-only dependency does not ship) is not built, so such a dependency counts as shipped, which fails closed.

The report is recomputed on every read, and the apps.check jera job judges every application once a minute (CHECK_EVERY; the timer only queues the job) against the data installed now, so a bundle imported or a refresh landed at noon lights the lamps by one past.

Where the lamp shows.

• The board: GET /-/flow.svg (chapter 4) carries a band WATCHED APPLICATIONS on top of the flow at every zoom, one box per application with data-lamp and each product's lamp, linked to apps/<name> (also in FlowSvgReply.links). The console's Overview and Flow pages show it.
• The console: Security → Watched applications (/apps) lists every application with its lamp, a Download OpenVEX button and the Declare form; /apps/<name> shows the reason, the call path, the dependency path, the VEX words and "Told" (who was notified, when).
• The doors: GET /-/apps (the list), GET /-/apps/<name> (the report, schema holger.watched-app/v1: lamp, headline, per product its hits with advisory, package, reach, why and call path, the graph's commit and signer, notified), GET /-/apps/<name>/vex (OpenVEX: one not_affected statement per product and advisory whose functions it does not call). An Auditor or a Repository Admin reads them. Over gRPC AdminService.WatchedApp returns the same report; holger-cli app <name> prints it.

Who is told

When a lamp turns red, Holger tells once per turn (notified.json beside the application's state keeps the record): an holger::alarm log line, a 503 audit record, and on a metered build with passkey sign-in a mail to each owner through the relay the box was fed (apps::PostmanNotify; a box with no relay says so, and the log line and the audit row are still written). The report's notified field and the console's "Told" section show it. No new mailer was added: it is the sign-in mail path.

Online: the upstream Holger

On an online Holger (intel: (offline: false), a build with intel-refresh) the advisory data is kept fresh by the intel.refresh job, once at boot and then daily: OSV, NVD and KEV, and since 3.2 the popular-names lists and registry records of the supply section (chapter 5). The box signs the set with its own refresh signer and lands it through the import door. The application's CI reaches the Holger over the network and PUTs each commit's graph. The red lamp mails the owners when the box has a relay.

Test: server/lib/tests/a_watched_application_is_lit_by_what_it_calls.rs, a_called_vulnerable_function_is_red_and_an_uncalled_one_yellow_online_and_airgapped, with the real records RUSTSEC-2021-0003, -2020-0071, -2021-0145 and -2022-0042 and two real cargo lens graph-package files (tests/fixtures/watched_apps/graphs/HOWTO.txt); the moniker impl#[SmallVec]insert_many(). matches RustSec smallvec::SmallVec::insert_many.

Airgapped: the same code, no network

On an airgapped Holger (network: Offline, intel: (offline: true)) nothing dials. The graph comes from the on-prem CI over the LAN, the same PUT. The advisory data comes as a signed holger.intel/v1 bundle (chapter 5):

• On the medium. The main and twin media of an estate carry a dated intel bundle signed by Holger's release signer, and the jera job intel.bundled imports it at first boot into the empty database (chapter 8). The box has advisory data before anyone signs in.
• Carried later. A newer bundle goes in through any import door: the inbox <data_root>/intel.inbox, the console's "Import newer" on the Overview line, holger intel import, POST /-/intel/import. It is verified, held to the anti-rollback floor and audited before it is live. A bundle older than the one installed is refused.

The lamp is then judged against that data, offline. Data past block_after_days (30) makes the lamp unknown; the Overview's CVE-data line turns yellow after 7 days and red with no data. The owners are told by the alarm log line and the audit row; the mail path needs a relay the box was fed, which an airgapped box usually has not.

The same test runs both ways: the loop over Net::Online and Net::Offline in a_called_vulnerable_function_is_red_and_an_uncalled_one_yellow_online_and_airgapped.

Where the watcher sits in the two setups

In the streaming setup the lamp lives on the internet Holger: the CI sends the graph there, and the on-prem downstream serves the packages (chapter 1). In the CDS setup the secure side cannot reach the internet Holger, so an application whose lamp must light inside the secure zone is watched by a full holger there (with the scanners and network: Offline), fed by the carried intel bundle; the plain holger-downstream serves the packages but watches nothing (holger-apps-7).

Refusals

Coming: Holger's own Heimdall job that builds the graph from the declared repository; the scope analyzer; the store's own malware verdicts and lens findings as lamp input; the graph for npm, pip and Maven applications (plan row H32-SCIP-REVIVE).