nordisk/znippy

Package-aware plugins — guide

Class: guide · audience: a techie writing a plugin or driving a handler. The what is in plugins-design.md; siblings iceberg-guide.md · signing-guide.md.

f0 — writing a plugin (the whole workflow)

Two moves ship an ecosystem handler:

1. Implement the six trait methods of ZnippyPlugin (znippy-common/src/plugin.rs): meta, run_command, extract_metadata, schema_fields, matches_path, extract_batch.
2. Register it in znippy-cli/src/handlers.rs::builtin_handlers(). Done — the CLI autodiscovers it via meta(); there is no name table to touch.

A native plugin builds twice: cdylib (native host) + rlib (compiled to WASM).

f1 — filling the trait

struct MavenPlugin;

impl znippy_common::ZnippyPlugin for MavenPlugin {
    fn meta(&self) -> HandlerMeta {                       // the discovery record
        HandlerMeta { name: "maven".into(), aliases: vec!["mvn".into()],
                      ecosystem: "maven".into(), extensions: vec!["jar".into()],
                      /* subcommand API … */ }
    }
    fn matches_path(&self, p: &Path) -> bool { p.extension() == Some("jar".as_ref()) }
    fn schema_fields(&self) -> Vec<Field> { vec![/* only maven's columns */] }
    fn extract_metadata(&self, path: &Path, data: &[u8]) -> RecordFields {
        // read ONLY pom.xml out of the jar — filter before explode:
        let pom = ljar::decompress_jar_filter(data, "pom.xml");
        pom::parse(&pom).unwrap_or_else(|_| filename_fallback(path))   // never panic
    }
    fn extract_batch(&self, items: &[BatchItem]) -> _ { /* optional zero-copy, once */ }
    fn run_command(&self, cmd: &str, args: &[String]) -> _ { /* `coords`, etc. */ }
}

Rule of thumb: parsers must fall back to filename parsing on garbage or oversized input, never panic (mirror the *_bomb_falls_back_to_filename tests).

f2 — driving a handler from the CLI

--format selects a single ecosystem; one archive = one package type. Entry points: znippy-cli/src/lib.rs Commands::Handlers (:516), Commands::Run (:520).

f3 — reading the columns back (typed views)

Each ecosystem's columns come back through its typed view — round-trip per ecosystem:

let archive = ZnippyArchive::open("out.znippy")?;
for row in archive.maven_view()? {        // or rust_view() / python_view()
    println!("{} {}", row.group_id, row.version);   // maven's own columns, out of the union
}

Views recover only their ecosystem's branch of the DenseUnion. Round-trip coverage lives in plugin_read_api_test.rs.

f4 — znippy-plugin-git: sizing and reading the exploded cache

The git store keeps a read cache of unpacked objects beside the verbatim packs. It has a byte budget, it evicts, and none of it is data — every row is re-derivable from the pack by resolve::materialize_at. What the design of that is: plugins-design.md §"what a push costs on DISK". What you do with it:

Choose what a push explodes eagerly

ZNIPPY_GIT_EXPLODE=commits   # DEFAULT: commits and tags. Trees and blobs are lazy.
ZNIPPY_GIT_EXPLODE=graph     # + trees eagerly. Fast closure walks, 3.24x the pack on linux.
ZNIPPY_GIT_EXPLODE=full      # everything eagerly. The 22.4x shape; a bench arm, not a setting.
ZNIPPY_GIT_EXPLODE=off       # nothing eagerly; every pack is re-queued on reopen.

An unknown value is an error, never a silent default: a typo that quietly disables a paid tier is not noticed until a bill is wrong.

Size the cache

ZNIPPY_GIT_EXPLODE_BUDGET=8G          # per repository
ZNIPPY_GIT_EXPLODE_BUDGET=unlimited   # a deliberate answer, not an absent one

Unset takes min(10 % of the filesystem holding the store, 8 GiB). In-process: GitStore::set_explode_budget(bytes) — it takes effect at the next rotation, never refuses a read and never deletes on the caller's thread.

Read the band

let band = store.explode_cache_use();
println!("{}", band.band());          // "3.1G/8G"
band.eager_bytes;   // the graph pool — never evicted
band.lazy_bytes;    // what reads have cached
band.direct_reads;  // reads served by inflating straight from the pack, nothing cached
band.evicted_bytes; // what the budget has already reclaimed, measured before the unlink

exploded_cache::use_all() gives (store root, CacheUse) for every open store in the process without taking a store lock — the shape a dashboard tick wants; it is the same Weak registry progress::snapshot_all uses.

Empty it

GitStore::drop_explode_cache() unlinks every lazy segment and keeps the graph pool. That is what repo gc calls.

Do not expect every read to be cached

The cache admits on rebuild cost, not on being read. A whole object is one inflate of a mapped byte range — microseconds — so shallow (depth ≤ CHEAP_DEPTH) and small (< CHEAP_BYTES) objects are read directly and nothing is written. On a real repository that is about six reads in ten. A test that asserts "the second read is served from the table" must ask direct_reads how many were declined, or it is asserting that the policy is switched off.