nordisk/dwarves

Release & publish traps — three live ones (2026-06-21)

Hard-won during the nornir 0.4.54 publish + the facett warehouse-landing fix. Each is a real trap that looked like a different bug. Read this before the next release publish / before debugging a "server build is red" report. Companions: release-for-idiots.md, release-rehearsal.md, running-as-a-server.md.

TL;DR

Trap 1 — private = true ≠ "don't publish to crates.io"

workspace_nordisk/release/nornir.toml had:

# private = true ⇒ SSH-cloned, NOT public → must NEVER publish to crates.io
[repo.nornir]
private = true

That comment is wrong for nornir. nornir's git is SSH-private (git@codeberg.org:nordisk/nornir.git), but the crate is published on crates.io (cargo install nornir works; 0.4.51 → 0.4.54 are all there). The private flag conflates the two. With it set, release publish nornir hits the private-repo gate and hard-refuses before doing anything.

Fix: private = false for nornir. holger / korp / ordning are genuinely private (no crate) and stay private = true. The distinction is: is there a crates.io crate? — not is the git repo public?

Trap 2 — the fork-promote gate over-blocks via publish=false bench crates

release publish nornir --bump patch --dry-run printed:

⏸ held from crates.io (6): iceberg-datafusion, iceberg-datafusion-arrow58,
                            nornir, skade, skade-katalog, znippy-iceberg
Publish (deps-first): nornir-airgap, nornir-saga, nornir-testmatrix, nornir-robotui

i.e. it would publish the sub-crates but NOT the nornir crate itself.

Why

patch_fork_blockers (src/release/doctor.rs:229) walks every Cargo.toml under the repo (depth 4) looking for [patch.crates-io] entries whose spec is a path/git fork. It does not skip publish = false crates. So these bench manifests get scanned:

• skade/bench/Cargo.toml → [patch.crates-io] iceberg-datafusion = { path = "vendor/iceberg-datafusion" }
• facett/bench/Cargo.toml → [patch.crates-io] iceberg = { path = "../../iceberg-arrow58" }

promote_blocked_crates then folds them through the coarse repo-level transitive closure (a repo that produces or depends on a blocked crate has all its crates blocked) — so skade → skade-katalog → znippy-iceberg → nornir all get held, even though none of the publishable library crates touch a strip-on-publish fork.

The truth the gate misses

nornir's real deps use the published fork by name, not a [patch]:

iceberg        = { package = "iceberg-arrow58", version = "0.9.1", path = "../skade/vendor/iceberg-arrow58" }
skade-katalog  = { version = "0.1.10", path = "../skade" }
skade          = { version = "0.4.12", path = "../skade/skade", default-features = false }

On publish the path is stripped and the version= resolves from crates.io — iceberg-arrow58 0.9.1, skade 0.4.12, skade-katalog 0.1.10 are all live. So nornir IS publishable; the hold is spurious.

Workaround that worked (nornir 0.4.54)

Bypass the spurious hold with cargo directly — but keep the --verify LAW:

# 1. bump [package] version in nornir/Cargo.toml by hand (0.4.53 → 0.4.54), commit
# 2. dry-run = full verify-build vs crates.io, no upload:
cargo publish -p nornir --dry-run     # ends "Uploading … / aborting upload due to dry run"
# 3. real publish (default --verify re-builds; refuses upload if deps don't resolve):
cargo publish -p nornir
# 4. tag + push
git tag -a v0.4.54 -m "…" && git push origin HEAD && git push origin v0.4.54

cargo publish defaults to --verify, so this is self-protecting: if the published deps didn't actually resolve, the verify build fails and nothing is uploaded. (Never --no-verify — that shipped the broken 0.4.45 that got yanked.)

The real fix (TODO, deferred)

Make the gate skip publish = false crates in patch_fork_blockers / compute_promote_block, so bench-only [patch.crates-io] forks don't hold the publishable crates. Then release publish nornir works through the tool, no hand-bump. Until then, the cargo path above is the escape hatch.

⛔ NEVER REMOVE — the arrow59 fork wiring (load-bearing)

⚠ REWRITTEN 2026-08-24. Every row of the old table pointed at a tree that no longer exists — vendor/iceberg-arrow58 (no such directory), a skade/bench [patch.crates-io] iceberg-datafusion (deleted, skade/bench's own comment records it), and a facett/bench [patch.crates-io] block (facett/bench has none). A "NEVER REMOVE" list naming things that are already gone is worse than no list: it makes a release lane preserve fiction and distrust the parts that are real. Checked line by line against the tree and the live registry.

Before reading Trap 3, know what is NOT a bug. These are deliberate, required wiring for the arrow-59 Iceberg stack. Do not "clean them up." Removing any of them breaks the build against arrow-59.

The ONLY [patch.crates-io] entries that ARE bugs are ones pointing a non-fork, already-published crate (e.g. nornir-testmatrix) at an absolute dev-drive path — see Trap 3. Distinguish by the crate: an arrow59 fork (iceberg*) → KEEP; a normal published crate patched to an absolute path → bug.

RENAMING is the mechanism — a [patch] cannot do this job

cargo publish strips [patch.crates-io]. A uniquely-named crate survives. That is why all three skade forks are renamed and published under their own global names, and it is the difference between wiring that works for a consumer on crates.io and wiring that only ever worked on this box:

• iceberg-arrow59 → crates.io 0.9.10 ✓ (lib name stays iceberg)
• iceberg-datafusion-arrow59 → crates.io 0.9.9 ✓ (lib iceberg_datafusion)
• iceberg-catalog-rest-arrow59 → crates.io 0.9.6 ✓ (lib iceberg_catalog_rest)

skade is the single canonical publisher of all three global names. Contrast knut's vendor/orc-rust-arrow59 and vendor/spark-connect-arrow59, which keep upstream's package name (orc-rust, spark-connect) and are therefore path-only — fine in a checkout, impossible to publish.

⚠ The old names iceberg-arrow58 (0.9.10) and iceberg-datafusion-arrow58 (0.9.9) are still on crates.io and are NOT referenced by anything in this tree. Do not resurrect them from this document's history; the live names end in 59.

So the vendored forks are part of the publish set — whenever their arrow-59 content changes they must be bumped and re-published to crates.io FIRST, before any consumer (skade / skade-katalog / znippy-iceberg / nornir / facett) that pins iceberg-arrow59 = "X" can publish. Strip the path on publish → the version= must point at a live crates.io version, or the consumer's verify-build fails. Never let a consumer reference a fork version that isn't on crates.io yet.

There is no datafusion fork, and none is needed

datafusion 55.0.0 on crates.io is the arrow-59 line. Anything in this constellation still saying "datafusion has no arrow-59 release", pinning = "54", or bridging to an apache/datafusion git rev is expired — that bridge was deleted from knut/Cargo.toml on 2026-08-24, and dwarves-gui moved 54 → 55 the same day after --features dataview-sql was measured un-compilable (3× E0308, arrow-59 types into an arrow-58 engine).

Trap 3 — committed dev-path [patch.crates-io] breaks the server build

The nornir-server builds each monitored repo from its own clone, as the isolated nornir system user, under /home/nornir/.nornir/workspaces/<ws>/git/<repo>. It cannot and must not reach the dev checkout (/run/media/rickard/T9/git/…). That isolation is the whole point — see running-as-a-server.md and the box note (the dev box IS oden; the server is a separate, sandboxed user).

So a committed patch like this (found in facett AND skade):

[patch.crates-io]
nornir-testmatrix = { path = "/run/media/rickard/T9/git/nornir/crates/nornir-testmatrix" }

makes the server's test run <repo> die at dependency resolution:

[build] <repo>::build fail — error: failed to load source for dependency `nornir-testmatrix`

It is NOT a cargo-PATH problem, NOT a server-install problem — it's the build trying to reach a path that doesn't exist in the server's sandbox.

Fix: delete the patch. nornir-testmatrix is on crates.io (0.2.2, carries functional_status + the testmatrix feature), so the plain nornir-testmatrix = { version = "0.2", optional = true } dep resolves. Verified: after removing it from facett (commit 81ad532), the server's test run facett went green — 702 rows landed in the warehouse.

• ✅ facett fixed (2026-06-21).
• ⛔ skade/Cargo.toml:80 still carries the same absolute-path patch — PENDING. It will break skade's server build identically until removed.

How to catch it before it bites

• nornir release gate path-patches <repo> — flags [patch.crates-io] blocks.
• nornir release gate path-dep-versions <repo> — path-deps must carry version=.
• Rule of thumb: a committed [patch.crates-io] with an absolute path (or any path outside the repo) is a bug. Dev-only overrides belong in a gitignored Cargo.toml-adjacent override or are simply not committed; the published crate must resolve everything from crates.io.

The one principle behind Traps 2 & 3

The published/served artifact must resolve every dependency from crates.io. A path = is a dev convenience that gets stripped (publish) or is unreachable (server sandbox). Carry the real version= (and package= for renamed forks) so that stripping the path leaves a working manifest — and never commit a patch that only works on one developer's drive.

Trap 4 (2026-08-30, lane AR) — the tool you were told to use is not on the wire

dwarves-server mcp starts, answers initialize, serves three funnel/autonom tools, and silently does not list the five release tools. There is no error, no warning, no log line. Two independent causes, and you must tell them apart before doing anything else:

1. Built without --features release. The tools are not in the binary at all. The installed fat CLI shows the same disease from the other side: ~/.local/bin/dwarves (v0.1.6, built 2026-08-30) lists a release subcommand in --help — clap always compiles the subcommand list — and running it prints dwarves release needs the release leaf: rebuild with --features release. The DECLARED atom in .nornir/build-variants.toml carries release + rehearse; the installed artifact did not. Build through the door, never by a bare cargo build --release.
2. The router defect (release-design.md f7, fixed 2026-08-30). The tools ARE in the binary — their descriptions are in the rodata, strings finds them — and the wire still does not advertise them, because #[tool_handler]'s default router expression is Self::tool_router() and the release routes were summed onto the instance field.

Telling them apart takes one command:

strings -a $(command -v dwarves-server) | grep -c 'live crates.io skip set'
# 0 → cause 1 (rebuild with the feature).   1 → cause 2 (rebuild past the fix).

The general shape is the one this constellation keeps meeting: a capability that is declared, compiled, unit-tested and unreachable. The unit test here asserted router.has_route("release_plan") and was green for 18 days while tools/list returned three names. Assert through the door the caller uses, not the structure behind it.

Trap 5 (2026-08-30, lane AR) — a floor raise re-drifts its own crate

release_gates tells you to raise an understated floor; you raise it; the crate you edited is now drifted at an unchanged version, because its Cargo.toml is one of the files release_drift compares against the published tarball. Measured here: raising nornir-funnel-embed's floor on nornir-warehouse-embed (0.1.1 → 0.1.3) moved nornir-funnel-embed itself from skip to drifted at 0.1.2 and forced its own bump to 0.1.3.

This is correct behaviour, not a bug, and it terminates — but only if you re-run release_drift after every manifest edit instead of after the last one. The cascade walks up the dependency order: bump the leaf, raise the consumer's floor, bump the consumer.

The second half of the same trap: after the bump, release_drift still prints the word drifted for a crate whose local version is now ahead of published_max (dwarves local 0.1.7 vs published 0.1.6), even though its own drift_means string says drift is "at the SAME version". The authoritative signal is release_gates — the StaleSourceDrift correctness blocker is what cleared. Read the gate, not the word.

Trap 6 (2026-09-02, lane DR) — crates.io answers "too big" with a bare 503

The live constellation cascade halted on:

error: failed to publish korp v0.1.0 … failed to get a 200 OK response, got 503

503 reads as "the registry is down"; the operator retries. It is not down. crates.io returns the same bare 503 for an OVERSIZED upload — no size, no limit, no reason in the body — and korp's .crate was 12.8 MiB against the 10 MiB limit. Three publish attempts and about an hour were spent before a local cargo package said so.

Two things follow from that, and both are now in dwarves release:

1. Size is measured BEFORE any upload, in the same pass that computes the skip and drift verdicts. An over-limit crate refuses the whole cascade the way a drift halt does, and the refusal names the biggest directories and files inside the .crate — the part that would have saved the hour. Automatic under --execute; --check-size makes a dry-run do it too. release.rs: probe_package_sizes / size_gate.
2. A 503 that still arrives is translated, not re-printed: over the limit ⇒ "this 503 IS the size refusal, waiting cannot clear it" and the step does not retry; within the limit ⇒ "reads as transient — retrying", mirroring the 429 path; never measured ⇒ it says so and names --check-size. release.rs: explain_503.

The limit itself is not a magic number: 10 MiB is a documented constant with its date and source (CRATES_IO_MAX_UPLOAD_BYTES), because no registry endpoint publishes it and a staff-granted per-crate increase is invisible to the API. Override it with --max-upload-mib <N>, DWARVES_MAX_UPLOAD_BYTES_<ALIAS> or DWARVES_MAX_UPLOAD_BYTES — never by editing the constant. A non-crates.io --registry with no override has an honestly unknown limit and refuses nothing (guessing crates.io's number at holger would refuse uploads holger accepts).

The sibling trap in the same halt — "files in the working directory contain changes"

cargo counts every uncommitted file under the package directory, so Cargo.lock churn, another lane's work and untracked scratch all read like a blocker. The size pass already holds the exact package file list (it came out of the same tarball walk), so the verb now prints which uncommitted files are actually IN the .crate — and the runner's dirty-tree refusal names those files instead of a count. release.rs: dirty_files_in_package.