Your browser sucks, JavaScript disabled! Gunnar's console needs JavaScript and WebAssembly. Git over SSH on port 2222 does not.

nordisk/skade

.nornir/arrow58-for-dummies.md

Arrow 58 on skade — the absolutely-no-context guide

You opened the skade repo, ran cargo build, and it pulled crates from folders called iceberg-arrow58 next door. You want to know: what is this, will it break, and when does it go away? Three questions, three answers. No data-lake knowledge assumed — for that start at dummy-guide.md. The full, precise writeup is releases.md; this page is the "I just need the gist" version.

The 30-second version

skade reads and writes data using a library called Arrow (the in-memory columnar format) and Parquet (the on-disk file format). There's a new major version of both — 58, up from 57. skade now uses 58.

The snag: skade sits on top of iceberg-rust, and the newest published iceberg-rust (0.9.1) is still stuck on Arrow 57. Arrow majors don't mix — the whole stack has to agree on one number. So we made two temporary copies of the iceberg crates with the version bumped to 58, and told skade to use those copies instead of the ones from the internet:

# skade/skade/Cargo.toml — the magic lines
[patch.crates-io]
iceberg            = { path = "../../iceberg-arrow58" }
iceberg-datafusion = { path = "../../iceberg-datafusion-arrow58" }

[patch.crates-io] means "when something asks for iceberg, give it the crate in this folder instead of the one from crates.io." That's all.

"How do I build skade on Arrow 58?"

You already did — there's nothing extra to do. Just make sure the two sibling folders exist next to your skade checkout:

git/
├── skade/                          ← you are here
├── iceberg-arrow58/                ← needed
└── iceberg-datafusion-arrow58/     ← needed (only for the `sql` feature)

Then, from the detached skade/skade/ workspace (not the repo root — the root is the skade-katalog workspace):

$ cd skade/skade
$ cargo check                 # core writer/reader
$ cargo check --features sql  # + DataFusion SQL

Both finish clean. The only noise is three warning: use of deprecated … to_naive_date lines coming from inside the iceberg-arrow58 fork — Arrow 58 renamed a function. They're warnings, not errors, and they're upstream iceberg's code, not skade's. Ignore them.

Gotcha: cargo check -p skade from the repo root fails with package ID specification 'skade' did not match any packages. That's expected — skade lives in its own detached workspace under skade/skade/. cd in first.

"Where's the perf patch? Did we lose it?"

No. skade's SQL has long carried a speed hack: it makes DataFusion read every data file in parallel (one CPU core per file) instead of one-file-at-a-time on a single core — roughly 8× faster on a 32-core machine. That hack lives in two files inside the datafusion fork:

iceberg-datafusion-arrow58/src/physical_plan/scan.rs   ← the patch (look for "LOCAL PATCH")
iceberg-datafusion-arrow58/src/table/mod.rs

It used to be vendored at skade/bench/vendor/iceberg-datafusion (still on disk for reference). When we made the Arrow 58 fork, we copied the patch over verbatim. It's only active behind skade's sql feature, so a plain cargo build never touches it.

"Does this affect the published skade?"

No. [patch.crates-io] is a local-only trick — cargo publish deletes it automatically. So the skade you'd get from crates.io still depends on the normal iceberg 0.9.1 on Arrow 57. Only your local builds, tests, and benchmarks get Arrow 58 and the perf patch. nornir's release path doesn't see any of it either (nornir takes skade with default-features = false, so no DataFusion at all).

"When do I delete the forks?"

The moment upstream iceberg-rust ships a release built on Arrow 58 (probably 0.10). Then:

  1. Open skade/skade/Cargo.toml.
  2. Delete the two [patch.crates-io] lines above.
  3. Bump the iceberg / iceberg-datafusion version requirements to the new release.
  4. Delete the iceberg-arrow58 and iceberg-datafusion-arrow58 folders.

One caveat for the second fork: even after Arrow 58 lands upstream, keep the perf patch until iceberg-datafusion also ships parallel per-file scanning natively. Two separate conditions — details in releases.md and the fork's own .nornir/README.md.

Where to go next