nordisk/skade

Releases

New here? The plain-English version — "how do I build skade on arrow 58, where's the perf patch, when do I drop the fork" — is arrow58-for-dummies.md. The forks document themselves in ../../iceberg-arrow58/.nornir/README.md and ../../iceberg-datafusion-arrow58/.nornir/README.md.

arrow 58 upgrade (2026-06-20) — and the iceberg-datafusion perf patch (INTACT)

skade moved its data plane from arrow 57 → 58 as part of the nordisk constellation upgrade. Because iceberg-rust (and iceberg-datafusion) 0.9.1 still pin arrow ^57.1 upstream, the move rides on local [patch.crates-io] forks (stripped by cargo package/publish, so the published skade is unchanged):

# skade/skade/Cargo.toml
arrow-array = "58"   # was 57
arrow-schema = "58"
arrow-cast   = "58"
parquet      = "58"
datafusion   = "54"  # was 52.2 (datafusion 53+ is the first arrow-58 line)

[patch.crates-io]
iceberg            = { path = "../../iceberg-arrow58" }
iceberg-datafusion = { path = "../../iceberg-datafusion-arrow58" }

The perf patch is NOT lost — it moved into the arrow-58 fork

skade's SQL surface (sql feature) has long carried a performance patch to iceberg-datafusion: a parallel, per-file partitioned table scan. Upstream iceberg-datafusion 0.9.1 exposes the whole table as UnknownPartitioning(1) — a single DataFusion partition, so the scan reads every data file on one stream, serially. The patch instead exposes one partition per data file, so execute(partition) reads just that file and DataFusion fans the scan across all cores (~8× faster on a 32-core box). It's two files: src/physical_plan/scan.rs and src/table/mod.rs.

• Before (arrow 57): vendored at skade/bench/vendor/iceberg-datafusion (still on disk for reference).
• Now (arrow 58): re-applied verbatim into the fork ../iceberg-datafusion-arrow58 (sibling of the skade repo). Verified by diffing the vendored copy against stock 0.9.1 — upstream has not adopted output partitioning, so the patch is carried as-is. Drop it the moment iceberg-datafusion ships output partitioning upstream.

What was a datafusion fix vs. the perf patch (don't conflate them)

Two separate things landed in the arrow-58 fork:

1. The perf patch (above) — parallel per-file scan. This is the important, intentional behaviour change; it is arrow-version-agnostic in spirit but lives in an arrow-bound crate, so it's pinned to the fork.

2. datafusion 52 → 54 API-compat fixes (mechanical, NOT behaviour, NOT arrow):

   • as_any() was removed from CatalogProvider/SchemaProvider/TableProvider/ ExecutionPlan/PhysicalExpr (downcast is now provided) — removed every fn as_any override.
   • ExecutionPlan::properties() now returns &Arc<PlanProperties> — each plan's plan_properties field became Arc<PlanProperties>, wrapped in Arc::new.
   • logical_expr::Cast field data_type → field: Arc<Field> — use *c.field.data_type().

Scope / isolation

• The patch + datafusion live only behind skade's sql feature (dep:datafusion + dep:iceberg-datafusion). They are not in nornir's release path: nornir depends on skade with default-features = false, so its lockfile pulls zero datafusion / iceberg-datafusion. The arrow-58 move and the perf patch are independent concerns from nornir's release gate.
• skade core (writer/reader/catalog) needed zero arrow 57→58 source changes — only the manifest version bumps + the patched forks.