nordisk/knut

Þund vs Spark — the definitive scope of "complete thund"

The question this doc answers: what does it take for knut-thund's NATIVE backend (DataFusion + skade/Iceberg) to be a COMPLETE replacement for Spark on knut's workloads — so a user can drop Spark entirely for these pipelines.

Two gap axes are catalogued here:

• GAP 1 — SDP-over-Connect drop list (streaming/quality knobs Spark Declarative Pipelines cannot express). Status: closed — see the last section. Every item on SparkBackend::lowering_report's drop list is now executed by the native backend with a real row-level test.
• GAP 2 — Databricks + Delta Lake features BEYOND OSS Spark + Iceberg. This is the forward-looking scope: for each proprietary/Delta feature, does thund-native already have it, NEED it, or is it OUT-OF-SCOPE (Databricks- proprietary, not required for knut's workloads)? Iceberg (via skade / iceberg-rust) already has equivalents for most of Delta's table features — we build on those, we do not reinvent Delta.

Date: 2026-07-29. Sources: Delta Lake docs (delta.io), Apache Iceberg docs (iceberg.apache.org), Databricks platform docs + 2025/2026 platform notes.

0. What "complete" means here

"Complete" is not "reimplement Databricks." Databricks is a managed platform whose value is largely operational (a cluster manager, a governance plane, a proprietary vectorized engine). knut already has its own substrate (skade's all-core gatling data plane, an embedded Iceberg catalog, korp's orchestration). So "complete thund" = the pipeline-authoring + execution surface knut needs is fully served by the native backend, with Iceberg as the table format. Anything that is a managed-platform concern, or a Delta-format-lock-in concern, is out-of-scope by construction — Iceberg is the chosen format.

Three verdicts are used below:

• HAVE — thund-native (DataFusion + skade/Iceberg) already does it, or gets it for free from the format/engine.
• NEED — genuinely required to be a complete Spark alternative for knut's workloads, and not yet wired. Goes on the worklist.
• OUT-OF-SCOPE — Databricks-proprietary or Delta-format-specific; Iceberg and/or knut's own substrate is the answer, or knut's workloads don't use it.

1. OSS Spark surface (the baseline thund must match)

These are the Spark capabilities knut's pipelines actually lean on. This is the floor — below "complete" is not negotiable.

Verdict on the OSS baseline: thund-native already covers it for knut's workloads. The one real gap is connector breadth, and that is scoped to the sources knut actually ingests, not Spark's entire connector zoo.

2. Databricks features BEYOND OSS Spark

These are the proprietary, managed-platform features. Almost all are OUT-OF-SCOPE — they are either a managed service, or the exact thing knut's own substrate replaces.

Verdict on Databricks-proprietary: nothing here is a thund gap. The one that looks like a feature — DLT — is precisely what thund already is. Everything else is a managed service or a proprietary engine that knut's own substrate (DataFusion + skade) already stands in for.

3. Delta Lake table features vs Iceberg (via skade) — the real catalogue

This is the substantive part. Delta and Iceberg are competing table formats; knut has chosen Iceberg (skade). For each Delta feature, the question is: does Iceberg have the equivalent (so skade gives it to us), does thund-native need to WIRE it, or is it Delta-lock-in we don't want?

Verdict on Delta-vs-Iceberg: Iceberg (skade) already provides equivalents for essentially every Delta table feature — often the original of it (position deletes predate deletion vectors; hidden partitioning predates liquid clustering; field-id evolution beats column mapping). The remaining thund work is wiring the IR to skade's existing Iceberg surface, not building new format features and emphatically not adopting Delta.

4. Prioritized worklist — what remains for "complete thund"

Ordered by leverage for knut's actual workloads. (GAP-1 items are DONE — §5.)

1. Iceberg write-path via skade for the native backend. Today the native backend materialises to in-process MemTables + file sinks; wire dataset OutputType + iceberg format to a real skade table write (append + snapshot commit). This is the single biggest lever — it turns "runs the computation" into "runs the computation AND lands it in the lakehouse." Blocker to note: the skade version collision (published 0.9.8 vs the local fork 0.9.1) must be resolved before knut-thund can add a skade dep; knut-thund does not depend on skade today (only graphar-flight does), so this is a clean add once the version is pinned.
2. Partition-transform write (bucket/truncate/months/…). The IR + the partition_transforms capability already model it and check gates it; the native writer must honor transforms when landing to Iceberg (the liquid- clustering-equivalent). Depends on (1).
3. Position-delete / MERGE-emits-deletes onto Iceberg. thund's CDC merge currently produces a merged snapshot; to be a true incremental lakehouse sink it should emit Iceberg position/equality deletes (the deletion-vector analog) via skade rather than rewriting. Depends on (1).
4. Snapshot / time-travel read knob. An IR "read table AS OF snapshot/tag" source, lowering to skade snapshot reads. Cheap once (1) lands.
5. Change Data Feed OUT. If a knut workload needs to emit a row-level changelog from an Iceberg table (not just consume CDC in), add an incremental changelog scan over skade snapshot diffs. Demand-driven — only if a workload needs it.
6. Table maintenance (compaction / snapshot expiry). OPTIMIZE-equivalent; belongs in skade, scheduled by korp. Not a thund runtime concern.
7. Connector breadth — scope strictly to knut's real sources (Kafka, object-store, Iceberg). Not a goal to match Spark's connector zoo.

Explicitly OUT-OF-SCOPE (do not build): Photon, Unity Catalog governance, DBSQL/serverless, Predictive Optimization/IO, identity columns, column mapping, and the Delta format itself. Iceberg + skade + DataFusion is the substrate.

5. GAP-1 status — SDP drop list, now run by Native (DONE)

SparkBackend::lowering_report enumerates every knob SDP-over-Connect drops. Each is now executed by the native backend with a real, passing, row-level test. The report strings themselves now name the native counterpart ("RUN BY NATIVE: …"), so the "dropped by SDP / run by Native" status is self-documenting.

Honest caveats (not overclaimed):

• Update mode is an upsert delta (changed/new rows re-emitted), NOT a retraction stream — the native run/event surface carries no negative-row retract marker. For an updating sink this is the correct changed-row feed; true retract streams are a separate, larger engine feature and are not claimed.
• ProcessingTime fires one trigger per arriving micro-batch; the wall-clock interval_ms is advisory because the in-process synthetic source has no real-time clock. Against a live Kafka source the trigger cadence would follow real arrivals.
• The native backend RUNS these knobs in-process (MemTable/file materialisation). Landing results into an Iceberg table via skade is worklist item (1) above — that is the boundary between "native runs the pipeline" and "native is a full lakehouse sink."