deploy/spark-connect/README.md · nordisk/knut

Local Spark 4.x Connect server (with SDP pipelines) for the knut viewer

A real, local, runnable Spark Connect server so the live pipeline viewer (knutpipe watch / knutpipe pipeline / knutpipe gui) can be exercised end-to-end without an external cluster.

The stock Apache Spark 4.1.2 GA image (the blessed default; pairs with iceberg-spark-runtime-4.1_2.13:1.11.0) already ships both the spark-connect_2.13 and spark-pipelines_2.13 jars, so a plain start-connect-server.sh exposes:

the SQL / job-progress path (knutpipe watch --sql …), and
the Spark Declarative Pipelines command family (spark.pipelines — CreateDataflowGraph / DefineOutput / DefineFlow / StartRun, SPARK-51727) that knutpipe pipeline … drives.

No image build is required.

Bring it up — pure Rust, zero shell

The primary bring-up is the knut-spark-server binary. It drives the whole container lifecycle (pull / create / start / logs / stop) through podman's REST API via the bollard client — no shell script, no podman/docker CLI, no std::process::Command. It talks to the rootless podman socket ($DOCKER_HOST, else unix://$XDG_RUNTIME_DIR/podman/podman.sock).

# Prerequisite (one time): the podman API socket must be enabled.
systemctl --user enable --now podman.socket

# Build the manager (gated behind the `server` feature so the default stays lean):
cargo build -p knut-pipelines --features server --bin knut-spark-server

knut-spark-server up       # pull-if-absent, start detached, wait, print "ready: sc://localhost:15002"
knut-spark-server status   # running / port
knut-spark-server logs     # stream server logs (follow)
knut-spark-server stop     # stop + remove the container

up returns while leaving the server running, so knutpipe watch / pipeline / gui can connect. If the podman socket isn't up, up prints the systemctl --user enable --now podman.socket hint and exits non-zero — it does not shell out to enable it.

Override the image with KNUT_SPARK_IMAGE=<name:tag> and the published port with PORT=<n>.

Optional declarative alternative (compose)

If you prefer a declarative spec over the Rust manager, compose.yaml brings the same server up (YAML, not shell):

podman compose -f deploy/spark-connect/compose.yaml up -d
podman compose -f deploy/spark-connect/compose.yaml down

Spark Connect listens on sc://localhost:15002 (default port).

Drive it end-to-end

Build the live client once (the grpc feature carries its own tonic 0.14 tree and runs protoc over the vendored proto/):

cargo build -p knut-pipelines --features grpc --bin knutpipe

1. A full Declarative Pipeline (bronze → silver), streaming run events

knutpipe pipeline sc://localhost:15002 \
  --spec crates/knut-pipelines/examples/pipeline.json \
  --storage file:///tmp/knut_pipeline

Each flow streams QUEUED → PLANNING → STARTING → RUNNING → COMPLETED, then Run is COMPLETED. See sample-run.log for a captured run against this server.

2. A Spark SQL job, streaming job progress

knutpipe watch sc://localhost:15002 \
  --sql "SELECT count(*) AS n FROM range(0, 20000000, 1, 16)"

Progress snapshots (n/m tasks, per-stage bars) stream as the job runs. See sample-watch-sql.log.

3. The desktop viewer (live node graph)

cargo run -p knut-pipelines --features live --bin knutpipe -- \
  pipeline sc://localhost:15002 --spec crates/knut-pipelines/examples/pipeline.json --gui

Set KNUT_SCREENSHOT=/path/to/shot.png to have the GUI save a frame (needs a display / compositor; headless CI without a GPU surface will not produce one).

Integration test (testcontainers, podman-driven)

crates/knut-pipelines/tests/integration_spark_pipelines.rs boots this server via testcontainers, points PipelinesClient::connect at it, defines the demo dataflow graph, starts a run, and asserts PipelineRunEvents stream back. It is gated behind the grpc feature and #[ignore], so the default cargo test stays server-free.

# Prereq: podman socket up
systemctl --user start podman.socket

cargo test -p knut-pipelines --features grpc \
  --test integration_spark_pipelines -- --ignored --nocapture

The test defaults to the blessed docker.io/apache/spark:4.1.2-… GA image; set KNUT_SPARK_IMAGE=<name:tag> to override — e.g. …:4.2.0-… for the 4.2.0 GA line, or a locally built localhost/knut-spark-iceberg:latest if you also want the Iceberg lake catalog (see containers/spark-iceberg/Containerfile).

What works vs. what you still need

Works, proven here (see the sample logs): a real Spark Connect server, the full SDP pipelines run streaming PipelineRunEvents, and the SQL job-progress path. The blessed default is now Spark 4.1.2 GA, whose stock spark-pipelines jar registers the spark.pipelines gRPC handler; the sample logs captured below were taken on the 4.2.0 GA line (still selectable via KNUT_SPARK_IMAGE). (Pipelines are not present in the 4.0.0 GA image; use the 4.1.2 GA default, the 4.2.0 GA line, or any 4.1+/4.2 build that ships the pipelines jar.)
You still need: podman (or docker) and the ~1.3 GB Spark image pull, plus a --features grpc build of knutpipe. To materialize a pipeline you must pass a writable --storage URI (a file:///… path inside the container is fine for the demo). For the GUI screenshot path you need a real display surface.