nordisk/skade

skade-flight

Apache Arrow Flight SQL over a skade Iceberg warehouse. DataFusion SQL in, streamed Arrow out. The door Power BI, the Arrow Flight SQL ODBC driver, ADBC (adbc_driver_flightsql) and DataFusion clients connect to.

This is skade remote: embedded callers link skade and query in-process; remote callers hit this server. There is no second query engine — skade is DataFusion.

skade-flight --warehouse /var/lib/skade/warehouse --listen 0.0.0.0:10009 \
             --token "$(cat ~/.skade/token)" --basic pbi:hunter2

What a client gets

Bad SQL is INVALID_ARGUMENT with the planner's message; everything else is INTERNAL.

Naming

SELECT * FROM t resolves in skade's default namespace (main). skade.<namespace>.<table> always works. Unquoted identifiers are lowercased, quoted ones preserved — DataFusion's rules, and what GetSqlInfo declares.

Performance shape

• One session for the life of the process, behind an ArcSwap: readers never lock.
• Timed refresh (--refresh-secs, default 5): a session older than that is rebuilt on the next GetFlightInfo / GetTables, so a new commit shows up within the window and a burst of Power BI refreshes shares one rebuild. The rebuild reads catalog pointers, not data.
• Streaming end to end: batches leave as scan partitions complete.
• --partitions (default: cores) sets DataFusion's target_partitions.

Auth

The bearer token comes from the ONE nordisk ladder (nornir-service): --token → $SKADE_FLIGHT_TOKEN → ~/.skade/token. Empty everywhere means the server is open and says so on stderr. --basic USER:PASS (or $SKADE_FLIGHT_BASIC) is the credential the handshake exchanges for the token. TLS / mTLS: --tls-cert, --tls-key, --tls-client-ca with the tls feature.

Deploying (from the first Windows test, 2026-09-09)

• Clone over SSH (ssh://git@git.gunnar.rs:2222/nordisk/skade.git); nordisk repos are SSH-only, so a tester needs a deploy key. One clone builds it (detached workspace, in-repo path deps), about 20 min cold.
• Do not run the binary as root to get around a bind mount: use --install and give the data root (SKADE_FLIGHT_DATA, default ~/.skade) to the service user. redb needs write access to catalog.redb even to serve reads (redb 2 has no read-only open). In a container the bound warehouse must be owned by the uid the door runs as: rootless podman run --userns keep-id -v $WH:/data/wh …, or a named volume owned by the image's service user. Measured 2026-09-10 on oden: as uid 10055 over a host-owned bind the binary now refuses BY NAME before redb speaks (skade-flight runs as uid 10055 and cannot write /data/wh (owner 0:0 mode 775) … with both fixes in the text, skade_flight::dataroot); the same run with --userns keep-id creates catalog.redb and serves.
• Plaintext works with the ODBC driver (useEncryption=false); on a tailnet-exposed server run with --tls-cert/--tls-key and flip the DSN to useEncryption=true.
• The handshake sends NO HandshakeResponse message, on purpose: Dremio's ODBC driver build hangs until its deadline if one is queued. The bearer rides the authorization response header, which is what every client (arrow-cpp, arrow-rs, arrow-go) authenticates with.

Lifecycle door

skade-flight --unit                 # print the systemd unit --install writes
skade-flight --install [--system] [--env K=V]...
skade-flight --start | --stop | --status | --uninstall

Rendered and installed through skidbladnir. Never copy the binary by hand, never write a drop-in.

As a library

let wh = skade::Warehouse::open("/var/lib/skade/warehouse").await?;
let engine = Arc::new(skade_flight::Engine::new(wh).await?);
let svc = skade_flight::SkadeFlightService::new(engine, Arc::new(Auth::bearer(token)));
// add FlightServiceServer::with_interceptor(svc, skade_flight::auth::interceptor(auth)) to your router

korp mounts it this way beside its Cypher Flight SQL service (knut's graphar-flight): two dialects, two services, one wire.

Locking

The warehouse's redb catalog is single-writer. skade-flight opens it like skade-katalog serve does — run one of them per warehouse directory, not both. (A consumer that wants both SQL and Iceberg-REST on one warehouse mounts this service into the same process.)

Tests

tests/tpch_flight_sql.rs is TPC-H through the door as a GATE, not a report: all 22 queries (bench/tpch_shared.rs's texts, by inclusion) over tpchgen data. tpch_22_execute_over_flight_sql runs at SF 0.01 in the ordinary suite; tpch_sf1_answers_are_canonical_over_flight_sql is #[ignore] and gates every row count, Q1's A/F count_order (1 478 493) and Q6's revenue (123 141 078.2283) against the dbgen answer sets tpchgen bundles — measured on oden 2026-09-10: 8 661 245 rows loaded in 2.4 s, 22/22 canonical, 2.1 s total, median 75 ms, slowest Q18 0.3 s (release):

TMPDIR=~/scratch/cargo/skade-b5/tpch-tmp \
  cargo test --release --test tpch_flight_sql tpch_sf1 -- --ignored --nocapture

q15_spec_create_view_runs_or_is_refused_by_name_never_an_internal_error sends Q15 as the spec writes it (CREATE VIEW revenue0 …; SELECT …; DROP VIEW) one statement at a time and accepts two outcomes only: the three run through a session view and the SELECT is one row, or a refusal that names the view and the CTE rewrite. Before 5c4ef64 the door answered Internal error: … register_table does not support tables with data — the test was written red on that and went green when session views landed.

cargo test runs a real tonic server on an ephemeral port and drives it with the stock FlightSqlServiceClient: introspection, SELECT * FROM "t", filter/order/limit folding, aggregates, a two-table join, prepared statements, bad SQL, refused writes, a late commit becoming visible, and the Basic → bearer handshake with wrong-password refusal.

Part of the nordisk constellation.

ODBC dialect shim

Power Query's SQL92 generator and the Flight SQL ODBC driver forward ODBC escapes verbatim: SELECT TOP n, {d '…'} {t '…'} {ts '…'}, {fn UCASE(x)}, {oj …}, {escape '\'}. odbc.rs rewrites them before planning (~60 ODBC scalar functions mapped, TIMESTAMPADD/TIMESTAMPDIFF for every SQL_TSI unit, CONVERT(x, SQL_T) → CAST). Plain SQL is returned borrowed, untouched.

What the ODBC driver needs, and gets

From the driver sources (dremio/flightsql-odbc, apache/arrow's ODBC donation, GizmoSQL's fork): every schema field carries ARROW:FLIGHT:SQL:TYPE_NAME, PRECISION, SCALE, IS_SEARCHABLE (the driver dereferences them unchecked, and Power Query folds only on columns whose TYPE_NAME is in GetXdbcTypeInfo); the three bitmask infos the driver reads first-bit-wins carry ONE bit; the function lists use ODBC names; the handshake always answers with a Bearer header; GetPrimaryKeys answers Iceberg identifier fields; the FK commands answer empty sets with the spec schema.

Datatype matrix

tests/datatype_matrix.rs writes one table with every Arrow type the Iceberg bridge accepts and drives 14 × 7 cells over the wire: wire type, values, predicate, ORDER BY, CAST to VARCHAR, aggregates, ODBC type row. Every cell is a functional_status row; cargo test --features testmatrix drains them and SKADE_FLIGHT_MATRIX_OUT=<file.jsonl> writes them. Reds that are not ours are named in KNOWN_RED with the reason, and the test fails if one of them turns green unannounced.

Power BI matrix

tests/powerbi_matrix.rs — 129 statement shapes Power Query's ODBC path emits (navigator, projection, filter, sort, top/limit, aggregate, join, cast, string, date, numeric, plus 38 documented verbatim from Microsoft's diagnostics and captured traces: CRLF formatting, "ITBL"/"OTBL" derived tables, cast("id" as DOUBLE) = 3, the Date.From timestampadd(timestampdiff) nesting, DirectQuery limits). 129/129. Two matrix reds were vendored-Iceberg bugs and are fixed there: no Double→Float datum conversion (no float column predicate could push down) and no Double→Long (Power Query's numeric equality shape failed at scan).

The first run of the matrix found that no predicate on a FLOAT column could be pushed down (the vendored Iceberg Datum::to had no Double→Float arm). Fixed in vendor/iceberg-arrow59.

Benchmark

cargo run --release --example flight_bench          # 2M rows, 8 clients
SKADE_FLIGHT_BENCH_ROWS=20000000 cargo run --release --example flight_bench

One BenchResult JSON line per arm on stdout (the nordisk envelope), a table on stderr. Arms: plan latency, GetSqlInfo and GetTables round trips, engine-only scans (no wire), raw-frame scan (no client decode), full and projected scans, filter pushdown, aggregate, GROUP BY, and 8 concurrent full scans.

2M rows × 6 columns, 16 cores, loopback, both peers at 1 MiB HTTP/2 frames:

The single-stream ceiling is HTTP/2 framing: 16 KiB frames gave 23M rows/s, 1 MiB frames 40M. The client's advertised frame size governs DoGet.