nordisk/edda

nornir-flight — design

The constellation's ONE generic Apache Arrow Flight transport seam (LAW #5, reuse). Use-case-agnostic: raw Flight verbs (do_put / do_get / do_exchange) over RecordBatch, plus schema-agnostic FlightSink / FlightSource traits. Every real Flight use in nordisk — the build-info stream, skade-flight, a future knut Flight-shuffle — is an independent consumer on top of this transport, not a fork of it.

This is slice 1 of the build-info-flight design (workspace_nordisk/.nornir/build-info-flight-stream-design.md §f1) — the load-bearing dependency the rest builds on.

f0 — the whole

   producer ──do_put──►┌─ AUTH ─┐──►┌─ NornirFlightService ─┐──►  FlightSink::put
   (RecordBatch)       │ bearer │   │  do_put / do_get       │
   consumer ◄─do_get───┤ (reuse)│◄──┤  do_exchange           │◄──  FlightSource::get
                       └────────┘   └────────────────────────┘
                                     FlightServer::serve(routes, addr)
                                     over nornir_service::serve loop
                                     (TLS variant: serve_with_tls over
                                      nornir_service::serve_with_tls)

The seam is a pipe. It frames RecordBatches on/off the wire (Arrow-IPC), enforces the shared bearer auth, and routes bytes through two traits. It knows nothing about what the batches mean.

f1 — what is shared vs. what is not

• Shared (this crate): the wire — the tonic Flight bootstrap, framing + backpressure, and the three verbs, composing nornir-service's bearer-auth interceptor and serve loop. Auth is NOT re-rolled: FlightServer wraps nornir_service::auth_interceptor and drives nornir_service::serve.
• NOT shared: the schema and the meaning of the bytes. graphar-flight (knut) is a separate Flight use — Cypher → Arrow Flight-SQL — and shares nothing with this but the wire. Its server.rs was the working precedent this bootstrap mirrors, not a dependency. Slice 1 does not touch knut.

f2 — the chosen design (only the chosen one; alternatives → reasoning)

• Raw FlightService, not Flight-SQL. Build-info / skade-flight stream record batches; they do not need SQL statements, prepared statements or a schema registry. So this implements the raw arrow_flight FlightService trait with only do_put/do_get/do_exchange functional; the discovery/action verbs return UNIMPLEMENTED.
• Auth composed, not re-implemented. FlightServiceServer::with_interceptor takes nornir_service::auth_interceptor(token) directly (both are tonic 0.13, so the Request/Status/Interceptor types unify). The interceptor already no-ops on an empty token, so "empty = open" holds with no branch — the server always installs it. arrow-flight is pinned to 56.x precisely because it is the arrow line that rides tonic 0.13, matching nornir-service.
• Two traits, one default store. FlightSink (a do_put target) and FlightSource (a do_get source) are the plug-in points; MemFlightStore implements both (keyed by the descriptor path / ticket bytes) as the reference impl and a real single-box sink.
• do_exchange = store-then-echo. The bidirectional verb decodes the incoming batches, delivers them to the sink, then echoes them back byte-identically — the offload/round-trip seam.
• Routing key. A producer stamps a FlightDescriptor path (put/exchange) or a Ticket (get); the server extracts the key (path joined by /, else the cmd/ticket bytes as UTF-8) and hands it to the sink/source.
• TLS composed, not re-rolled (feature tls). FlightServer::serve_with_tls and serve_with_tls_and_auth take the caller's tonic::transport::ServerTlsConfig and hand it to nornir_service::serve_with_tls — the SAME shared TLS serve loop holger-server terminates on (LAW #5); the transport does not hand-roll Server::builder().tls_config(_)…. The client mirror is FlightConn::connect_tls (a ClientTlsConfig over https://…). All four are behind the tls feature (nornir-service/tls + tonic/tls-ring), so the plaintext consumers never pull in tonic's TLS stack. The plaintext serve/serve_with_auth/connect paths are untouched (L2, additive).

f3 — acceptance (this slice)

• Round-trip: do_put N batches → do_get and do_exchange return them byte-identical (tests/roundtrip.rs).
• Auth via the nornir-service interceptor: missing token rejected, correct accepted, empty = open.
• functional_status rows for serve / serve_with_tls / do_put / do_get / do_exchange, gated #[cfg(feature = "testmatrix")] (release strips them).
• TLS round-trip (feature tls): do_put → do_get/do_exchange over a self-signed TLS channel returns batches byte-identical, and bearer auth still guards the verb under TLS (missing token rejected) — tests/tls_roundtrip.rs, cert generated in-process with rcgen (no openssl subprocess, L9).
• HEAVY bench arm nornir-flight.put_get_throughput (feature bench) — put/get MiB/s + rows/s over a localhost Flight round-trip, host cores + peak memory stamped by the shared nornir-bench harness.

f4 — reuse map

f5 — deferred (later slices / other agents)

• Streaming FlightSink (batch-at-a-time backpressure) — today put takes the whole Vec<RecordBatch> after the client stream is decoded.

• Live subscribe-by-ticket over do_exchange (control: cancel/seek) — the viz live-tail consumer (design §f5).

• mTLS client-auth on the Flight bootstrap — serve_with_tls terminates server-side TLS today; verifying a client cert (ServerTlsConfig::client_ca_root

  • a CN/SAN policy) is a natural follow-up, matching holger's mTLS-CN model.

• graphar-flight dedup onto this seam (a separate reuse win; do not touch knut in slice 1) — blocked by the arrow56/tonic0.13 vs arrow58/tonic0.14 pin, NOT by TLS.