nordisk/knut

graphar-flight — reasoning

Class: reasoning. The why, the trade-offs, the rejected alternatives, and the arrow-58 history. The descriptive shape we actually run is in flight-design.md, which by rule mentions none of this. Reuse: flight.uses = [falkordb, graphar] — see falkordb, graphar.

Why registered-queries-are-tables

The whole point is that a generic driver can read knut. ADBC/ODBC Flight SQL clients — Power BI's connector, adbc_driver_flightsql — don't speak Cypher; the only query they emit against a table they discovered is SELECT * FROM "<table>". So the design meets them there: a registered query is advertised as a table in GetTables, and parse_select_from + resolve_table rewrite that trivial SELECT back to the underlying Cypher. We deliberately did not build a SQL parser — anything that isn't a single-table SELECT … FROM is taken as Cypher verbatim, so a Cypher-aware client keeps full power while a dumb one still gets its table. The ticket carries the resolved Cypher, so DoGet never re-resolves.

Why one AuthConfig, two enforcement points

A tonic Interceptor runs before routing and cannot see which Flight method is being called. The Flight SQL handshake is itself just another call passing through the interceptor, carrying a Basic header — but the per-call path also needs to accept the Bearer token clients replay afterwards. If the interceptor demanded Bearer, the Basic handshake would be rejected before it could mint a token; if it demanded Basic, replayed bearer calls would fail. So the single acceptance rule is Bearer OR Basic, and both the interceptor and do_handshake consult the same AuthConfig (check_header is literally what the handshake calls). One config means the two can never drift out of agreement about who is allowed in. mTLS is orthogonal — it's transport identity, set on the tonic server, and composes with either header method. An open server installs no interceptor at all, so the zero-auth path stays byte-identical.

Why a bearer-token set

A single shared secret can't be rotated without a flag-day: the instant you change it, every client still holding the old one breaks. Making the accepted tokens a list turns rotation into a safe overlap: add the new token (both now valid) → migrate clients at leisure → drop the old. The handshake always issues the first token, so "the current token" is well-defined even mid-rotation. with_named_bearer_tokens goes further — several tokens can resolve to one authz identity, so an analyst role rotates its secret without its authorization changing. A one-element set is exactly the old single-token behavior, which is why with_bearer_token still exists and existing servers see no change.

Why authenticate → Identity, then authorize

Authentication answers who; authorization answers what may they run. Folding them together would mean re-deriving the caller at each Cypher entry point from the raw header. Instead the interceptor resolves an Identity once (Token(name) / Basic(user) / Anonymous) and stashes it as a request extension; the service reads it back and asks the Authorizer about the resolved query name. That split is why authz can key on a friendly table name (register_named) rather than a raw Cypher string, and why the default AllowAll is a genuine no-op — an existing server that sets no authorizer runs the identical code path with an always-true check. Denials become permission_denied at exactly the five entry points where a Cypher query enters (both statement paths, both prepared reads, and prepared-statement creation) — there is no sixth door.

Why execute-once (the H9 schema cache)

Auto-schema mode has a structural problem: GetFlightInfo must know the result shape to describe it, and the only way to learn an unregistered query's shape is to run it (FalkorDB returns column names in the response header, but only in a real reply). Naively you'd run it again in DoGet — two identical, possibly expensive graph queries per client read. Instead the GetFlightInfo batch is parked in a bounded SchemaCache keyed by resolved Cypher, and the matching DoGet removes-and- streams it. The cache is bounded (SCHEMA_CACHE_CAP = 16) precisely because a client can call GetFlightInfo and never follow up (or vanish mid-flight); when full, an arbitrary stale entry is evicted, since these are short-lived hand-offs where any abandoned one is fair game. Registered (typed) queries skip all of this — their schema is known without executing, so they never cache and always execute fresh in DoGet.

The honest gotcha: chunked export is slice-and-write, not pull-stream

export_cypher_chunked looks like streaming but is not a pull-stream, and the reason is upstream, not us. FalkorDB's GRAPH.QUERY is a single RESP request/response: the server serializes the entire result set into one reply with no server-side cursor, so query_auto always materializes the full result as one in-memory RecordBatch. There is no API to pull the result in pages — the read side cannot be bounded below the whole result. What chunking does bound is the write side: the fetched batch is zero-copy-sliced (RecordBatch::slice shares the Arrow buffers — no row copy) into chunk_rows-sized batches, each written and dropped before the next, so peak encoder memory (parquet row groups, CSV/JSON string buffers) and per-file size are bounded, and read_chunk-per-file streams the result back chunk by chunk. True end-to-end streaming would need a FalkorDB cursor API that RESP doesn't expose. Calling this out matters because the function's shape invites the assumption that huge results never fully land in memory — they do, once, on fetch.

History: the arrow-58 convergence (the FFI bridge is gone)

The export → Iceberg path (feature skade) once had to cross an Arrow major-version boundary: knut built on arrow-56, skade on arrow-57, and arrow_array::RecordBatch from two different major versions are distinct, incompatible types. bridge_56_to_57 was a real bridge then — a C Data Interface FFI transmute that reinterpreted the batch across the ABI without copying buffers. That whole hazard is now gone: knut and skade share a single Arrow 58 tree, so skade::arrow_array::RecordBatch is arrow_array::RecordBatch (same crate, same major). bridge_56_to_57 survives only as an identity clone — a plain clone() (Arc bump, no buffer copy) — kept as a named seam so the export path and its round-trip test read the same, and so the name in the git history still resolves to something. The temp-parquet variant append_arrow_to_iceberg_via_parquet is retained purely as the comparison baseline for the two skade benchers (knut.skade_iceberg_export vs …_export_parquet); everywhere else the zero-copy path is the default. The skade feature stays off by default because it pulls a heavy Iceberg dependency tree; it depends on skade from crates.io, with no path coupling to the sibling repo, and its tests run without a container (skade's catalog is an embedded redb warehouse in a tempdir).