nordisk/edda

nornir-flight — for idiots

What is it? A pipe for tables. You put Arrow tables (RecordBatch) in one end; someone else pulls the exact same tables out the other end, even from another machine. That's it.

Why does it exist? Lots of nordisk things want to stream tables over the network: build logs/metrics, graph query results, warehouse shuffles. Instead of each one writing its own network code, they all use this ONE pipe. (That's the "don't repeat yourself" law.)

The three moves (Arrow Flight calls them "verbs"):

• do_put — send tables in. ("Here, hold these.")
• do_get — pull tables out by a name. ("Give me the ones called build/42.")
• do_exchange — send and get back in one call. ("Take these and hand them right back.")

The pinky-promise: what you put is what you get — byte-for-byte. If that ever breaks, the test put_then_get_roundtrip_byte_identical turns red.

Locks on the door (auth): the server can require a password (a "bearer token"). No password or the wrong one → the door stays shut (UNAUTHENTICATED). An empty password means the door is open (fine for your own laptop). The lock itself is borrowed from nornir-service — we did not build a new lock.

Sealed tube (TLS): turn on the tls option and the pipe becomes an encrypted tube — nobody snooping the wire can read your tables. The lock (the password) and the tube (the encryption) are both borrowed from nornir-service; we didn't build new ones. Test it with cargo test -p nornir-flight --features tls.

What we did NOT do: we did not teach this pipe anything about what the tables mean. It moves bytes. The thing that knows what a "build log" row is lives somewhere else and just uses the pipe.

Try it:

cargo test -p nornir-flight
cargo run --release -p nornir-flight --features bench --example nornir-bench

The first proves the pipe is honest; the second measures how fast it is.