Apache Arrow Flight SQL for High-Throughput Telemetry: Columnar Zero-Copy vs gRPC vs MQTT (2026)

A historian query returns forty million sensor readings. The database engine answers in under a second, yet the dashboard waits half a minute. The time is not spent in the query. It is spent turning a columnar result into rows, serialising each row, pushing it through a driver, and rebuilding columns on the client so a dataframe library can use it. The transport is quietly the bottleneck of most analytics stacks, and telemetry makes it worse because telemetry is wide, numeric and enormous.

Apache Arrow Flight SQL attacks that cost directly. It moves data between systems in the same columnar layout both sides already use in memory, over gRPC, with parallel endpoints. This matters in 2026 because the engines that dominate modern telemetry analytics, including DataFusion-based time series databases and DuckDB, already speak Arrow natively. Converting to rows and back is pure overhead.

You will leave with a precise mental model of Arrow, Flight and Flight SQL, a runnable pyarrow server and client sketch, a worked bytes-on-the-wire comparison against row-oriented protobuf messages, an honest comparison with gRPC and MQTT, and a rule for when Flight is the wrong tool.

What this covers: the Arrow memory and IPC format, the Flight RPC surface, Flight SQL and ADBC, parallel endpoints, batching and backpressure, security, comparisons with gRPC and MQTT, failure modes, and a decision rule.

Context and Background

Telemetry pipelines usually have two very different legs. The first is ingest: thousands of small devices push small messages over unreliable links. The second is analytics access: a handful of heavy consumers, such as notebooks, feature pipelines, dashboards and export jobs, pull millions of rows at once. Engineers routinely use one protocol family for both legs, and that is where trouble starts. MQTT is excellent at the first leg and poor at the second. JDBC and ODBC were designed for the second leg decades before columnar engines existed.

The classic database driver path is row-oriented. A server holding columnar data reconstructs rows, encodes them in a wire format, the client driver decodes them into row objects, and the application then pivots those rows back into arrays for numpy, pandas or Polars. Each stage copies and allocates. The Arrow project documents this cost as the motivation for Flight, and the Flight SQL specification describes a protocol for database access built on Flight so that results stay columnar end to end (Arrow Flight SQL specification).

Arrow itself is a language-independent columnar memory format with a defined binary layout. Because the layout is standardised, a buffer written by a C++ process can be read by Python, Rust or Java without parsing, which is the “zero-copy” idea: the receiver interprets bytes in place rather than decoding them into a different representation. That property is what we mean by columnar telemetry transport in this post.

Flight is the RPC layer defined on top of gRPC for moving Arrow data, and Flight SQL adds a standard set of SQL-oriented commands (execute a query, list tables, prepare a statement) so that database vendors can implement one protocol and clients can reuse one driver. ADBC, Arrow Database Connectivity, is the client-side API standard that wraps this, giving applications a JDBC-like interface that returns Arrow instead of rows (ADBC documentation). If you are choosing the engine behind the endpoint, our comparison of Polars, DuckDB and DataFusion covers how each handles Arrow, and the storage side is covered in Parquet vs ORC vs Lance vs Vortex.

One framing point before we go further. This post is not an argument that Flight replaces MQTT or gRPC. It is an argument about where each belongs. The honest summary is that Flight SQL wins on the bulk read path, gRPC wins on service control calls, and MQTT wins on the constrained device edge. Figure 1 shows the placement.

Reference Architecture: Where Flight SQL Fits in a Telemetry Pipeline

Direct answer: Apache Arrow Flight SQL is a gRPC-based protocol that streams query results and bulk uploads as Arrow record batches, in the same columnar layout engines use in memory. It belongs on the analytics read and bulk-ingest path between servers, not on the constrained device link, where MQTT remains the better fit.

Figure 1: MQTT carries device traffic to a broker, an ingest consumer builds Arrow batches, and Flight SQL serves bulk reads to analytics clients and parallel workers.

The pipeline in Figure 1 has four zones. Sensors and PLCs speak whatever the field requires. An edge gateway normalises them and publishes over MQTT, usually at QoS 1 so that the broker acknowledges delivery. A consumer subscribes, accumulates messages into columnar batches, and writes them to a columnar store. The Flight SQL server sits in front of that store and answers analytics clients.

The reason to accumulate before writing is structural. A single MQTT message carries one device’s reading or a small payload. Columnar formats only pay off when many values of the same type sit next to each other, because that is what enables vectorised execution, compression and bulk copies. The ingest consumer is therefore the place where row-shaped events become column-shaped batches.

The four layers: Arrow, IPC, Flight, Flight SQL

It helps to separate four things that are often blurred together.

Arrow columnar format defines how an array of a given type is laid out in memory: a validity bitmap for nulls, an offsets buffer for variable-length data, and a contiguous data buffer. Arrow IPC defines how to serialise sequences of such arrays, called record batches, as messages, with a schema message first. Arrow Flight defines RPC methods that carry IPC messages over gRPC. Arrow Flight SQL defines a command vocabulary, expressed as protobuf messages, carried inside Flight.

Each layer is independently useful. You can write IPC files without Flight. You can implement a Flight service with no SQL at all, which is common for custom telemetry services. Flight SQL is the layer you adopt when you want interoperability with existing clients and drivers.

Why the layout makes copies unnecessary

Consider a column of float64 temperature readings. In Arrow it is a contiguous buffer of 8-byte values plus an optional validity bitmap. The IPC specification recommends aligning buffers to 64 bytes, with 8-byte alignment as the required minimum, so that consumers can use SIMD instructions on the received memory (Arrow columnar format). When those bytes arrive from a socket into an aligned allocation, the receiver can wrap them as an array directly. No per-value parse happens.

Compare that with a row protocol. Each row is a message containing a timestamp, a sensor identifier and a value. The decoder must walk the message, read field tags, decode varints, and write the result into an array. The work scales with the number of values, and it scales with a large constant.

Figure 2: An IPC stream carries a schema message, optional dictionary batches, and record batch messages whose bodies are aligned buffers for validity, offsets and data.

Figure 2 shows the structure. The metadata is a flatbuffer, which can be read without parsing, and the body is a sequence of raw buffers whose offsets and lengths the metadata describes. A reader that already holds the schema can map each buffer to an array in constant time per column, independent of the number of rows. That is the central performance claim of columnar transport, and it is an asymptotic argument, not a benchmark.

One practical caveat. Zero-copy is a property of the format, not a guarantee of the end-to-end path. TLS encryption, gRPC framing, compression and the kernel network stack still copy or transform bytes. Flight reduces the serialisation and deserialisation work, which is typically the dominant CPU cost for large numeric results, but it does not eliminate every memcpy. Be precise when you describe it to stakeholders: the claim is that no row-by-row encode or decode occurs.

Deeper Analysis: Flight RPC, Flight SQL, ADBC and Parallel Endpoints

Direct answer: Flight defines a small set of gRPC methods: GetFlightInfo describes where and how to fetch a dataset, DoGet streams it out, DoPut streams it in, and DoExchange is bidirectional. Flight SQL layers SQL commands on top, and ADBC wraps the client side so applications receive Arrow results through a familiar connect, execute, fetch API.

The Flight RPC surface

The Flight service exposes a defined set of methods in its protobuf service definition. Handshake lets a client and server exchange credentials and obtain a token. ListFlights enumerates available datasets. GetFlightInfo takes a descriptor, which is either a path or an opaque command, and returns a FlightInfo. GetSchema returns only the schema. DoGet takes a ticket and streams back record batches. DoPut streams batches from client to server. DoExchange opens a bidirectional stream. DoAction and ListActions carry custom operations that do not fit the data-streaming model (Arrow Flight RPC documentation).

The separation between GetFlightInfo and DoGet is the most important design decision in the protocol, and it is the one newcomers miss. GetFlightInfo does not move data. It returns a FlightInfo containing the schema, optional row and byte counts, and a list of endpoints. Each endpoint carries a ticket, an opaque token that DoGet will honour, and optionally a list of locations, meaning the hosts that can serve that ticket. If no location is given, the client is expected to return to the same server.

This two-step design enables three things. First, planning and data movement are decoupled, so a coordinator can plan a query and hand out tickets while workers serve the data. Second, a result can be split into several endpoints that the client fetches in parallel. Third, endpoints can point at different machines, so the coordinator is not on the data path.

Figure 3: The coordinator plans the query and returns tickets. The client then fetches endpoints in parallel directly from workers.

Flight SQL: commands inside Flight

Flight SQL does not add new gRPC methods. It reuses Flight and packs protobuf command messages into the descriptor or action body, wrapped in a protobuf Any. The specification defines commands such as CommandStatementQuery to run a SQL string, CommandStatementUpdate for statements that return a row count, CommandPreparedStatementQuery for prepared statements, and a family of metadata commands such as CommandGetTables, CommandGetCatalogs, CommandGetDbSchemas, CommandGetSqlInfo and CommandGetPrimaryKeys. Prepared statements are created and closed using Flight actions (Flight SQL specification).

The query flow therefore looks like this. The client sends GetFlightInfo with a descriptor whose command is a CommandStatementQuery. The server plans the query, and returns a FlightInfo with one or more endpoints. The client calls DoGet on each endpoint with its ticket. Result batches stream back. For writes, the client uses DoPut with a command such as CommandStatementIngest, which the specification added for bulk ingestion; if you rely on it, check that your server version implements it, because support is optional per command.

Optionality is a real portability concern. Flight SQL servers advertise capabilities through CommandGetSqlInfo, and a conforming client should consult it rather than assuming support for transactions, prepared statements or particular SQL dialect features. The protocol standardises the transport and command shapes, not the SQL dialect. A query that works against a DataFusion-based server may not parse on another engine.

ADBC: the client side

ADBC is an API standard with a driver model, analogous to JDBC and ODBC, but where the result type is Arrow. Applications load a driver, open a connection, create a statement and fetch an Arrow table or a stream of record batches. Drivers exist as native libraries that can be loaded by language bindings. The Flight SQL driver speaks the Flight SQL protocol to any compliant server, and separate drivers exist for databases such as PostgreSQL and SQLite, where the driver converts the database’s native protocol into Arrow (ADBC drivers).

Two points deserve emphasis. First, ADBC is not Flight SQL. Flight SQL is a wire protocol; ADBC is a client API. When the backing driver is not Flight SQL, ADBC still gives you a uniform API but the speed-up depends on how natively the database produces columnar data. A PostgreSQL ADBC driver, for example, still receives PostgreSQL’s wire format and converts it. Second, the existence of an ADBC driver for your database does not mean the network path is columnar. Check what the driver actually does.

For Python, the adbc_driver_flightsql package provides a DB-API 2.0 compatible interface plus Arrow-specific methods such as fetch_arrow_table. That means a notebook can run a query and receive a pyarrow Table, then hand it to Polars or DuckDB without a row pivot.

Parallel endpoints and why they matter for telemetry

Telemetry results are typically partitioned already, by time range, device group or file. A Flight server can map partitions to endpoints directly. Suppose a query touches eight hourly partitions. The server returns eight endpoints. A client with eight worker threads, or a distributed framework with eight tasks, fetches them concurrently, each over its own gRPC stream.

This is Flight’s answer to the single-connection ceiling of JDBC. A single gRPC stream is bounded by one TCP connection’s congestion window and by single-thread serialisation on both ends. Parallel endpoints multiply the achievable throughput roughly by the number of independent streams until some other resource, usually network bandwidth or server disk, saturates. That is an architectural expectation rather than a measured figure; real scaling depends on your network and storage.

Parallel endpoints also change failure semantics. Each ticket is independently retryable if the server makes tickets idempotent. A client that loses one stream can re-issue DoGet for that ticket alone. Many servers cannot guarantee this, since a ticket may encode a position in a cursor that is consumed on read, so test retry behaviour explicitly rather than assuming it.

A runnable pyarrow sketch

The following is a minimal Flight service built with pyarrow, plus a client that fetches data. It uses the plain Flight API, not the SQL layer, because that is what ships in pyarrow’s pyarrow.flight module and it demonstrates the mechanisms: descriptors, endpoints, tickets, DoGet and DoPut. This code has not been executed for this article; treat it as an illustrative sketch and test it against your installed pyarrow version.

import threading
import numpy as np
import pyarrow as pa
import pyarrow.flight as fl

SCHEMA = pa.schema([
    ("ts", pa.timestamp("us")),
    ("sensor", pa.dictionary(pa.int32(), pa.string())),
    ("value", pa.float64()),
])

def make_batch(n, start_us):
    ts = pa.array(start_us + np.arange(n) * 1000, type=pa.timestamp("us"))
    sensor = pa.array(np.random.choice(["t1", "t2", "t3"], n)).dictionary_encode()
    value = pa.array(np.random.randn(n) * 2 + 20)
    return pa.record_batch([ts, sensor, value], schema=SCHEMA)

class TelemetryServer(fl.FlightServerBase):
    def __init__(self, location):
        super().__init__(location)
        self.store = {}          # partition name -> list of batches
        self.lock = threading.Lock()

    def get_flight_info(self, context, descriptor):
        name = descriptor.path[0].decode()
        with self.lock:
            parts = [p for p in self.store if p.startswith(name)]
        endpoints = [fl.FlightEndpoint(p.encode(), []) for p in parts]
        return fl.FlightInfo(SCHEMA, descriptor, endpoints, -1, -1)

    def do_get(self, context, ticket):
        with self.lock:
            batches = self.store[ticket.ticket.decode()]
        return fl.GeneratorStream(SCHEMA, iter(batches))

    def do_put(self, context, descriptor, reader, writer):
        part = descriptor.path[0].decode()
        batches = [chunk.data for chunk in reader]
        with self.lock:
            self.store.setdefault(part, []).extend(batches)

if __name__ == "__main__":
    server = TelemetryServer("grpc://0.0.0.0:8815")
    server.serve()

import pyarrow.flight as fl
from concurrent.futures import ThreadPoolExecutor

client = fl.connect("grpc://localhost:8815")

# Write: stream batches into a partition with DoPut
desc = fl.FlightDescriptor.for_path("plant1-2026110508")
writer, _ = client.do_put(desc, SCHEMA)
for i in range(100):
    writer.write_batch(make_batch(10_000, i * 10_000_000))
writer.close()

# Read: plan, then fetch every endpoint in parallel
info = client.get_flight_info(fl.FlightDescriptor.for_path("plant1"))

def fetch(endpoint):
    c = fl.connect("grpc://localhost:8815")  # or endpoint.locations[0]
    return c.do_get(endpoint.ticket).read_all()

with ThreadPoolExecutor(max_workers=len(info.endpoints)) as pool:
    tables = list(pool.map(fetch, info.endpoints))

Read the code for the mechanisms rather than the style. The server never serialises rows. The dictionary-encoded sensor column sends each distinct string once per batch dictionary and then small integer indices. The client builds nothing; read_all assembles record batches into a Table whose buffers are the received bytes.

Worked Example: Bytes on the Wire for Row Messages vs Columnar Batches

To make the layout argument concrete, take an illustrative workload: one record batch of 10,000 readings, each with a microsecond timestamp, a sensor identifier drawn from a few hundred devices, and a float64 value. The numbers below are derived arithmetic from format definitions, not measurements.

In Arrow, the timestamp column is 8 bytes per value, so 80,000 bytes. The value column is 8 bytes per value, so another 80,000. The sensor column, dictionary-encoded with int32 indices, is 4 bytes per value, so 40,000 bytes, plus the dictionary itself, perhaps a few kilobytes for a few hundred short strings. Validity bitmaps are one bit per value, about 1,250 bytes each, and can be omitted entirely when a column has no nulls. The total is roughly 200 KB of body, plus a small flatbuffer header, so about 20 bytes per reading.

In a row-oriented protobuf stream carrying one message per reading, each message has a field tag per field, a varint or fixed-width timestamp, a string or integer sensor field, and a double. Protobuf encodes a double as 8 bytes plus a 1-byte tag, a timestamp as a varint that for microsecond epoch values takes around 8 or 9 bytes plus a tag, and a short string sensor name as a length byte plus its characters plus a tag. That puts a typical message near 25 to 35 bytes, before the per-message length prefix in a repeated field or the per-message framing in a stream. So the byte count differs by perhaps a third, and not by an order of magnitude. Wire size is not where columnar wins.

The win is CPU. Decoding the protobuf stream means roughly 30,000 field reads for 10,000 messages and 10,000 object or struct allocations unless the library uses arena allocation, followed by a pivot into arrays. Decoding the Arrow batch means reading one metadata flatbuffer and mapping three or four buffers. The work no longer depends on the row count apart from the memory copy from the network.

There is a second win that the bytes hide: compression behaves differently. Columns of timestamps with a regular cadence, or floats from a slow-moving process, compress better than interleaved rows because similar values are adjacent. Arrow IPC supports buffer-level compression with LZ4 frame and Zstandard, applied per buffer (Arrow IPC format). The cost is CPU on both ends, and you give up the strict zero-copy read for compressed buffers because they must be decompressed. On a fast local network, uncompressed is often right; across a WAN link, compressing usually pays. Measure on your own data.

The dictionary encoding choice also affects streaming. Dictionaries are sent as separate messages. In a long-lived stream with a growing vocabulary of sensor names, the sender uses dictionary deltas or replaces dictionaries, and the receiver must support whichever mode is used. If you do not control both ends, prefer sending plain strings or integer device keys and joining names later.

Flight SQL vs gRPC with Protobuf vs MQTT for Telemetry

Direct answer: Use MQTT to move small messages from constrained devices to a broker, gRPC with protobuf for typed service-to-service calls, and Arrow Flight SQL to move large columnar result sets or bulk batches between servers. They solve different problems, and most production telemetry stacks use two or three of them together.

Figure 4: Choose by position in the pipeline. Device links favour MQTT, bulk analytics reads favour Flight SQL, and control calls favour plain gRPC.

What each protocol is optimised for

MQTT is a publish-subscribe protocol with a small header, three delivery levels (QoS 0, 1 and 2), retained messages, last-will messages and, in version 5, features such as shared subscriptions and user properties. It runs over a persistent TCP connection, tolerates intermittent links through session persistence, and has a minimal fixed header, which is why it dominates constrained device networks. Its unit is the single message. The MQTT specification allows very large payloads, but brokers typically enforce limits far lower, and the protocol has no notion of schema, columns, query or streaming a result set (MQTT 5.0 specification, OASIS).

gRPC with protobuf gives typed, schema-first RPC over HTTP/2, with unary and streaming calls, deadlines, cancellation and flow control per stream. It is the right default for service interfaces. For bulk numeric data it is mediocre if you model each record as a message, because per-message overhead and row-wise encoding apply. gRPC also has a default maximum receive message size of 4 MB in most implementations, so large payloads must be chunked or the limit raised. Flight sits on gRPC and inherits its HTTP/2 flow control, deadlines and TLS, but replaces the payload model with Arrow batches.

Flight SQL is built for the large, columnar, read-heavy case. Its unit is the record batch, its control flow is plan then fetch, and its schema travels once at the start of a stream.

Dimension	MQTT	gRPC + protobuf rows	Arrow Flight / Flight SQL
Unit of transfer	One message	One message, or stream of messages	Record batch of thousands of rows
Schema handling	None; payload is opaque or Sparkplug protobuf	Compiled protobuf schema	Arrow schema sent once per stream
Messaging model	Pub/sub via broker	RPC, client/server	RPC, client/server with parallel endpoints
Best at	Device uplink, fan-out, flaky links	Typed service calls	Bulk analytics read, bulk ingest
Query support	None	Application defined	Standard via Flight SQL commands
Consumer CPU for numeric data	Application decode per message	Per-field decode	Near constant per batch
Delivery guarantees	QoS 0/1/2, session persistence	Per call, retries by client	Per stream, retries by client
Typical weakness	Large payloads, analytics	Row overhead on bulk	Poor fit for tiny messages and device links

Why not just MQTT for everything?

Because MQTT has no backpressure model aligned with analytics. A subscriber that cannot keep up either loses messages at QoS 0 or forces the broker to queue, and broker queues have finite memory and policies for dropping. Analytics consumers pulling history need to say “give me the next block when I am ready”, which is exactly what gRPC stream flow control provides. MQTT also has no query: to replay last Tuesday you need another system.

The reverse is also true. Running Flight to every field device is unrealistic. A Flight client needs HTTP/2, a gRPC stack and TLS, which is heavy for microcontrollers, and gRPC connections do not tolerate drops as gracefully as MQTT sessions designed for intermittent cellular links. For constrained telemetry design, our post on MQTT in industrial settings applies; for brokers, the MQTT 5 shared subscription feature lets you scale the ingest consumer horizontally.

Where Flight beats generic gRPC

If you already use gRPC with protobuf, you can model a batch as a message with repeated fields per column, repeated double value, which is a columnar protobuf encoding and gets part of the benefit. That is a legitimate design. What Flight adds is interoperability and tooling. Every Arrow-aware engine can consume the stream without a custom decoder, nullability and nested types follow a standard, the two-phase plan-and-fetch with parallel endpoints is already defined, and Flight SQL clients already exist. The trade-off is that you adopt Arrow’s type system and a larger dependency.

Batching, Backpressure, Auth and Operations

Batch size is the main tuning knob

A record batch is the unit of work on both ends, so its size controls three competing costs. Small batches, such as a few hundred rows, incur per-batch metadata and gRPC message overhead, and starve vectorised kernels. Very large batches use more memory on both sides, increase latency to first row, and risk exceeding gRPC message limits. Arrow guidance and engine defaults commonly sit in the range of thousands to low millions of rows per batch, with many engines defaulting near the tens of thousands; the right number depends on row width and cache sizes, so treat this as a starting range and tune against your workload. A practical method is to target batch bodies in the low megabytes, below the configured message limit, then adjust by measuring throughput and tail latency together.

For ingest, batching has a freshness cost. If your consumer waits to accumulate 100,000 readings, data is invisible to queries until the batch closes. A common compromise is a dual trigger: flush when the batch reaches a row count or after a time interval, whichever comes first. The time trigger bounds staleness; the row trigger bounds memory. Very small flushes caused by the time trigger produce many tiny files in the store, which hurts query performance later and calls for compaction.

Backpressure

Flight inherits HTTP/2 stream-level flow control through gRPC. When a DoGet consumer reads slowly, its receive window fills, the server’s write blocks, and the server’s producer thread is throttled. This is true backpressure and it is one of Flight’s advantages over a push system. Two caveats apply. First, some server implementations buffer a number of batches ahead of the socket, so memory use can rise by the buffer depth times the batch size per active stream. Second, if the server generates results into an unbounded queue before the writer, backpressure never reaches the producer. When you implement a server, confirm that the producer is pulled by the writer, as with a generator stream, not pushed.

For DoPut the direction reverses: a slow server throttles the client writer. Ingest clients should treat a blocked write_batch as a signal to spill to local disk or a durable queue, not as an error to retry blindly.

Authentication, TLS and authorisation

Flight supports TLS via the grpc+tls:// scheme and mutual TLS through client certificates. It also defines a Handshake-based authentication flow in which the client sends credentials and receives a token that is attached to subsequent calls, plus middleware hooks for bearer tokens and custom headers. Flight SQL clients commonly send credentials with HTTP basic authentication on the first call and then a bearer token. The detailed behaviour is implementation-specific, so read your server’s documentation.

Authorisation is not part of the protocol. The server must decide, per command and per ticket, whether the caller may read the data. This is where tickets deserve care. A ticket is an opaque byte string the server issues, and if it merely encodes a partition name, a client that learns the name of another tenant’s partition can forge a ticket. Sign tickets, bind them to the authenticated identity, give them a short lifetime, or re-check authorisation inside DoGet instead of trusting the ticket. Also remember that endpoint locations are fetched by the client directly, so workers must authenticate the client too and not rely on the coordinator having done so.

Operating Flight in production

Load balancing needs thought. gRPC uses long-lived HTTP/2 connections, so a layer-4 balancer pins a client to one backend for the connection’s lifetime. Parallel endpoints help here, because the server controls which locations it returns and can spread tickets across workers. Layer-7 proxies must support HTTP/2 and gRPC streaming with long timeouts, and default idle or request timeouts on proxies will cut long DoGet streams. Observability should track per-stream throughput, batch sizes, time to first batch, and aborted streams, since a slow query shows up as a stalled stream rather than a slow response.

Engines in the Ecosystem: What Speaks Flight SQL

The practical question is whether the systems you already run support the protocol. As of this writing, several do, but support differs in completeness, so verify the version you deploy.

InfluxDB 3 is built on the FDAP stack, named for Flight, DataFusion, Arrow and Parquet, and its documentation describes querying through Flight SQL, using DataFusion as the query engine over Parquet storage (InfluxData FDAP architecture). For telemetry teams this is the most direct example: a time series database where Arrow Flight is the primary bulk query interface and client libraries return Arrow tables.

Apache DataFusion is a Rust query engine that operates on Arrow in memory. The DataFusion project provides building blocks and examples for exposing a Flight SQL service, which makes it a common base for custom servers. I did not verify the current feature completeness of those examples in this run, so check the repository before depending on a specific command.

DuckDB consumes and produces Arrow efficiently within a process, including zero-copy scans of Arrow tables from Python. Reaching a remote Flight SQL endpoint from DuckDB depends on extensions, and I have not verified a particular extension’s maturity here, so confirm before planning around it. A frequent pattern that does work is fetching Arrow data via ADBC into a pyarrow Table and querying that table in DuckDB or Polars with no copy.

Dremio and several other analytics products expose Flight endpoints, and the Arrow Flight SQL JDBC driver lets JVM tools connect to any compliant server, which gives legacy BI tools a path, though through JDBC’s row API the columnar advantage is partly lost. That is an important limitation: the speed benefit depends on both ends staying columnar.

The common theme is that Flight SQL is strongest when the server is already columnar and the client is a dataframe or analytics engine. It is weakest when either side forces a row pivot.

Trade-offs, Gotchas, and What Goes Wrong

Small messages are a bad fit. Flight’s per-call overhead, gRPC stream setup and Arrow schema message are amortised over thousands of rows. Sending one reading per DoPut is slower than almost anything. If you cannot batch, you do not want Flight.

Schema evolution is manual. An Arrow stream has one schema. Adding a sensor channel mid-stream means a new stream. Telemetry sources whose fields drift need either a wide nullable schema, a long-form layout of timestamp, key and value, or schema versioning in the descriptor. Long-form is flexible but gives up much of the columnar density, since the key column repeats.

Time zones and units. Arrow timestamps carry a unit, second to nanosecond, and an optional time zone string. Mixed units between producer and consumer cause silent scale errors if code casts carelessly. Fix the unit in the schema contract, and test for it.

Memory pressure from large results. read_all materialises the whole result. For a billion rows, iterate over batches and process incrementally. The same holds on the server: building a full result before streaming negates backpressure and invites out-of-memory kills.

Partial failure. A DoGet stream that dies after 70 percent of the data has delivered 70 percent. Flight offers no built-in resume. Unless you design tickets with offsets or make queries deterministic, the client must restart the endpoint, so idempotent reads and per-endpoint retry are your responsibility.

Version and feature drift. Flight SQL commands are optional and have been extended over releases, for example the bulk ingest command is more recent than the core query commands. Client and server library versions differ in what they support, so pin versions and test the commands you use.

Not a message bus. Flight has no pub-sub, no durable queue, no consumer groups. Teams sometimes try DoExchange for streaming fan-out and rebuild a worse broker. Pair Flight with a broker or log, as in Figure 1, not instead of one.

Security surface. gRPC endpoints exposed to analysts invite misuse: unbounded queries, giant result sets, ticket forgery. Apply per-user quotas, query timeouts and result-size limits at the server.

Practical Recommendations

Start by drawing the two legs of your pipeline separately. Keep MQTT, or whatever your devices already speak, for the uplink. Insert a batching consumer that produces Arrow record batches and writes them to a columnar store with a sensible partition layout. Put a Flight SQL endpoint in front of the store only if you have heavy bulk readers; if your consumers are mostly dashboards issuing small aggregate queries, a normal HTTP API may be simpler.

When you do adopt Flight, prefer a server whose engine is natively Arrow, such as a DataFusion-based or DuckDB-based service, so no hidden row conversion appears. Use ADBC in clients so the application code is independent of the specific driver. Design partitions to become endpoints. Test the whole path with your own data, because the gain depends on row width, null rates, compression and network.

A short checklist:

• Confirm the ingest path batches at least thousands of rows per record batch, with a time-based flush trigger.
• Fix the Arrow schema contract: units, time zone, nullability, dictionary or plain strings.
• Return multiple endpoints for partitioned results and let clients fetch in parallel.
• Enforce TLS, authenticate in every worker, and sign or re-validate tickets.
• Set message size limits, query timeouts and per-user result caps.
• Stream on both sides; never materialise a full result unless it fits comfortably in memory.
• Pin client and server library versions and test each Flight SQL command you rely on via CommandGetSqlInfo.
• Measure end to end, including serialisation CPU, with your own data before and after.

Frequently Asked Questions

What is Apache Arrow Flight SQL?

Apache Arrow Flight SQL is a protocol for database access built on Arrow Flight, which in turn runs over gRPC. A client sends SQL commands such as query, prepared statement and metadata requests, and the server returns results as Arrow record batches. Because results stay columnar, clients avoid row-by-row decoding. Servers can also split results across multiple endpoints so clients fetch in parallel.

Is Arrow Flight faster than gRPC with protobuf?

Flight is gRPC underneath, so the difference is the payload. For large numeric datasets, Arrow batches avoid per-row encoding and decoding and map directly into the buffers engines use, which cuts CPU. For small messages or a few fields, protobuf is fine and often simpler. The gain depends on data shape and network, so benchmark with your own telemetry rather than trusting general claims.

Can Arrow Flight replace MQTT for IoT telemetry?

No. MQTT is designed for constrained devices, flaky links, small messages and publish-subscribe fan-out, while Flight needs a full gRPC stack and works best on large batches between servers. The usual design uses MQTT to reach a broker, an ingest consumer to build Arrow batches, and Flight SQL for analytics reads and bulk transfers downstream.

What is the difference between ADBC and Flight SQL?

Flight SQL is a wire protocol between a client and a database server. ADBC is a client API standard, comparable to JDBC or ODBC, that returns Arrow data. ADBC has a Flight SQL driver that speaks the protocol, but it also has drivers for other databases that convert their native formats. Using ADBC does not by itself guarantee a columnar network path.

Does zero-copy mean no copies at all?

No. Zero-copy describes the Arrow format: a receiver can interpret received bytes as arrays without decoding them into a new representation. The network stack, TLS, gRPC framing and any compression still touch the bytes. The saving is the removal of serialisation and deserialisation work and extra allocations, which is typically the dominant CPU cost for large numeric results.

When should I not use Arrow Flight SQL?

Skip it when messages are small and cannot be batched, when clients are microcontrollers or on lossy cellular links, when you need publish-subscribe semantics or durable queues, or when your consumers are legacy tools that force row conversion through JDBC anyway. In those cases MQTT, a broker or a plain gRPC or HTTP API is the better fit.

Further Reading

• Polars vs DuckDB vs DataFusion: Arrow dataframe engine comparison for the engines behind your endpoint.
• Parquet vs ORC vs Lance vs Vortex columnar file formats for the storage layer Flight reads from.
• Iceberg catalogs: Polaris vs Nessie vs Unity for table metadata behind lakehouse telemetry.
• Arrow Flight SQL specification and Arrow Flight RPC, the primary sources for this post.

By Riju — about