Fast result transfer (Arrow)¶
Dosi can move query results as Apache Arrow — columnar end to end for engines whose wire format is already columnar. The rows are the same; what changes is how much encoding and parsing sits between the warehouse and your client. This page covers which engines support it, what to configure, and which APIs get faster. Connector setup is in Connect a warehouse.
Which engines support it¶
| Engine | Arrow path |
|---|---|
| DuckDB | in-process, zero-copy |
| ClickHouse | FORMAT ArrowStream over the same HTTP connection |
| StarRocks, Doris | Arrow Flight SQL (opt in per profile, see below) |
| Databricks | ARROW_STREAM results |
| Snowflake, BigQuery | not yet — both execute normally over their REST APIs and return rows |
| Every other connector | rows only — their wire formats are row-oriented, so converting to Arrow on the client would gain nothing |
Do I need to change my configuration?¶
Mostly no. Existing connection files keep working unchanged; the row path stays the default everywhere and behaves identically.
| Engine | Config change to get Arrow |
|---|---|
| DuckDB | none — in-process Arrow is the engine's native path |
| ClickHouse | none — same HTTP profile; the adapter requests FORMAT ArrowStream itself |
| Databricks | none beyond the normal profile (host, warehouse, password: = PAT) |
| StarRocks | add one key: arrow_flight_port: 9408 (the FE arrow_flight_port; needs SR ≥ 3.5.1). Key present = Flight SQL; key absent = MySQL wire |
| Doris | add one key: arrow_flight_port: 8070 (the FE arrow_flight_sql_port; Doris ≥ 2.1.5 recommended). Same opt-in rule |
Example:
datasources:
prod-sr:
type: starrocks
host: sr.internal
port: 9030 # MySQL-wire port, still used as fallback/seeding
arrow_flight_port: 9408 # presence opts this profile into Arrow Flight SQL
username: osi
password: ${SR_PASSWORD}
database: analytics
There is deliberately no result_format: knob: each adapter has one
native decode, and the egress format is chosen per request (Accept header /
CLI flag), not per datasource.
Build features¶
The release binaries already include every Arrow path; the -lean binaries
include all of them except in-process DuckDB. Building from source, these
Cargo features turn them on:
| Feature | What it turns on | In default build? |
|---|---|---|
exec-duckdb |
in-process libduckdb, Arrow-native | yes (--no-default-features = the lean build, which shells out to a duckdb CLI) |
exec-http-arrow |
ClickHouse FORMAT ArrowStream |
no (implies exec-http) |
exec-flightsql |
StarRocks/Doris Arrow Flight SQL client | no |
exec-databricks |
Databricks Statement Execution API, inline JSON rows | no |
exec-databricks-arrow |
Databricks ARROW_STREAM + EXTERNAL_LINKS |
no (implies exec-databricks) |
exec-all |
all of the above | — |
dosi-server arrow |
Arrow IPC responses on /v1/query/execute |
yes (server default) |
Which APIs get faster?¶
POST /v1/query/executewithAccept: application/vnd.apache.arrow.stream— the headline win. Batches stream from the warehouse adapter straight into the HTTP body as Arrow IPC: no per-value JSON encoding on the server, no JSON parsing in the client, and columnar consumers (Polars/pandas/DataFusion/ another dosi) read it zero-parse. The JSON response stays byte-identical for existing clients. Benefit scales with result size: negligible for a 10-row aggregate, decisive for high-cardinality group-bys and detail/export queries (10⁵+ rows).dosi query --execute --format arrow— Arrow IPC on stdout for piping into DuckDB/Polars/anything, skipping the table/JSON rendering entirely.- StarRocks/Doris large reads — Flight SQL reads bypass FE row forwarding (client pulls columnar results from BEs in parallel); vendors measure 20–160x vs the MySQL protocol on large result sets. Small aggregates: ~no change.
- ClickHouse — server-side Arrow serialization (lz4-framed) replaces JSON encode/decode on both ends; also removes float/decimal text round-trips.
- DuckDB (default engine) — in-process zero-copy replaces spawning a CLI
per query and parsing its JSON: lower latency on every
--execute, and in-memory databases persist across statements in one session. - What does NOT get faster: compile-only APIs (
/v1/query/compile,explain) — they never touch results; small metric queries on row-wire engines (MySQL/TiDB/Postgres/Trino) — unchanged by design.
Every Arrow path is checked against the test corpus before it ships: Arrow results ≡ row results ≡ the DuckDB reference.