Skip to content

Fast result transfer (Arrow)

Dosi can move query results as Apache Arrow — columnar end to end for engines whose wire format is already columnar. The rows are the same; what changes is how much encoding and parsing sits between the warehouse and your client. This page covers which engines support it, what to configure, and which APIs get faster. Connector setup is in Connect a warehouse.

Which engines support it

Engine Arrow path
DuckDB in-process, zero-copy
ClickHouse FORMAT ArrowStream over the same HTTP connection
StarRocks, Doris Arrow Flight SQL (opt in per profile, see below)
Databricks ARROW_STREAM results
Snowflake, BigQuery not yet — both execute normally over their REST APIs and return rows
Every other connector rows only — their wire formats are row-oriented, so converting to Arrow on the client would gain nothing

Do I need to change my configuration?

Mostly no. Existing connection files keep working unchanged; the row path stays the default everywhere and behaves identically.

Engine Config change to get Arrow
DuckDB none — in-process Arrow is the engine's native path
ClickHouse none — same HTTP profile; the adapter requests FORMAT ArrowStream itself
Databricks none beyond the normal profile (host, warehouse, password: = PAT)
StarRocks add one key: arrow_flight_port: 9408 (the FE arrow_flight_port; needs SR ≥ 3.5.1). Key present = Flight SQL; key absent = MySQL wire
Doris add one key: arrow_flight_port: 8070 (the FE arrow_flight_sql_port; Doris ≥ 2.1.5 recommended). Same opt-in rule

Example:

datasources:
  prod-sr:
    type: starrocks
    host: sr.internal
    port: 9030                # MySQL-wire port, still used as fallback/seeding
    arrow_flight_port: 9408   # presence opts this profile into Arrow Flight SQL
    username: osi
    password: ${SR_PASSWORD}
    database: analytics

There is deliberately no result_format: knob: each adapter has one native decode, and the egress format is chosen per request (Accept header / CLI flag), not per datasource.

Build features

The release binaries already include every Arrow path; the -lean binaries include all of them except in-process DuckDB. Building from source, these Cargo features turn them on:

Feature What it turns on In default build?
exec-duckdb in-process libduckdb, Arrow-native yes (--no-default-features = the lean build, which shells out to a duckdb CLI)
exec-http-arrow ClickHouse FORMAT ArrowStream no (implies exec-http)
exec-flightsql StarRocks/Doris Arrow Flight SQL client no
exec-databricks Databricks Statement Execution API, inline JSON rows no
exec-databricks-arrow Databricks ARROW_STREAM + EXTERNAL_LINKS no (implies exec-databricks)
exec-all all of the above —
dosi-server arrow Arrow IPC responses on /v1/query/execute yes (server default)

Which APIs get faster?

  • POST /v1/query/execute with Accept: application/vnd.apache.arrow.stream — the headline win. Batches stream from the warehouse adapter straight into the HTTP body as Arrow IPC: no per-value JSON encoding on the server, no JSON parsing in the client, and columnar consumers (Polars/pandas/DataFusion/ another dosi) read it zero-parse. The JSON response stays byte-identical for existing clients. Benefit scales with result size: negligible for a 10-row aggregate, decisive for high-cardinality group-bys and detail/export queries (10⁵+ rows).
  • dosi query --execute --format arrow — Arrow IPC on stdout for piping into DuckDB/Polars/anything, skipping the table/JSON rendering entirely.
  • StarRocks/Doris large reads — Flight SQL reads bypass FE row forwarding (client pulls columnar results from BEs in parallel); vendors measure 20–160x vs the MySQL protocol on large result sets. Small aggregates: ~no change.
  • ClickHouse — server-side Arrow serialization (lz4-framed) replaces JSON encode/decode on both ends; also removes float/decimal text round-trips.
  • DuckDB (default engine) — in-process zero-copy replaces spawning a CLI per query and parsing its JSON: lower latency on every --execute, and in-memory databases persist across statements in one session.
  • What does NOT get faster: compile-only APIs (/v1/query/compile, explain) — they never touch results; small metric queries on row-wire engines (MySQL/TiDB/Postgres/Trino) — unchanged by design.

Every Arrow path is checked against the test corpus before it ships: Arrow results ≡ row results ≡ the DuckDB reference.