Skip to content

dosi-engine vs MetricFlow: performance benchmark

Apples-to-apples comparison of the dosi CLI against the legacy MetricFlow engine. Reproduce with:

cargo build --release
python3 scripts/bench_mf_vs_osi.py \
    --mf-model ~/src/metricflow/metricflow/test/fixtures/model_yamls/simple_model \
    --mf-python .venv-mf/bin/python \
    --seed scripts/diff_seed_simple_model.sql \
    --execute --json bench_results.json

Timestamped run reports (with raw JSON) are archived under tests/benchmarks/; latest: 2026-07-11.

Methodology

Both engines process the same semantic model (the MetricFlow simple_model fixtures, converted 1:1 to OSI by scripts/mf_fixtures_to_osi.py) and the same queries against the same seeded DuckDB database. The query matrix is the CI differential-suite matrix — result-set equality between the two engines for these queries is already enforced by scripts/diff_python.py, so the benchmark measures the speed of equivalent work.

measurement dosi-engine MetricFlow
cold CLI one dosi query process: spawn → load model → compile → print SQL one python subprocess: interpreter start + imports + model parse + MetricFlowClient.explain() — the same work mf query --explain does, minus click/config overhead (a bound favorable to MetricFlow)
warm API same full-process invocation (osi has no daemon mode; the bare process-spawn floor is reported separately) in-process explain() loop after a warmup call — interpreter, imports, and model parse all already paid
execute dosi query --execute --db <file> (cold, full process) in-process query() loop (warm) and cold subprocess
peak RSS max resident set size of the cold subprocess (wait4 rusage) same

Cold runs are interleaved A/B, 5 timed runs after 1 discarded warmup; warm loops are 30 iterations. Medians reported.

Environment

  • 2026-07-11, dosi-engine f850dfd, cargo build --release
  • AWS EC2, 4 vCPU Intel Xeon Platinum 8175M @ 2.50GHz, 15 GiB RAM, Linux (al2023)
  • MetricFlow: Datus fork, editable install, CPython 3.12.12 (uv venv)
  • DuckDB CLI v1.4.4; seed: scripts/diff_seed_simple_model.sql

Results

osi process-spawn floor (dosi --version): 2.0 ms median.

SQL compilation — cold CLI (process spawn → SQL printed)

query dosi MetricFlow speedup
simple_agg_by_dim (bookings ⟨is_instant⟩) 10.6 ms 2,321.9 ms 220x
three_metrics_no_dim 10.1 ms 2,403.5 ms 237x
ratio_metric (booking_fees_per_booker) 10.4 ms 2,309.2 ms 222x
expr_metric (views_times_booking_value) 10.4 ms 2,338.8 ms 226x
simple_agg_by_time (bookings ⟨ds⟩) 10.5 ms 2,289.9 ms 219x

MetricFlow cold-start breakdown: ~1,004 ms imports + ~742 ms model parse + ~161 ms client init before any query compiles — ~1.9 s of fixed overhead per CLI invocation. dosi pays ~2 ms of process spawn and parses the model in the remaining ~8 ms.

SQL compilation — warm API (everything resident)

query dosi (full process) MetricFlow (in-process) speedup
simple_agg_by_dim 10.6 ms 107.6 ms 10.2x
three_metrics_no_dim 10.1 ms 218.9 ms 21.6x
ratio_metric 10.4 ms 105.3 ms 10.1x
expr_metric 10.4 ms 205.0 ms 19.8x
simple_agg_by_time 10.5 ms 111.4 ms 10.6x

This is the most generous comparison for MetricFlow — its number is pure in-process compile time with all imports and model parsing already paid, while dosi still re-reads and re-compiles everything in a fresh process. dosi remains 10–22x faster.

End-to-end execution on DuckDB (compile + run + print rows)

query dosi cold MF cold speedup MF warm (in-process) vs dosi cold
simple_agg_by_dim 34.2 ms 2,321.1 ms 68x 127.0 ms 3.7x
three_metrics_no_dim 36.2 ms 2,428.1 ms 67x 233.1 ms 6.4x
ratio_metric 35.7 ms 2,368.8 ms 66x 122.7 ms 3.4x
expr_metric 33.5 ms 2,441.2 ms 73x 235.9 ms 7.0x
simple_agg_by_time 32.9 ms 2,309.1 ms 70x 132.8 ms 4.0x

With execution, DuckDB query time (~20 ms via the CLI shell-out, identical work for both engines) becomes a shared floor; osi's end-to-end cold run is still 3–7x faster than MetricFlow's warm in-process path.

Peak memory (cold process, max RSS)

mode dosi MetricFlow
compile ~15.5 MB ~157 MB
execute ~26.5 MB ~162 MB

Caveats

  • simple_model is small (~25 datasets/metrics). MetricFlow's ~742 ms model parse scales with model size; osi's full 10 ms budget includes its parse.
  • The MetricFlow "cold" path skips the real mf CLI's click startup and Datus config resolution, so real mf query invocations are somewhat slower than measured here.
  • Group-by spelling differs for the time-dimension case (ds for MF, bookings_source.ds for osi) — same semantic query, engine-native syntax.
  • Numbers are single-machine (shared EC2 box); treat relative ratios, not absolute times, as the finding.

Arrow IPC vs JSON: result serialization

How much the columnar egress path (POST /v1/query/execute with Accept: application/vnd.apache.arrow.stream, see rest-api.md) saves over the default JSON body for the same query. Reproduce with:

cargo build --release -p dosi-server
python3 scripts/bench_arrow_vs_json.py            # human table
python3 scripts/bench_arrow_vs_json.py --json out.json

The script is self-contained: it generates a high-cardinality synthetic DuckDB dataset (5M-row fact table, 200k distinct user_id) and a matching OSI model in a temp dir, launches the release dosi-server against them, and runs each result set through both Accept formats.

Methodology

One running server, one query, two Accept headers — the only thing that changes is the response encoding. For each format we measure three things:

measurement what it captures
wire MB bytes of the HTTP response body (median)
e2e ms wall time from request send to full body received — query exec + server serialize + transfer (median)
parse ms client-side cost to turn the raw body into a usable structure: json.loads vs pyarrow.ipc.open_stream(...).read_all() (median)

Query execution over the 5M-row table is identical work on both paths, so it forms a shared floor in e2e ms (isolated by the SMALL row, where the payload is tiny and serialization is negligible). wire MB and parse ms have no shared floor — they are pure format cost. Three result sizes show the advantage scaling with the result, not the scan. 2 warmup + 7 timed runs per cell, medians reported.

Environment

  • 2026-07-14, dosi-engine 2e23173 (working tree), cargo build --release
  • AWS EC2, 4 vCPU Intel Xeon Platinum 8259CL @ 2.50GHz, 15 GiB RAM, Linux (al2023)
  • DuckDB CLI v1.4.4; client CPython 3.9 with pyarrow 21.0.0

Results

result set format rows wire MB e2e ms parse ms
SMALL — 365 rows × 3 cols JSON 365 0.03 1483.5 0.3
Arrow 365 0.01 1463.1 0.2
LARGE — 200k rows × 3 cols JSON 200,000 10.63 2051.8 198.4
Arrow 200,000 6.50 1684.0 1.4
WIDE — 200k rows × 4 cols JSON 200,000 13.99 2749.3 242.7
Arrow 200,000 8.90 1707.9 2.0

Reading the large result sets (Arrow vs JSON):

LARGE (200k × 3) WIDE (200k × 4)
wire — bytes on the network 1.6x smaller 1.6x smaller
parse — client decode ~140x faster (1.4 vs 198 ms) ~120x faster (2.0 vs 243 ms)
e2e — including the shared exec floor 1.2x faster 1.6x faster
e2e minus the ~1.46 s exec floor (serialize + transfer + parse only) ~2.7x faster (221 vs 589 ms) ~5.2x faster (245 vs 1286 ms)

Interpretation

  • Parse is the headline. JSON makes the client re-parse every value out of a string; Arrow hands over typed column buffers with essentially zero decode (1–2 ms for 200k rows vs 200–240 ms). This gap does not shrink in a release build — it is the client's, and it is structural.
  • Serialization is the server-side win. Once the exec floor is removed, Arrow spends ~2.7–5.2x less time on the format itself: no per-row materialization and no per-value JSON string encoding — record batches stream straight from the executor through the IPC writer into the body. The wider the row, the bigger the gap (WIDE > LARGE).
  • Benefit scales with result size. At 365 rows there is no measurable difference — the payload is a rounding error against query execution. The columnar path is worth opting into precisely when the result is large.
  • Wire size is a steady ~1.6x. Real deployments can layer HTTP compression on top; the parse and serialize wins are independent of it.

Caveats

  • Single-machine, loopback transport (no real network RTT or bandwidth limit), so wire MB understates the transfer-time advantage a remote client would see. Treat relative ratios as the finding.
  • The synthetic dataset is numeric/low-width-string; JSON's disadvantage grows with wider rows and more string columns, shrinks for a few wide numeric columns.
  • e2e ms carries the DuckDB scan of all 5M rows on every request (no result cache); the isolated "minus exec floor" row is the cleaner read of format cost, but the floor is only approximately constant across group-by cardinalities.