Warehouse connectors¶
Dosi executes by SQL pushdown: it compiles a metric query to dialect SQL and the
warehouse runs it. A connector handles connection setup, statement submission,
and normalizing results into a ResultSet — columns plus rows of
Null | Bool | Int | Float | Str, dates as ISO-8601 strings — so results compare
equal across engines.
This section is the setup guide. Start here for the connection-profile format that every connector shares, then follow the routing table below to your database's own page. Arrow-native result transfer (DuckDB, ClickHouse, StarRocks/Doris, Databricks) is described in Arrow.
Let your agent connect¶
With the Dosi plugin in your coding agent, you don't write the connection yourself. Say "connect to my data", or name the source ("connect to our Snowflake, account xy12345"), and the agent:
- finds what is already there. It reads a dbt
profiles.ymlor a Cube.envthroughdosi migrate conf, connects a local.duckdbor.sqlitefile directly, or writes a warehouse entry from the connector's template, asking only for the fields that are not secret; - keeps the password out of the conversation. It writes a
${NAME}placeholder and tells you where the real password goes: indosi-connections.yamlitself, which it adds to.gitignore, or asNAME=...in~/.config/dosi/secrets.env. It never asks for a password in the chat and never opens these files; - tests the connection with
dosi connection testand reports it in one line, or, when it fails, what happened, why, and the next step.
To change a connection later, ask the agent. It cannot read the file, so it tells you which line to edit. The quickstart shows the whole flow on a sample database.
The rest of this page is the file the agent writes: read it to set up a connection manually, or to review what the agent wrote.
Connection profiles¶
Connection profiles use the Datus agent.yml
datasources: vocabulary. Dosi runs queries through its own connectors and
shares the common configuration vocabulary with Datus. Each connector page
defines its transport and authentication settings; Redshift Data API profiles
use target and identity keys in place of TCP host/password fields.
--connections <path> (env DOSI_CONNECTIONS) accepts either a full agent.yml
(profiles under services.datasources) or a standalone file with the same map
under a top-level datasources:.
datasources:
local-duck:
type: duckdb
uri: duckdb:///warehouse.duckdb # omit for in-memory
prod-sr:
type: starrocks
host: sr.internal
port: 9030
username: dosi
password: ${SR_PASSWORD} # ${VAR} / ${VAR:-fallback} from the environment
database: analytics
default: true # used by --execute without --connection
Run it:
--execute without --connection uses the profile marked default: true if
there is one, else local DuckDB (--db <file>, or in-memory).
Where the file is discovered¶
Without --connections, the first path that exists wins:
| # | Path |
|---|---|
| 1 | $DOSI_CONNECTIONS |
| 2 | ./dosi-connections.yaml |
| 3 | ./osi-connections.yaml (pre-rename fallback) |
| 4 | ~/.config/dosi/connections.yaml |
| 5 | ~/.config/osi/connections.yaml (pre-rename fallback) |
Each legacy osi- path sits directly after its dosi- counterpart, not at |
|
the end of the list. A Datus agent.yml (./conf/agent.yml, |
|
~/.datus/conf/agent.yml) is not discovered; name it with --connections to |
|
| use it. |
Keys every connector shares¶
| Key | Type | Required | Default | Notes |
|---|---|---|---|---|
type |
string | yes | — | Selects the connector, case-insensitively. Use a value from the routing table below; postgresql and greenplum are also accepted and use the postgres connector and SQL dialect. pg and opengauss are not aliases. |
default |
bool | no | false |
At most one profile in the file may set it. Used by --execute without --connection. Compile-only query and select also use its dialect when no explicit dialect, connection or local database is selected. |
Every other key is per-connector — its page lists the ones that connector reads, and the ones it parses and ignores.
Secrets and ${VAR}¶
Any string value may reference the environment as ${VAR} or
${VAR:-fallback}. For password: and private_key_file_pwd: the exact form
matters:
- A value written exactly as
${VAR}resolves lazily — the file loads with the variable unset, and the error (carrying anexporthint) fires only when that connection is used. - A value that merely contains
${VAR}resolves when the file loads. - A plaintext secret works, and warns on stderr:
warning: connection "x" has a plaintext password in the connections file; prefer password: ${VAR}.
From dbt or Cube¶
If dbt (or MetricFlow) or Cube already connects to your warehouse,
dosi migrate conf writes the entries for you,
then dosi connection test checks one:
dbt. profiles.yml is found the way dbt finds it — $DBT_PROFILES_DIR,
then ./profiles.yml, then ~/.dbt/profiles.yml. Every profile × target
becomes one entry named <profile>.<target>; the target your
dbt_project.yml's profile selects is marked default.
{{ env_var('X') }} becomes ${X} and {{ env_var('X', 'd') }} becomes
${X:-d}; any other Jinja refuses the target with unsupported_config.
Cube. CUBEJS_DB_* settings are read from ./.env (never a parent
directory) and the environment. CUBEJS_DATASOURCES=default,events with
CUBEJS_DS_EVENTS_DB_* gives one entry per data source, named
cube.<name>; cube.default is marked default. A value set in the
environment is written as ${NAME}; a value only in .env is copied. A
driverFactory in cube.js / cube.py builds connections in code and is
reported, not migrated.
dbt type / Cube CUBEJS_DB_TYPE |
Becomes | Refused with unsupported_auth |
|---|---|---|
snowflake |
snowflake (key pair: private_key_path) |
password, inline private_key, any authenticator other than snowflake_jwt (browser SSO, OAuth, MFA, Okta) |
bigquery |
bigquery (keyfile, keyfile_json, CUBEJS_DB_BQ_KEY_FILE / _CREDENTIALS) |
method: oauth and other token methods, no key at all |
databricks / databricks-jdbc |
databricks (SQL warehouse from http_path / httpPath, token) |
no token (OAuth) |
postgres, mysql, tidb, starrocks, doris, oracle |
the same type |
— |
duckdb |
duckdb (path made absolute) |
MotherDuck (md:) is unsupported_dialect |
trino |
trino |
method other than none; a password or token |
athena |
Amazon Athena | exec-athena |
clickhouse |
clickhouse over HTTP(S) |
— (driver: native is unsupported_config) |
anything else (redshift, sqlserver, spark, ...) |
not written: unsupported_dialect |
Interactive sign-in is refused because nobody is at the keyboard when an agent runs dosi; switch that target to a key or token and run the command again.
Values. The report never prints them. In the file, literal values —
passwords included — are copied verbatim, the file is written owner-only
(0600), and when it sits inside a git work tree its path is added to that
tree's .gitignore. Re-running replaces entries of the same name and leaves
every other entry alone; the entry already marked default keeps the role —
also when its settings are replaced — and no imported target takes it over.
A Datus agent.yml is never rewritten — pass --out for a standalone file.
What a broken entry does¶
A profile that fails to parse — an unset ${VAR}, an unknown type: — never
takes the file down. Its error is stored and replayed only when that name is
used, so one bad entry cannot stop the profiles you are actually running.
Unknown keys are ignored: Datus adapters keep private settings in the same file. The pre-1.0 spellings are the exception — they are rejected with a migration hint rather than silently doing nothing:
| Rejected | Use instead |
|---|---|
dialect: |
type: |
user: |
username: |
password_env: |
password: ${VAR} |
private_key_passphrase: |
private_key_file_pwd: |
private_key_passphrase_env: |
private_key_file_pwd: ${VAR} |
path: |
uri: duckdb:///<path> |
url: |
uri: |
a top-level connections: list |
a datasources: map keyed by name |
Connectors¶
type: |
Page | Cargo feature | How it talks to the engine | Arrow-native |
|---|---|---|---|---|
duckdb |
DuckDB | exec-duckdb (default) |
in-process, shipped library | yes |
sqlite |
SQLite | exec-duckdb |
local file, read-only, through DuckDB | yes |
mysql |
MySQL | exec-mysql |
MySQL wire protocol | no |
tidb |
TiDB | exec-mysql |
MySQL wire protocol | no |
starrocks |
StarRocks | exec-mysql, exec-flightsql |
MySQL wire, or Arrow Flight SQL | opt-in |
doris |
Apache Doris | exec-mysql, exec-flightsql |
MySQL wire, or Arrow Flight SQL | opt-in |
postgres |
PostgreSQL | exec-postgres |
Postgres wire protocol | no |
hologres |
Hologres | exec-hologres |
Postgres wire protocol | no |
oracle |
Oracle Database | exec-oracle |
OCI through ODPI-C | no |
hana |
SAP HANA | exec-hana |
SAP HDBODBC | no |
clickhouse |
ClickHouse | exec-http, exec-http-arrow |
HTTP interface | opt-in |
trino |
Trino | exec-http |
REST /v1/statement |
no |
snowflake |
Snowflake | exec-snowflake |
SQL API v2 over HTTPS | no |
bigquery |
BigQuery | exec-bigquery |
jobs.query REST over HTTPS |
no |
databricks |
Databricks | exec-databricks, exec-databricks-arrow |
Statement Execution API | opt-in |
redshift |
Amazon Redshift | exec-redshift |
Data API through AWS SDK | no |
What your build includes¶
Published binaries and the dosi-engine wheel ship every connector. A
hand-built binary defaults to DuckDB only; ask for the rest with
--features exec-all, or name them individually. A connector that is missing
says so:
this build has no postgres executor (feature "exec-postgres" not enabled)
hint: rebuild with --features exec-postgres
Troubleshooting¶
These come from the connections file itself, before any connector is reached.
| Message | Cause |
|---|---|
no connections file found; pass --connections <path>, set DOSI_CONNECTIONS, or create ./dosi-connections.yaml |
None of the seven discovery paths exist. |
<path> has no datasources: map (nor services.datasources) |
The file parsed, but the profiles are somewhere else in it. |
<path> uses the removed connections: list format |
Pre-1.0 file; move the entries under a datasources: map. |
datasource "x" is missing required field "type" |
Every profile needs type:. |
datasource "x": unknown dialect "pg" |
Use postgres, postgresql or greenplum; pg is not a supported alias. |
datasource "x": "user" is the legacy osi-connections spelling |
See the rejected-spellings table above. |
<path> marks 2 datasources as default: a, b |
At most one default: true. |
datasource "x": environment variable "PW" is not set |
Export it, or write ${PW:-fallback}. |
no connection named "x" — with hint: available: … |
Typo in --connection; the hint lists the names the file defines. |
Next¶
- CLI reference —
--connections,--connection,--execute - Arrow — which connectors stream Arrow, and what it buys
- REST API — the same profiles, served over HTTP