Skip to content

Databricks connector

Databricks is reached over the Statement Execution API (/api/2.0/sql/statements) against a SQL warehouse, with a personal access token for auth — no driver, no SDK.

The API is stateless per request, so the catalog (and optional schema) ships with every statement rather than being set once with USE. A statement that does not finish inside the initial wait is polled to a terminal state. Gated by exec-databricks, plus exec-databricks-arrow for the Arrow path.

Connection profile

datasources:
  databricks:
    type: databricks
    host: dbc-xxxx.cloud.databricks.com   # a pasted https://…/ URL is tolerated
    warehouse: e9678899fc3ffa09           # SQL warehouse id, from Connection details
    password: ${DATABRICKS_TOKEN}         # a dapi… personal access token
    catalog: workspace                    # Unity Catalog catalog
    schema: analytics                     # optional default schema

Parameters

Key Type Required Default Notes
type string yes — databricks
host string yes — The workspace hostname. A pasted https://…/ URL is trimmed for you.
warehouse string yes — The SQL warehouse id (a hex string), not its display name.
password string yes — The personal access token. Prefer ${VAR}.
catalog string no — Unity Catalog catalog, sent as request context. Folded into database: internally, so either key works.
schema string no — Default schema for queries.
default bool no false See connection profiles.

Parsed and ignored on this connector: username, port, sslmode, sslrootcert, arrow_flight_port, account, role, compat_mode. uri: is not merely ignored — it is rejected with an error telling you to use host + warehouse + catalog.

Authentication

A personal access token in password:, sent as a bearer token. Tokens start with dapi. Keep it in the environment rather than in the file.

Arrow-native results

With exec-databricks-arrow, results are requested as ARROW_STREAM with EXTERNAL_LINKS: each chunk is a self-contained Arrow IPC stream on a presigned cloud-storage URL, fetched without the workspace token (the URL is already signed) and decoded once. No re-typing is needed — Databricks exports real Date32/Timestamp Arrow types. See Arrow.

Limitations

  • No 3-argument DATEDIFF in the portable form. Databricks has the 3-argument function but wants a bare keyword (DAY), not the quoted unit the cross-engine form uses, so a metric written that way is refused for this dialect.
  • The catalog must be writable if you want Dosi to create anything in it; the sample catalog shipped with a workspace is read-only.
  • Unquoted identifiers come back lower-cased.

Timestamps arrive ISO-8601 with a T and a trailing Z and are canonicalized for you to the same zone-less form the other engines produce; dates already arrive ISO.

Verify the connection

$ dosi query --model model.yaml \
    --metrics revenue --group-by orders.status --execute --connection databricks

A stopped SQL warehouse is the usual first failure: the statement waits while the warehouse starts, which can exceed the poll budget on a cold start.

Troubleshooting

Message Cause and fix
connection "x" (databricks) needs host The workspace hostname.
connection "x" (databricks) needs warehouse (the SQL warehouse id) Copy the id from the warehouse's Connection details tab.
connection "x" (databricks) needs password (a Personal Access Token) The hint says it: set password: ${DATABRICKS_TOKEN} to a dapi... PAT.
datasource "x": uri "…" is not supported for databricks; use host + warehouse + catalog This connector takes discrete keys only.
Databricks statement did not finish within the poll budget Usually a cold warehouse; start it, or raise the warehouse's auto-stop threshold.
HTTP 401: <body> / HTTP 403: <body> Expired or revoked token, or no rights on the catalog.
bad Databricks JSON: <e> / bad Databricks chunk JSON: <e> Unexpected response body — usually a proxy in front of the workspace.
bad Arrow IPC: <e> / reading Arrow chunk: <e> Arrow path only: a presigned chunk URL could not be fetched or decoded.

Reference