Amazon Athena connector¶
Dosi compiles the athena dialect and executes it through the AWS Athena API.
Use a SQL engine v3 workgroup. The exec-athena feature is included in the full
and lean official connector sets and in Python wheels.
Division and AVG promote their input to DOUBLE to match the DuckDB reference
semantics, including window averages. This avoids Athena rounding decimal ratios
and means to the input scale (for example, 30 / 180 becoming 0.17 for scale 2).
These operations produce approximate floating-point results.
Connection profile¶
The same datasource vocabulary configures the Python datus-athena adapter
and native Dosi execution. Standalone Dosi accepts datasources:; a Datus
agent.yml places this map under services:.
datasources:
lake:
type: athena
aws:
region: us-east-1
profile: analytics
workgroup: analytics
catalog: AwsDataCatalog
database: sales
query_timeout_seconds: 600
result_timeout_seconds: 60
cancel_timeout_seconds: 10
max_rows: 10000
max_result_bytes: 16777216
Relations use catalog.database.table, database.table, or a table name
resolved in the profile's database. Set these names in model dataset sources.
Athena has no additional schema level.
Parameters¶
| Key | Default | Meaning |
|---|---|---|
aws.region |
SDK configuration | AWS region. |
aws.profile |
SDK default credential chain | Named profile, including refreshable AssumeRole and credential-process profiles. |
aws.ca_bundle |
SDK trust roots | Optional PEM CA bundle; certificate and hostname verification remain enabled. |
workgroup |
primary |
Enabled SQL engine v3 workgroup. |
catalog |
AwsDataCatalog |
Query execution catalog. |
database |
unset | Default database for unqualified sources. |
output_location |
unset | Optional s3://bucket/prefix/, subject to workgroup policy. |
timeout_seconds |
30 |
Budget for one SDK operation, including its retries. |
query_timeout_seconds |
600 |
Submission and query-state budget. |
result_timeout_seconds |
60 |
Budget for fetching all result pages. |
cancel_timeout_seconds |
10 |
Additional bounded recovery/cancellation budget; zero disables cleanup calls. |
poll_interval_seconds |
1 |
Query-state polling interval. |
max_rows |
10000 |
Maximum complete result rows. |
max_result_bytes |
16777216 |
Maximum UTF-8 cell payload bytes. |
page_size |
1000 |
Rows per API page, from 1 to 1000. |
Flat region, profile_name, and ca_bundle spellings are also accepted.
Conflicting flat and nested settings are errors. AWS credentials remain in SDK
providers and refresh automatically; the datasource contains identity selectors.
Results and cancellation¶
Customer S3 results and Athena managed results both use GetQueryResults.
Workgroup-enforced settings take precedence, and managed results need no client
output_location. Result encryption is configured on the workgroup.
An outer Python/server deadline also bounds execution. Expiry signals cooperative
cancellation; cleanup may take up to cancel_timeout_seconds in addition to
the execution budget. Dosi retains one idempotency token per submission, including
recovery after an uncertain response. Recovery and cancellation are best effort
when AWS cannot be reached.
JSON results and execution errors include execution_info, one entry per SQL
query. Entries carry the query execution ID, state, scanned bytes, timing,
engine version and cancellation flags. cancellation_requested records an
attempt; cancellation_confirmed requires the CANCELLED terminal state.
Arrow schemas carry the same array in dosi.execution_info.
Results that exceed either budget fail explicitly. Exact numerics wider than the shared numeric representation retain their decimal text. A result row limit controls returned data; use the workgroup's per-query scanned-byte cutoff to bound scans.
Verify the connection¶
Disposable AWS acceptance¶
scripts/aws/athena_environment.py create <run-dir> creates an isolated bucket,
Glue database, S3/managed-results workgroups with a 10 MiB scan cutoff, and a
scoped test role. No EC2 instances or provisioned capacity are created.
populate uploads tiny Parquet fixtures to one Glue database per scenario,
named <database>_<scenario namespace>. Table names, models, seeds and reviewed
DuckDB expected results remain unchanged. destroy stops remaining queries, deletes resources and
writes a live deletion audit. These commands use AWS CLI credentials.
Run the complete canonical corpus against DuckDB, then Athena, with automatic cleanup even on population or query failure:
uv pip install -r scripts/aws/requirements-athena.txt
cargo build -p dosi-testkit --features exec-athena
python scripts/aws/athena_environment.py run target/athena-acceptance
cargo run -p dosi-testkit -- accept-reports --report target/athena-acceptance/athena.json
The run writes separate DuckDB/Athena Row and Arrow receipts, a fixture inventory,
query histories and cleanup-audit.json. An interrupted process can be reclaimed
with python scripts/aws/athena_environment.py destroy <run-dir>.
The weekly workflow uses the same command with --ignore-seals. It requires a
dedicated DOSI_TEST_ATHENA_AWS_ROLE_ARN repository variable for an OIDC role
allowed to manage this template's disposable resources and Glue namespaces;
without it, the dashboard records an unexecuted report. Do not grant these
permissions to the EC2 runner role or expose them to PR workflows.
The existing testkit accepts DOSI_TEST_ATHENA_DATABASE,
DOSI_TEST_ATHENA_WORKGROUP, and optional DOSI_TEST_ATHENA_CATALOG. Set
DOSI_TEST_ATHENA_FIXTURES_READY=1 only after loading the isolated fixtures.
DOSI_TEST_ATHENA_DATABASE is the run's base name; testkit appends each scenario's
namespace. Receipts can be accepted directly without rewriting or bootstrapping
fixtures. Real API tests are opt-in:
cargo test -p dosi-exec --no-default-features --features exec-athena \
--test athena_live -- --ignored
The MCP acceptance test sends a protocol cancellation notification after the
real query enters RUNNING, confirms CANCELLED and permit release, then runs
another query through the same executor: