Extension versions and compatibility¶

Every DATUS payload may declare the version it was written against in v, and list in requires the keys an older engine must not drop — both are described with the envelope. Any engine reports the version it implements: dosi info, GET /v1/capabilities, or dosi_engine.DATUS_EXT_VERSION. This page is the policy behind that number and the record of what each version added.

Versioning policy¶

Every key this engine reads is registered with the version that introduced it and what it costs a consumer to ignore it. The registry is built into the engine and is the source of truth: the version gate, the basic-mode warning text, dosi info and GET /v1/capabilities all read it at runtime, so none of them can drift from it. The change log below is written by hand against that registry.

Ignore-impact levels, ordered by increasing danger — note that failing loudly ranks safer than quietly returning different numbers:

Impact	Meaning
`inert`	Presentation metadata the engine never consumed. Results are identical.
`degraded`	No wrong numbers — the request fails loudly instead (a structured error, or a query name that stops resolving).
`documented`	Numbers change, but predictably and in a documented direction (LEFT vs INNER, NULL vs 0).
`silent`	Numbers are wrong and the caller cannot tell. Must be listed under `requires`.

Three shipped keys are silent: measure (D-MEASURE, 1.4), time (D-FORMAT, 1.6) and cardinality (D-CONFORM, 1.8) — see the change log. Every other key is degraded or documented, so basic mode remains coherent for models that use none of them. A model that uses one must list it under requires, which is what stops a datus-mode engine that does not implement the key (an older Dosi, another vendor's reader of the envelope) from silently computing a different number: it fails at the version gate with datus_ext_key_required instead. Basic mode is a different case — it never reads the payload at all, requires included, and only warns ignored_vendor_extension; a model with a silent key should not be run in basic mode.

When to bump:

MINOR — purely additive: a new key, or a new accepted value for an existing key that no old payload could have used. Old payloads keep working unchanged; a new payload on an old engine gets datus_ext_version_ahead and loses only the new key. A new key whose impact is silent is still a minor bump, but the producer must list it under requires.
MAJOR — only when an existing key changes meaning, is renamed, or is withdrawn. Every payload declaring an older v is then rejected per changed key, with an instruction for updating the YAML.
Neither — engine-internal work that changes no key's contract. A version with no key behind it would make the registry's invariants vacuous.

Change log¶

Each released minor and the keys it added. The machine-readable form of this table is capabilities().history — dosi info --format json or GET /v1/capabilities.

Version	Date	Keys added	Impact if ignored
1.0	2026-07-11	`join_type` (D-JOIN), `fill_nulls_with` (D-FILL)	`documented`, `documented`
1.1	2026-08-02	`time_dimension` (D-TIME), `time_granularity` (D-GRAIN), `dataset` (D-DATASET)	`degraded`, `documented`, `degraded`
1.3	2026-08-09	(none — in-family growth of `window`)	(inherits `window`'s `documented`)
1.4	2026-08-12	`derive` (D-DERIVE M1), `measure` (D-MEASURE)	`degraded`, `silent`
1.5	2026-08-26	`params` (D-PARAM M1)	`degraded`
1.6	2026-09-02	`time` (D-FORMAT)	`silent`
1.7	2026-09-11	`is_dimension` (D-DIM)	`documented`
1.8	2026-09-16	`cardinality` (D-CONFORM)	`silent`
1.9	2026-09-16	(none — in-family growth of `window`)	(inherits `window`'s `documented`)
1.10	2026-09-23	`lod` (D-LOD; also the `lod` family of `derive`)	`degraded`
1.11	2026-10-07	(none — in-family growth of `lod`)	(inherits `lod`'s `degraded`)
1.12	2026-10-25	`concept_params` (D-PARAM, on the mapping document); `params` grows onto the `field` carrier	`degraded`

1.0 (2026-07-11): initial — D-JOIN (relationship join_type, implemented: the join is emitted as INNER or LEFT accordingly), D-FILL (metric fill_nulls_with, implemented 2026-07-12: the metric output is COALESCEd; overrides the S-FAN-6 count default, any metric kind).
1.1 (2026-08-02): time & attribution — D-TIME (dataset/metric time_dimension, implemented: a dataset's primary time dimension and a metric's own, the reserved metric_time query name, per-branch time-range fallback), D-GRAIN (field time_granularity, implemented: finer-grain requests → grain_too_fine), D-DATASET (metric dataset, implemented: COUNT(*) home attribution). Basic-mode ignored_vendor_extension warnings now cover all five carrier objects (previously only Relationship and Metric).
1.1 (2026-08-04, no version bump): the envelope version gate itself — v is now read and enforced (envelope), requires is introduced, each key carries a declared ignore-impact, and the whole registry is published through dosi info, GET /v1/capabilities, and dosi_engine.DATUS_EXT. No key was added or changed, so the version stays at 1.1. Payloads without a v — which is what the Datus agent emits today — are unaffected.
1.2 (2026-08-06): window metrics — D-WINDOW (metric window, implemented: the WindowStage step in --explain; pop/rolling/cumulative sugars + general offset/frame form, cumulative reset with scan expansion + post-window trim; ignore-impact documented — the metric falls back to its plain base aggregate; spec in window-extension.md). Initially DuckDB-executed; the all-dialect enablement (per-dialect offset-shift lowerings, verified on live warehouses) landed as a follow-up. In-family additions (same minor, fail-closed on engines predating them): mixed families / mixed shifts / mixed resets-with-start relaxed, and require_full_window on finite frames (2026-08-06) — see window-extension.md#restrictions.
1.3 (2026-08-09): window families & functions — grows the window key in-family, no new registry row (capabilities().history still ends at 1.2; the version signals the payload dialect). Additions: the rank family (row_number / rank / dense_rank / ntile / percent_rank / cume_dist over the metric value), the value family (first_value / last_value / nth_value), offset direction: "forward" (LEAD — next-period reference), frame order / partition / units / second modifiers (value-ordered and RANGE frames, partition modes, two-argument inputs), and registry levels W2 (stddev_pop/samp, var_pop/samp) + W3 (covar_pop/samp, corr). Every addition fails closed on a 1.2 engine: unknown top-level family keys fall into the exactly-one-family error, unknown in-family keys are rejected, unknown function names fail enum parse — and a declared "v": "1.3" trips the version gate besides. Spec: window-extension.md (family sections + function registry).
1.4 (2026-08-12): derived metrics & measure naming — D-DERIVE M1 (derive key, implemented: filter + compose families, metric references resolved with cycle detection, the expression equivalence gate, compile-time conformed dimensions, and derive metadata on every host surface; reagg / cross-dataset filter predicates / window.base are reserved and reject not_implemented naming their milestone — note window.base previously fell into the silently-ignored unknown-key bucket and now fails closed) and D-MEASURE (measure key, implemented: explicit measure naming, signature-keyed dedup and collision checks unchanged — the fix for the non-ASCII stem collision, and the first silent-impact key: emit it under requires). M2 (2026-08-19, in-family — no new keys, no version bump): window.base flips from reserved to functional (windows over leaf / filter metrics), compose accepts window members (stage-tier lowering: arithmetic over window output) and filter members (flatten tier). A 1.4 payload using these shapes compiles on an M2 engine and fails closed (not_implemented / rejection) on the M1 engine — never a silent misread. Nested compose (in-family — no new keys, no version bump): a compose member may itself be a compose, expanded inline at compile time. An older engine rejects the shape with the same invalid_datus_extension it always did, so this too fails closed. The metric stays on the flatten tier, and derive_members becomes the transitive leaf list with folded coefficients; a window anywhere in the tree keeps the combination not_implemented.
1.6 (2026-09-02): time storage formats — D-FORMAT (field time block, implemented: the column is parsed per dialect on the grouping side, and filter bounds are folded into the stored format before the SQL is built, with the grain inferred from the layout). time.granularity joins the flat time_granularity as an equal spelling — no deprecation, but the two may not disagree. The second silent-impact key: emit it under requires, so a datus-mode engine that lacks it fails at the version gate (basic mode never reads requires; see the envelope). A consumer that drops it does not fail loudly; it assumes a native date column and, for a month- or year-stored one, returns an empty result with no diagnostic. See D-FORMAT for configuration and limitations. Scope: string and integer layouts plus the native temporal types; epoch_seconds / epoch_millis are deferred because from_unixtime reads the session timezone and that needs its own round.

In-family addition (2026-09-08, still 1.6): time.format accepts continuous ordered fixed-width calendar prefixes with arbitrary fixed literal text. Filtering still folds date bounds to stored constants; grouping extracts components into a canonical date/time before dialect parsing. Existing layouts remain compatible. Older Datus engines reject new layouts rather than silently interpret them. - 1.7 (2026-09-11): field roles — D-DIM (field is_dimension, implemented: read by list_dimensions and by attribution's candidate list). The model can finally say that a column is a measurement, an identifier, free text or PII, so it stops being offered as a grouping dimension. It is a recommendation signal only: an explicitly named group_by is honored either way and no compiled SQL changes, hence documented impact and no requires entry. Where the key is absent the engine infers the role from model structure alone — keys, relationship columns and numerically aggregated columns — and never probes the warehouse. See D-DIM. - 1.8 (2026-09-16): conformed fact-to-fact relationships — D-CONFORM (cardinality key on a Relationship, implemented: conformed classes in the join graph, per-branch resolution of group-by / filter / time-range columns and of metric_time to each fact's own spelling, merge on the output name). A many_to_many edge is never joined; only its paired columns are shared. The third silent-impact key: emit it under requires — without it the edge is a many→one join and duplicate-sensitive aggregates fan out. See D-CONFORM. - 1.9 (2026-09-16): a ranked scorecard — grows the window key in-family the way 1.3 did, so there is no new registry row. Four additions driven by one shape.

frame.scope: "partition" states the frame as the entire partition instead of an offset from the current row, lowering to <func>(…) OVER (PARTITION BY …) with no ORDER BY and no frame clause. It is the second window function beside a rank — "how many rows am I ranked against" — which a ranked scorecard needs and which no combination of the existing keys could express: count with preceding: "unbounded" is a running count, and units: "range" still stops at the current row's peer group. scope is mutually exclusive with preceding, reset, require_full_window, units and order, each of which only describes an ordered frame; writing one is rejected rather than ignored.

window.base widens: a ratio or a compose can now be the series a window reads, so ranking a rate or taking its period-over-period is expressible. A cross-fact compose's series only becomes a value after the branch merge, so the window stage sits on the merge — the branches aggregate separately and their CTEs ride beside the stage's base. Such a base has a shared time axis only when the facts' dates are paired (D-CONFORM); otherwise the families that need one refuse on conformance.

Fails closed on a 1.8 engine twice over: it rejects the new keys inside frame and order as unknown, and the payload's "v": "1.9" trips the version gate. See window-extension.md.

Nested windows and NULL placement complete the shape. order.nulls: "first" | "last" pins where NULLs sort in a window ordering — left unset, each warehouse applies its own default and they disagree (DuckDB and PostgreSQL last ascending, MySQL-family and StarRocks first), which a ratio base turns from theoretical into routine. The generator emits the clause only where it differs from the target's default, and where a dialect has no such clause at all (MySQL, TiDB) the placement lowers to a leading CASE WHEN … IS NULL sort key — so the rendering varies by dialect and the meaning does not. And window.base goes one step further than a ratio: it accepts a compose over another window's output, so a second window can read the first one's. Each level lowers to one more SQL stage; depth is capped at 3 and level 2+ must declare partition. The guards that matter run across levels, not within one: the output trim stays outside the outermost stage, and a partition-global metric at any level refuses to share a query with a scan-expanding one at any other. Also in-family: a stage-tier compose may have a compose member (it stays a member rather than being inlined, since on that tier each member is its own series). - 1.10 (2026-09-23): levels of detail — D-LOD, one grammar on two carriers: the lod key on a Field and the lod family of the existing derive key on a Metric (implemented).

On a field, fixed(base, dims…) makes a per-entity aggregate groupable, filterable and re-aggregatable through an ordinary query (the LodRollup step in --explain; a D-FORMAT rollup key keeps its storage encoding and is parsed once, outside). When the view needs more than the rollup — a dimension outside its keys, a row-level field reading the value, several key families — the broadcast shape (LodBroadcast) joins the detail rows with one derived table per key family; an aggregate over a pinned value counts each of its keys once per group. No keys is the table-scoped FIXED, broadcast onto every row.

On a metric, include groups the inner stage by the query's own group-by plus the declared dimensions and collapses it with the declared outer aggregate; the grain and any D-FORMAT parse apply inside. exclude attaches a coarser aggregation of the same fact to the view's rows — LEFT JOIN on the dimensions they share, CROSS JOIN on none — recomputed rather than windowed so a COUNT(DISTINCT) base is right too. fixed is the field's declaration hosted on a metric so it can attach to the view: a table-scoped or key-scoped total computed before the dimension filter, which is what Tableau's {SUM([Sales])} is as a denominator and what exclude() cannot mean (LOD metrics). A conformed pairing whose endpoints are two facts' own fixed fields aligns them on one axis: each fact rolls up for itself and the branches merge, the facts are never joined.

Filter placement follows Tableau's order of operations. The where clause is a dimension filter — applied to a fixed rollup's output when it names only the rollup's keys, to the detail rows otherwise, never inside the rollup — while time_range is the context filter that scopes every rollup. A row-level field that reads only D-LOD values and constants aggregates once per key and can filter a reaggregation at that key grain. A calculated classification such as customer_margin >= 5000 AND customer_orders >= 6 therefore filters the fixed values before they are reaggregated. If the predicate reads a different fixed key family, it is evaluated where that family is available, before the grid is reduced.

Basic mode drops a D-LOD field (aggregate_field_dropped) and fails closed on a D-LOD metric, whose expression is the nested aggregate its declaration describes (nested_aggregate), rather than computing one level of it. See Understand LOD. - 1.11 (2026-10-07): levels of detail across datasets — grows the lod key in-family, no new registry row. A field's fixed base may aggregate another dataset than the field's own when every key corresponds to a column of it through a declared relationship: a many→one edge between the two in either direction, a many→one path from the base to the host, or a D-CONFORM pairing (a rollup over a scan of its own). The rollup is LEFT JOINed onto the host's rows on plain equality and a count reads 0 where nothing matched, so > 0, = 0 and >= n are the semi-join, the anti-join and a threshold; the context layer reaches the rollup through the base's own edges or the pairing, and a conjunct it cannot read there is left out (across datasets). A 1.10 engine fails closed: it refuses the base as aggregating another dataset. - 1.12 (2026-10-25): parameters on fields and derived subtypes. params grows onto the field carrier — a field expression's :name holes, the field.literal slot, bound by patching the field for the whole query and echoed under bindings (field parameters) — and concept_params lands on the Ossie mapping document, declaring the placeholders of a derived subtype's derived_by rule (subtype thresholds). Alongside, the semantics contract gains S-ATTR-3′: a CASE WHEN <guard> THEN <arg> END aggregate may read, in its guard, one dataset the arg's dataset reaches many→one — which is what lets a D-DERIVE filter on a fact metric be written where: "Customer is PremierCustomer". An older engine fails closed: concept_params is an unknown key, a :name in a field is not a column, and "v": "1.12" trips the version gate.

Not yet implemented (roadmap)¶

The queued RFCs are all additive, so each lands as one minor bump with its own registry rows. Order may change; every entry occupies a minor of its own.

Minor	RFC	Keys	Carrier	Impact	`requires`?
≥1.12	`rfc-semi-additive-metrics` §4	`semi_additive`	Metric	`silent` — an end-of-period balance gets summed across snapshots	yes
≥1.12	`rfc-minimal-declarations` §3 D-DISTINCT-STATE	`distinct_state`	Field / Dataset	`silent` — a materialized bitmap column is aggregated as if it were an ordinary column	yes
≥1.7	`rfc-derived-metrics` M3-M4	`derive` reagg + the general cross-dataset filter (`window.base` and compose window/filter members shipped in-family at 1.4/M2; the one-dimension guard shipped at 1.12 as S-ATTR-3′)	Metric	`degraded`/error — reserved shapes reject `not_implemented` today, never silently compute	no
—	`window-extension.md` rank/share/streak families, W2+ functions	future `window` shapes (in-family additions, fail-closed on older engines)	Metric	`degraded`	no

Three silent keys have shipped: measure (D-MEASURE) at 1.4, time (D-FORMAT) at 1.6 and cardinality (D-CONFORM) at 1.8. If you author models against a Datus agent that emits silent-impact keys, keep an eye on the requires list it produces: that is what stops an older engine from quietly returning wrong numbers, and it only works because requires itself shipped at 1.1 — ahead of the first key that needs it.