Five commerce questions with one LOD model¶
This walkthrough answers five everyday questions about buyers, payments and refunds using one semantic model and three raw tables. It demonstrates fixed fields, equal-weight reaggregation, include, exclude, fixed metrics, composition and independent parameterized windows. The data is entirely synthetic. Concepts are explained in Understand LOD; rules are in the LOD authoring reference.
Run the example¶
Install Dosi and the DuckDB CLI, and use datus mode, the default. Python 3.10+ is needed only for the comparison script, which has no third-party dependencies.
The example files ship as examples/lod-commerce in the Dosi release
archive. If the examples directory reported by dosi info has no
lod-commerce, upgrade Dosi. The installer places
examples under ~/.local/share/dosi/examples; if you unpacked the release
yourself, set DOSI_EXAMPLES to its examples/ directory.
The directory contains the complete model.yaml, seed.sql with table
definitions and all synthetic rows, queries.json, expected.json, five
independent raw-SQL checks, a verification script and a README. For an installed release, copy the
example to a working directory before creating the database:
export DOSI_EXAMPLES=~/.local/share/dosi/examples
cp -R "$DOSI_EXAMPLES/lod-commerce" ./lod-commerce
Alternatively, from the root of a source checkout containing this guide:
Then initialize and check the working copy:
cd lod-commerce
dosi --version
duckdb commerce.duckdb < seed.sql
dosi validate --model model.yaml --strict
python3 verify_lod_commerce.py --case-dir . --variants baseline --report report.json
Use a new database file when seeding. The last command executes exactly the
five semantic queries below and compares their full results with independent
raw SQL. Omit --variants baseline to repeat them across five data variants
(25 comparisons); no model changes are made. report.json contains actual
results, emitted SQL and validation output. Tables below use the business
column names recorded in queries.json; that manifest explicitly maps the
CLI's metric/field names to those labels. Row order is immaterial.
Data and business definitions¶
The baseline has 7 users, 12 paid orders and 9 refund applications. The
snapshot is complete through May 31, 2030. Amounts are CNY DECIMAL(12,2).
All periods are half-open: April is April 1 inclusive through May 1 exclusive.
| Table | One row represents | Raw fields |
|---|---|---|
users |
one registered user | user_id, signup_date, region, acquisition_channel |
orders |
one successfully paid order | order_id, user_id, paid_date, paid_date_key, channel, paid_amount |
refunds |
one refund application at its current status | refund_id, order_id, user_id, requested_date, resolved_date, resolved_date_key, status, requested_amount |
The three IDs are their respective table's primary keys. order_id repeats
in refunds because an order may have multiple partial refunds. All orders
belong to registered users; a refund's user agrees with its order's buyer.
paid_amount remains the original payment, even after refunds. Multiple
orders by one user on one date count as separate purchases.
acquisition_channel is the user's signup source (ads, organic, or
referral), distinct from the order channel (app or web). Region is a
current attribute. There is no acquisition cost or multi-touch attribution.
Refund statuses are SUCCEEDED, REJECTED and PENDING. Only a successful
application's requested amount is refunded. For successful applications,
resolved_date is the success date; pending applications have no completion
date. The integer date keys encode YYYYMMDD and preserve NULL. Successful
refunds never exceed the corresponding payment. Dates and amounts are not
precomputed business metrics.
The model declares two ordinary many-to-one relationships: orders to users, and refunds to users. It does not join the two detail tables. User U07 has no baseline orders and must not enter paying-user denominators or the April-buyer recall list.
How the model separates calculation grains¶
| Purpose | Model definition | Why this grain |
|---|---|---|
| Classify buyers | orders.buyer_order_count = fixed(order_count, orders.user_id) |
one purchase count per buyer |
| Equal-weight lifetime spending | orders.buyer_lifetime_paid = fixed(paid_total, orders.user_id); mean_lifetime_paid averages the field |
each buyer contributes once |
| Purchase frequency in each channel | AVG(include(order_count, orders.user_id)) and MAX(include(order_count, orders.user_id)) |
add buyer to the displayed channel |
| Daily site baseline | site_amount = fixed(paid_total, orders.paid_date) |
retain date while ignoring the app display filter |
| Selected channel's monthly total | app_month_amount = exclude(paid_total, orders.paid_date) |
remove date, retain channel and its filter |
| Recall assessment | fixed fields hosted on users, with order/refund filter bases | independent windows, one output row per user |
These expressions describe DATUS declarations. The complete model also carries
the required core SQL: a fixed field repeats its base aggregate, include
wraps that aggregate in AVG or MAX, and fixed/exclude metrics use the SUM
wrapper. Two compose metrics divide the daily payment by the daily site
baseline and channel monthly total, with NULLIF returning NULL for a zero
denominator. No nested LOD or preaggregated source is needed.
Shared datasets, relationships and base metrics¶
The complete model.yaml and seed.sql are in the example directory copied
in Run the example.
The model below is the common starting point: source fields, real primary keys,
two many-to-one relationships and ordinary base metrics. The time range routes
through orders.paid_date, declared as a time dimension.
Expand the shared model
version: 0.2.0.dev0
semantic_model:
- name: commerce
datasets:
- name: users
source: users
primary_key:
- user_id
fields:
- name: user_id
expression: {dialects: [{dialect: ANSI_SQL, expression: user_id}]}
datatype: String
dimension: {}
- name: region
expression: {dialects: [{dialect: ANSI_SQL, expression: region}]}
datatype: String
dimension: {}
- name: acquisition_channel
expression: {dialects: [{dialect: ANSI_SQL, expression: acquisition_channel}]}
datatype: String
dimension: {}
- name: orders
source: orders
primary_key:
- order_id
fields:
- name: order_id
expression: {dialects: [{dialect: ANSI_SQL, expression: order_id}]}
datatype: String
custom_extensions:
- vendor_name: DATUS
data: '{"v":"1.11","is_dimension":false}'
- name: user_id
expression: {dialects: [{dialect: ANSI_SQL, expression: user_id}]}
datatype: String
dimension: {}
- name: paid_date
expression: {dialects: [{dialect: ANSI_SQL, expression: paid_date}]}
datatype: Date
dimension:
is_time: true
- name: paid_date_key
expression: {dialects: [{dialect: ANSI_SQL, expression: paid_date_key}]}
datatype: Integer
custom_extensions:
- vendor_name: DATUS
data: '{"v":"1.11","is_dimension":false}'
- name: channel
expression: {dialects: [{dialect: ANSI_SQL, expression: channel}]}
datatype: String
dimension: {}
- name: paid_amount
expression: {dialects: [{dialect: ANSI_SQL, expression: paid_amount}]}
datatype: Decimal
custom_extensions:
- vendor_name: DATUS
data: '{"v":"1.11","is_dimension":false}'
- name: refunds
source: refunds
primary_key:
- refund_id
fields:
- name: refund_id
expression: {dialects: [{dialect: ANSI_SQL, expression: refund_id}]}
datatype: String
custom_extensions:
- vendor_name: DATUS
data: '{"v":"1.11","is_dimension":false}'
- name: order_id
expression: {dialects: [{dialect: ANSI_SQL, expression: order_id}]}
datatype: String
custom_extensions:
- vendor_name: DATUS
data: '{"v":"1.11","is_dimension":false}'
- name: user_id
expression: {dialects: [{dialect: ANSI_SQL, expression: user_id}]}
datatype: String
dimension: {}
- name: resolved_date_key
expression: {dialects: [{dialect: ANSI_SQL, expression: resolved_date_key}]}
datatype: Integer
custom_extensions:
- vendor_name: DATUS
data: '{"v":"1.11","is_dimension":false}'
- name: status
expression: {dialects: [{dialect: ANSI_SQL, expression: status}]}
datatype: String
dimension: {}
- name: requested_amount
expression: {dialects: [{dialect: ANSI_SQL, expression: requested_amount}]}
datatype: Decimal
custom_extensions:
- vendor_name: DATUS
data: '{"v":"1.11","is_dimension":false}'
relationships:
- name: orders_to_users
from: orders
to: users
from_columns:
- user_id
to_columns:
- user_id
- name: refunds_to_users
from: refunds
to: users
from_columns:
- user_id
to_columns:
- user_id
metrics:
- name: paid_total
expression:
dialects:
- dialect: ANSI_SQL
expression: SUM(orders.paid_amount)
- name: order_count
expression:
dialects:
- dialect: ANSI_SQL
expression: COUNT(DISTINCT orders.order_id)
- name: buyers
expression:
dialects:
- dialect: ANSI_SQL
expression: COUNT(DISTINCT orders.user_id)
- name: refund_requested_total
expression:
dialects:
- dialect: ANSI_SQL
expression: SUM(refunds.requested_amount)
The next five sections show the definitions to add for each question. Merge
their fields into the named dataset and their metrics into the model's
existing list; do not replace a dataset or create a second dataset with the same
name. After all additions, these definitions form the complete commerce model.
When using the installed model.yaml, they are already present.
1. How much do repeat buyers contribute?¶
Through May 31, classify each buyer as once or repeat, count buyers once,
and sum every payment in each group. The cutoff is context so it scopes both
the fixed classification and the payment sum. The query uses ordinary
row-based metrics grouped by a fixed-derived field.
Model definitions. Add these fields to orders.fields. The shared order_count
counts distinct orders; the fixed field turns it into one count per buyer, and
buyer_group turns that count into a grouping dimension. Orders contain only
successful payments, so every buyer present has at least one order. This CASE
is not a classification for users with no purchases.
datasets:
- name: orders
fields:
- name: buyer_order_count
expression:
dialects:
- dialect: ANSI_SQL
expression: COUNT(DISTINCT orders.order_id)
datatype: Integer
custom_extensions:
- vendor_name: DATUS
data: '{"v":"1.11","is_dimension":false,"lod":{"expr":"fixed(order_count, orders.user_id)"}}'
- name: buyer_group
expression:
dialects:
- dialect: ANSI_SQL
expression: CASE WHEN buyer_order_count = 1 THEN 'once' ELSE 'repeat' END
datatype: String
dimension: {}
Query.
dosi query --model model.yaml --format json \
--metrics buyers,paid_total \
--group-by orders.buyer_group \
--context-filter 'orders.paid_date < DATE '"'"'2030-06-01'"'"'' \
--execute --db commerce.duckdb
| buyer_group | buyers | paid_amount |
|---|---|---|
| once | 3 | 500 |
| repeat | 3 | 900 |
paid_total is labeled paid_amount by the manifest. U01, U02 and U03 are
repeat buyers. Refunds do not reduce these original payment amounts.
2. Which acquisition channels bring higher-spending buyers?¶
Average each paying user's lifetime spending equally within signup channel. This asks about buyer spending, not average order size or acquisition ROI. A user with four orders must not receive four times the weight.
Model definitions. Add the field to orders.fields and the metric to the
model's metrics. The fixed key is the buyer, while the query groups by
acquisition channel through orders_to_users. An ordinary aggregate over this
fixed field gives each buyer one contribution within each displayed channel.
Using AVG(orders.paid_amount) would instead average orders. Hosting the field
on orders keeps the population to paying buyers; a user-based model would need
an explicit decision about users without orders.
datasets:
- name: orders
fields:
- name: buyer_lifetime_paid
expression:
dialects:
- dialect: ANSI_SQL
expression: SUM(orders.paid_amount)
datatype: Decimal
custom_extensions:
- vendor_name: DATUS
data: '{"v":"1.11","is_dimension":false,"lod":{"expr":"fixed(paid_total, orders.user_id)"}}'
metrics:
- name: mean_lifetime_paid
expression:
dialects:
- dialect: ANSI_SQL
expression: AVG(orders.buyer_lifetime_paid)
Query.
dosi query --model model.yaml --format json \
--metrics mean_lifetime_paid \
--group-by users.acquisition_channel \
--context-filter 'orders.paid_date < DATE '"'"'2030-06-01'"'"'' \
--execute --db commerce.duckdb
| acquisition_channel | mean_lifetime_paid |
|---|---|
| ads | 400 |
| organic | 120 |
| referral | 180 |
For ads, the two buyers spent 500 and 300, so their mean is 400. Non-buying registrations are excluded because the metric starts from orders.
3. How often do app and web buyers purchase?¶
Within April and each order channel, count orders per buyer, then calculate the mean and maximum of those counts. A buyer using both channels belongs to both populations. Two orders on the same day remain two orders.
Model definitions. Add both metrics to metrics. Unlike the previous
question, purchase counts must follow the current channel and April range.
include adds the buyer key to the channel grouping, then AVG or MAX summarizes
those counts. A fixed buyer count would ignore an ordinary channel filter and
would not express this channel-specific purchase frequency.
metrics:
- name: mean_orders
expression:
dialects:
- dialect: ANSI_SQL
expression: AVG(COUNT(DISTINCT orders.order_id))
custom_extensions:
- vendor_name: DATUS
data: '{"v":"1.11","derive":{"type":"lod","expr":"AVG(include(order_count, orders.user_id))"}}'
- name: max_orders
expression:
dialects:
- dialect: ANSI_SQL
expression: MAX(COUNT(DISTINCT orders.order_id))
custom_extensions:
- vendor_name: DATUS
data: '{"v":"1.11","derive":{"type":"lod","expr":"MAX(include(order_count, orders.user_id))"}}'
Query.
dosi query --model model.yaml --format json \
--metrics mean_orders,max_orders \
--group-by orders.channel \
--start-time 2030-04-01 \
--end-time 2030-05-01 \
--time-dimension orders.paid_date \
--execute --db commerce.duckdb
| channel | mean_orders | max_orders |
|---|---|---|
| app | 1.5 | 2 |
| web | 1 | 1 |
Both include metrics share the same added dimension. They run together in this query; ordinary sales metrics belong in another query.
4. What does the daily app sales report show?¶
For April app order dates, show daily app sales, all-site daily sales, app's
share of that day, the app monthly total, and the day's share of that total.
The ordinary app filter scopes the numerator and exclude subtotal. The
fixed daily site baseline retains web sales. The April time range scopes all
three calculations. Use the ungrained orders.paid_date grouping required
by this keyed fixed metric, even though the stored field is already a date.
Model definitions. Add these four metrics to metrics; they share
paid_total from the common model. site_amount fixes the calculation by date
and ignores the ordinary app filter. app_month_amount excludes date from the
view, retaining channel and its filter. Both still obey the April context.
The compose metrics divide the ordinary daily payment by each baseline and
return NULL on a zero denominator. The core expressions expand the metric
references and use the required nested aggregate representation.
metrics:
- name: site_amount
expression:
dialects:
- dialect: ANSI_SQL
expression: SUM(SUM(orders.paid_amount))
custom_extensions:
- vendor_name: DATUS
data: '{"v":"1.11","derive":{"type":"lod","expr":"fixed(paid_total, orders.paid_date)"}}'
- name: app_month_amount
expression:
dialects:
- dialect: ANSI_SQL
expression: SUM(SUM(orders.paid_amount))
custom_extensions:
- vendor_name: DATUS
data: '{"v":"1.11","derive":{"type":"lod","expr":"exclude(paid_total, orders.paid_date)"}}'
- name: daily_site_share
expression:
dialects:
- dialect: ANSI_SQL
expression: SUM(orders.paid_amount) / NULLIF(SUM(SUM(orders.paid_amount)), 0)
custom_extensions:
- vendor_name: DATUS
data: '{"v":"1.11","derive":{"type":"compose","expr":"paid_total / NULLIF(site_amount, 0)"}}'
- name: app_month_share
expression:
dialects:
- dialect: ANSI_SQL
expression: SUM(orders.paid_amount) / NULLIF(SUM(SUM(orders.paid_amount)), 0)
custom_extensions:
- vendor_name: DATUS
data: '{"v":"1.11","derive":{"type":"compose","expr":"paid_total / NULLIF(app_month_amount, 0)"}}'
Query.
dosi query --model model.yaml --format json \
--metrics paid_total,site_amount,daily_site_share,app_month_amount,app_month_share \
--group-by orders.channel,orders.paid_date \
--where 'orders.channel = '"'"'app'"'"'' \
--start-time 2030-04-01 \
--end-time 2030-05-01 \
--time-dimension orders.paid_date \
--execute --db commerce.duckdb
| channel | paid_date | app_amount | site_amount | daily_site_share | app_month_amount | app_month_share |
|---|---|---|---|---|---|---|
| app | 2030-04-01 | 260 | 260 | 1 | 570 | 260 / 570 |
| app | 2030-04-02 | 160 | 500 | 0.32 | 570 | 160 / 570 |
| app | 2030-04-03 | 150 | 240 | 0.625 | 570 | 150 / 570 |
paid_total is labeled app_amount. The last column is shown as an exact
fraction for readability; the CLI returns numeric ratios. Dates without
app orders are absent. The subtotal metric's name reflects this report;
its definition follows the query's retained channel and filters.
5. Who bought in April but did not return in May?¶
Return one row per such user with region, April payments, May order count and successful April refunds, measured by completion date. Refunds are information for assessing the recall list, never an exclusion criterion.
The root is users. Cross-dataset fixed fields read orders and refunds via the declared user-key relationships. Eligibility uses a positive April order count and zero May orders, so a zero-value paid order still counts as a purchase. Missing refund SUM becomes zero explicitly; missing COUNT already means zero. A unique user root needs no DISTINCT.
Model definitions. Add these fields to users.fields and the four filter
metrics to metrics. The shared relationships map each user's key to orders
and refunds separately; the two fact tables never join each other's detail rows.
The user's key fixes the output grain, not an order or refund identifier.
Each period belongs to a filter base: reference orders and payments share
reference_start/reference_end; observation orders use separate names, as do
refunds. The SQL expressions below contain those parameters' default values,
while derive.where contains the placeholders bound by the query. The field
expressions repeat their bases. Missing COUNT is zero; missing refund SUM is
NULL, so only the display field explicitly applies COALESCE.
datasets:
- name: users
fields:
- name: reference_orders
expression:
dialects:
- dialect: ANSI_SQL
expression: COUNT(DISTINCT CASE WHEN orders.paid_date_key >= 0 AND orders.paid_date_key < 99991231 THEN
orders.order_id END)
datatype: Integer
custom_extensions:
- vendor_name: DATUS
data: '{"v":"1.11","is_dimension":false,"lod":{"expr":"fixed(reference_order_count, users.user_id)"}}'
- name: reference_paid
expression:
dialects:
- dialect: ANSI_SQL
expression: SUM(CASE WHEN orders.paid_date_key >= 0 AND orders.paid_date_key < 99991231 THEN orders.paid_amount
END)
datatype: Decimal
custom_extensions:
- vendor_name: DATUS
data: '{"v":"1.11","is_dimension":false,"lod":{"expr":"fixed(reference_paid_amount, users.user_id)"}}'
- name: observation_orders
expression:
dialects:
- dialect: ANSI_SQL
expression: COUNT(DISTINCT CASE WHEN orders.paid_date_key >= 0 AND orders.paid_date_key < 99991231 THEN
orders.order_id END)
datatype: Integer
custom_extensions:
- vendor_name: DATUS
data: '{"v":"1.11","is_dimension":false,"lod":{"expr":"fixed(observation_order_count, users.user_id)"}}'
- name: window_refunded_raw
expression:
dialects:
- dialect: ANSI_SQL
expression: SUM(CASE WHEN refunds.resolved_date_key >= 0 AND refunds.resolved_date_key < 99991231 AND refunds.status
= 'SUCCEEDED' THEN refunds.requested_amount END)
datatype: Decimal
custom_extensions:
- vendor_name: DATUS
data: '{"v":"1.11","is_dimension":false,"lod":{"expr":"fixed(window_refund_amount, users.user_id)"}}'
- name: window_refunded
expression:
dialects:
- dialect: ANSI_SQL
expression: COALESCE(window_refunded_raw, 0)
datatype: Decimal
custom_extensions:
- vendor_name: DATUS
data: '{"v":"1.11","is_dimension":false}'
metrics:
- name: reference_order_count
expression:
dialects:
- dialect: ANSI_SQL
expression: COUNT(DISTINCT CASE WHEN orders.paid_date_key >= 0 AND orders.paid_date_key < 99991231 THEN orders.order_id
END)
custom_extensions:
- vendor_name: DATUS
data: |-
{
"v": "1.11",
"derive": {
"type": "filter",
"base": "order_count",
"where": "orders.paid_date_key >= :reference_start AND orders.paid_date_key < :reference_end"
},
"params": [
{
"name": "reference_start",
"type": "int",
"default": 0,
"min": 0,
"max": 99991231
},
{
"name": "reference_end",
"type": "int",
"default": 99991231,
"min": 0,
"max": 99991231
}
]
}
- name: reference_paid_amount
expression:
dialects:
- dialect: ANSI_SQL
expression: SUM(CASE WHEN orders.paid_date_key >= 0 AND orders.paid_date_key < 99991231 THEN orders.paid_amount
END)
custom_extensions:
- vendor_name: DATUS
data: |-
{
"v": "1.11",
"derive": {
"type": "filter",
"base": "paid_total",
"where": "orders.paid_date_key >= :reference_start AND orders.paid_date_key < :reference_end"
},
"params": [
{
"name": "reference_start",
"type": "int",
"default": 0,
"min": 0,
"max": 99991231
},
{
"name": "reference_end",
"type": "int",
"default": 99991231,
"min": 0,
"max": 99991231
}
]
}
- name: observation_order_count
expression:
dialects:
- dialect: ANSI_SQL
expression: COUNT(DISTINCT CASE WHEN orders.paid_date_key >= 0 AND orders.paid_date_key < 99991231 THEN orders.order_id
END)
custom_extensions:
- vendor_name: DATUS
data: |-
{
"v": "1.11",
"derive": {
"type": "filter",
"base": "order_count",
"where": "orders.paid_date_key >= :observation_start AND orders.paid_date_key < :observation_end"
},
"params": [
{
"name": "observation_start",
"type": "int",
"default": 0,
"min": 0,
"max": 99991231
},
{
"name": "observation_end",
"type": "int",
"default": 99991231,
"min": 0,
"max": 99991231
}
]
}
- name: window_refund_amount
expression:
dialects:
- dialect: ANSI_SQL
expression: SUM(CASE WHEN refunds.resolved_date_key >= 0 AND refunds.resolved_date_key < 99991231 AND refunds.status
= 'SUCCEEDED' THEN refunds.requested_amount END)
custom_extensions:
- vendor_name: DATUS
data: |-
{
"v": "1.11",
"derive": {
"type": "filter",
"base": "refund_requested_total",
"where": "refunds.resolved_date_key >= :refund_start AND refunds.resolved_date_key < :refund_end AND refunds.status = :refund_status"
},
"params": [
{
"name": "refund_start",
"type": "int",
"default": 0,
"min": 0,
"max": 99991231
},
{
"name": "refund_end",
"type": "int",
"default": 99991231,
"min": 0,
"max": 99991231
},
{
"name": "refund_status",
"type": "string",
"default": "SUCCEEDED",
"allowed": [
"SUCCEEDED",
"REJECTED",
"PENDING"
]
}
]
}
Query.
dosi select --model model.yaml --format json \
--from users \
--fields users.user_id,users.region,users.reference_paid,users.observation_orders,users.window_refunded \
--where 'users.reference_orders > 0 AND users.observation_orders = 0' \
--param reference_start=20300401 \
--param reference_end=20300501 \
--param observation_start=20300501 \
--param observation_end=20300601 \
--param refund_start=20300401 \
--param refund_end=20300501 \
--param refund_status=SUCCEEDED \
--limit 10000 \
--execute --db commerce.duckdb
| user_id | region | april_paid | may_orders | april_refunded |
|---|---|---|---|---|
| U02 | south | 200 | 0 | 30 |
| U03 | north | 200 | 0 | 0 |
| U05 | south | 40 | 0 | 0 |
| U06 | east | 160 | 0 | 0 |
The manifest maps reference_paid, observation_orders and window_refunded
to the month-specific display labels. The model fields themselves remain
reusable. Three distinct parameter pairs control the reference, observation
and refund windows, and refund_status controls the refund state. The
reference count and payment share their parameter names. Changing the refund
window changes the displayed refund amount without changing eligibility.
Do not add a query-wide time range that intersects these independent windows.
Detail queries have a maximum limit of 10000; this small demonstration uses that limit explicitly. It is not a bulk export or a pagination example.
What this example proves¶
The five queries exercise the central LOD forms together in one model, with separate queries where combinations are restricted. They also expose common mistakes: counting refund applications as orders, weighting users by their order counts, filtering the all-site denominator to app, and intersecting independent observation windows.
They do not exhaust every supported key shape, null case, cross-fact alignment, time encoding, error or warehouse dialect. The rule guide contains those boundaries. The verification script checks the same five questions with changed amounts, repeated orders, new dates and date-boundary records; it does not add more public business scenarios.