Ontologies (experimental)¶

Experimental

This page documents a feature that is still being shaped. Flags, output and error payloads may change without notice, and the page is deliberately not linked from the rest of the documentation.

Dosi can load an Ossie ontology alongside — or instead of — a core Ossie model. An ontology adds no information a vendor extension could not carry. What it adds is that one model can bear several of them, and they may disagree: operations means the departure airport by "airport delay", ground handling means the arrival one. Both are right, and each is a separate ontology over the same tables.

Everything below is the dosi CLI. Examples use the fixtures in the repository: fixtures/ossie-mini-flights (two concepts), fixtures/aviation (a hand-written model plus an ontology) and fixtures/ossie-flights (the upstream reference ontology).

Loading an ontology¶

These flags apply to every command:

Flag	Env	Meaning
`--ontology <path>`	`DOSI_ONTOLOGY`	Ossie ontology file
`--mapping <name>`	—	Which of several `ontology_mappings` to use; required when there is more than one
`--model <path>`	`DOSI_MODEL`	A core model — or an ontology file, recognized by its top-level `ontology:` key

Model-first and ontology-first¶

	Command	Truth	The other layer
ontology-first	`--ontology o.yaml`	the ontology's embedded `semantic_model`	—
model-first	`--model m.yaml --ontology o.yaml`	`m.yaml`	the ontology adds concepts, roles and value domains

With both, --model wins and no consistency check runs: the embedded model only has to make the ontology's mapping expressions resolvable, and a hand-written model carrying more fields is a superset, not a conflict. Where --model already declares a join with the same columns as an ontology link, the concept walks the model's relationship instead of adding a second one.

What loading produces¶

The ontology is lowered to an ordinary core Ossie model, which then goes through the same validate → compile → plan path a hand-written model does. A ConceptMap is kept beside it; it only names things — a concept path to the dataset, field and join path it resolves to.

Ontology construct	What it becomes
`identify_by` with an object mapping	the dataset's `primary_key`, unless the model already declares one
Entity → Value relationship	a name for an existing field; temporal value types mark it `is_time`
Entity → Entity, `ManyToOne` / `OneToOne`	a relationship named `<owner>_<link>`, e.g. `flight_departs_from`
Entity stored on another entity's table	no join, no primary key, not a query root — see Limitations
Unary relationship with a filtered mapping	a boolean field `<owner>_<name>`, reachable as `Concept.name`
`derived_by: AGG[<value> WHERE <correlation> GROUP BY <concept>]`	a metric `<owner>_<relationship>` — see Derived metrics

What is not lowered is reported as a note: line on stderr on every call:

$ dosi describe --ontology fixtures/ossie-flights/ontology.yaml
note: City.state: State has no table of its own; reached without a join
note: Airport.city: City has no table of its own; reached without a join
note: Flight.registers_latitude_series: n-ary relationship recorded as unmapped
...

Browsing: describe and paths¶

`dosi describe [concept]`¶

With no argument, the concept catalog:

$ dosi describe --ontology fixtures/ossie-mini-flights/ontology.yaml
ontology MiniFlights
Two airports' worth of ontology, kept small enough to read whole.
concept  identity  out  in  props  root
Airport  code      0    2   3      yes
Flight   id        2    0   2      yes

out and in count links, props counts properties, and root says whether a query can start from the concept.

With a concept name, everything needed to write a query against it:

$ dosi describe FlightAlert --model fixtures/aviation/model.yaml \
    --ontology fixtures/aviation/ontology.yaml
concept FlightAlert
identity: id  (root: flight_alerts)

properties
path                            type                    domain                   time
FlightAlert.id                  AlertId (String)
FlightAlert.type                AlertType (String)
FlightAlert.priority            AlertPriority (String)  one of [P0, P1, P2, P3]
FlightAlert.status              AlertStatus (String)    one of [OPEN, RESOLVED]
FlightAlert.created_at          DateTime                                         yes
FlightAlert.resolved_at         DateTime                                         yes
FlightAlert.resolution_comment  String

links out
name    role    to      multiplicity  join
flight  flight  Flight  ManyToOne     yes

metrics by group-by
group by        metrics
FlightAlert.id  alert_count, open_alert_count, flights_with_open_alert, maintenance_alerts

time dimensions: FlightAlert.created_at, FlightAlert.flight.actual_arrival, FlightAlert.flight.actual_departure, FlightAlert.flight.scheduled_arrival, FlightAlert.flight.scheduled_departure, FlightAlert.resolved_at

Section	What it tells you
`properties`	Every mapped property as a concept path, with its value type, closed value domain and time flag
`links out`	Name, role, target and multiplicity. `join: none` means the target is stored on this table
`links in (query from)`	Concepts that link here — the side to root a query on
`metrics reachable`	Metrics the ontology derived, with the group-by and the edges (`via`) that give them their meaning
`metrics by group-by`	The model's own metrics this concept can group, each under a group-by that walks from the concept the metric lives on. Every listed group-by is planned before it is printed
`time dimensions`	Time properties on the concept and one hop out, for a time range
`unmapped`	Relationships declared but not lowered: n-ary, or links with no mapping
`not queryable`	Properties bound to no column (row-level `derived_by`, mostly)

Every path in the tables can be pasted into select or query unchanged. That is the contract that makes the view worth calling.

--format json returns the same content: concept, identity, root, properties (path, type_name, domain, domain_summary, is_time, mapped), links, incoming (query_from), metrics_reachable (name, metric, grain, group_by, via), groupable_metrics (group_by, from, metrics), time_dimensions_reachable, default_projection and unmapped. The catalog is ontology, description, requires and concepts (name, identity, links_out, links_in, properties, queryable, summary).

Without an ontology, describe with no argument lists datasets and metrics instead, and naming a concept is an error:

$ dosi describe Airport --model fixtures/ossie-mini-flights/model.yaml
error: this model has no ontology, so "Airport" is not a concept; pass --ontology <path>, or use `list` for datasets and metrics

`dosi paths --from <concept> --to <concept>`¶

Every named path between two concepts, up to three hops, with the roles walked and the pairs the ontology declares equivalent:

$ dosi paths --from Flight --to Airport --model fixtures/aviation/model.yaml \
    --ontology fixtures/aviation/ontology.yaml
paths Flight → Airport
path                              roles                        hops  multiplicity  join
Flight.arrives_at                 destination                  1     ManyToOne     yes
Flight.departs_from               origin                       1     ManyToOne     yes
Flight.follows_route.destination  follows_route → destination  2     ManyToOne     yes
Flight.follows_route.origin       follows_route → origin       2     ManyToOne     yes
equivalent: Flight.departs_from ≡ Flight.follows_route.origin  (Flight.follows_route.origin == origin)
equivalent: Flight.arrives_at ≡ Flight.follows_route.destination  (Flight.follows_route.destination == destination)

Two paths to one target is exactly the case a bare dataset.field reference cannot resolve (ambiguous_join_path); the role names are what separate them. Equivalences come from relationship requires of the form Concept.link.role == role. JSON adds unique: true when exactly one path exists.

Concept paths in queries¶

With an ontology loaded, select and query accept concept spellings wherever they take a field:

Where	Spelling
`select --from`	a concept name: `FlightAlert`
fields, `--group-by`	`Concept.property`, `Concept.link.property`, `Concept.link.link.property`
`--where`	the same paths; string literals are never rewritten
`--time-dimension`	a time property path: `Delay.flight.scheduled_departure`
`--metrics`	a derived metric's concept path: `Airport.average_departure_delay`

Each resolves to the dataset, field and relationship path the planner already understood, so the SQL is identical to the dataset spelling:

$ dosi select --model fixtures/aviation/model.yaml \
    --ontology fixtures/aviation/ontology.yaml \
    --from FlightAlert \
    --fields "FlightAlert.id,FlightAlert.priority,FlightAlert.flight.arrives_at.code" \
    --where "FlightAlert.status = 'OPEN' AND FlightAlert.priority = 'P1'"
SELECT flight_alerts.alert_id AS alert_id, flight_alerts.priority AS priority, airports.airport_code AS flight_alerts_flight__flights_arrives_at__airport_code FROM main.flight_alerts AS flight_alerts LEFT JOIN main.flights AS flights ON flight_alerts.flight_id = flights.flight_id LEFT JOIN main.airports AS airports ON flights.destination_airport_code = airports.airport_code WHERE flight_alerts.status = 'OPEN' AND flight_alerts.priority = 'P1' LIMIT 100

Output columns keep the planner's names, and --order takes those names, not concept paths.

Derived metrics¶

The correlation path of a derived_by rule is what gives the metric its meaning. Two such metrics can compile to byte-identical SQL and still mean different things:

$ dosi query --ontology fixtures/ossie-mini-flights/ontology.yaml \
    --metrics Airport.average_departure_delay --group-by Airport.code --execute --db mini.duckdb
code  airport_average_departure_delay
ATL   16
ORD   30
DCA   20
SFO   32
4 rows

$ dosi query --ontology fixtures/ossie-mini-flights/ontology.yaml \
    --metrics Airport.average_inbound_departure_delay --group-by Airport.code --execute --db mini.duckdb
code  airport_average_inbound_departure_delay
ATL   30
DCA   16
SFO   35
ORD   10
4 rows

Both are AVG(FLIGHT.dep_delay). Named by its concept path, each metric groups through its own edge — the first through the departure airport, the second through the arrival airport. Two consequences are deliberate:

Asking for the model metric by its flat name (airport_average_departure_delay) names no path, so grouping by Airport.code is ambiguous_join_path: the model alone cannot tell the two apart.
Asking for both at once names two paths that disagree, and the ambiguity is reported rather than resolved to one side.

A rule is lowered whole or not at all. A WHERE that is not a correlation (WHERE Flight.departure_delay > 10), or a correlation over a link that does not resolve, leaves the rule as text with a note:; no metric is emitted.

When a group-by is rooted on the wrong side¶

A metric grouped through a path that starts on a concept it cannot reach is refused by the planner. With an ontology loaded, the refusal names the repair — the same path, walked from the concept the metric lives on:

$ dosi query --model fixtures/aviation/model.yaml --ontology fixtures/aviation/ontology.yaml \
    --metrics delayed_flights --group-by Flight.arrives_at.code
error: InvalidRelationshipPath: a relationship path pins this query to dataset "flights", which has no many→one path to "delays" — where measure "delays_flight_id_count_distinct" of metric "delayed_flights" lives
  candidates: Delay.flight.arrives_at.code
  hint: root the group-by on the concept the metric lives on: --group-by Delay.flight.arrives_at.code

Every candidate is planned before it is offered; at most three, shortest first. Under --format json they are candidates and suggested_retry.

Value domains¶

A ValueType whose requires enumerates a closed set makes a filter literal checkable before the query is compiled:

$ dosi select --model fixtures/aviation/model.yaml --ontology fixtures/aviation/ontology.yaml \
    --from FlightAlert --fields FlightAlert.id --where "FlightAlert.status = 'unresolved'"
error: value_not_in_domain: "unresolved" is not a value of FlightAlert.status; its domain is a closed set
  allowed: OPEN, RESOLVED

{
  "code": "value_not_in_domain",
  "violations": [
    {
      "reference": "FlightAlert.status",
      "literal": "Open",
      "allowed": ["OPEN", "RESOLVED"]
    }
  ]
}

This is what profiling the warehouse cannot do: a probe finds the values a column holds today, but only the declaration asserts the set is closed.

What is checked, exactly:

Enumerations written X == 'A' OR X == 'B', whose every value is a literal. A rule comparing to anything else (Status == Other.status) contributes no domain.
Predicates of the form <path> = '<literal>' and <path> IN ('a', 'b'), with '' read as a quote inside the literal. Text inside a string literal is never taken for a reference.
Ranges (1 <= X, X <= 10) are shown by describe but not checked, and neither are numeric literals. Cross-field, cross-relationship and model-level constraints are carried verbatim and enforced by nothing.

Lineage¶

dosi lineage draws the metric lineage graph — physical tables → datasets → atomic metrics → derived metrics — and, with an ontology loaded, the concepts on top of it. The graph itself is documented on Lineage; this section is what the ontology adds.

$ dosi lineage view --ontology fixtures/ossie-mini-flights/ontology.yaml
$ dosi lineage view --ontology fixtures/ossie-flights/ontology.yaml --no-open --out flights.html
flights.html
$ dosi lineage dump --ontology fixtures/ossie-mini-flights/ontology.yaml > graph.json

view goes straight from the model to one HTML page — it does not read a dump — and the page is self-contained: the graph is inlined, nothing is fetched, and it opens over file://. It opens in the default browser, or, with --no-open or no browser, prints the path. The note: lines from loading still go to stderr.

The page has two views. Lineage lays the lanes out left to right with a concept lane inserted between datasets and metrics; an entity with no table of its own has a dashed border, and each grain edge is labelled with its path. Concepts shows the entities as an ER diagram, properties folded into each card, parallel relationships drawn apart. Selecting a node in either view shows its detail and its edges; clicking an edge goes to the other end, switching views when it has to.

What the ontology adds¶

Nothing in the plain graph moves: the four lanes, the seven edge kinds and every node and edge id are unchanged, and a model with no ontology has no ontology key at all. An ontology adds:

Added	Where	What
entity nodes	`ontology.nodes`	one per entity concept, id `concept:<Name>`
`mapping` edges	`ontology.edges`	dataset → concept
`relationship` edges	`ontology.edges`	concept → concept
`concept` object	metric nodes in `nodes`	what a `derived_by` rule says about the metric: `path`, `measure`, `group_by`, `via`, `realized_by`

The two edge kinds are lineage — where a concept's data is. The concept object is semantics, which is why it sits on the node beside dimensions rather than being an edge of its own.

One field is new on every graph, ontology or not: each entry in a dataset node's detail.fields carries label — the field's label in the model, else its name.

Value types are not nodes; they appear on their entity as properties[].concept. Overlay edges point at plain-graph nodes (dataset:, metric:) and at entity nodes (concept:), so a renderer looks endpoints up in nodes and ontology.nodes together.

Entity nodes¶

{
  "id": "concept:Airport", "kind": "entity", "name": "Airport", "label": "Airport",
  "mapping": "table", "dataset": "AIRPORT", "key_columns": ["code"], "identity": ["code"],
  "properties": [
    { "name": "code", "concept": "AirportCode", "base": "String",
      "field": "AIRPORT.code", "is_time": false, "derived": false }
  ]
}

Field	Meaning
`mapping`	what the ontology mapping landed on. `table`: a dataset of its own, one row per object. `denormalized`: no table of its own — its columns sit on another entity's table, whose rows they are a projection of (`City` on `AIRPORT`, where two Californian airports make `State` CA twice). `unmapped`: nothing it declares resolves to a column
`dataset`, `key_columns`	the dataset and key it is identified by — only when `mapping` is `table`
`identity`	the relationships named in `identify_by`
`properties[]`	`name`; `concept`, the value type; `base`, the built-in it reduces to; `field`, as `dataset.field`, absent when unmapped; `is_time`; `derived`, computed by `derived_by`; `domain`, the closed value set when there is one; `description`
`unmapped`	relationships declared but not lowered — the list `describe` prints

description appears when the ontology gives one. A key with nothing to say is absent, not null.

Edges¶

Kind	Source → target	Payload	Id
`mapping`	`dataset:D` → `concept:C`	`via`: `identity` (the entity's own dataset) or `property` (a dataset its properties are denormalized onto — one edge per such dataset)	`mapping:dataset:D->concept:C`
`relationship`	`concept:A` → `concept:B`	`name`, `role`, `multiplicity`, `description`; `realized_by`, the `join:` edge ids it walks; `no_join`, true when the target sits on the same table (then `realized_by` is empty); `equivalences`, the relationship's `requires`, verbatim	`relationship:A.<name>`

The `concept` object on a metric node¶

Only metrics the ontology derived have one. A metric written by hand in the model (model-first) keeps its aggregates edge from its dataset and gains nothing.

Field	Meaning
`path`	the concept spelling, e.g. `Airport.average_departure_delay` — paste it into `query --metrics`
`measure`	the value the rule aggregates, as a concept path: `Flight.departure_delay`
`group_by`	the concept the rule groups by
`via`	the `relationship:` ids walked from the measured concept to the grain
`realized_by`	the same path as plain-projection `join:` ids

The two metrics from Derived metrics:

{ "id": "metric:airport_average_departure_delay", "concept": {
    "path": "Airport.average_departure_delay", "measure": "Flight.departure_delay",
    "group_by": "Airport", "via": ["relationship:Flight.departs_from"],
    "realized_by": ["join:flight_departs_from"] } }
{ "id": "metric:airport_average_inbound_departure_delay", "concept": {
    "path": "Airport.average_inbound_departure_delay", "measure": "Flight.departure_delay",
    "group_by": "Airport", "via": ["relationship:Flight.arrives_at"],
    "realized_by": ["join:flight_arrives_at"] } }

In the plain graph each metric has one in-edge, dataset:FLIGHT → metric, and nothing else; via is what tells them apart. The viewer draws it as an arrow from the grain concept to the metric, labelled with the path.

--redact-sql leaves the ontology block unchanged — it holds no SQL. In Python, Engine(ontology_path=…) loads the ontology and lineage() returns the same graph.

Names: dosi's, not the specification's¶

The overlay's kinds and fields are dosi's lineage vocabulary. The Apache Ossie ontology specification defines concepts, relationships, roles, derived_by, requires and ontology mappings; it has no grounds, measures, grain or grounding. How each maps back:

In the graph	From the specification
entity node; `concept.path`	a concept of `type: EntityType`. `Concept.relationship` is how the specification identifies a relationship
`relationship` edge	a binary Entity → Entity relationship. It is not called `link`: in the specification a link is one instance of a relationship — one tuple — not the relationship itself
`equivalences`	the relationship's `requires`, verbatim
`mapping` edge; the node's `mapping` field	the concept mappings — object mappings and `identify_by` — that place an entity on a dataset. Its three values are dosi's reading of what those mappings landed on, which the specification does not itself distinguish
`concept.measure`, `concept.group_by`, `concept.via`	one aggregate `derived_by` rule, `AGG[<value> WHERE <correlation> GROUP BY <concept>]`, split into the value, the `GROUP BY` concept and the `WHERE` correlation path. Measure and grain are the semantic-layer words, not the specification's
`realized_by`, `no_join`	nothing in the ontology — the joins lowering produced

Note also that the specification has no notion of a metric at all: what it declares is a derived relationship (Airport.average_departure_delay relates an Airport to a Delay). dosi lowers that into a core-OSI metric, which is why the graph has a metric node to hang the concept object on.

The aggregate form is not a grammar the specification's text defines either: its derivation examples are logical rules (Person.parent_of(descendant), EXISTS ( Person.earns )). The AGG[…] rules dosi lowers are the ones in the upstream example examples/flights.yaml, which is the fixtures/ossie-flights ontology.

Errors¶

Refusals while loading an ontology exit non-zero with one or more codes, each naming the construct and, where there is one, a fix:

Code	Meaning
`ontology_parse`	The document's top-level shape is not an ontology
`ontology_unknown_reference`	A referenced concept, relationship, role or mapping does not exist
`ontology_federation_unsupported`	Several `ontology_mappings` and no `--mapping` to pick one
`ontology_mapping_unresolved`	A mapping expression names a dataset or field the model lacks
`ontology_expression_unsupported`	A mapping expression is not a plain qualified column, optionally `WHERE ( … )`-filtered
`ontology_mapping_depth_mismatch`	A link-mapping node sits at a depth that does not match its relationship's arity
`ontology_multiplicity_required`	A relationship that must become a join declares no `multiplicity`
`ontology_concept_spans_datasets`	One concept's mappings land on more than one dataset
`ontology_identity_unresolved`	`identify_by` names something that cannot identify
`ontology_name_collision`	A concept name collides with a dataset or relationship name
`ontology_type_unresolved`	`extends` forms a cycle, or a value type never reaches a built-in
`ontology_recursion_unsupported`	A `derived_by` rule references the relationship it defines

$ dosi describe --ontology fixtures/ossie-mini-flights/ontology.yaml --mapping nope
error: ontology could not be lowered:
  ontology_unknown_reference: no ontology mapping named "nope" (ontology_mappings)
    candidates: mini_flights_mapping

At query time an ontology adds value_not_in_domain, and fills candidates on invalid_relationship_path. Everything else is the ordinary query error set.

Limitations¶

Denormalized concepts are not query roots. A concept stored on another concept's table — State, City and Market on AIRPORT — is reachable through its owner (Airport.city.name) but select --from State is refused, because its rows would repeat once per airport until roots can deduplicate.
n-ary relationships and row-level derived_by (MINUTES(a - b)) are not lowered; they appear under unmapped / not queryable.
Range domains are not enforced (see Value domains).
--order takes output column names, not concept paths.
paths stops at three hops.
Search is not a dosi command. Retrieval needs an index and storage; dosi compiles rather than stores, and describe output is the raw material an orchestration layer indexes.
The REST API does not load ontologies. The Python binding loads one (Engine(ontology_path=…)), but only lineage() uses it: compile and select there do not accept concept paths.