Skip to content

Lineage

A semantic model answers "what is revenue?" once; the lineage graph shows where that answer comes from. Dosi projects a compiled model into a graph of four lanes:

lane node edge into it
physical table main.orders —
dataset orders reads_table
atomic metric revenue aggregates from each dataset it reads
derived metric net_revenue derive_base, compose_member, window_base, window_second from the metrics it builds on

Datasets are joined by join edges, one per relationship. The graph is a pure function of the model: the same model always produces the same bytes, with no timestamps and nothing about the machine that produced it.

Getting it

$ dosi lineage dump --model model.yaml > graph.json
$ dosi lineage view --model model.yaml            # opens a page in your browser

from dosi_engine import Engine
graph = Engine(model_path="model.yaml").lineage()
graph = Engine(model_path="model.yaml").lineage(redact_sql=True)

view writes one self-contained HTML file — no network access, opens over file:// — with the lanes left to right, badges on the nodes, edges styled by kind, a name filter, and a panel that shows a node's detail and lets you click through its edges. The derived lane is drawn as several columns, one per step of the derive chain: a derived metric sits one column to the right of the rightmost metric it is built from, so a compose chain reads left to right instead of collapsing into one column wired to itself. Without a browser (--no-open, or a server with no display) it prints the file's path instead.

redact_sql keeps the shape and replaces every SQL text — dataset sources, field expressions, metric expressions, derive predicates — and every opaque extension payload with "<redacted>". Names are never redacted.

What a node carries

Every node has an id (table:, dataset:, metric: prefixed), a name, a label, badges, and a detail block whose shape depends on the kind.

  • Dataset: the source (table name, or the query and its pretty-printed form), the tables it reads, keys, the primary time dimension, and every field with its expression, the physical columns behind it, and its label.
  • Metric: the authored expression and dialect, the datasets and physical columns it reads, the dimensions worth grouping it by, its time dimension, parameters, window and derive declarations, and its semantics — shape, attribution strategy, additivity.

dimensions is what dosi list dimensions --metric recommends, in the same order: primary and foreign keys, and the columns some metric aggregates, are left out, because a graph that lists every reachable field makes the reader sift them out by hand. A name that is not listed is still groupable — the judgement withholds a recommendation, it never blocks an explicit group-by.

label is the display name. Apache Ossie defines one for fields (label on a field, the same property the Databricks, dbt and other converters read as the display name) and none yet for datasets or metrics, so a dataset or metric label always equals its name. Dosi never invents a title: label == name means nobody gave one.

With an ontology

Load an Apache Ossie ontology and the graph gains an ontology block: one node per entity, and edges that tie the concepts to the lanes below and above them.

$ dosi lineage view --ontology flights.yaml
edge from → to what it says
mapping dataset → concept the dataset an entity is mapped onto — the one it is identified from, or, for an entity with no table of its own, the datasets its properties are denormalized onto
relationship concept → concept a navigable relationship between two entities, with its role and multiplicity; realized_by names the join edges it walks

Both are lineage: where a concept's data is. What an ontology's derived_by rule says about a metric is semantics, so it rides on the metric node, where v2 already keeps dimensions and time_dimension:

"concept": {
  "path": "Airport.average_departure_delay",   // the concept spelling
  "measure": "Flight.departure_delay",         // the value it aggregates
  "group_by": "Airport",                       // its grain
  "via": ["relationship:Flight.departs_from"], // the path that correlates them
  "realized_by": ["join:flight_departs_from"]
}

via is the reason the overlay exists. An ontology can derive two metrics with byte-identical SQL — an airport's average departure delay, and the average departure delay of the flights arriving at it — that differ only in which relationship correlates a flight to the airport. In the plain graph they are indistinguishable; here each names its own path.

path is what describe prints and what query and select accept, so the graph and the query tools speak the same names. Metrics written by hand in the model have no concept; their lineage is the aggregates edge, and the dataset it starts from carries the mapping edge to its entity.

Concept nodes say what their mapping landed on: table (a dataset of its own, one row per object), denormalized (no table of its own; its columns sit on another entity's, whose rows they are a projection of), or unmapped (nothing it declares resolves). The viewer draws the second two with a dashed border.

The overlay carries no SQL, so redact_sql leaves it unchanged. A model with no ontology has no ontology key at all — the payload is exactly what it was before ontologies existed.

Python

Engine accepts an ontology in both directions the CLI does:

Engine(ontology_path="flights.yaml")                       # the embedded model is the model
Engine(model_path="model.yaml", ontology_path="flights.yaml")  # the model wins; the ontology supplies concepts
Engine(ontology_text=yaml_text, mapping="prod_mapping")     # text, and one of several ontology_mappings

lineage() then includes the overlay. Constructs the ontology declares but Dosi cannot lower (an n-ary relationship, an unmapped concept) are reported as RuntimeWarnings at construction, the same channel as compile warnings. An ontology that cannot be parsed or lowered at all raises ModelError with code ontology; the ontology options are keyword-only.

See also