Data lineage: tracing a number in your dashboard back to its source

Data lineage is the route a number travelled: which source it came from, which transformations it passed through, and which reports it ends up in.

You use it in two directions. Forward, to see what breaks when you change a source. Backward, to trace a figure in a dashboard to the point where it was created.

In a marketing stack that point is almost always a tag on your website or app.

A developer renames a parameter on the purchase event in Google Tag Manager, from value to order_value, because that read more clearly. The tag fires, GTM shows green, the preview looks fine.

GA4 ignores the unknown parameter and stops filling its own revenue field. Two weeks later someone notices the revenue chart has collapsed while the shop is trading normally.

Then the hunting starts. Is it the tag? The GA4 model? The BigQuery export? The dbt model? The dashboard? Without lineage you open all five layers by hand, and by then the GTM change is no longer the first place anyone looks. In ecommerce the mistake often sits in the items array, which we covered separately.

With lineage you go the other way: start at the chart, walk the chain back, and land on the tag that changed within minutes.

dbt, for everything inside your warehouse. Because models point at each other with ref(), dbt knows the dependencies without you writing them down. dbt docs generate followed by dbt docs serve gives you a clickable graph from source to mart. dbt ls --select stg_ga4__events+ lists everything downstream of that model on the command line, which is what you want before you break something.

BigQuery, for what happens outside dbt. Query history in INFORMATION_SCHEMA.JOBS shows which tables were read by which queries, including the ones your BI tool sends. The history only goes back a limited period, so treat it as an investigation tool rather than an archive.

SELECT
  user_email,
  COUNT(*) AS jobs,
  MAX(creation_time) AS last_seen
FROM `region-eu`.INFORMATION_SCHEMA.JOBS,
UNNEST(referenced_tables) AS t
WHERE creation_time > TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
  AND t.dataset_id = 'marts_marketing'
  AND t.table_id = 'mart_channel_performance'
GROUP BY user_email
ORDER BY jobs DESC

Google Cloud can also record lineage for BigQuery through Dataplex, which captures the relationships between tables for you.

The last step to the site or app: your measurement plan. No tool knows the link between the button on your checkout page and the event that follows from it. That lives in your tagging documentation, or it lives nowhere.

With ten to fifteen models and one dashboard, a well-kept measurement plan is enough. A sheet listing each event, its parameters, the tag that sends it and the report it lands in does the job.

Lineage software gets interesting once you have dozens of models, several sources that cross over, and people using each other's models without discussing it. Past that point you cannot hold the chain in your head.

Buy the tool before you are there and you end up maintaining documentation about a stack you could see in one glance.

• Keep one measurement plan listing each event, its parameters, where the tag fires and where the event is eventually used.
• Build transformations with ref() in dbt, never with hard-coded table names. The dependency graph then comes for free.
• Run dbt ls --select <model>+ before renaming or dropping a column, so you know what hangs underneath it.
• Point BI reports at the marts. Any report reading raw data sits outside your lineage.
• Record tag changes in GTM's version history with a description that names the reason, so a break can be tied to a date.
• Once your model count moves into the dozens, look at a lineage tool. Below that, a measurement plan is cheaper and faster.