Data Lineage Architecture with OpenLineage and Marquez: Column-Level Lineage and Impact Analysis (2026)

A finance analyst reports that revenue in the weekly dashboard dropped eleven percent overnight. The pipeline is green, every test passed, and nobody changed the dashboard. Three hours later someone finds that an upstream team renamed a currency column and a downstream join silently started matching fewer rows. The data was wrong and nothing was broken, which is the worst kind of incident.

Data lineage is the record that makes this kind of failure traceable in minutes instead of hours. It tells you which jobs produced which datasets, from which inputs, and, at its best, which source column ended up in which output column. In 2026 the open, vendor-neutral way to collect that record is OpenLineage, an event specification, paired with Marquez, its reference server for storing and querying the resulting graph.

This article treats lineage as an architecture problem, not a catalog feature. You will leave with a mental model of the event schema, a reference topology, working code for emitting and querying events, a clear picture of what column-level lineage can and cannot tell you, and an honest list of failure modes.

What this covers: the state of lineage tooling, the OpenLineage event model and facets, the Marquez storage and API design, column-level lineage semantics, impact analysis workflows, runnable examples, trade-offs, and a practical rollout checklist.

Context and Background

Lineage used to be something you reconstructed after the fact. Early catalogs scraped SQL query logs, parsed them offline, and drew boxes and arrows. That approach still exists, and it works for warehouses where every transformation is a SQL statement the warehouse itself logged. It breaks down in mixed estates: Spark jobs writing Parquet, Airflow tasks calling Python, dbt models compiled to SQL, Flink jobs reading Kafka. No single log contains the whole story.

The alternative is runtime lineage: each tool reports what it did while it did it. That is the premise of OpenLineage, an open standard under the LF AI and Data Foundation. Instead of one tool scraping everyone else, every tool emits a small JSON event at the start and end of each run, describing the job, the run, and the datasets read and written. A backend collects the events and assembles the graph. The spec is formalized as JSON Schema, with an OpenAPI definition for HTTP-based implementations, according to the project’s specification document.

Marquez is the reference backend. It is a Java service backed by PostgreSQL that accepts OpenLineage events, versions the metadata, and serves lineage queries plus a web UI. At the time of writing, the most recent Marquez release I could verify on the project’s releases page was 0.50.0, which added a data observability dashboard for event statistics, a GET /api/v1/jobs endpoint, and timezone-aware timestamps. The page showed day and month but not the year, so check the release date before quoting it. The OpenLineage client and integration project (the repository holds the spec and the Spark, Flink and dbt integrations) listed 1.53.0 as its latest release, again without a visible year. Treat both version numbers as a snapshot; pin whatever you deploy.

Why does this matter now? Three pressures converge. Regulators and auditors increasingly ask “where did this number come from” for AI training sets and financial reports. Data contracts and schema registries, which I covered in data contracts and schema evolution across streaming and the lakehouse, define what should flow between teams but cannot show what actually does. And orchestrators have standardized on emitting lineage natively, so the cost of collecting it has dropped sharply.

Commercial catalogs such as Collibra, Atlan, Alation and DataHub, and the cloud-native lineage services, all ingest OpenLineage in some form or compete with it. The open spec’s practical value is that producers and consumers decouple. You can swap Marquez for another backend without touching a single Airflow DAG.

A note on terminology. “Lineage” is overloaded. Table-level lineage says that table B was built from tables A1 and A2. Column-level lineage says that B.revenue came from A1.price multiplied by A2.quantity. Operational lineage adds run history: which run, when, did it succeed. Business lineage maps technical assets to glossary terms. OpenLineage and Marquez cover the first three well and leave the fourth to catalogs. This article concentrates on the first three.

The Reference Architecture: Producers, Transport, Backend, Consumers

The shortest accurate description of a modern data lineage architecture is this: producers emit events, a transport carries them, a backend stores and indexes them, and consumers query the resulting graph. Every design decision is about where each of those four responsibilities lives and how failure in one is prevented from breaking the others.

Figure 1: End-to-end data lineage architecture. Integrations in each tool emit OpenLineage events through a transport to Marquez, which persists them in PostgreSQL and serves the lineage graph to the UI and to impact-analysis automation.

Figure 1 shows the topology. Four producers (Airflow provider, Spark listener, dbt wrapper, Flink listener) share one transport, the transport posts to a single Marquez API, and Marquez writes to PostgreSQL. Downstream, the graph API and UI feed humans and automation.

Producers: lineage is emitted at the point of knowledge

The tool that executes the work is the only component that knows exactly what it read and wrote. A Spark listener sees the logical plan. An Airflow operator knows its inputs and outputs, and for SQL operators it can parse the statement. dbt knows the compiled model graph. This is why OpenLineage ships integrations inside each tool instead of a central scraper.

The cost is that coverage equals integration quality. Spark has the richest support, including column lineage collected through its listener mechanism, according to the project’s integrations matrix. The same page states that the matrix is incomplete, and it listed no column-lineage capability for Airflow, Flink, Hive, Trino or dbt. Do not infer absence from silence, though: Airflow’s provider includes a SQL parser for supported operators, and the OpenLineage dbt integration works from dbt artifacts. Test each integration against your own workloads, because capability tables lag behavior.

Transport: decouple emission from delivery

Producers must never fail a data job because the lineage backend is down. The standard answer is a transport abstraction in the client. HTTP is the default and simplest. Kafka is a common choice for high-volume estates because it buffers bursts and lets several consumers read the same event stream, for example Marquez plus a security tool. Console and file transports exist for debugging. The 1.53.0 release notes mention configurable retries for the GCP Lineage and GCS transports, which tells you cloud-managed backends are first-class targets.

An architectural rule worth adopting: emission is fire-and-forget with bounded retries, and a lineage failure logs a warning but never fails the task. Treat lineage as observability, not as a dependency.

Backend: Marquez as the system of record

Marquez accepts events at POST /api/v1/lineage, normalizes them into namespaces, jobs, runs and datasets, versions each entity, and persists everything in PostgreSQL. It keeps a distinction that matters: a job version changes when the job’s definition (source location, code) changes, and a dataset version changes when a run produces output with a new schema or data state. That versioning lets you ask not only “what feeds this table” but “what fed it last Tuesday, before the refactor”.

Consumers: the graph is only useful if something reads it

Humans read the web UI. Automation reads the API. The consumers that pay for the whole system are impact analysis in change review, incident triage, and audit evidence. If nobody queries the graph, lineage collection becomes shelfware, which is the most common failure of lineage programs in practice.

Why events and not a scraped graph

There is a design argument underneath all of this that is worth making explicit, because it is the thesis of this article. Scraped lineage is a derived view with unbounded staleness and unknowable precision. Event-based lineage is an append-only fact log: this run, at this time, read these inputs and wrote these outputs. From a log of facts you can derive any graph you like, including graphs for a past date, and you can audit the derivation. From a derived graph you cannot recover the facts.

That is the same reasoning behind event sourcing in application design. Marquez stores the events and builds the graph, which is why the graph can be rebuilt after a bug fix in the parsing logic, and why it can answer temporal questions. When you evaluate lineage tools, ask whether the primary store is the facts or the picture.

OpenLineage defines a small core and an extension mechanism. The core is three entities. A Run is one execution of a job, identified by a UUID, with start and end times. A Job is the process definition, named within a namespace. A Dataset is an abstract representation of data, named by its physical location. These definitions come directly from the OpenLineage specification.

Figure 2: The OpenLineage RunEvent and its facets. The column lineage facet hangs off an output dataset and separates DIRECT contributions from INDIRECT influences.

A RunEvent ties them together. It carries an event type (START, COMPLETE, FAIL, ABORT, and in the published schema also RUNNING and OTHER), an event time, the run, the job, the input datasets, the output datasets, a producer URI naming the emitting client, and a schemaURL pointing at the schema version. The spec page I fetched explicitly requires a START plus a terminal event (COMPLETE, FAIL or ABORT) and treats further events as optional. I could not confirm RUNNING and OTHER from that page, so verify them against the schema version you pin.

Identity: namespaces and names

Identity is where lineage graphs succeed or fail. A dataset is identified by a namespace plus a name, and the convention is that the pair encodes the physical location. For a Postgres table it might be namespace postgres://db.internal:5432 and name public.orders. For an S3 object it is s3://bucket and a key prefix. For Kafka it is the broker list and topic name.

If two tools name the same table differently, the graph splits into two disconnected components, and every cross-tool lineage question returns nothing. This is the single most common reason a proof of concept looks good in one tool and fails when stitched across Airflow, Spark and dbt. The OpenLineage project publishes a naming specification precisely for this. Adopt it, write down your conventions, and test with a deliberate cross-tool path before you roll out widely.

Jobs have the same problem in a different form. A job’s name should be stable across runs. If your orchestrator embeds a timestamp or a run suffix in the job name, every run becomes a new job and the graph fragments. Airflow uses the DAG id and task id, which is stable by design.

Facets are metadata attached to runs, jobs or datasets. Each is identified by name, and emitting a facet with an existing name replaces the earlier instance entirely, per the spec. Standard facets include, on runs, nominalTime, parent and errorMessage; on jobs, sourceCodeLocation, sourceCode, sql and ownership; on datasets, schema and dataSource, among others.

The parent run facet deserves attention because it solves a real problem. An Airflow task launches a Spark job. Both emit events. Without a link they look like unrelated runs. The parent facet lets the Spark run declare “I was launched by this Airflow run”, so the graph and the run history can be joined. In practice you propagate the parent identifiers through the Spark configuration or environment, which is a configuration chore that is easy to skip and expensive to retrofit.

Custom facets follow a naming rule: {prefix}{name}{entity}Facet in PascalCase for the type, and {prefix}_{name} in camelCase for the key when attached. The spec gives BigQueryStatisticsJobFacet and the key bigQuery_statistics as an example. Custom facets carry their own _schemaURL, and versioned schema URLs are expected to be immutable, pinned to a git SHA instead of a branch. That immutability rule matters for audit: an event written today must still validate in five years.

Facets are where you put organization-specific context without forking the spec: a data classification tag, a cost center, a PII flag, a contract identifier. Because consumers ignore facets they do not understand, you can add them without breaking anyone.

A minimal event, by hand

Here is a complete lineage hand-off you can run against a local Marquez. The event says a job named daily_orders_agg read raw.orders and wrote mart.daily_revenue.

# Start Marquez locally (verify flags against the repo README for your version)
git clone https://github.com/MarquezProject/marquez && cd marquez
./docker/up.sh --seed   # API on :5000, UI on :3000 in the default compose setup

RUN_ID=$(python3 -c "import uuid;print(uuid.uuid4())")

curl -s -X POST http://localhost:5000/api/v1/lineage \
  -H 'Content-Type: application/json' \
  -d '{
    "eventType": "START",
    "eventTime": "2026-10-30T08:00:00Z",
    "run": {"runId": "'"$RUN_ID"'"},
    "job": {"namespace": "analytics", "name": "daily_orders_agg"},
    "inputs":  [{"namespace": "postgres://db:5432", "name": "raw.orders"}],
    "outputs": [{"namespace": "postgres://db:5432", "name": "mart.daily_revenue"}],
    "producer": "https://example.com/my-producer/1.0",
    "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent"
  }'

curl -s -X POST http://localhost:5000/api/v1/lineage \
  -H 'Content-Type: application/json' \
  -d '{
    "eventType": "COMPLETE",
    "eventTime": "2026-10-30T08:04:10Z",
    "run": {"runId": "'"$RUN_ID"'"},
    "job": {"namespace": "analytics", "name": "daily_orders_agg"},
    "inputs":  [{"namespace": "postgres://db:5432", "name": "raw.orders"}],
    "outputs": [{"namespace": "postgres://db:5432", "name": "mart.daily_revenue"}],
    "producer": "https://example.com/my-producer/1.0",
    "schemaURL": "https://openlineage.io/spec/1-0-5/OpenLineage.json#/definitions/RunEvent"
  }'

The schema version in that URL is illustrative; use the version your client library emits. Notice how little is required. A job name, a run id, two dataset names, two timestamps. The richness comes from facets, and the discipline comes from consistent naming.

Marquez Internals, Column-Level Lineage and Impact Analysis

With the event model in hand, the interesting questions are what Marquez does with an event, how column-level lineage changes the graph, and how you turn it into decisions.

What happens when an event arrives

Marquez treats each incoming event as a transaction against its relational model. It upserts the namespace and job, creates or updates the run, creates dataset records and a new dataset version when the output state changed, links the run to its input and output dataset versions, and stores facets. Column lineage rows are derived from the output dataset’s columnLineage facet.

Figure 3: Event lifecycle. The orchestrator triggers the client, the client sends START and COMPLETE events to Marquez, and Marquez upserts jobs, runs and dataset versions and stores column lineage rows in PostgreSQL.

Two properties follow from this design. First, the backend is order-tolerant in practice but not magic: a COMPLETE event that arrives without a START still gets recorded, but start-time-dependent views will be incomplete. Second, because PostgreSQL is the store, the write path scales with the database. For a typical estate of thousands of jobs, the load is modest. For very high event volumes, such as a streaming engine emitting frequent RUNNING events, plan for write amplification and put a queue in front.

The 0.50.0 release’s data observability dashboard is relevant here. It surfaces OpenLineage event statistics by source, dataset and job, which is exactly what you need to spot a producer that has gone silent or started emitting malformed events. Silent producers are the quiet killer of lineage programs: the graph looks fine but a whole team’s pipelines stopped reporting two weeks ago.

Reading the graph

The primary query is the lineage graph around a node. A node id encodes the entity type and its identity, for example dataset:postgres://db:5432:mart.daily_revenue or job:analytics:daily_orders_agg. The API returns the nodes and edges within a depth. From memory, and subject to verification against your version’s OpenAPI document, the calls look like this:

# Table-and-job level lineage, 3 hops either direction
curl -s "http://localhost:5000/api/v1/lineage?nodeId=dataset:postgres://db:5432:mart.daily_revenue&depth=3" | jq '.graph[].id'

# Column-level lineage around one column
curl -s "http://localhost:5000/api/v1/column-lineage?nodeId=datasetField:postgres://db:5432:mart.daily_revenue:revenue&depth=3" | jq '.'

I could not fetch the live Marquez API reference during research, so confirm endpoint paths, the datasetField node id format and parameters in the OpenAPI spec bundled with your version before depending on them. The release notes I verified do confirm that column lineage queries exist (0.47.0 fixed a column lineage query for symlink datasets) and that 0.49.0 added job-to-job lineage.

Table-level lineage says that daily_revenue depends on orders. That is true of every column in the table and therefore tells you little. Column-level lineage narrows the claim: daily_revenue.revenue depends on orders.amount and orders.currency, while daily_revenue.order_day depends on orders.created_at.

OpenLineage represents this with the columnLineage dataset facet on an output dataset. Per the facet documentation, it has a fields map from each output column to its inputFields, each naming a source namespace, dataset name and column, plus a list of transformations. It also has a dataset-level dataset list for dependencies that affect the whole output, such as a filter that decides which rows exist.

Each transformation has a type of DIRECT or INDIRECT, a subtype, an optional description, and a masking boolean. This is the part most explanations skip, and it is what makes the facet useful for decisions.

Type	Subtype	Meaning
DIRECT	IDENTITY	Value copied as is
DIRECT	TRANSFORMATION	Computed from the source value within the same row
DIRECT	AGGREGATION	Computed across multiple input rows
INDIRECT	JOIN	Column used in a join condition
INDIRECT	GROUP_BY	Column used for grouping
INDIRECT	FILTER	Column used in a filter predicate
INDIRECT	SORT	Column used for ordering
INDIRECT	WINDOW	Column used to define a window
INDIRECT	CONDITIONAL	Column used in IF, CASE WHEN or COALESCE

DIRECT means the output value is derived from the input. INDIRECT means the input shaped the output without supplying its value. A WHERE currency = 'USD' clause makes currency an INDIRECT FILTER input to every column of the output, because it determines which rows exist. This distinction is what lets you answer “which downstream values would change if this column’s contents changed” differently from “which downstream values would change if this column disappeared”.

The masking flag marks inputs that were obfuscated, for example hashed. For privacy tracking, a masked path means the raw value does not propagate even though the dependency exists.

The legacy representation gotcha

One detail from the docs deserves emphasis. With Spark’s spark.openlineage.columnLineage.datasetLineageEnabled option at its default of false, all dependencies, including the indirect ones, are placed under fields and the dataset-level list is empty. The documentation states this produces a nearly cartesian set of INDIRECT links and recommends enabling the option. If your column lineage graph from Spark looks like a dense hairball where every output depends on every filter column, this is why. Turning the option on moves row-selection dependencies to the dataset level, where they belong, and the per-column graph becomes readable.

{
  "eventType": "COMPLETE",
  "job": {"namespace": "analytics", "name": "daily_orders_agg"},
  "outputs": [{
    "namespace": "postgres://db:5432",
    "name": "mart.daily_revenue",
    "facets": {
      "columnLineage": {
        "_producer": "https://example.com/my-producer/1.0",
        "_schemaURL": "https://openlineage.io/spec/facets/1-2-0/ColumnLineageDatasetFacet.json",
        "fields": {
          "revenue": {
            "inputFields": [
              {"namespace": "postgres://db:5432", "name": "raw.orders", "field": "amount",
               "transformations": [{"type": "DIRECT", "subtype": "AGGREGATION", "description": "SUM", "masking": false}]}
            ]
          }
        },
        "dataset": [
          {"namespace": "postgres://db:5432", "name": "raw.orders", "field": "status",
           "transformations": [{"type": "INDIRECT", "subtype": "FILTER", "description": "status = paid", "masking": false}]}
        ]
      }
    }
  }]
}

The facet schema URL above comes from the documentation I fetched. The other fields (eventTime, run, producer, schemaURL) are omitted here for brevity but are required in a real event.

From graph to impact analysis

Impact analysis is the question “if I change X, what breaks”. With only table-level lineage the answer is a long list, most of which is false alarms, and teams learn to ignore it. Column-level lineage makes the answer specific enough to act on.

Figure 4: Impact analysis workflow. A proposed schema change triggers a column lineage query, results are filtered by DIRECT versus INDIRECT, and the surviving jobs and owners drive notifications and CI gating.

The workflow in Figure 4 has five steps. Take a proposed change, say altering the type of raw.orders.amount. Query the column lineage API for the downstream closure of that column. Classify each downstream edge as DIRECT or INDIRECT. For DIRECT edges, values will change or break, so these are blocking. For INDIRECT edges, the effect is on which rows appear, which is still significant for FILTER, JOIN and GROUP_BY, but is usually reviewable instead of blocking. Finally, resolve owners from the ownership job facet or an external registry, and notify them or gate the deployment.

The classification step is the original insight here. Most lineage tools collapse DIRECT and INDIRECT into one edge type. The OpenLineage taxonomy gives you the vocabulary to set different policies. A reasonable starting policy: block on DIRECT dependents of a column being dropped or retyped; require acknowledgment for INDIRECT JOIN or GROUP_BY dependents; ignore INDIRECT SORT dependents.

Here is a runnable sketch of the downstream-closure query in Python. It assumes the API shapes described above, so treat it as a starting point to adapt after checking your version’s OpenAPI document.

import requests, sys

MARQUEZ = "http://localhost:5000"

def downstream_columns(namespace, dataset, column, depth=5):
    node = f"datasetField:{namespace}:{dataset}:{column}"
    r = requests.get(f"{MARQUEZ}/api/v1/column-lineage",
                     params={"nodeId": node, "depth": depth}, timeout=15)
    r.raise_for_status()
    graph = r.json().get("graph", [])
    out = []
    for n in graph:
        data = n.get("data", {})
        # field names and nesting are version dependent; inspect a real response first
        out.append({"id": n.get("id"),
                    "inputs": [i.get("field") for i in data.get("inputFields", [])]})
    return out

if __name__ == "__main__":
    impacted = downstream_columns("postgres://db:5432", "raw.orders", "amount")
    for item in impacted:
        print(item["id"])
    sys.exit(1 if impacted else 0)  # CI gate: non-zero means review required

Wire this into a pull request check on your schema repository: if the changed column has downstream dependents, post the list as a comment and require an acknowledgment. This is where lineage stops being documentation and becomes a control.

Deeper Analysis: Capture Mechanics, Scale and Precision

The architecture diagram suggests that lineage is plumbing. The real engineering difficulty is in precision, coverage and cost.

How each integration actually knows the lineage

Different producers derive lineage by different mechanisms, and the mechanism determines both accuracy and blind spots.

Spark listens to query execution through its listener interface and inspects the logical plan, which gives it exact input and output datasets and, with column lineage enabled, expressions mapping input attributes to output attributes. It sees what the engine actually executed, not what the code suggested, so it handles dynamic SQL and generated code. Its blind spots are anything outside the plan: a UDF is opaque, so a column computed in a Python UDF typically shows its input columns as dependencies without detail on the logic.

Airflow uses a provider package that extracts lineage per operator. For SQL operators it parses the statement to find tables and columns. For other operators, the author can declare inlets and outlets or implement extraction hooks. The provider ships a list of supported classes, and operators outside it emit job-level events without dataset detail. A PythonOperator that internally reads a file and writes a table is invisible unless you annotate it. This is the largest coverage gap in most estates, because the tasks that matter most are often custom Python.

dbt knows the compiled model graph and its artifacts, so table-level lineage between models is reliable. Column-level lineage from dbt requires parsing the compiled SQL, which is feasible for plain SQL and harder with heavy Jinja macros or warehouse-specific syntax. For how dbt and its alternatives structure models, tests and incremental builds, see the comparison of dbt and SQLMesh for transformation, CI testing and incremental models.

SQL parsing in general is the common thread. Static parsing of SQL text infers lineage without running anything, which is cheap and works at CI time, but it cannot resolve SELECT * without schema knowledge, cannot see through dynamically built queries, and needs a dialect-aware parser. The 1.53.0 notes include “parenthesized join support in SQL lineage extraction”, a reminder that parser coverage grows one syntax construct at a time. Expect gaps in unusual syntax.

Precision versus recall, and why the dataset-level list matters

A lineage collector can err in two directions. Missing a real dependency (low recall) means impact analysis says “safe” when it is not. Reporting a dependency that does not exist (low precision) means impact analysis cries wolf until people stop reading it. For governance use cases, recall matters more; for developer-facing change review, precision matters more, because noise kills adoption.

The Spark legacy representation discussed earlier is a precision failure mode created by a modeling choice: spreading dataset-wide dependencies onto every column. Enabling the dataset-level list fixes it without losing any information. The general lesson is to check whether an integration distinguishes “affects the value” from “affects which rows exist”. If it does not, the column graph will be over-connected.

Event volume and sizing

I will not give throughput benchmarks, because I have no verified published numbers for Marquez and inventing them would mislead you. What can be derived from first principles is the shape of the load. Batch estates emit at least two events per run (START and terminal), so the daily event count is roughly twice the number of daily job runs. An estate with 5,000 task runs a day produces on the order of 10,000 events a day, a trivial write load for PostgreSQL. Event size is dominated by facets, particularly large sql, sourceCode and schema facets, and by wide tables with many columns in the column lineage facet. A table with 400 columns and several inputs per column yields a facet with thousands of entries.

Streaming and micro-batch engines change the arithmetic. If a streaming job emits an event per micro-batch, the volume is the number of batches, which can reach tens of thousands per day per job. For these, either configure the integration to emit sparingly, aggregate before sending, or route through Kafka so the backend can consume at its own pace. Measure with your own estate: count events per day per producer in a staging environment for a week, then project.

Retention matters too. Because Marquez versions datasets and keeps runs, the database grows with history. Decide up front how long run history is useful. Lineage structure changes slowly, but run records pile up, and partitioning or pruning old run data is the standard remedy.

Orchestration context

Lineage and orchestration are intertwined: the orchestrator is the best source of run-level truth and the natural place to emit parent run facets. If you are choosing or migrating orchestrators, lineage support is a legitimate selection criterion. I compared scheduling models and developer experience in Airflow vs Dagster vs Prefect for data orchestration. On lineage specifically, Airflow’s provider package is the most mature OpenLineage producer I am aware of from documentation, while the OpenLineage integrations page I fetched made no mention of Dagster at all, so check current Dagster-side support yourself instead of assuming it.

Marquez versus the alternatives

A decision matrix helps because “should we run Marquez” is the wrong first question. The first question is what you need lineage for.

Need	Marquez	Catalog with OpenLineage ingest	Warehouse-native lineage
Collect lineage across Spark, Airflow, dbt	Strong, reference implementation	Strong, depends on product	Weak outside that warehouse
Column-level lineage storage and API	Supported	Product dependent	Often strong for SQL inside warehouse
Business glossary, ownership workflow	Minimal	Strong	Limited
Operational run history and job versions	Strong	Varies	Limited
Self-hosting cost and ops burden	You run Java service plus PostgreSQL	Vendor or you	None
Vendor lock-in risk	Low	Moderate, event format is portable	Higher

My read: Marquez is the right backend when you want an open, inspectable store for run-level and column-level lineage and are willing to operate a small service. A catalog is the right front-end when business users need search, glossary and workflow. Because producers speak OpenLineage, the choices are not exclusive. Run Marquez as the lineage store today and add a catalog later that reads the same events.

Trade-offs, Gotchas, and What Goes Wrong

Honest limits first. Lineage is a model of what tools reported, not a guarantee of what happened to the data.

Coverage gaps are invisible. A job that does not emit events does not appear as a gap; it appears as the absence of an edge, which looks identical to “no dependency”. The graph cannot tell you what it does not know. Mitigate with a completeness check: compare the list of jobs your orchestrator ran against the jobs Marquez saw, and alert on the difference. The event statistics in recent Marquez versions help.

Naming drift fragments the graph. Covered above, and worth repeating because it is the most common failure. A trailing slash, a different host alias or a case difference creates two datasets. Add a lint step that checks emitted namespaces against an allow-list.

Column lineage depends on parser fidelity. SELECT *, UDFs, dynamic SQL, pivots and complex macros will produce incomplete or coarse column lineage. A fall-back is common: when the parser cannot resolve columns, the producer emits table-level lineage only. That is correct behavior, but it means the column graph is a subset of the truth. Do not present it to auditors as complete.

Facet replacement semantics bite. Because emitting a facet with an existing name replaces the earlier one entirely, two producers describing the same dataset with the same facet name will overwrite each other. If both Spark and dbt emit a schema facet for a table with slightly different content, the last writer wins and the version history churns. Decide which producer owns which dataset’s metadata.

Backends are not transactional with your data. An event can claim an output dataset was written even if the write failed later, depending on when the integration emits. Prefer integrations that emit the terminal event only after the commit, and treat FAIL and ABORT events as first-class so the graph does not show phantom data.

Sensitive data leaks into lineage. The sql and sourceCode facets can contain literal values, table names that reveal business structure, and in bad cases credentials embedded in query text. Lineage stores are therefore security-relevant. Restrict access, scrub literals where the integration allows it, and review what your producers put in facets before enabling them broadly. Also note that the 1.53.0 notes mention Jackson upgrades for several CVEs, a reminder to patch the client libraries running inside your Spark drivers.

Operational weight. Marquez is a stateful service with a database. Someone must back it up, upgrade it and monitor it. The 0.50.0 notes show the db-migrate command now runs separately from HTTP server startup, which is the safer pattern for rolling upgrades because you can run migrations as a distinct step. Adopt that separation in your deployment pipeline.

Anti-pattern: lineage as a one-time project. Teams stand up Marquez, wire up three pipelines, demo it, and stop. Within a quarter new pipelines are uninstrumented and the graph is stale. Lineage must be a default of your platform templates, not an opt-in.

Anti-pattern: treating the graph as authoritative for deletion. “Nothing depends on this table according to lineage” is a hypothesis, not proof. Ad hoc notebook queries, BI tools reading directly and external consumers may not emit events. Use lineage to find who to ask, then confirm with query logs and a deprecation period.

Practical Recommendations

Start narrow and wire up the paths that hurt. Pick one critical data product, such as the revenue mart, and instrument every step from raw ingestion to the dashboard source. A complete chain through three tools proves the naming conventions and parent run linkage and gives you a real impact analysis demo, which is worth more than broad but disconnected coverage.

Decide naming before code. Write down the namespace and dataset naming convention, aligned with the OpenLineage naming spec, and enforce it in a shared client configuration. Adopt a stable job naming scheme tied to your orchestrator’s identifiers.

Put the transport behind a queue if you can. HTTP straight to Marquez is fine for batch estates of modest size. If you have streaming producers or several consumers, emit to Kafka and let Marquez and any other consumer read from it.

Then make lineage a control, not a picture. The step that creates organizational value is the pull request check from the previous section, plus incident runbooks that begin with a lineage query. Pair it with the contract layer: contracts declare intended interfaces, lineage verifies actual dependencies, and the gap between the two is your risk register. The earlier article on data contracts and schema evolution covers the contract half.

A rollout checklist:

Pin versions of the OpenLineage clients and Marquez, and record them in infrastructure code.
Publish a naming convention document and a lint check for namespaces.
Enable the Spark dataset-level column lineage option before analyzing any column graph.
Propagate parent run facets from orchestrator to engine jobs.
Make lineage failures non-fatal for data jobs, with bounded retries.
Alert on producers that go silent, using event statistics.
Scrub or restrict sql and sourceCode facets and lock down API access.
Add a CI gate that queries column lineage for any schema change.
Review coverage monthly: orchestrator job list versus lineage job list.
Define retention for run history and test restores of the PostgreSQL database.

Frequently Asked Questions

What is data lineage and why does it matter?

Data lineage is a record of how data moves and transforms: which jobs read which inputs and wrote which outputs, and at finer grain which source columns feed which output columns. It matters for debugging bad numbers, assessing the impact of schema changes before they ship, proving provenance to auditors, and finding who owns an upstream problem. Without it, tracing an incident means asking around and reading code across several tools.

What is the difference between OpenLineage and Marquez?

OpenLineage is an open specification for describing lineage as events, plus client libraries and integrations that emit them. Marquez is a separate project that acts as the reference backend: it receives OpenLineage events, stores them in PostgreSQL, versions the metadata, and exposes an API and web UI. You can use OpenLineage without Marquez, sending events to another compatible backend, and Marquez can serve as one consumer among several.

What is column-level lineage?

Column-level lineage maps each output column to the specific input columns it depends on, instead of saying only that one table depends on another. In OpenLineage it is carried by the columnLineage dataset facet, which also classifies each dependency as DIRECT (the value is derived from the input) or INDIRECT (the input influences which rows exist, such as a join, filter or sort), with subtypes like AGGREGATION, FILTER and JOIN.

Does OpenLineage support Airflow, Spark and dbt?

Yes. The project publishes integrations for Apache Airflow, Apache Spark, dbt, Apache Flink, Apache Hive, Trino and Presto, according to its integrations page, though that page warns its capability matrix is incomplete. Spark is listed with column lineage support. Capability varies by tool and version, so test each integration on your own jobs, especially for column-level detail and for custom Python operators.

Is Marquez production ready?

Marquez is a maintained open source project with regular releases, and the verified recent releases show active work on jobs, lineage and observability features. Whether it is ready for you depends on operations: you run a Java service and a PostgreSQL database, so you need backups, upgrades and monitoring. I have no verified published benchmark of its scaling limits, so load test with your own event volumes before committing.

How do I use lineage for impact analysis?

Query the column lineage API for the downstream closure of the column you plan to change, then split dependents into DIRECT and INDIRECT. Treat DIRECT dependents of a dropped or retyped column as blocking, and INDIRECT join, group or filter dependents as review items. Resolve owners from ownership metadata and attach the list to the pull request, ideally as an automated CI check.