Data Contracts and Schema Registries: Schema Evolution for Streaming and Lakehouse Pipelines (2026)

A producer team renames a column from cust_id to customer_id. The unit tests pass, the service deploys on a Tuesday afternoon, and nothing alarms. Three days later the finance dashboard shows revenue per customer collapsing to zero, a feature store silently serves nulls to a fraud model, and a nightly Spark job writes a week of corrupted partitions. No component crashed. Every one of them did exactly what it was told, using a shape of data that quietly stopped being true.

Data contracts are the discipline that prevents this: an explicit, versioned, machine-checkable agreement between the team that produces a dataset and the teams that depend on it. A contract covers more than field names and types. It also carries semantics, quality rules, service levels and a named owner, and it is enforced at several points between the producer’s pull request and the lakehouse table.

What this covers: what a contract contains, how a schema registry and its compatibility modes work, how Avro, Protobuf and JSON Schema differ in evolution, where to enforce the contract, how Iceberg field IDs make table evolution safe, a CI example, a dual-publish workflow for breaking changes, dead-letter handling and the failure modes a schema can never catch.

Context and Background

For most of the history of data engineering, the interface between operational systems and analytics was an accident. A database table existed, someone built a nightly extract against it, and the extract’s author became a silent consumer of every column. The producing team had no idea the dependency existed. Change-data-capture and event streaming made the problem faster rather than smaller, because the coupling now propagates in seconds. If you run CDC pipelines, our walkthrough of change data capture with Debezium shows how a single ALTER TABLE on an operational database flows straight into every downstream topic.

Schema registries were the first structural answer. A registry stores versioned schemas centrally, hands each one a numeric ID, and lets serializers embed that ID in every message instead of the full schema. The registry also checks each new version against earlier ones and rejects incompatible changes before any data is written. Confluent Schema Registry popularized the model for Kafka, and several other registries expose compatible APIs.

A schema alone is not a contract, though. The Confluent documentation on data contracts for Schema Registry defines a data contract as a formal agreement between an upstream and a downstream component on the structure and meaning of data in motion, and says the schema is only one part of it. The remaining parts are integrity constraints, metadata, rules and versioned evolution. That broader view is the one this article adopts.

At the batch and lakehouse end, the open-source community has converged on a vendor-neutral file format. The Open Data Contract Standard (ODCS), maintained by the Bitol project under the Linux Foundation’s AI and Data umbrella, describes a dataset in a single YAML document. The Data Contract CLI documentation lists v3.1.0 as the current version at the time of writing and uses ODCS as its native format. We will use it for the human-readable half of the contract, while the registry carries the machine-enforced half on the wire.

The motivation is economic, not aesthetic. Every broken pipeline costs engineering hours on the consumer side, but the cost is paid by people who did not make the change. A contract moves that cost back to the moment of change, where it is cheapest to fix. If you are comparing the table layer your contract will land on, our posts on Apache Iceberg versus Paimon and on Iceberg catalogs cover the storage side.

Data Contracts Reference Architecture: Where Each Control Sits

A data contract architecture is a set of enforcement points along the path from producer code to lakehouse table. The producer’s CI pipeline checks schema compatibility and lints the contract file, the schema registry rejects incompatible versions at registration, serializers embed schema IDs in each message, optional broker validation rejects unregistered IDs, and stream processors route bad records to a dead-letter topic before writing to the table.

Figure 1: Enforcement points for data contracts, from the producer pull request to the lakehouse table. Each control catches a different class of failure.

The diagram shows one contract document feeding CI, a registry sitting beside the topic as the source of schema truth, and a stream processor that splits traffic into good records for the lakehouse and rejects for a dead-letter topic. No single control is sufficient. The registry cannot see semantic drift, CI cannot see a producer that bypasses the pipeline, and a lakehouse table cannot recover data it never received.

What a contract actually contains

A useful contract has five layers, and each one fails differently when it is missing.

Schema is the structural layer: field names, types, nullability, nested structure. This is the part machines can verify exhaustively, and it is the part registries enforce natively.

Semantics answers what the fields mean. Is amount in cents or dollars, is event_time the moment the user clicked or the moment the server logged it, is status = 3 documented anywhere? ODCS gives this a home through descriptions, logical types and authoritative definitions that link a field to a shared business concept by URL. Semantics are mostly prose and links, so enforcement is partial at best.

Quality rules are executable assertions about values: order_total >= 0, country belongs to an allowed set, customer_id is never null, row counts per hour stay within a band. In ODCS a quality rule can carry a SQL query and a threshold such as mustBeBetween. In Confluent’s model, rules attach to the schema as a rule set with CEL expressions evaluated at serialization time.

Service levels define freshness, retention and delivery expectations: the data lands within five minutes of the event, history is retained for 400 days, a topic’s partition key is stable. ODCS exposes SLA properties for exactly these. Consumers need them to size their own alerting, because a consumer cannot distinguish “no new data” from “pipeline broken” without a freshness promise.

Ownership names the team and the escalation path. A contract with no owner is a document, not an agreement. ODCS includes team and role information, and the practical value is that a failed check can page a human instead of an anonymous mailing list.

The registry as the system of record for wire format

On Kafka the registry has three jobs. It stores schemas under named subjects, with each registration creating a new numbered version. It assigns each schema a globally unique ID that producers embed in a small header at the start of the message payload. And it enforces a compatibility mode per subject, or globally, whenever someone registers a new version.

The subject name is how versions are grouped. The default strategy ties a subject to a topic, so the value schema for topic orders lives in subject orders-value. Alternative strategies group by record type, which lets one topic carry several event types, at the cost of making compatibility checks apply per record type rather than per topic. Choose the strategy deliberately before you have data, because changing it later effectively starts the version history over.

Because consumers fetch schemas by ID and cache them, the registry is on the read path only for the first message of each schema. That keeps latency impact small, but it also means a registry outage mainly affects new schema registrations and cold-starting clients, not steady-state throughput. Run it highly available anyway, since your CI pipeline and every restarting consumer depend on it.

Contract versus schema: the layering that avoids duplication

A common mistake is to maintain the same truth in three places: an Avro file, an ODCS YAML and a dbt model. The better layering is to pick one authoritative source and generate the others. The Data Contract CLI can import from and export to many formats, including Avro, Protobuf, JSON Schema, SQL DDL and dbt, according to its documentation, so a contract file can be the source and the Avro schema a generated artifact. Teams that prefer schema-first development reverse the flow and import an existing Avro or Protobuf definition into a contract that adds the semantics, quality and SLA fields the schema cannot hold. Either direction works. What does not work is hand-editing both and hoping they agree.

Schema Registry Compatibility Modes and Schema Evolution Rules

Schema evolution is safe only relative to a stated promise about who upgrades first. A schema registry encodes that promise as a compatibility mode. BACKWARD means consumers on the new schema can read data written with the previous one. FORWARD means consumers on the old schema can read data written with the new one. FULL means both. Transitive variants check against every earlier version, not just the latest.

Figure 2: Choosing a compatibility mode decides who must upgrade first, and the transitive flag decides how far back the check reaches.

The seven modes and what they promise

The Confluent documentation lists seven settings: BACKWARD, BACKWARD_TRANSITIVE, FORWARD, FORWARD_TRANSITIVE, FULL, FULL_TRANSITIVE and NONE. The default is BACKWARD, not BACKWARD_TRANSITIVE. That default is deliberate for Kafka, because the common recovery operation is to rewind a consumer to the start of a topic and reprocess. A consumer on the newest schema must therefore be able to read everything still retained, which is what backward compatibility guarantees.

The upgrade order follows from the definitions. Under BACKWARD, upgrade every consumer before any producer emits the new schema. Under FORWARD, upgrade producers first, and make sure old data is no longer something consumers will read, then upgrade consumers. Under FULL, the two sides can move independently, which is why FULL is the mode that scales best across many teams that cannot coordinate release trains.

The transitive flag matters more than it looks. Non-transitive checking compares a candidate only to the latest version. That allows a slow drift in which each step is compatible with its predecessor but version 5 cannot be read by a reader built for version 1. The documentation notes explicitly that schemas can be incrementally compatible without being transitively compatible. If any consumer can be rewound across several versions, or any long-retention topic spans many versions, use a transitive mode. The Confluent guidance for Protobuf goes further and recommends BACKWARD_TRANSITIVE as best practice, because adding a new message type is not forward compatible.

A worked example in Avro

Consider an OrderPlaced event at version 1.

{
  "type": "record",
  "name": "OrderPlaced",
  "namespace": "com.example.orders",
  "fields": [
    {"name": "order_id", "type": "string"},
    {"name": "customer_id", "type": "string"},
    {"name": "amount_cents", "type": "long"}
  ]
}

Version 2 adds a currency code. Avro has no optional keyword. A field is optional when it is a union with null, listed first, with a default of null, or when it has any other default value.

{
  "type": "record",
  "name": "OrderPlaced",
  "namespace": "com.example.orders",
  "fields": [
    {"name": "order_id", "type": "string"},
    {"name": "customer_id", "type": "string"},
    {"name": "amount_cents", "type": "long"},
    {"name": "currency", "type": "string", "default": "USD"}
  ]
}

Is this change backward compatible? A v2 reader decoding a v1 message finds no currency in the data, falls back to the default and succeeds. So yes. Is it forward compatible? A v1 reader decoding a v2 message ignores the unknown field, which Avro schema resolution permits. So yes, and the change is FULL compatible. Remove the default and the answer changes: the v2 reader has no value to use for old messages, so BACKWARD fails, while FORWARD still passes. That single keyword is the difference between a safe change and a rejected registration.

Now the rename from the opening story. In Avro a rename is a delete plus an add. Deleting customer_id and adding customer_ref with a default passes BACKWARD, because the new reader supplies the default, but old consumers still expecting customer_id have no default to fall back on, so FORWARD fails. Avro does provide aliases to let a reader map an old field name to a new one, but support depends on the library and registry compatibility checks may not treat an alias as a rename. Treat renames as breaking changes unless you have tested the full path.

Avro vs Protobuf vs JSON Schema evolution

The three formats solve the same problem with different physical mechanisms, and the differences decide which compatibility mode is realistic. Avro vs Protobuf is the comparison teams most often get wrong, so it deserves a table.

Property	Avro	Protobuf	JSON Schema
Wire identity of a field	Position resolved by schema; reader needs writer schema	Field number	Field name in text
Reader needs writer schema	Yes, always	No, tolerates unknown fields	No
Add optional field	Needs default, then FULL safe	Safe	Depends on policy
Remove field	Safe if it had a default	Safe, but reserve the number	Depends on policy
Rename field	Breaking without aliases	Safe on the wire, breaking for JSON mapping and code	Breaking
Change field type	Only listed promotions	Mostly unsafe; some integer types interchangeable	Breaking in most cases
Required fields	Field without default behaves as required	Proto3 has none	`required` keyword
Typical registry advice	BACKWARD or FULL	BACKWARD_TRANSITIVE	Choose policy first

Avro always resolves a reader schema against the exact writer schema, which is why the registry embeds an ID in every message. That design gives Avro compact payloads and strict, checkable rules about defaults, but it makes defaults the load-bearing part of every change. Protobuf identifies fields by number, so names can change freely on the wire, and unknown fields are skipped by default. Its hazards are different: reusing a number or changing a type reinterprets old bytes as something else.

The Protocol Buffers best-practice guidance is blunt on this. It says never to reuse a tag number, to reserve the numbers of deleted fields (for example reserved 2, 3;), not to change a field’s type, not to add required fields, and not to change a field’s default value because clients and servers can disagree during a rollout. It also recommends that the first enum value be an UNSPECIFIED default with tag zero, so that an unset or unknown value decodes to something safe.

syntax = "proto3";
package com.example.orders;

message OrderPlaced {
  reserved 4;            // was: legacy_channel, removed in v3
  reserved "legacy_channel";

  string order_id = 1;
  string customer_id = 2;
  int64  amount_cents = 3;
  string currency = 5;   // added in v2, new number, never reused
}

JSON Schema is the awkward third option. The Confluent documentation states that JSON Schema does not explicitly define compatibility rules, so the registry derives them from a compatibilityPolicy, either LENIENT or STRICT, and in STRICT mode from whether the content model is open or closed through additionalProperties. The consequence is counterintuitive. In STRICT mode with a closed model (additionalProperties: false), adding an optional field is backward compatible but not forward compatible, because an old reader rejects the unknown property. In an open model, the same change flips: it is forward compatible and not backward compatible, per the documented tables. The identical edit has opposite compatibility depending on a flag that is easy to overlook.

Which breaking changes look harmless

Registries judge structure, so several changes pass every check and still break consumers. Widening a numeric type from int to long is a documented backward-compatible promotion in Avro, yet a consumer that stores the value in a 32-bit column overflows on the first large value. Adding a new enum symbol is allowed in many configurations, yet consumers with a switch statement and no default branch throw. Changing the unit of amount from dollars to cents changes nothing structurally and everything semantically. Tightening a value range, say requiring quantity > 0 where zero was previously legal, is invisible to a schema. These cases are why the structural check is necessary and never sufficient, and why the later sections spend time on quality rules and semantic drift.

Choosing a mode in practice

For event topics on Kafka with replay requirements, start with BACKWARD_TRANSITIVE or FULL_TRANSITIVE. Reserve plain BACKWARD for topics with short retention where nobody replays. Use FORWARD only when you control producers and have a reason to let consumers lag, such as a long consumer-side validation window, and remember the documentation’s warning that FORWARD requires anticipating future changes. Never leave NONE enabled in production. A topic set to NONE has a schema registry in name only, and the first incompatible registration will not be noticed until a consumer fails.

Two operational details deserve attention. A compatibility mode set through the REST API is global and overrides properties-file settings, so audit the effective configuration per subject rather than trusting a config file. And Kafka Streams and Flink have their own constraints on which modes are safe, since stateful jobs deserialize with a single schema; check the vendor documentation for your runtime, because the published guidance has some inconsistency between sections.

Enforcing Data Contracts: CI, Registry, Broker and Stream Processor

Enforcement works best as defense in depth, with the cheapest and earliest check doing most of the work. A contract that is only validated at runtime tells you about a break after data is already wrong. A contract that is only validated in CI misses producers that bypass the pipeline. The practical design uses four enforcement points, each catching what the others cannot.

Point one: producer CI

The earliest check runs when the schema change is proposed. The registry exposes a compatibility endpoint that tests a candidate schema against a subject’s existing versions without registering it. A CI job calls it, and the pull request fails if the answer is false. This is the single highest-value control, because it fails in minutes, in front of the person who made the change, with the context to fix it.

#!/usr/bin/env python3
"""CI gate: fail the build if a schema change violates registry compatibility."""
import json, os, sys, requests

REGISTRY = os.environ["SCHEMA_REGISTRY_URL"]
AUTH = (os.environ["SR_KEY"], os.environ["SR_SECRET"])

def check(subject: str, schema_path: str, schema_type: str = "AVRO") -> bool:
    schema = open(schema_path).read()
    resp = requests.post(
        f"{REGISTRY}/compatibility/subjects/{subject}/versions/latest",
        headers={"Content-Type": "application/vnd.schemaregistry.v1+json"},
        data=json.dumps({"schema": schema, "schemaType": schema_type}),
        auth=AUTH, timeout=10,
    )
    if resp.status_code == 404:        # subject does not exist yet: first version
        return True
    resp.raise_for_status()
    return resp.json().get("is_compatible", False)

if __name__ == "__main__":
    failures = []
    for subject, path in [("orders-value", "schemas/order_placed.avsc")]:
        if not check(subject, path):
            failures.append(subject)
    if failures:
        print(f"INCOMPATIBLE schema change for: {', '.join(failures)}")
        sys.exit(1)
    print("All schema changes are compatible.")

Two refinements make this gate trustworthy. First, run it against the registry that holds production schemas (or a read-only mirror), not a developer’s local one, otherwise the check compares against nothing. Second, make the compatibility mode itself part of the contract and set it from code, for instance with a PUT /config/{subject} call in the deployment pipeline, so nobody can quietly loosen it in a console.

Point two: the contract file and the Data Contract CLI

Registry compatibility knows nothing about quality rules or SLAs. A contract file in version control fills that gap. The Data Contract CLI documentation describes a lint command that validates a contract against the ODCS JSON Schema, a test command that connects to a data source and checks that the actual data matches the contract’s schema and quality expectations, and import and export commands for converting to and from formats such as Avro, JSON Schema, Protobuf, SQL DDL and dbt. Its supported sources include Kafka, Snowflake, Databricks, BigQuery, Postgres, S3 and local Parquet and Delta files, among others.

An ODCS contract for our order stream looks roughly like this. Field names follow the structure shown in the ODCS documentation; check the current specification before copying, since the standard is still evolving and v3.1.0 tightened several definitions.

apiVersion: v3.1.0
kind: DataContract
id: urn:datacontract:orders:order-placed
name: order-placed
version: 2.1.0
status: active
description:
  purpose: One event per accepted customer order, published at checkout.
team:
  name: checkout-platform
  members:
    - username: owner@example.com
      role: owner
servers:
  - server: production
    type: kafka
    host: broker.example.com:9092
    topic: orders.placed.v2
schema:
  - name: order_placed
    properties:
      - name: order_id
        logicalType: string
        required: true
        primaryKey: true
      - name: customer_id
        logicalType: string
        required: true
      - name: amount_cents
        logicalType: integer
        required: true
        description: Order total in minor currency units, tax included.
        quality:
          - type: sql
            description: Totals are never negative
            query: SELECT COUNT(*) FROM order_placed WHERE amount_cents < 0
            mustBe: 0
slaProperties:
  - property: freshness
    value: 5
    unit: m

The value of this file is not the YAML. It is that amount_cents now has a description that states the unit, a rule that rejects negative totals, a named owner and a freshness target, all reviewable in a pull request. The CLI’s test command can run the quality query against the live source on a schedule, which turns the document into a continuously checked agreement.

A caution on verification: the CLI documentation describes lint, test, import and export explicitly. Commands for breaking-change detection and CI integration exist in the project, but I have not verified their exact behaviour here, so confirm them against the current command reference before building a gate around them.

Point three: the registry and the serializer

At runtime the serializer embeds a schema ID in each message, and with auto-registration disabled, a producer whose schema is not already registered fails fast instead of creating a new version. Turning off auto-registration in production is one of the cheapest hardening steps available: only the CI pipeline, with review, registers schemas. Developers get fast feedback from the compatibility check, and production producers can never invent a version.

Confluent’s data contract rules go beyond structure. A rule has a kind, either CONDITION for validation or TRANSFORM for modifying data, plus a mode, a type such as CEL, and an action on failure. Documented failure actions are ERROR (the default), DLQ and NONE, though the documentation notes that the non-Java clients did not yet support the DLQ action when it was written. Rules also run only on the root schema, not on referenced schemas, and the documentation lists Kafka Connect on Confluent Cloud, Flink SQL and ksqlDB as not executing them. These rules require Confluent Enterprise or Confluent Cloud with the Advanced Stream Governance package. A condition rule such as a CEL check on a field’s length therefore makes sense when your producers are Java services on a supported platform, and less sense for a heterogeneous estate.

Point four: the broker

Broker-side schema ID validation is often misunderstood. According to the Confluent Platform documentation, it checks that the schema ID in a record’s wire format is registered under a valid subject, with the subject derived from the topic name by default. It does not inspect the payload, so it cannot verify that the bytes actually match the schema, and null values and keys, including tombstones, pass unchecked. It is enabled per topic with confluent.value.schema.validation and confluent.key.schema.validation, both false by default, and it requires Confluent Server on Confluent Platform 5.4.0 or later with an enterprise license. Open-source Apache Kafka has no equivalent built in.

What it buys you is protection against the worst class of bug: a producer writing raw JSON or an unregistered Avro blob into a governed topic. That is a real failure mode when one team’s script writes “just this once” to production. It buys nothing against a registered but semantically wrong message, so it complements the other controls rather than replacing them.

Point five: the stream processor and the dead-letter topic

The last gate is the consumer that moves data into the lakehouse. A well-built processor deserializes, applies the contract’s quality rules, and routes failures to a dead-letter topic with enough context to replay them: the original bytes, the schema ID, the topic and offset, the rule that failed, and a timestamp. Fail-stop is sometimes right, for instance when a poison message signals a systemic producer bug, but for isolated bad records a dead-letter queue keeps the pipeline flowing while preserving evidence. If your processing relies on transactional guarantees, remember that routing to a DLQ must itself be part of the same atomic unit; our article on Kafka exactly-once semantics and transactions explains how to keep reject-and-commit consistent.

Lakehouse Schema Evolution: Iceberg Field IDs and the Table Layer

Table formats treat evolution as a metadata operation rather than a rewrite. In Apache Iceberg, every column has a unique integer field ID stored in table metadata, and data files reference columns by that ID, not by name or position. A rename, reorder or drop therefore changes metadata only, and old files stay readable without being rewritten.

Figure 3: A breaking change handled as a migration. The producer dual-publishes while consumers move, then retires the old topic.

Why field IDs make evolution safe

Iceberg’s documentation describes schema evolution as supporting add, drop, rename, reorder and type promotion operations, and states that these changes are independent and free of side effects, without rewriting table data. The correctness claim in the docs is specific: because each column is tracked by a unique ID, a dropped column’s ID is never reused for a new column, so old files cannot be misread as containing the new column’s data. Formats that track columns by name or position can resurrect old values when someone drops a column and later adds a different one with the same name. That is a silent corruption bug, and field IDs eliminate it by construction. For exact type-promotion rules, read the Iceberg table specification; as a general matter the supported promotions are widening ones such as int to long, float to double and increasing decimal precision, and narrowing is not allowed.

-- Spark SQL against an Iceberg table: all metadata-only operations
ALTER TABLE lake.orders.order_placed ADD COLUMN currency string;
ALTER TABLE lake.orders.order_placed RENAME COLUMN cust_ref TO customer_id;
ALTER TABLE lake.orders.order_placed ALTER COLUMN amount_cents TYPE bigint;  -- int to bigint widening
ALTER TABLE lake.orders.order_placed DROP COLUMN legacy_channel;

Notice what the table layer does not tell you. A rename in the table is safe for the table, but every downstream view, dashboard query and feature definition that references the old name still breaks. Field IDs protect storage integrity, not consumer expectations. The table format guarantees that the data you wrote is still readable, and the contract guarantees that the data means what consumers think it means. Those are different promises.

Mapping a stream schema to a table schema

The path from a Kafka topic to an Iceberg table is where drift often enters. When the stream processor writes to the table, it needs a policy for what happens when the stream schema changes. There are three common policies. Strict mode fails the job on any difference between the incoming schema and the table, which is safe and noisy. Additive mode automatically adds new nullable columns to the table when the stream gains an optional field, and fails on anything else. Manual mode routes records with unknown fields to the dead-letter topic until a human updates the table.

For contract-governed topics, additive mode is usually the right balance, provided the registry already guarantees the stream only changes in compatible ways. A FULL_TRANSITIVE topic can add or remove optional fields, which maps cleanly onto adding nullable table columns and leaving removed fields as nulls for new rows. The table then becomes a faithful superset history of the stream. Avoid automatic type promotion at the table layer unless the contract explicitly allows it, since a widening that is harmless in Avro can be a breaking change for a column that downstream SQL casts.

Our comparison of Iceberg and Paimon discusses how streaming-first table formats handle schema changes differently, which matters if your lakehouse is fed by Flink rather than a micro-batch job.

Handling Breaking Changes: Dual Publish and Migration Rules

Some changes cannot be made compatible. A field changes meaning, a key is redefined, a type is narrowed, or two events are merged into one. For these, the honest answer is that the change is a new version of the contract, and the migration is a project with a start date, an end date and an owner. Figure 3 above shows the shape of the workflow.

The dual-publish pattern

The producer creates a new topic, for example orders.placed.v2, with the new schema, and keeps publishing to the old topic as well. Both topics receive every event for a defined migration window. Consumers move one at a time, each on its own schedule, and each one confirms through a tracked ticket or a registry-side consumer inventory that it has moved. When the last consumer has migrated, the producer stops writing to v1 and, after the retention period lapses, deletes it.

Dual publishing has real costs. The producer must derive both shapes from the same internal event, and a bug in the down-conversion can make the two topics disagree. Write the conversion once, test it with property-based tests that round-trip representative samples, and compare per-key counts between the two topics in a reconciliation job during the window. Publishing to two topics also breaks atomicity unless you use transactions or an outbox, so a crash between the two writes can leave v2 ahead of v1. The outbox pattern, writing both messages from one database transaction, avoids that.

Set the window in writing. A migration with no deadline never finishes, and the old topic becomes a permanent tax. Pick a period based on the slowest consumer’s release cycle, add a safety margin, and have the producer team publish the retirement date in the contract’s changelog. The contract’s status field is a natural place to mark the old version as deprecated.

Migration rules as an alternative

Confluent’s data contract feature offers a different approach for schemas that evolve through major versions. Migration rules, in modes UPGRADE, DOWNGRADE and UPDOWN, transform messages between schema major versions so that a producer on one version and a consumer on another can still interoperate. The documentation gives the example of a JSONata UPGRADE rule that renames a field and a matching DOWNGRADE rule that reverses it, and notes that rules chain across multiple versions. This lets a rename be a managed change without a second topic. It runs client-side, so it depends on supported client libraries and on the enterprise or Stream Governance packaging noted earlier. If you can use it, it shortens migrations considerably. If you cannot, dual publish is the portable answer.

Expand and contract for non-breaking-looking changes

For changes that are technically compatible but behaviorally risky, such as a new enum value or a new default, a two-phase rollout works well. In the expand phase the schema gains the new element and consumers are updated to handle it, while producers still emit the old behavior. In the contract phase, once monitoring confirms every consumer is ready, producers begin emitting the new behavior. The registry sees both phases as compatible. The ordering is enforced by humans and by consumer-side tests, which is why a contract needs an inventory of who consumes what.

Where Contracts Fail: Trade-offs, Gotchas, and What Goes Wrong

A data contract is a set of detectable promises, and the interesting failures are the ones outside the detectable set. The most common problems are semantic drift that no schema can express, compatibility modes set to match convenience rather than the rollout reality, and contracts that exist as documents but are not wired into any gate.

Figure 4: Rejected records go to a dead-letter topic for triage and replay, and a drift monitor watches values that pass every structural check.

Semantic drift is invisible to schemas

A field named status keeps its type of string while the team starts using a new value, or event_time silently changes from client time to server time, or amount moves from dollars to cents. The schema is unchanged and the registry approves. Detecting this requires value-level monitoring: distribution checks on categorical columns, range and null-rate checks on numerics, freshness and volume checks per partition, and a comparison to a baseline. Quality rules in the contract cover the cases you anticipated. Drift monitoring covers the ones you did not. Budget for both, and send alerts to the contract owner rather than to the consumers who find the problem first.

Compatibility mode misuse

Several anti-patterns recur. Setting NONE during a migration and forgetting to restore it. Using non-transitive BACKWARD on a long-retention topic so a consumer rewound to the start fails on a version from two years ago. Choosing FORWARD because it sounds safer without realizing it forces producers to upgrade first and constrains what consumers can rely on. Deleting a subject’s old versions from the registry to make a failing check pass, which removes the history that the transitive check needed. And treating a green registry check as proof of safety, when it only proves structural compatibility.

Costs of being strict

Strict contracts slow producers. A team that must go through review, a compatibility gate and a notification step for every field addition will, under pressure, route around the system, for example by stuffing new data into an untyped metadata map field. That field then becomes an ungoverned side channel carrying exactly the information that mattered. The remedy is to make the compliant path fast: a schema change with a passing check and no breaking impact should merge without human approval from consumers, and only breaking changes should require negotiation.

Over-specification is the opposite failure. A contract that pins every field as required and every enum as closed makes legitimate evolution a breaking change. Prefer optional fields with defaults, include an explicit unknown value in enums, and avoid putting consumer-specific derived fields in the producer’s contract.

Registry and platform dependencies

The registry becomes critical infrastructure. If CI depends on it, an outage blocks deployments. If serializers cannot reach it and the schema is not cached, restarting consumers cannot decode. Plan for high availability, caching and a documented break-glass procedure. Be careful with vendor-specific features as well: Confluent’s rule sets, broker-side validation and some governance functions depend on particular editions and client languages, so a contract design that relies on them is portable only as far as your platform is.

Contracts do not remove the need for people

Tooling can reject an incompatible schema. It cannot decide that a column should be deprecated, or negotiate a cutover date with a team that is mid-reorg. Many contract programs stall not on technology but on ownership: nobody is accountable for a dataset because it was created by a pipeline that nobody owns. The next section addresses that directly.

Practical Recommendations

Start where the leverage is highest and the effort is lowest, then widen. The sequence that works across teams is: register schemas for your most-consumed topics, gate changes in CI, lock down production registration, add quality rules to the same contract, and only then invest in drift monitoring and migration tooling.

On the organizational side, assign a named owner per contract and make the producing team accountable for it, because the producer is the only party able to prevent a break. Keep a registry of consumers, even a lightweight one, since you cannot run a safe deprecation without knowing who depends on what. Agree a lightweight change classification: additive and compatible changes need no negotiation, behavior-affecting changes need notice, breaking changes need a migration plan with a date. Many teams review contracts quarterly, the same way they review access, and retire contracts whose consumers have gone.

A checklist you can adopt this week:

Every production topic has a registered schema and an explicit compatibility mode, with FULL_TRANSITIVE or BACKWARD_TRANSITIVE for replayable topics.
Auto-registration is disabled for production producers; only CI registers schemas.
CI runs a compatibility check against the production registry on every schema change.
Each contract names an owner, a freshness target and at least one executable quality rule.
Protobuf tag numbers and names are reserved when fields are deleted; Avro additions have defaults.
Stream processors route invalid records to a dead-letter topic with replay metadata.
Lakehouse tables use format-level field IDs, and downstream views are tested against renames.
A drift monitor watches value distributions on high-impact fields.
Breaking changes follow a written dual-publish or migration-rule plan with an end date.
The consumer inventory is reviewed before any topic is retired.

Frequently Asked Questions

What is a data contract?

A data contract is a versioned, enforceable agreement between a data producer and its consumers that specifies the structure, meaning, quality expectations, service levels and ownership of a dataset or stream. The schema is only one part. Contracts are checked automatically, for example in CI and at the schema registry, so that breaking changes are caught before they reach consumers.

What is the difference between a data contract and a schema registry?

A schema registry stores versioned schemas and enforces structural compatibility rules when new versions are registered. A data contract is broader: it adds semantics, quality rules, service levels and ownership, and is often expressed in a format such as the Open Data Contract Standard. The registry is one enforcement mechanism for the schema part of a contract, not the whole thing.

Which compatibility mode should I use for Kafka topics?

BACKWARD is the Confluent default and fits topics where consumers upgrade before producers. For topics that consumers can replay from the beginning, or that retain many versions, use a transitive mode such as BACKWARD_TRANSITIVE or FULL_TRANSITIVE so every retained version stays readable. FULL lets producers and consumers upgrade independently. Avoid NONE in production because it disables the check entirely.

Is Avro or Protobuf better for schema evolution?

Neither wins outright. Avro resolves reader and writer schemas, so it requires defaults for safe additions and removals and is strict about checkable rules, which suits registry-driven governance. Protobuf identifies fields by number, so renames are safe on the wire, but reusing numbers or changing types silently corrupts data. Both work well when you follow their evolution rules and gate changes in CI.

How do Iceberg field IDs help with schema evolution?

Iceberg assigns each column a unique integer ID stored in table metadata, and data files reference columns by ID rather than name or position. Adding, dropping, renaming and reordering columns are therefore metadata-only operations, and a dropped column’s ID is never reused, which prevents old data from being misread as a new column. Downstream queries that reference old names still need updating.

How should I handle a breaking schema change?

Treat it as a migration rather than an edit. Publish the new version on a new topic while continuing to publish the old one, track each consumer as it moves, reconcile the two topics during the window, then retire the old topic on a published date. Where your platform supports it, registry migration rules can transform between major versions and reduce the need for a second topic.