Data Contracts and Schema Evolution

A data contract is an agreement between the producer of data (a service, an app, a source database) and its consumers about the data's schema, meaning and quality: which fields exist, their types and units, which are required, and how changes will be made. Schema evolution is how that schema changes over time without breaking consumers. Together they move data problems from 3 a.m. pipeline failures to a review comment on the producer's pull request.

Why pipelines break

Most data pipelines consume data their owners don't control. A backend team renames amount to amount_paise, changes a status enum, or starts sending nulls, and a pipeline three teams away silently produces wrong numbers. The producer didn't know anyone depended on the field; the consumer didn't know it was changing. Contracts make the dependency explicit.

What a contract contains

• Schema: field names, types, nullability and nested structure, in a machine-readable form (Avro, Protobuf or JSON Schema).
• Semantics: what each field means, its units and currency, valid values for enums, and the event's trigger ("emitted once when payment is captured").
• Keys and ordering: the unique ID for deduplication and the partition key that defines ordering.
• Quality expectations: freshness, completeness and volume ranges the consumer can rely on.
• Ownership and versioning: who owns it, how to request changes, and how breaking changes are announced.

# order_completed.v2 (illustrative)
owner: checkout-team
key: order_id            # unique, used for deduplication
partition_key: customer_id
fields:
  order_id:     { type: string, required: true }
  customer_id:  { type: string, required: true }
  amount_minor: { type: long, required: true, unit: "minor currency units" }
  currency:     { type: string, required: true, format: ISO-4217 }
  completed_at: { type: timestamp, required: true, meaning: "event time at checkout service" }
  coupon_code:  { type: string, required: false }
compatibility: backward
freshness: "available in the stream within 60 s, p99"

Compatible and breaking changes

Compatibility modes describe the direction: backward compatible means new readers can read old data; forward compatible means old readers can read new data; full means both. Event streams with long retention usually need at least backward compatibility, because consumers replay old messages with new code.

Enforcing contracts

• Schema registry: producers register schemas with a registry that rejects incompatible versions, and messages carry a schema ID. This is common with Kafka and Avro or Protobuf.
• CI checks: a producer's pull request fails if it changes the schema incompatibly or removes a field with known consumers.
• Runtime validation: pipelines validate incoming data against the contract and quarantine violations, as in data quality.
• Versioned events: for unavoidable breaking changes, publish order_completed.v3 alongside v2, migrate consumers, then retire v2 on an announced date.

Database sources and CDC

When data comes from change data capture on a service's database, the database schema becomes an accidental public contract: any column rename breaks the pipeline. Two common fixes are the outbox pattern, where the service publishes deliberate, versioned events instead of exposing its tables, and a thin, owned interface view over the tables that the producer promises to keep stable.

Start contracts where breakage is most expensive: the handful of events and tables behind revenue, billing and core product metrics. Contracting every table at once usually stalls.