Event Schema Evolution with Avro and a Schema Registry in Kafka

How Avro writer and reader schemas make event schema evolution possible

Event schema evolution is the practice of changing the structure of an event over time without breaking the services that produce or consume it. This section is for teams whose services publish and consume shared events and deploy independently. In that setup, some consumers will always run older or newer code than the producer.

Avro handles this with two schemas. The writer schema is the one the producer used to encode the record. The reader schema is the one the consumer's code expects. When they differ, Avro's schema resolution reconciles them:

• Fields are matched by name, not by position.
• A field in the reader schema that is missing from the data takes the reader's default value. If it has no default, resolution fails.
• A field in the data that the reader schema doesn't declare is skipped.

Consider an OrderPlaced record:

A v2 consumer reading v1 data gets channel = "web" and couponCode = null. A v1 consumer reading v2 data skips both new fields. For a union, the default must match the union's first branch, which is why null comes first.

Avro binary data contains no field names or tags. Values are written in schema order, so the bytes can't be decoded without the exact writer schema. The binary format alone gives you no compatibility. The resolution step does: it maps the writer's layout onto the reader's expectations. That is what lets producers and consumers upgrade at different times.

Copy# pip install fastavro
import io
from fastavro import parse_schema, schemaless_writer, schemaless_reader

V1 = parse_schema({"type": "record", "name": "OrderPlaced", "namespace": "com.example.orders",
    "fields": [
        {"name": "orderId", "type": "string"},
        {"name": "amountCents", "type": "long"},
    ]})

V2 = parse_schema({"type": "record", "name": "OrderPlaced", "namespace": "com.example.orders",
    "fields": [
        {"name": "orderId", "type": "string"},
        {"name": "amountCents", "type": "long"},
        # New field with a default, so v2 readers can still decode v1 data
        {"name": "channel", "type": "string", "default": "web"},
        # Optional field: "null" is listed first because the default (null) must match the first branch
        {"name": "couponCode", "type": ["null", "string"], "default": None},
    ]})

def encode(schema, record):
    # Raw Avro binary with no field names, only values in writer-schema order
    buf = io.BytesIO()
    schemaless_writer(buf, schema, record)
    return buf.getvalue()

def decode(data, writer_schema, reader_schema):
    # Resolution step: map the writer's layout onto the reader's expected fields
    return schemaless_reader(io.BytesIO(data), writer_schema, reader_schema)

# Old producer, new consumer: missing fields are filled from the reader's defaults
v1_bytes = encode(V1, {"orderId": "ord-1001", "amountCents": 4599})
print(decode(v1_bytes, V1, V2))
# {'orderId': 'ord-1001', 'amountCents': 4599, 'channel': 'web', 'couponCode': None}

# New producer, old consumer: fields the reader doesn't declare are skipped
v2_bytes = encode(V2, {"orderId": "ord-1002", "amountCents": 1250,
                       "channel": "mobile", "couponCode": "SPRING10"})
print(decode(v2_bytes, V2, V1))
# {'orderId': 'ord-1002', 'amountCents': 1250}

This example uses fastavro to encode OrderPlaced records with the v1 and v2 schemas and decode each with the other version, showing that the reader's defaults fill missing fields and that fields the reader does not declare are skipped.

What the Avro schema registry does at produce and consume time

The registry is what makes resolution practical across services that never share a deployment. It stores schemas under subjects, keeps every version, and gives each distinct schema a numeric ID. When a new version is registered, the registry checks it against earlier versions under that subject's compatibility mode and rejects it if the check fails.

The message flow

On produce, the serializer looks up the record's schema in the registry, or registers it if auto-registration is enabled. It receives an ID and writes a short header with that ID ahead of the Avro payload. The full schema never travels with the message.

On consume, the deserializer reads the ID, fetches the matching writer schema, and resolves it against the reader schema compiled into the consumer.

Both clients cache schemas by ID, so steady-state traffic rarely calls the registry. The cache does not remove the dependency, though:

• A new schema ID still requires a registry call.
• So does a fresh consumer instance with an empty cache.
• So does any deploy that registers a new version.

If the registry is down during a rollout, producers with new schemas cannot publish, and new consumers cannot decode messages.

Subject naming

The subject naming strategy decides what the compatibility check actually compares.

• One schema per topic. The subject is derived from the topic, so every version of that topic's event is checked against the previous one.
• Subject per record type. Several event types can share a topic, which keeps related events in a single partition order, for example OrderCreated before OrderCancelled.

The ordering benefit of a shared topic has costs:

• The registry checks each type only against its own history. Nothing stops an unexpected type from appearing on the topic unless you enforce it separately.
• Consumers have to dispatch on the type and handle types they don't recognize.

Copy PRODUCER                  SCHEMA REGISTRY                  CONSUMER
+----------------+      +----------------------+      +------------------+
| app record     |      | subject: orders-value|      | reader schema    |
|      |         |      |  v1 -> id 7          |      | (compiled in)    |
|      v         |      |  v2 -> id 12         |      |        ^         |
| serializer     |--1-->|  compat check on     |<--4--| deserializer     |
| cache by ID    |<--2--|  each registration   |--5-->| cache by ID      |
+----------------+      +----------------------+      +------------------+
        |                                                      ^
        | 3: [ID header][Avro payload]                         |
        v                                                      |
+------------------------------------------------------------------------+
|                          KAFKA TOPIC: orders                           |
|  partition 0: OrderCreated [id 12] --> OrderCancelled [id 15] --> ...  |
+------------------------------------------------------------------------+

 1  look up schema, or register it if auto-registration is enabled
 2  schema ID returned; registration rejected if compat check fails
 3  message carries the ID header and Avro payload, not the full schema
 4  fetch writer schema by ID (cache miss only)
 5  writer schema returned; resolved against the reader schema

 Registry still required for: new schema IDs, cold consumer caches,
 and any deploy that registers a new version.

The producer serializer registers or looks up a schema and writes its ID before the Avro payload; the consumer fetches the writer schema by ID on a cache miss and resolves it against its reader schema.

Choosing Kafka schema compatibility modes by deployment order

Each mode answers one question: which side of the topic is allowed to deploy first.

BACKWARD means the new schema can read data written with the previous one, so consumers upgrade before producers. FORWARD means old readers can decode data from the new schema, so producers ship first. FULL permits either order. That flexibility matters when the producing and consuming teams deploy on independent schedules.

The non-transitive modes compare only against the latest version. Version 3 can be compatible with version 2 while still being incompatible with version 1. That gap is harmless if a topic keeps a few hours of data. It causes failures when a topic has long retention, is compacted or gets replayed. A consumer that resets to the earliest offset decodes every historical writer schema, not only the previous one. For those topics, use BACKWARD_TRANSITIVE or FULL_TRANSITIVE.

Strictness has a cost. FULL_TRANSITIVE turns most changes into "add an optional field with a default." Teams then start creating OrderV2 topics or packing data into string blobs to get around the check, which is worse than a looser mode. A practical default is BACKWARD_TRANSITIVE for retained or shared topics. Reserve FULL variants for topics whose producers and consumers deploy independently.

NONE is acceptable for short-lived or internal topics with a single owning team that deploys producer and consumer together. Avoid it on any topic another team reads, or any topic that might be replayed.

Which changes produce backward compatible events

…whose consumers you don't control.

Avro matches writer and reader fields by name, so every rule follows from one question: when a field is missing on one side, can the reader still fill it in?

You can declare an alias on the reader so it accepts the old field name. This only helps readers that carry the alias. Older readers still don't know the new name. Registry compatibility checks may not treat aliases the way the Avro runtime does, so test alias-based renames against the registry before relying on them.

Enums need the same care. A new reader handles old data fine after you add a symbol. An old reader that receives the unknown symbol fails, unless its reader schema declares an enum default. That default must already be deployed to consumers before any producer writes the new symbol.

Optional fields are unions with null listed first and a null default. Making every field optional makes nearly any change pass compatibility checks. The cost is a weaker contract: the schema no longer guarantees anything is present, so every consumer has to repeat null checks and validation the schema used to enforce.

Copy{
  "type": "record",
  "name": "OrderPlaced",
  "namespace": "com.example.orders",
  "doc": "Version 2 of the schema. Each field's doc says why the change from version 1 is or is not safe.",
  "fields": [
    {
      "name": "order_id",
      "type": "string",
      "doc": "Unchanged required field. Required fields that already exist are fine. Only new required fields break readers."
    },
    {
      "name": "amount_cents",
      "type": "long",
      "doc": "Was int in version 1. New readers promote old int data to long. Old int readers cannot read long data."
    },
    {
      "name": "currency",
      "type": "string",
      "default": "USD",
      "doc": "New field with a default. A reader of version 1 events fills in USD because the writer never wrote the field."
    },
    {
      "name": "customer_ref",
      "type": "string",
      "aliases": ["customer_id"],
      "doc": "Renamed from customer_id. The alias helps only readers that carry it. Old readers do not know the new name. Test this rename against the registry check before deploying."
    },
    {
      "name": "status",
      "type": {
        "type": "enum",
        "name": "OrderStatus",
        "symbols": ["PENDING", "PAID", "CANCELLED", "UNKNOWN"],
        "default": "UNKNOWN"
      },
      "doc": "The enum default maps symbols this reader does not know to UNKNOWN. Deploy this reader schema to consumers before any producer writes a new symbol."
    },
    {
      "name": "coupon_code",
      "type": ["null", "string"],
      "default": null,
      "doc": "Optional field: a union with null listed first and a null default. Consumers must handle the null case themselves."
    }
  ]
}

An Avro record schema for version 2 that shows the safe evolution patterns: a field added with a default, an int-to-long promotion, an alias-based rename, an enum with a default symbol, and an optional union-with-null field.

Enforcing compatibility in CI before schemas reach the registry

Whichever mode you choose, it only protects you if every schema is checked before a producer can use it. Move the check out of production and into the pull request.

Keep schemas in version control and test every change

Store .avsc files in the repository alongside the code that produces them, with a fixed mapping from file to subject name. On each pull request, CI finds the changed schemas and tests each one against the registry. The common REST endpoint for this is POST /compatibility/subjects/{subject}/versions/latest. It returns whether the candidate is compatible under the subject's configured mode. Fail the build if any schema comes back incompatible. If CI cannot reach the production registry, run the same check against a registry seeded with the stored schema history. Under transitive modes, the check must cover every prior version, not just the latest.

Register from a controlled step after merge

After the merge, a single pipeline job registers the new versions with POST /subjects/{subject}/versions. That job owns the write credentials. Application producers should have read access only.

Producer serializers usually default to auto.register.schemas=true. In development this saves time, because you can iterate on a schema without any ceremony. In production, the same default lets any deployed build register whatever schema it was compiled with. That schema then becomes the subject's latest version, and nobody reviewed it. Set auto.register.schemas=false in production. Optionally, set use.latest.version=true so the serializer looks up the registered schema rather than registering its own. Defaults and how these flags interact differ between registry clients, so confirm the details against your registry's documentation.

Copy# producer-prod.properties
# Production producer settings for an Avro serializer backed by a schema registry.
# Goal: the application may read schemas from the registry but never write them.

bootstrap.servers=kafka-1.prod.internal:9092,kafka-2.prod.internal:9092
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=io.confluent.kafka.serializers.KafkaAvroSerializer

schema.registry.url=

# Use a read-only registry account. Only the post-merge pipeline job holds
# credentials that can call POST /subjects/{subject}/versions.
basic.auth.credentials.source=USER_INFO
basic.auth.user.info=orders-producer-ro:REPLACE_WITH_SECRET_FROM_VAULT

# Do not let a deployed build register the schema it was compiled with.
# With auto-registration on, an unreviewed schema could become the subject's
# latest version as soon as the first record is produced.
auto.register.schemas=false

# Serialize with the latest version already registered for the subject
# (the one CI checked and the pipeline registered) instead of registering
# the producer's own copy. Confirm how this flag behaves with your client
# version, and whether extra settings are needed to make the serializer
# check the local schema against that version, in your registry's docs.
use.latest.version=true

# Development override (producer-dev.properties), for comparison:
#   schema.registry.url=
#   auto.register.schemas=true
#   (fast iteration against a throwaway local registry; nothing to review)

Production producer configuration that turns off automatic schema registration, serializes against the latest registered version, and uses read-only registry credentials, so new schemas can enter the registry only through the reviewed post-merge pipeline step.

Handling breaking changes deliberately and what the registry cannot catch

Sometimes a change cannot be made compatible, and the registry will rightly reject it. There are three ways to handle that.

The cheapest is expand and contract. Add the new field next to the old one and populate both. Move consumers to the new field, then remove the old one in a later release once nothing reads it. Dual-writing to two topics or two schemas works on the same principle. It gives consumers a window to switch, but the producer has to keep both outputs consistent for that whole period.

The heaviest option is a new versioned topic, such as orders.v2. It isolates the break completely, but the operational cost is real:

• You have to coordinate cutover across every consumer team.
• Consumer group offsets do not carry over between topics. Each consumer must choose a starting point on the new topic and tolerate gaps or duplicates around the switch.
• The old topic can only be retired after you confirm it has no remaining readers.

The registry also has a limit you need to plan around. It checks structure, not meaning. Suppose amount changes from cents to dollars, or a timestamp moves from UTC to local time. The schema is identical, so the change passes, and consumers still break.

Defaults can hide breakage in a similar way. When a consumer reads an older event that lacks a newer field, it gets the default. A default like 0 or "USD" looks plausible, so the consumer processes a wrong value without any error.

To reduce these risks:

• Put units and semantics in field names, such as amount_cents, and in schema docs.
• Run consumer-driven contract tests against real sample events.
• Where a default would mislead, make the field nullable and reject or alert on null instead of guessing.

Alternatives, and the impact on cost and engineering time

Avro with a registry is one of several options. They differ in how much the tooling enforces and what must be available at runtime.

The costs come early and keep running. You have to set up the registry, run it as a highly available service, add compatibility checks to CI, and accept that strict modes make some changes take more steps. Clients cache schemas, so a short registry outage mostly affects new schema IDs. Plan for it anyway.

The savings come later and are larger. Incompatible changes fail in a pull request instead of showing up as consumer errors in production. Most changes ship without coordinated deploys. Replaying old events is safer because every event points to the schema it was written with.

Illustrative estimate (assumed numbers, not a benchmark): savings = incidents avoided × hours per incident − upkeep hours. Assume 4 incidents avoided per year at 12 hours each, against 30 hours of upkeep: 4 × 12 − 30 = 18 hours saved per year.

Adopt this approach when independently deployed teams share events or when you rely on replays. Skip it for a few tightly coupled services that always deploy together, or for an early prototype whose schemas change every day.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to event schema evolution, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.