Event Schema Evolution: Versioning Microservice Messages Safely

Why event schemas are harder to change than APIs

Event schema evolution is the practice of changing the structure of messages on a topic without breaking the services that produce or consume them. It looks like ordinary API versioning, but one property of events makes it harder: the message outlives the code that wrote it.

A synchronous API call is short-lived. The request and response exist for milliseconds, and both sides are running at the same moment. If you control the client and the server, you can coordinate a deploy, add a version to the path, or keep an old endpoint alive until traffic drops to zero.

An event is persisted. It may sit on a topic for days under a retention policy, indefinitely in a compacted topic, or permanently in an event store. A consumer deployed next quarter may replay it from offset zero. Another team may subscribe to the topic without telling you. The producer cannot do a lockstep deploy with every reader, because some readers do not exist yet and others are not under its control.

That turns schema change into a question of ordering. Either consumers upgrade first and must tolerate data they have not seen, or producers upgrade first and must keep writing data that old readers can still parse. Choosing a direction is half the work. The other half is enforcing it mechanically so a single careless deploy cannot violate it.

You need this discipline when a topic has more than one consumer, long retention, or any replay requirement. If one service writes a short-lived queue that one other service reads, coordinated deploys may be enough.

Compatibility modes: deciding who upgrades first

A compatibility mode is a rule the registry enforces on every new schema version. Each mode fixes which side of the topic can safely deploy first.

Backward compatibility suits topics where consumers are few and under your control. Upgrade them, then let producers emit the new shape. Forward compatibility suits the reverse case: one producer and many consumers you cannot coordinate. The producer ships first, and old readers skip what they don't recognise. Full compatibility removes the ordering problem, which matters when several teams deploy on independent schedules.

The non-transitive modes compare only adjacent versions. That is not the same as comparing against every version. A field can be made optional in v2 and then removed in v3, and each step passes the check. A v3 consumer still fails on v1 records. If a topic retains history, is compacted, or can be replayed from the beginning, consumers will read old records. Use a transitive mode for those topics.

Full transitive compatibility has a cost. Suppose a field was mistyped or a unit was wrong. The fix you need is usually a breaking change, and the registry will reject it permanently. The usual way out is a new event type, together with migration work on both sides.

How Avro schema resolution works, and how Protobuf differs

Avro never decodes a message with a single schema. The consumer holds the reader schema it was compiled against, and it fetches the writer schema the producer used, usually by an ID embedded in the message. Avro binary contains no field names or tags, so the bytes cannot be parsed at all without the writer schema. Resolution then matches fields by name:

• A field in the writer but not the reader is skipped.
• A field in the reader but not the writer takes the reader's default.
• A reader field with no writer match and no default causes a resolution error.

Take three versions of OrderPlaced:

• v1: orderId, amount.
• v2: adds currency with default "USD". A v2 consumer reading v1 events fills in "USD". A v1 consumer reading v2 events ignores currency. Both directions work.
• v3: adds region with no default, and renames amount to totalAmount. A v3 reader cannot resolve old events because there is no value for region and no field called totalAmount. A declared alias can bridge the rename, but without one the old data is unreadable.

Protobuf identifies fields by number. Names never go on the wire, so a rename is harmless. Changing a field's number, or reusing a number for a different meaning, silently corrupts data. When you delete a field, mark its number and name reserved so that nobody can reuse them.

Both formats have costs. You need code generation, registry lookups, and tooling just to inspect a message. JSON avoids all of that, but its compatibility rules exist only as team convention, and conventions drift.

Copy// Avro v1: writer schema for events already on the topic
{"type": "record", "name": "OrderPlaced", "fields": [
  {"name": "orderId", "type": "string"},
  {"name": "amount",  "type": "long"}
]}

// Avro v3, broken: a v3 reader cannot resolve v1 data
//   region      -> not in writer, no default  => resolution error
//   totalAmount -> no writer field by that name => resolution error
{"name": "region",      "type": "string"},
{"name": "totalAmount", "type": "long"}

// Avro v3, compatible: every new reader field has a way to get a value
{"type": "record", "name": "OrderPlaced", "fields": [
  {"name": "orderId",     "type": "string"},
  {"name": "totalAmount", "type": "long", "aliases": ["amount"]},
  {"name": "currency",    "type": "string", "default": "USD"},
  {"name": "region",      "type": "string", "default": "unknown"}
]}

// Protobuf: identity is the field number. Names never reach the wire.
syntax = "proto3";
message OrderPlaced {
  // Field 4 (region) was deleted. Reserving the number and the name
  // stops anyone from reusing them with a different meaning.
  reserved 4;
  reserved "region";

  string order_id     = 1;
  int64  total_amount = 2;  // renamed from amount: safe, number unchanged
  string currency     = 3;  // never renumber: old bytes would decode as the wrong field
}

Annotated Avro schema versions show why a v3 reader fails on v1 events and how aliases and defaults fix it, alongside a Protobuf message where field numbers carry identity and a deleted field is reserved.

Safe versus breaking changes: a practical catalog

Whichever format you pick, most proposed changes fall into one of three groups. A registry can check the first two. Nothing checks the third.

Defaults hide missing data

Defaults are what make the first two rows safe, and they also cost you information. Suppose you add discount_cents with a default of 0. When a consumer reads an old event, it gets 0 and cannot tell whether no discount applied or the field did not exist yet. Aggregations over historical data will then quietly understate discounts. If absence matters to downstream logic, make the field a nullable union with a null default. Consumers then have to handle "unknown" explicitly.

Semantic breaks

Changing amount from dollars to cents, or letting user_id refer to accounts instead of people, keeps every type intact. No schema tool can catch this. Treat a change in meaning as a breaking change: introduce a new field name instead of reinterpreting the existing one.

Implementing a schema registry and CI compatibility checks

A schema registry stores every version of a schema under a subject, usually one per topic or event type, and assigns each version an ID. Producers embed that ID in each message, so consumers can fetch the exact writer schema needed for resolution. When a producer registers a new version, the registry checks it against the subject's configured compatibility mode and rejects it if the check fails. That rejection is the registry's main value.

Copy  PR ──► CI: compatibility check ──► merge
                   │
                   ▼
 Producer ──register/lookup──► Registry ◄──fetch by ID── Consumer
    │                                                      ▲
    └──────── message [schema ID + payload] ──► Broker ────┘

Run the compatibility check in CI, before merge, against the registry's latest version for the target subject. Fail the build on incompatibility. Store schema files in the producing team's repository, give them code owners, and review them like public interfaces, because they are public interfaces. Without the CI check, the first rejection happens at deploy time, in production.

The registry does not validate meaning, and it does not stop a producer from skipping it. It also cannot repair data already written. Contract tests and documentation are still required for semantic changes.

The registry is also a runtime dependency. Clients cache schemas, which hides short outages, but cold starts and new versions still need it. An outage, a wrong subject-naming strategy, or lost registry state can stall producers and leave consumers unable to decode messages they have already received. Back up the registry's state and monitor it like any production database.

Copy#!/usr/bin/env bash
# CI step: fail the build if the proposed schema is incompatible with the
# latest registered version of its subject. Run on every PR that touches schemas/.
# The script only checks compatibility. Registration happens later, at deploy time.
set -euo pipefail

# Placeholder values. Credentials come from CI secrets and are never committed.
REGISTRY_URL="${REGISTRY_URL:-"
SUBJECT="orders.order-placed-value"
SCHEMA_FILE="schemas/order_placed.avsc"

# The registry expects the schema as an escaped JSON string inside {"schema": ...}.
payload=$(jq -n --rawfile schema "$SCHEMA_FILE" '{schema: $schema}')

# Test the candidate against the subject's configured compatibility mode.
# --fail makes HTTP errors (e.g. 404 for a misspelled or missing subject) fail
# the build instead of passing silently.
response=$(curl --fail --silent --show-error \
  -u "${REGISTRY_USER}:${REGISTRY_PASSWORD}" \
  -X POST \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data "$payload" \
  "${REGISTRY_URL}/compatibility/subjects/${SUBJECT}/versions/latest")

if [ "$(echo "$response" | jq -r '.is_compatible')" != "true" ]; then
  echo "Schema ${SCHEMA_FILE} is NOT compatible with latest ${SUBJECT}; blocking merge." >&2
  exit 1
fi

# Passing proves structural compatibility only. A field whose meaning changed
# still needs contract tests and documentation.
echo "Schema ${SCHEMA_FILE} is compatible with latest ${SUBJECT}."

A pre-merge CI script that sends a candidate schema to the registry's REST compatibility endpoint and fails the build if the subject's compatibility mode rejects the schema.

Handling breaking changes: new event types, dual-publishing and version fields

When a change cannot be made compatible, such as a field's meaning changing or a key restructuring, stop evolving the existing schema and publish a new contract. Usually this means a new event type or topic, such as OrderPlacedV2. The producer dual-publishes both versions while consumers migrate one by one. You retire the old stream only after its last consumer is gone, which you confirm through consumer-group lag or subscription metrics rather than by asking around.

Dual-publishing doubles the write path, and the two writes are not atomic. If the V2 publish succeeds and the V1 publish fails, the two streams disagree, and consumers on different versions act on different facts. Derive both events from one committed record, for example through a transactional outbox, so a failed publish can be retried instead of lost. The other risk is organisational. Migrations stall, and the V1 stream can live for years. Set a retirement date and give it an owner before the first V2 message ships.

An in-payload version field avoids new topics, but it pushes conditional parsing into every consumer. A consumer that forgets to check the field will misread new messages without raising an error.

Designing for fewer breaking changes, replay, and engineering cost

Habits that keep changes additive

Most breaking changes are designed in long before they ship. Write consumers as tolerant readers: ignore unknown fields and default missing optional ones, so producers can add data without a coordinated deploy. Mark a field required only when no consumer could act without it, because that decision is the hardest to reverse. Make additive change the default, and treat removals and renames as exceptions that need review. Publish facts about what happened, such as "order placed, with total and currency", rather than a copy of a table row. Table-shaped payloads break whenever someone refactors the database. Business facts change far less often.

Replay and retention

If consumers can rebuild state by replaying a topic from the beginning, every schema version still in that topic is live. An upcasting layer can transform old events into the current shape on read. It adds per-event latency, though, and its code must be maintained for as long as those events are retained. Rewriting the log instead is risky, and for immutable or mirrored logs it is often impossible. Set retention and compaction with schema history in mind. Compaction keeps only the latest record per key, which can shrink the range of old versions you must support. It only works when the latest record per key is all a consumer actually needs.

Cost

Registries, CI checks and schema review cost visible effort up front. The savings are spread out and easy to undercount: fewer incidents, fewer lockstep deploys, and migrations that don't stall halfway. Count both sides when you decide how much rigor a topic deserves.

Copyfrom dataclasses import dataclass

@dataclass(frozen=True)
class OrderPlaced:
    # A business fact, not a copy of the orders table row.
    order_id: str
    total_cents: int
    currency: str
    coupon_code: str | None = None  # optional; older events never had it

def upcast_v1_to_v2(event: dict) -> dict:
    # v1 used "amount_cents" and implied a single currency (assumed USD here).
    # Keep this function as long as any v1 event is still retained in the topic.
    data = dict(event)
    data["total_cents"] = data.pop("amount_cents")
    data.setdefault("currency", "USD")
    data["schema_version"] = 2
    return data

UPCASTERS = {1: upcast_v1_to_v2}  # each step moves an event one version forward

def read_order_placed(event: dict) -> OrderPlaced:
    version = event.get("schema_version", 1)
    while version in UPCASTERS:  # upcast on read; the stored log is never rewritten
        event = UPCASTERS[version](event)
        version = event["schema_version"]
    # Tolerant reader: take the fields we need, default optional ones, ignore the rest.
    return OrderPlaced(
        order_id=event["order_id"],
        total_cents=event["total_cents"],
        currency=event["currency"],
        coupon_code=event.get("coupon_code"),
    )

# Replaying from the start of the topic means every retained version is live.
history = [
    {"schema_version": 1, "order_id": "ord-1001", "amount_cents": 4999},
    {"schema_version": 2, "order_id": "ord-1002", "total_cents": 1250,
     "currency": "EUR", "channel": "mobile"},  # unknown field is ignored
]
for raw in history:
    print(read_order_placed(raw))

A consumer that upcasts old OrderPlaced events on read and acts as a tolerant reader, so a full replay can handle every schema version still retained in the topic without rewriting the log.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to event schema evolution, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.