What is event versioning in Kafka, and why is it harder than API versioning?
Event versioning is the practice of changing the structure of messages on a Kafka topic without breaking the services that produce or consume them. Message schema evolution is the underlying mechanism: a set of rules about which changes to a payload's shape are allowed, so that data written under one schema can still be read under another.
With HTTP API versioning, a server can run /v1 and /v2 side by side, watch traffic drain from the old endpoint, and then delete it. Kafka offers no such clean cutover, because the log keeps old messages after they are consumed. Topics with long or unlimited retention hold months or years of payloads. Compacted topics keep the latest record per key indefinitely, so a customer record last written two schema versions ago remains the current value. Any consumer can reset its offset and replay from the beginning, and every historical format comes back with it.
Producers and consumers also deploy independently. A producer cannot know which consumers exist, how far behind they are, or when they will upgrade. A consumer may read data written before it existed, or after a producer change it has not been told about. Compatibility therefore becomes a long-lived contract, not a one-time migration.
This matters for any team whose topics are read by other services, or replayed to rebuild projections, populate new data stores, or restore state in event-sourced systems.
Who should use event versioning, and who can skip most of it?
Formal event versioning pays off when at least one of these is true:
- Topics are shared across teams. If the team that owns the producer does not own every consumer, nobody can coordinate releases. A compatibility contract replaces that coordination.
- Data is retained or replayed. Long retention, compaction, event sourcing and projection rebuilds all mean consumers will read old formats long after the producer has moved on.
- Consumers are unknown or external. Analytics pipelines, data lake sinks, and other organisations' services subscribe to topics without telling the producer. Treat those topics as public APIs.
- The platform team runs Kafka for many services. A registry with enforced compatibility rules and CI checks gives every team the same guardrails without each one building its own.
Teams can use much less of this process when a topic has short retention, a single consumer owned by the same team as the producer, and no replay requirement. In that case, deploying both sides together and letting old records age out is often enough. The checklist at the end of this article notes which steps you can drop in that situation.
How does event versioning work end to end?
A versioned event moves through four stages.
- Define and review the schema. The producing team writes the schema (Avro, Protobuf or JSON Schema) in a repository, reviews it like code, and runs a compatibility check against the registry in CI.
- Serialize with a version marker. At produce time, the serializer looks up the schema in the registry, gets its ID, and writes that ID into the record, typically as a small prefix before the encoded payload. Some teams also set a version header or include a version field. The record on the topic now identifies the exact schema it was written with.
- The registry enforces the rules. The registry stores every version of each subject and rejects new versions that break the subject's compatibility mode (backward, forward or full, optionally transitive). With auto-registration disabled for production producers, only schemas that passed CI can be used.
- Deserialize and adapt on read. The consumer reads the schema ID, fetches the writer's schema from the registry (and caches it), and decodes the payload. In Avro, the decoder then resolves the writer's schema against the consumer's reader schema, filling defaults and dropping unknown fields. In Protobuf, unknown field numbers are skipped. If the consumer needs a single current shape, an upcaster chain converts older versions before the record reaches domain code.
When a change cannot pass the compatibility rules, the same pipeline is used to publish a new event type or a new topic instead, as described later in this article. The sections below cover each stage in detail.
What do backward, forward and full compatibility mean for consumers?
Compatibility is always defined from the reader's side: given a schema a consumer holds and data written with another schema, can the consumer decode it without failing or misreading fields?
- Backward compatibility means consumers using the new schema can read data written with the old one. Adding a field is backward compatible only if it has a default, because the new reader fills that value in for old records. Deploy consumers first. They can handle both old and new payloads before any producer starts emitting the new shape.
- Forward compatibility means consumers still on the old schema can read data written with the new one. The old reader ignores fields it does not recognise. Removing a field is safe only if the old reader has a default for it. Deploy producers first. Lagging consumers keep working until they are upgraded.
- Full compatibility means both directions hold. Producers and consumers can be deployed in any order, which matters when different teams own each side and nobody coordinates releases.
| Mode | Guarantee | Deploy first | Typical safe change |
|---|---|---|---|
| Backward | New reader, old data | Consumers | Add field with default |
| Forward | Old reader, new data | Producers | Add field old readers ignore |
| Full | Both | Either | Add or remove fields that have defaults |
Each mode can be checked against only the latest schema or transitively against every earlier version. The difference shows up on topics with long retention or compaction. A consumer replaying from offset zero may meet version 1 records right after version 7 records. Version 7 being compatible with version 6 proves nothing about version 1, so replayable topics need transitive checks.
Which schema changes are safe and which are breaking?
The safest changes are additive and optional. A new field with a default lets old readers ignore it and new readers fill it in when they read old records. Almost everything else needs more care than it first appears.
| Change | Wire format still parses? | Effect on consumers |
|---|---|---|
| Add optional field with default | Yes | Safe in both directions |
| Remove a required field | Often no | Readers that expect it fail or misbehave |
| Rename a field | Sometimes | Readers see the old field removed and a new one added, so data is silently dropped |
| Change a type (string to int, int to string) | Varies by format | Deserialization errors or lossy coercion |
| Change units (cents to dollars) | Yes | Wrong results with no error |
| Reuse an enum value for a new meaning | Yes | Old logic runs on new semantics |
The last two rows are the dangerous ones. A schema checks structure, not meaning. If amount changes from cents to dollars, every compatibility check passes, yet a downstream billing service is now wrong by a factor of 100. When meaning changes, add a new field such as amount_minor_units or publish a new event type. Either way, the change becomes visible in the contract and is not buried in a producer's release notes.
Defaults carry a similar risk. A default of 0 or false lets an old record deserialize cleanly, but the consumer cannot tell "the value was zero" from "the value was never sent." For fields that drive business decisions, pick defaults that are clearly sentinels, or make absence explicit, so a missing value cannot pass as a real one.
How do Avro, Protobuf and JSON differ in schema evolution support?
Each format puts the compatibility contract in a different place, and that placement determines what tooling can check for you.
Avro resolves data at read time using two schemas: the writer's schema, which travels as a registry ID, and the reader's schema, which is compiled into the consumer. Fields are matched by name. To add a field safely, give it a default, for example {"name": "currency", "type": "string", "default": "EUR"}. A new reader that decodes an old record then fills in "EUR". If you add a field without a default, old data has nothing to resolve against and decoding fails. Because the reader always needs the writer's schema, Avro without a registry is impractical.
Protobuf identifies fields on the wire by number, not by name. Renaming a field is wire-safe, but if you reuse a number for a new field, old bytes get silently decoded into that new field. When you delete a field, reserve both its number and its name, for example reserved 4; reserved "discount_code";. The compiler then rejects any later attempt to reuse them. Old parsers skip unknown fields instead of failing, which makes additive changes safe in both directions.
Plain JSON carries no schema and enforces nothing. Safety depends on tolerant readers that ignore unknown keys, on naming conventions, and on contract tests that run real payloads against every consumer.
| Format | Fields matched by | Safe additive change | Safe removal | Built-in enforcement |
|---|---|---|---|---|
| Avro | Name, via reader/writer resolution | New field with default | Field had a default | Strong, with a registry |
| Protobuf | Field number | New number | Reserve number and name | Compiler and registry checks |
| JSON | Key string, by convention | Readers ignore unknown keys | Only after consumers stop reading it | None |
The choice of format is an architectural decision. Pick it based on how much compatibility checking you want enforced by machines and how much you are willing to leave to discipline.
// --- Avro: additive change (v2 of OrderPlaced) ---
// "currency" is new. Its default lets a v2 reader decode v1 records,
// which were written without this field.
{
"type": "record",
"name": "OrderPlaced",
"namespace": "com.example.orders",
"fields": [
{"name": "order_id", "type": "string"},
{"name": "amount_cents", "type": "long"},
{"name": "currency", "type": "string", "default": "EUR"}
]
}
// --- Protobuf: additive change plus a removed field ---
syntax = "proto3";
package example.orders;
message OrderPlaced {
string order_id = 1;
int64 amount_cents = 2;
string customer_id = 3;
// Field 4 (discount_code) was deleted. Reserve its number and its name
// so protoc rejects any later field that tries to reuse either one.
reserved 4;
reserved "discount_code";
// New field with a new number. Old parsers skip it as an unknown field.
string currency = 5;
}
The same OrderPlaced event evolved in two formats: Avro adds a field with a default so reader/writer resolution succeeds, and Protobuf adds a field under a new number and reserves the number and name of a deleted field.
How do you enforce consumer compatibility with a schema registry and CI?
Treat each event schema as a code artifact. It lives in a repository, has a named owning team, goes through review, and gets a version on every change. Then enforce compatibility in two places. In CI, a pipeline step tests the candidate schema against the registry's compatibility rule for that subject and fails the merge if the check fails. At produce time, production producers should not auto-register schemas. An unreviewed schema is then rejected at serialization instead of reaching a topic. A failed check stops a pull request while its author still has the context to fix it. An incident-driven fix happens hours later and pulls in consumer teams, replay work and possibly data repair.
Version information can live in several places:
| Location | Routing | Debugging |
|---|---|---|
| Serializer-embedded schema ID | Requires deserializing first | Opaque without registry access |
| Payload version field | Requires parsing the payload | Visible in any dump |
| Message header | Cheap; no payload parse | Visible if tooling shows headers |
Event type name (e.g. OrderPlacedV2) |
Simplest; can map to topics | Obvious, but versions multiply |
The registry is now on the critical path. Clients cache schemas by ID, so a running service can usually tolerate a registry outage. A cold start or a never-seen schema cannot. Run the registry with redundancy and monitor it like any other production dependency.
Compatibility checks validate structure, not meaning. Renaming the unit of amount from cents to dollars passes every check and still breaks consumers. Semantic changes need human review and explicit contract documentation.
#!/usr/bin/env bash
set -euo pipefail
REGISTRY_URL="${REGISTRY_URL:?REGISTRY_URL must be set by the CI environment}"
SUBJECT="orders.order-placed-value" # owned by the orders team
SCHEMA_FILE="schemas/orders/order_placed.avsc" # reviewed, versioned in this repo
payload=$(jq -Rs '{schema: .}' < "$SCHEMA_FILE")
response=$(curl -sS --fail -X POST \
-H "Content-Type: application/vnd.schemaregistry.v1+json" \
--data "$payload" \
"$REGISTRY_URL/compatibility/subjects/$SUBJECT/versions/latest")
if [ "$(echo "$response" | jq -r '.is_compatible')" != "true" ]; then
echo "Schema $SCHEMA_FILE is incompatible with $SUBJECT; fix before merging." >&2
echo "$response" >&2
exit 1
fi
echo "Compatibility check passed for $SUBJECT."
A CI script that posts a candidate schema to the schema registry's compatibility endpoint and blocks the merge when it violates the subject's rule.
How does event upcasting work, and when should you avoid it?
Upcasting puts all version handling in one place: the consumer's deserialization boundary. When a record arrives, the deserializer reads its version marker (a header, a schema ID, or a field) and passes the payload through a chain of small functions. Each function converts one version into the next, so v1 becomes v2 and v2 becomes v3. Each step only needs to know about its two neighbours, and adding v4 means writing one new step instead of touching every handler. Domain handlers only ever see the current shape. The read path is: raw bytes → deserialize at the stored version → upcast v1→v2 → upcast v2→v3 → handler.
The chain has real costs:
- Read-time overhead. Every old event is converted on every read. During a full replay of a large topic, that conversion runs for millions of records.
- Permanent maintenance. Any step you delete makes the records behind it unreadable, so the chain must last as long as the retained data does.
- Silent corruption on replay. A wrong default in v1→v2 changes the result of every rebuild that replays those records, and nothing fails loudly. Test each step with frozen fixtures captured from real historical payloads, not hand-written ones.
The alternative is to rewrite history by consuming the old topic, transforming the records, and producing them to a new topic. This removes the chain, but it breaks immutability. Offsets change, and the stored bytes are no longer the bytes that were originally published, which undermines audit and compliance guarantees. On a large retained topic, a rewrite is also a long migration that needs extra storage and a coordinated cutover. Prefer upcasting unless the chain itself has become the bigger risk.
import json
CURRENT_VERSION = 3
def v1_to_v2(e: dict) -> dict:
# v2 split "name" into first/last; v1 only ever stored a single full name
first, _, last = e.pop("name").partition(" ")
return {**e, "first_name": first, "last_name": last}
def v2_to_v3(e: dict) -> dict:
# v3 added "currency"; every v1/v2 order was EUR, so this default must stay exact
return {**e, "currency": e.get("currency", "EUR")}
UPCASTERS = {1: v1_to_v2, 2: v2_to_v3}
def deserialize(raw: bytes, headers: list[tuple[str, bytes]]) -> dict:
# Records produced before versioning carried no header, so treat them as v1
version = int(dict(headers).get("event-version", b"1"))
if version > CURRENT_VERSION:
raise ValueError(f"unknown future version {version}") # fail loudly, never guess
event = json.loads(raw)
while version < CURRENT_VERSION:
event = UPCASTERS[version](event)
version += 1
return event # domain handlers only ever see the v3 shape
def test_v1_fixture_upcasts_to_v3():
with open("fixtures/order_placed_v1.bin", "rb") as f:
raw = f.read()
with open("fixtures/order_placed_v1.expected_v3.json") as f:
expected = json.load(f)
assert deserialize(raw, [("event-version", b"1")]) == expected
A chained v1→v2→v3 upcaster at the consumer deserialization boundary that reads the version from a record header, rejects unknown future versions, and is checked against frozen fixtures captured from real historical payloads.
What are the alternatives when a breaking change is unavoidable?
When a change cannot be expressed as an optional field or an upcast, stop forcing it into the existing contract and publish a new one. You can add a new event type on the same topic, such as OrderPlacedV2 alongside OrderPlaced, or create a new topic. A new type keeps per-key ordering intact, but every consumer must handle both types. A new topic gives a clean contract, but ordering across the old and new topics is lost. Any consumer that needs full history then has to read both topics and merge them.
During migration, producers either dual-write to both contracts or keep writing the old one while a translating bridge republishes events in the new shape. Dual-writing from application code risks drift, because one write can succeed while the other fails. A bridge has a single source of truth, but it adds latency and can produce duplicates on retry, so consumers must be idempotent.
Track adoption through consumer group lag and offsets on each topic, not through announcements. Retire the old contract only when no group still reads it, its retention window has expired, and no replay or audit job depends on it.
| Option | Ordering | Full-history reads | Main risk | Development cost |
|---|---|---|---|---|
| New event type, same topic | Preserved per key | Single topic | Consumers branch on type | Low to moderate |
| New topic + dual-write | Split | Merge two topics | Drift between writes | Moderate |
| New topic + bridge | Split | Merge two topics | Duplicates, extra service | Moderate to high |
| Additive-only forever | Preserved | Single topic | Schema clutter, dead fields | Low now, rising later |
How can event versioning save engineering time?
Most of the time savings come from moving failures earlier, to a point where one person can fix them cheaply.
- Breaking changes fail in CI, not in production. The author of the pull request sees the incompatibility immediately and fixes it with full context. Without the check, the same mistake surfaces as deserialization errors or wrong data in another team's service, and several teams spend time diagnosing it.
- Releases need less coordination. Once the compatibility mode and deployment order are agreed, producer and consumer teams ship on their own schedules. There is no need for synchronised deploys or cross-team release meetings for routine additive changes.
- Replays and rebuilds become routine. With transitive checks and tested upcasters, rebuilding a projection or backfilling a new data store from offset zero is a normal operation. Nobody has to write one-off scripts to handle historical formats discovered mid-replay.
- Less data repair. Semantic mistakes and silently dropped fields are the most expensive failures, because they require finding and correcting bad downstream state. Contract review and new-field-for-new-meaning rules prevent many of them.
- Faster debugging. When every record carries a schema ID or version marker, an engineer investigating a bad message can see exactly which contract produced it instead of guessing from the payload.
As an illustrative estimate, with all inputs assumed rather than measured: suppose a breaking change reaches production and affects 4 consumer teams. Assume each team spends 3 engineer-hours diagnosing and patching, and the producing team spends 6 hours on rollback and data repair. That is 18 engineer-hours for one incident. If the same change is caught by a CI compatibility check, assume the author spends 30 minutes reworking the schema. Under these assumptions, each prevented incident saves roughly 17 engineer-hours, before counting the cost of wrong data reaching users. Your own numbers will depend on how many consumers a topic has and how hard bad state is to repair.
The process does cost time. Someone has to run the registry, maintain the CI step, write upcaster fixtures and review schemas. For a single-team topic with short retention, that overhead can exceed the savings, which is why the checklist below includes a lighter path.
Operational checklist for rolling out a Kafka schema change
Work through these steps in order. Each one catches a different class of failure.
-
Classify the change. Decide whether it is additive, a type change, a removal, or a semantic change that the schema cannot express, such as a field whose units changed. The category determines everything that follows. Semantic changes need a new field or event type even when the schema check passes.
-
Choose deployment order. Under backward compatibility, upgrade consumers first so they can read both old and new records before any new record exists. Under forward compatibility, upgrade producers first. If the topic has consumers you don't control, assume they are slow to upgrade and plan for the longer window.
-
Run compatibility checks in CI against the registry's configured mode for that subject. Fail the build on violations. A local check against your own last schema is not enough.
-
Test against recorded production samples of every version still within retention. Synthetic fixtures miss the odd records: nulls in "required" fields, payloads from a hotfixed producer, and upcaster edge cases.
-
Roll out gradually. Start with one producer instance or a low-traffic partition key, and keep rollback possible.
-
Verify. Watch consumer lag, deserialization errors and dead-letter volume per consumer group. Rising lag with no errors often means a retry loop.
When less process is enough: for an internal topic with short retention and a single consumer owned by the same team, you can skip steps 4 and 5. Deploy the consumer and producer together and let old records age out.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to event versioning kafka, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.