How does Kafka Schema Registry compatibility work on the wire?
A schema registry is a small service that stores versioned schemas and assigns each one a numeric ID. In the Confluent Schema Registry wire format, the producer's serializer does not embed the schema in each message. It registers the schema (or looks up its existing ID) and writes a five-byte prefix: one magic byte, currently zero, followed by a 4-byte big-endian schema ID. The Avro-encoded payload follows.
On the consumer side, the deserializer reads that ID, fetches the writer schema from the registry, and caches it. Avro binary encoding carries no field names or type tags, so the consumer cannot decode the bytes without the exact schema that produced them. Avro then resolves the writer schema against the consumer's own reader schema. Fields are matched by name. Writer fields the reader doesn't know are skipped. Reader fields missing from the writer are filled from their declared defaults. Certain type promotions, such as int to long, are allowed.
Resolution can fail. A reader field with no default that the writer never wrote is one example, and an incompatible type change is another. Compatibility rules exist to catch these failures at registration time, before any message reaches a consumer.
Teams adopt a registry for two reasons:
- Smaller messages. A five-byte prefix replaces a full schema on every record.
- One enforced contract per subject. Every service reads the same contract, instead of each team keeping its own copy.
Platform teams get the most from this. They run shared topics that several independently deployed services consume, and no single team controls when those consumers upgrade.
+----------------+ +-------------------------+
| Producer | 1. register schema | Schema Registry |
| (serializer) |------------------------->| versioned schemas per |
| |<-------------------------| subject, each with ID |
+----------------+ 2. schema ID | |
| | compatibility check |
| 3. produce record | at registration time |
v +-------------------------+
+---------------------------------------------+ ^ |
| Kafka topic record | | |
| +-------+---------------+-----------------+ | | |
| | magic | schema ID | Avro binary | | | |
| | 0x00 | 4 bytes | payload | | | |
| | 1 byte| big-endian | no field names | | | |
| +-------+---------------+-----------------+ | | |
+---------------------------------------------+ | |
| | |
| 4. consume record 5. fetch writer | | 6. schema
v schema by ID | | (cached)
+---------------------------------------------+ | |
| Consumer (deserializer) |-----+ |
| |<--------------+
| 7. Avro resolution, writer vs reader: |
| - match fields by name |
| - skip writer fields reader lacks |
| - fill missing reader fields from |
| declared defaults |
| - allowed promotions, e.g. int -> long |
+---------------------------------------------+
The producer registers its schema and prefixes each record with a magic byte and 4-byte schema ID, and the consumer fetches the writer schema by that ID and resolves it against its own reader schema.
Which compatibility mode matches your deployment order?
| Mode | Check against | Upgrade first | Safe changes |
|---|---|---|---|
| BACKWARD (default) | Latest version | Consumers | Delete fields, add fields with defaults |
| FORWARD | Latest version | Producers | Add fields, delete fields with defaults |
| FULL | Latest version | Either | Add or delete fields with defaults only |
*_TRANSITIVE |
Every prior version | Same as base mode | Same rules, applied to the whole history |
| NONE | Nothing | Coordinated release | Anything |
Non-transitive modes compare a new schema only with the latest registered version, so a chain of changes can pass each step and still break. Consider a record with field region:
- v1:
regionis astringwith no default. - v2:
regionis removed. This passes BACKWARD because a v2 reader ignores the extra field in v1 data. - v3:
regionis re-added as anintwith default0. Checked against v2, this passes because v2 data lacks the field and the default fills it.
A v3 consumer reading v1 records still finds a string where it expects an int, and deserialization fails. BACKWARD_TRANSITIVE would reject v3 at registration. Use transitive modes when consumers can read old data: long retention, compacted topics where the oldest value per key can be arbitrarily old, and consumers that replay from earliest.
Two trade-offs matter in practice. First, BACKWARD only works if consumers actually deploy before producers. The registry checks schemas, not rollout order, so that ordering has to come from your release process. Second, FULL_TRANSITIVE makes every added field permanent and optional with a default. A field can never become truly required at the schema level, so "required" has to be enforced in application validation.
#!/usr/bin/env bash
# CI gate: check a candidate schema against the subject's full history before it ships.
set -euo pipefail
SR_URL=""
SUBJECT="orders-value"
CT="Content-Type: application/vnd.schemaregistry.v1+json"
# Pin the subject to BACKWARD_TRANSITIVE so a re-added field must be
# compatible with v1, not only with the latest version.
curl -sf -X PUT -H "$CT" \
--data '{"compatibility": "BACKWARD_TRANSITIVE"}' \
"$SR_URL/config/$SUBJECT"
# Candidate v3: re-adds 'region' as int with a default.
# This passes against v2 but conflicts with v1, where region was a string.
SCHEMA='{"type":"record","name":"Order","namespace":"com.example","fields":[
{"name":"id","type":"string"},
{"name":"region","type":"int","default":0}]}'
# The schema is sent as an escaped JSON string inside the request body.
PAYLOAD=$(jq -n --arg s "$SCHEMA" '{schema: $s}')
# POST to /versions (no version number) to check against the versions
# the subject's compatibility level requires: every version under *_TRANSITIVE.
# verbose=true returns the reasons for any failure.
RESULT=$(curl -sf -X POST -H "$CT" --data "$PAYLOAD" \
"$SR_URL/compatibility/subjects/$SUBJECT/versions?verbose=true")
if [ "$(echo "$RESULT" | jq -r '.is_compatible')" != "true" ]; then
echo "Schema is incompatible with the history of $SUBJECT:" >&2
echo "$RESULT" | jq '.messages' >&2
exit 1
fi
echo "Schema is compatible with all prior versions of $SUBJECT."
A CI step that sets a subject to BACKWARD_TRANSITIVE and fails the build when the v3 schema, which re-adds region as an int, conflicts with any earlier version, including v1 where region was a string.
Which Avro schema changes are safe, and which break consumers?
Avro resolves a writer schema against a reader schema field by field, using names. Most rules follow from one question: when a reader meets data that lacks a field, or contains one it does not know, can it resolve the record without guessing?
| Change | Backward (new reader, old data) | Forward (old reader, new data) |
|---|---|---|
Add field with default |
Safe: reader fills the default | Safe: old reader ignores it |
| Remove field that had a default | Safe | Safe: old reader uses its default |
| Add field without default | Breaks: no value to fill | Safe |
Promote int → long (or float → double) |
Safe | Breaks: no narrowing |
Rename with aliases on reader |
Safe if client honours aliases | Breaks unless old reader has alias |
| Add enum symbol | Safe | Breaks unless enum declares a default |
| Add union branch | Safe | Breaks when the new branch appears |
Unions carry a trap that review rarely catches. A union's default must match its first branch. A field typed ["string", "null"] with "default": null reads as "optional string" to a reviewer, but null is not a valid string. Depending on whether your parser validates defaults, the registry rejects it, or it registers and fails later when a reader has to fill the missing field. Put "null" first whenever the default is null.
The registry checks structure, not meaning. Some changes pass every check and still break consumers. Examples include switching timeout from seconds to milliseconds, splitting what an enum symbol means, or reusing a removed field's name for different data. Old records then resolve cleanly into the wrong semantics.
{
"type": "record",
"name": "OrderEvent",
"namespace": "com.example.orders",
"doc": "v2 of OrderEvent. Each field's doc notes why its change from v1 resolves cleanly.",
"fields": [
{"name": "order_id", "type": "string"},
{"name": "quantity", "type": "long",
"doc": "Promoted from int in v1. Backward-safe because a new reader widens old ints. Not forward-safe because old readers cannot narrow a long."},
{"name": "customer_ref", "type": "string", "aliases": ["customer_id"],
"doc": "Renamed from customer_id. The alias lets a new reader match old data, but only if the client library honours aliases."},
{"name": "status",
"type": {"type": "enum", "name": "OrderStatus",
"symbols": ["PENDING", "PAID", "SHIPPED", "REFUNDED"],
"default": "PENDING"},
"doc": "The enum-level default gives readers a fallback when they meet a symbol they do not know."},
{"name": "coupon_code", "type": ["null", "string"], "default": null,
"doc": "null comes first because a union default must match the first branch. [\"string\", \"null\"] with default null looks optional in review but is invalid."},
{"name": "channel", "type": "string", "default": "web",
"doc": "New in v2. The default lets new readers fill the field when they read v1 records."},
{"name": "timeout_ms", "type": "long", "default": 30000,
"doc": "Unit is milliseconds. The registry would also accept a change to seconds, and that change would still break consumers."}
]
}
A v2 Avro schema that annotates each compatible change: an int-to-long promotion, an alias rename, an enum default, a null-first optional union, and a new field with a default. It also flags a unit change, which passes the registry's checks but still breaks consumers.
Should you use TopicNameStrategy or RecordNameStrategy for subjects?
The subject name strategy decides which schemas the registry compares with each other. Compatibility is checked within a subject, so the strategy you pick sets the boundaries of every check you run.
| Strategy | Subject name | What gets compared | Multiple event types per topic |
|---|---|---|---|
| TopicNameStrategy (default) | <topic>-key, <topic>-value |
Every schema written to that topic | No, unless you use a union |
| RecordNameStrategy | Fully qualified record name | That record type, across all topics | Yes |
| TopicRecordNameStrategy | <topic>-<fully qualified record name> |
That record type, within one topic | Yes |
TopicNameStrategy gives the strongest per-topic guarantee. Every schema on orders-value must be compatible with the previous ones, so a consumer of that topic can rely on one evolving shape. The cost is that it pushes you toward one event type per topic.
The record-based strategies let OrderPlaced and OrderCancelled share a topic, which preserves their relative ordering per key. The trade-off is that no rule constrains what the topic as a whole may contain. A producer can register a brand-new record type and start writing it without any check failing. Consumers must therefore handle unknown types explicitly, by skipping them, routing them to a dead-letter topic, or failing loudly.
Hold key schemas to a stricter mode than value schemas, for example FULL_TRANSITIVE, and treat key schemas as close to frozen. Kafka's default partitioner hashes the serialized key bytes, and compaction keeps the latest record per key. A change that the registry accepts as compatible can still change those bytes. Records for the same entity would then land on different partitions, and compaction would keep both versions.
You set compatibility per subject with PUT /config/{subject} and a body such as {"compatibility": "FULL_TRANSITIVE"}. Keep these settings in version control next to the schemas, apply them from CI, and review changes to them the same way you review schema changes. Relaxing a mode weakens every check that runs after it.
#!/usr/bin/env bash
# Apply per-subject compatibility from a file kept in version control.
# Run from CI after the change has been reviewed, like schema changes.
#
# schema-registry/compatibility.conf (reviewed in pull requests):
# orders-key FULL_TRANSITIVE
# orders-value BACKWARD_TRANSITIVE
# payments-key FULL_TRANSITIVE
# payments-value BACKWARD
set -euo pipefail
REGISTRY_URL="${REGISTRY_URL:-"
CONFIG_FILE="${1:-schema-registry/compatibility.conf}"
while read -r subject level; do
# Skip blank lines and comments.
[[ -z "${subject}" || "${subject}" == \#* ]] && continue
echo "Setting ${subject} -> ${level}"
# PUT /config/{subject} overrides the global default for one subject only.
curl --silent --show-error --fail \
-X PUT \
-H "Content-Type: application/vnd.schemaregistry.v1+json" \
--data "{\"compatibility\": \"${level}\"}" \
"${REGISTRY_URL}/config/${subject}"
echo
# Read it back so the CI log shows the level the registry now enforces.
curl --silent --show-error --fail \
"${REGISTRY_URL}/config/${subject}"
echo
done < "${CONFIG_FILE}"
A CI script that reads reviewed, version-controlled compatibility levels and applies them per subject through the registry's /config/{subject} REST endpoint, holding key subjects to FULL_TRANSITIVE.
How do you run schema evolution checks in CI before deploy?
Keep the .avsc files in the repository that owns the producer. The team that changes the record type then also owns the check, and the schema change appears in the same diff as the code that writes it. On every pull request, send each candidate schema to the registry's compatibility endpoint for its target subject (POST /compatibility/subjects/{subject}/versions/latest), or run your build tool's equivalent plugin goal. Fail the build if the response reports is_compatible: false. The test uses whatever mode the subject is configured with, so a transitive mode checks the candidate against every registered version.
Registering schemas belongs in a single pipeline step that runs after merge. Set auto.register.schemas=false in production producer configs. Otherwise any service that starts with an edited schema can register a new version that nobody reviewed. With auto-registration off, a producer whose schema is not yet registered fails at serialisation. That is the failure you want, because it is loud and happens early.
This setup has costs you should plan for:
- Registry dependency. CI needs network access to the registry and depends on its current state. A registry outage blocks merges, and a check against a staging registry that has drifted from production proves nothing.
- Merge races. Two PRs can each be compatible with the current latest version and still conflict with each other once both merge. To prevent this, serialise registration through a merge queue, or re-run the compatibility check inside the post-merge registration step.
The payoff is where the failure shows up. A rejected PR costs the author a few minutes with the diff in front of them. The same break found in production surfaces as consumer deserialisation errors and a stalled pipeline. The person on call then has to trace it back to someone else's deploy.
What should you do when a schema change is genuinely incompatible?
Treat a breaking change as a migration, not an edit. Publish the new schema under a new subject, usually by creating a new topic such as orders.v2, so its compatibility history starts clean. During the transition, either have producers dual-write to both topics, or run a translation consumer that reads orders and republishes converted records to orders.v2. A translation consumer keeps producer code simple but adds lag and another service to operate. Move consumers over one at a time. Retire the old topic only after its consumer lag is zero and nothing still reads it.
Setting a subject's compatibility to NONE is a last resort, because it switches off the only automated check for that subject. If you must do it:
- Announce a fixed window.
- Coordinate producer and consumer deploys inside that window.
- Restore the previous mode as soon as the new version is registered.
- Record what changed and why, in the same repository that holds the schemas.
Do not delete schema versions, soft or hard, to undo a mistake. Every message on a retained topic carries a schema ID in its header. Deleting the version from the registry leaves those IDs in the data, and any consumer that reads them, including one replaying from an earlier offset, can fail to resolve them.
Compatibility settings are mutable state. A change to them leaves no trace in the message stream. Restrict registry write access to the CI identity and a small operator group. Log every config and delete call so that a silent mode change can be traced to a person and a time.
When should you avoid a schema registry, and what are the alternatives?
A registry exists to protect consumers you don't control and can't redeploy alongside the producer. When one team owns both producer and consumer and ships them in the same release, that coordination problem mostly disappears. The same applies to short-lived topics such as backfill or migration staging, internal-only streams with one known reader, and prototypes whose shape changes daily. In those cases, a strict compatibility gate adds review friction before the contract is worth protecting.
| Approach | Where compatibility is checked | Main weakness |
|---|---|---|
| Full schema embedded in each message | Consumer, at read time | Larger messages; nothing stops a bad producer |
| Version field in payload | Consumer branching logic | Old-version handling piles up in consumer code |
| Schemaless JSON, consumer-side validation | Consumer, at runtime | Breakage surfaces in production, often as dead-letter traffic |
| Contract tests between services | CI of both services | Only covers consumers that publish contracts |
| Protobuf or JSON Schema | Format-specific rules (Protobuf field numbers; JSON Schema open vs. closed content) | Different evolution semantics; still needs some enforcement point |
Cost picture (illustrative estimate). Assume about three engineer-days to wire CI checks and subject configuration. Assume 40 schema changes a year at 15 minutes of review each. That totals roughly five engineer-days in the first year and well under two after that. Now assume one cross-team consumer outage costs one to two engineer-days, counting diagnosis, replay, and coordination. Under those assumptions, avoiding two or three such incidents a year covers the cost. If your topics have no independent consumers, you are unlikely to see those incidents, and the registry may never pay back.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to kafka schema registry compatibility, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.