How does Avro schema resolution work between writer and reader schemas?

Avro binary encoding carries no field names or type tags. A record is just its field values concatenated in the order the writer's schema declares them. A consumer therefore cannot decode a message without the exact schema used to write it. Confluent Schema Registry supplies that schema. The serializer prepends a magic byte and a 4-byte schema ID to every payload, and the deserializer uses that ID to fetch the writer schema.

Producer, registry and consumer exchanging schema IDs:

  1. Producer → Registry: registers or looks up the writer schema under a subject.
  2. Registry → Producer: returns schema ID 42.
  3. Producer → Kafka: sends [magic byte][42][Avro bytes].
  4. Consumer → Registry: requests schema 42 and caches it.
  5. Consumer: resolves writer schema 42 against its own reader schema, then decodes.

Resolution follows a few rules:

  • Name matching: fields are matched by name (or alias), not by position.
  • Extra writer fields: a field in the writer schema but not in the reader is skipped.
  • Missing writer fields: a field the reader expects but the writer lacks is filled from the reader's default. If the reader has no default, resolution fails.
  • Type changes: types must match or be a permitted promotion, such as int to long or string to bytes.

Every compatibility rule follows from these mechanics. A schema change is only "compatible" if every reader that will meet it can resolve it. When a consumer fails with a resolution error, the cause is a reader and writer pair that nobody checked. Any team producing to or consuming from a registry-backed topic shares this contract, whether or not they own the schema.

import io
import struct
from fastavro import parse_schema, schemaless_writer, schemaless_reader

writer_schema = parse_schema({
    "type": "record", "name": "User", "namespace": "com.example",
    "fields": [
        {"name": "id", "type": "int"},
        {"name": "name", "type": "string"},
        {"name": "internal_note", "type": "string"},  # reader lacks it, so it is skipped
    ],
})

reader_schema = parse_schema({
    "type": "record", "name": "User", "namespace": "com.example",
    "fields": [
        {"name": "id", "type": "long"},                 # int -> long is a permitted promotion
        {"name": "name", "type": "string"},
        {"name": "email", "type": "string", "default": ""},  # writer lacks it, so the default is used
    ],
})

buf = io.BytesIO()
buf.write(struct.pack(">bI", 0, 42))
schemaless_writer(buf, writer_schema, {"id": 7, "name": "Ada", "internal_note": "vip"})
payload = buf.getvalue()

magic, schema_id = struct.unpack(">bI", payload[:5])
record = schemaless_reader(io.BytesIO(payload[5:]), writer_schema, reader_schema)
print(schema_id, record)  # 42 {'id': 7, 'name': 'Ada', 'email': ''}

Using fastavro, this example encodes a record in the Confluent wire format and decodes it with a different reader schema. The read skips the extra writer field, fills the missing reader field from its default and promotes int to long.

Who should use compatibility modes as a deployment order contract?

Treating the compatibility mode as a statement about rollout order is worth the effort for any team where producers and consumers of a topic deploy independently. In practice that means:

  • Platform and data infrastructure teams that run the registry and set its global and per-subject compatibility levels. They decide which guarantees every topic starts with.
  • Producer service owners who change schemas. They need to know whether their change lets them deploy first or forces them to wait for consumers.
  • Consumer service owners, including teams that never edit a schema. A BACKWARD subject assumes they upgrade before producers do, so their release timing is part of the contract.
  • Teams running long-retention, compacted or replayed topics, such as event-sourced stores, CDC pipelines and analytics jobs that reprocess history. They are the readers most likely to meet old schema versions and the strongest case for _TRANSITIVE modes.
  • Release and SRE engineers who own deployment pipelines and can gate producer rollouts on consumer state.

It matters less when one team owns both ends of a topic and deploys them atomically, or when a topic is short-lived. The section on avoiding compatibility checks covers those cases.

What do BACKWARD, FORWARD and FULL compatibility actually check?

Each mode names which side of that pair must cope. BACKWARD means the new schema, as reader, can decode data written with the old one, so consumers can upgrade first. FORWARD means the old schema can decode data written with the new one, so producers can upgrade first. FULL requires both directions.

Take two changes: adding a field and removing one. A reader fills a missing field only from its own default. It silently skips a writer field it does not declare.

Change BACKWARD FORWARD FULL
Add field with default Pass Pass Pass
Add field without default Rejected: new reader has no value for old records Pass Rejected
Remove field that had a default Pass Pass: old reader uses its default Pass
Remove field without default Pass Rejected: old reader has no value for new records Rejected

Latest version vs. every version

Plain modes compare a candidate only against the latest registered version. The _TRANSITIVE variants compare it against every prior version in the subject. That difference shows up in this chain:

  • v1 has region as a string.
  • v2 drops it. This passes BACKWARD.
  • v3 re-adds region as an int with default 0. This also passes BACKWARD against v2.

Yet v3 cannot read v1 records, because a string cannot resolve to an int. A consumer replaying from offset zero, or reading a compacted topic that still holds v1 keys, fails. BACKWARD_TRANSITIVE would have rejected v3.

NONE disables the check entirely. The registry stores whatever you send, and any compatibility guarantee rests on team discipline.

#!/usr/bin/env bash
SR=
SUBJECT=orders-value
CT='Content-Type: application/vnd.schemaregistry.v1+json'

V3='{"schema":"{\"type\":\"record\",\"name\":\"Order\",\"fields\":[{\"name\":\"id\",\"type\":\"string\"},{\"name\":\"region\",\"type\":\"int\",\"default\":0}]}"}'

curl -s -X PUT -H "$CT" --data '{"compatibility":"BACKWARD"}' "$SR/config/$SUBJECT"
curl -s -X POST -H "$CT" --data "$V3" \
  "$SR/compatibility/subjects/$SUBJECT/versions/latest"
curl -s -X POST -H "$CT" --data "$V3" \
  "$SR/compatibility/subjects/$SUBJECT/versions/1"
curl -s -X PUT -H "$CT" --data '{"compatibility":"BACKWARD_TRANSITIVE"}' "$SR/config/$SUBJECT"
curl -s -X POST -H "$CT" --data "$V3" "$SR/subjects/$SUBJECT/versions"

Calls to the Schema Registry REST API showing that re-adding region as an int passes BACKWARD against the latest version but fails against v1, and that BACKWARD_TRANSITIVE rejects it at registration.

Which compatibility mode should you choose for your rollout order?

Read each mode as a statement about which side of the topic may run the new schema before the other side does.

Mode Guarantee Deploy first Sequence
BACKWARD New schema reads data written with the previous schema Consumers 1. Register v2. 2. Roll out all consumers. 3. Confirm no consumer still runs v1. 4. Roll out producers.
FORWARD Previous schema reads data written with the new schema Producers 1. Register v2. 2. Roll out producers. 3. Roll out consumers at any pace.
FULL Both directions hold Either side 1. Register v2. 2. Roll out producers and consumers in any order.

The registry checks schemas, not deployments. It has no idea which consumer versions are running. Under BACKWARD, suppose v2 adds a field with no default. If a producer ships v2 while one consumer group is still on v1, the registry has no reason to object, because the check already passed. That consumer then fails at deserialization or misreads records, and nothing warns you beforehand. The ordering is your release process's responsibility. Gate producer rollouts on consumer version metrics or on an explicit sign-off.

Choose a _TRANSITIVE variant when consumers read data older than the previous version. That covers long retention, compacted topics, replays from offset zero, and event-sourced stores rebuilt from full history. Non-transitive modes compare only against the latest version, so a reader on v5 is not guaranteed to decode v1 records.

FULL_TRANSITIVE has a real cost. Every field you add or remove must carry a default, and that requirement never expires. Over time, schemas fill up with nullable unions that exist only for compatibility. The schema stops telling readers which fields are actually required, so consumers end up writing defensive null checks for data that is always present in practice.

How does the deployment order contract work in practice?

The contract has three parts, and each is enforced by a different actor:

  1. The registry enforces schema shape. When a new version is registered, the registry runs the resolution rules from the first section in the direction the subject's mode specifies. BACKWARD tests "new schema as reader, old schema as writer". FORWARD tests the reverse. FULL tests both. The _TRANSITIVE variants repeat the test against every earlier version. If the test fails, registration is rejected and no producer can write with that schema.
  2. The mode tells you the safe rollout order. A passing check under BACKWARD means "consumers may move first". Under FORWARD it means "producers may move first". Under FULL either order is safe. The mode is the only place this ordering is written down, so read it before planning a release.
  3. Your release process enforces the order. The registry does not know which consumer and producer builds are running. Someone, or some pipeline, has to confirm that the side meant to go first has finished before the other side ships.

A typical BACKWARD change runs like this. The producer team opens a pull request with the new .avsc. CI runs the compatibility check against the subject. After merge, the pipeline registers the schema. Consumer teams deploy builds whose reader schema is the new version. These readers can still decode old records, so they are safe to run immediately. Once consumer version metrics or sign-offs show that no v1 reader remains, the producer deploys and starts writing v2 records. If the producer ships early, the registry will not stop it. The contract only holds if step 3 is real.

Which Avro schema changes are safe under each mode?

The matrix below covers the changes that come up most often. "Old reader" means a consumer still on the previous schema reading data written with the new one, which is what FORWARD protects. "New reader" reading old data is what BACKWARD protects. FULL requires both.

Change BACKWARD FORWARD FULL
Add field with default Safe Safe Safe
Add field without default Fails Safe Fails
Remove field that has a default Safe Safe Safe
Remove field without default Safe Fails Fails
Add union branch Safe Fails Fails
Remove union branch Fails Safe Fails
Add enum symbol Safe Safe only if the old enum has a default Same as FORWARD
Remove enum symbol Safe only if the new enum has a default Safe Same as BACKWARD
Promote int→long, float→double Safe Fails Fails
string↔bytes Safe Safe Safe
Rename field with alias on new schema Safe Fails Fails
Incompatible type change (string→int) Fails Fails Fails

Most of these results follow from one rule: resolution consults only the reader's schema. A reader can widen a value it receives, use its own default, or match a field through its own aliases. It cannot interpret a union branch, enum symbol or field name that its schema has never seen. Avro also ignores aliases declared on the writer's schema, so an old consumer cannot follow a rename.

Defaults can make a change pass the checker while the data still breaks. Suppose you rename amount_cents to amount and give the new field a default of 0. The schema passes, and every old record now reads as zero. Changing a field from milliseconds to seconds while keeping it a long also passes, because the type did not change. The registry compares structure only. It does not check meaning, units, or invariants between fields, so a reviewer has to catch these changes.

import io
from fastavro import parse_schema, schemaless_writer, schemaless_reader

v1 = parse_schema({
    "type": "record", "name": "Payment", "namespace": "com.example.billing",
    "fields": [
        {"name": "id", "type": "string"},
        {"name": "amount_cents", "type": "long"},
    ],
})

v2 = parse_schema({
    "type": "record", "name": "Payment", "namespace": "com.example.billing",
    "fields": [
        {"name": "id", "type": "string"},
        {"name": "amount", "type": "long", "default": 0},
    ],
})

buf = io.BytesIO()
schemaless_writer(buf, v1, {"id": "pay-1001", "amount_cents": 4599})
buf.seek(0)

record = schemaless_reader(buf, v1, v2)
print(record)  # {'id': 'pay-1001', 'amount': 0}  <- no error, but the value is wrong

A rename done as "remove field, add field with default" resolves without error, but every old record silently reads as an amount of 0.

How do subject naming strategies change what gets checked?

The registry never checks a schema in isolation. It checks it against earlier versions registered under the same subject, and the serializer's subject naming strategy decides which subject that is. The strategy is set per client with key.subject.name.strategy and value.subject.name.strategy. The default is io.confluent.kafka.serializers.subject.TopicNameStrategy, and the alternatives are RecordNameStrategy and TopicRecordNameStrategy in the same package.

Strategy Subject for a value Compatibility scope
TopicNameStrategy <topic>-value Every schema written to the topic
RecordNameStrategy <namespace>.<RecordName> One record type, across all topics
TopicRecordNameStrategy <topic>-<namespace>.<RecordName> One record type within one topic

Under the default, the topic carries a single version history. Suppose orders carries both OrderPlaced and OrderCancelled. Registering OrderCancelled after OrderPlaced checks one record against the other. Their fields share nothing, so the check fails under any mode other than NONE. Teams then often set NONE, which silently removes protection for every event type on the topic.

Record-based strategies give each type its own history, so OrderCancelled evolves only against earlier OrderCancelled versions. RecordNameStrategy shares that history across topics. That suits a type with one owner but couples unrelated topics. TopicRecordNameStrategy keeps types separate per topic.

The cost is governance. One subject per topic becomes many subjects per topic. Each subject can carry its own compatibility level and drift from the others. The registry also has no notion of who owns a subject. Before switching strategies, decide who approves changes to each record type. Also record the expected compatibility level somewhere reviewable, so it does not live only in registry state.

bootstrap.servers=kafka-1.internal.example:9092
schema.registry.url=

key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=io.confluent.kafka.serializers.KafkaAvroSerializer

value.subject.name.strategy=io.confluent.kafka.serializers.subject.TopicRecordNameStrategy

auto.register.schemas=false
use.latest.version=true

A producer configuration that switches value subject naming to TopicRecordNameStrategy, so each event type on a shared topic gets its own compatibility history, and turns off auto-registration so new subjects are created only through review.

How do you enforce schema compatibility in CI instead of at runtime?

The expected compatibility level should not live only in the registry's state. Keep it in version control, next to the schemas, and let the pipeline apply it.

Configuration as code

The registry has a global default (/config) and per-subject overrides (/config/{subject}). Set a conservative global level and override only the subjects that need something different. A PUT from the deploy job keeps the registry in line with the repository. If someone changes the level by hand, the next pipeline run either resets it or flags the drift.

Test, then register

For each changed .avsc file, the CI job posts the candidate schema to /compatibility/subjects/{subject}/versions/latest. If the response reports is_compatible: false, the job fails the build. This check runs the same resolution rules the registry would apply when the schema is registered. The difference is timing: the failure shows up in review, not on the first produce call. A brand-new subject has no latest version yet, so handle that case explicitly instead of treating it as an error.

Only after the change merges does the pipeline register the schema via POST /subjects/{subject}/versions.

Lock down producers

Set auto.register.schemas=false in the producers so a stray local schema can never become a new version in production. This adds friction:

  • Every schema change now needs a pipeline run before the producer deploys.
  • Local development needs its own registry or credentials that are allowed to auto-register.

Illustrative estimate (assumed numbers, not a benchmark):

Cost = occurrences × engineers × hours.

  • Assume a failed build costs 1 engineer × 0.25 h.
  • Assume a production incident costs 3 engineers × 4 h = 12 h.

At those numbers, catching one break in CI instead of production saves 11.75 engineer-hours.

How can it save engineering time?

The savings come from moving work earlier and removing coordination that would otherwise happen ad hoc:

  • Breaks surface in review, not in production. A failed compatibility check in CI is a red build on the author's own pull request. The same break found at runtime means consumer lag, dead-lettered or stuck partitions, and several people working out which schema version caused it. The illustrative estimate above shows how to size this for your own team. Substitute your own assumed incident and build costs.
  • Release planning becomes a lookup. Once teams agree that the mode defines rollout order, "who deploys first?" is answered by reading the subject's configuration. It no longer needs a meeting or a thread across producer and consumer teams for every schema change.
  • FULL removes ordering work entirely for subjects where teams accept the cost of mandatory defaults. Producers and consumers ship on their own schedules.
  • Transitive modes prevent the most expensive failures to debug. A replay or compaction failure caused by a v1 record meeting a v5 reader can appear months after the change that caused it. Rejecting that change at registration avoids an investigation across old versions.
  • Configuration as code avoids drift hunts. When the expected level lives in the repository and CI flags differences, nobody has to discover during an incident that a subject was set to NONE long ago.
  • Separate subjects per record type stop workarounds. With a record-based naming strategy, teams sharing a topic no longer fight a single version history or fall back to NONE to get unrelated event types registered.

The time is not free. Someone has to build the CI check, maintain per-subject configuration and run a local registry for development. Those costs are paid once or grow slowly. Incident costs recur with every unchecked change.

When should you avoid compatibility checks, and what are the alternatives for breaking changes?

Compatibility checks only verify structural resolution between schemas. Three situations fall outside what they can protect:

  • Intentional breaking changes. If you are removing a required field or changing a type, the checker is correctly telling you that existing consumers will break.
  • Semantic changes. Switching amount from cents to dollars, or changing a timestamp from UTC to local time, passes every mode because the type stays long.
  • Short-lived topics. Scratch topics, ephemeral test environments and single-producer, single-consumer pipelines deployed atomically can reasonably run with NONE.

The risky case is setting a live subject to NONE for an urgent fix. The override is per-subject configuration, so it outlives the incident. Months later, an unrelated change registers without any check, and nothing in the deploy flags it. If you must do this, record the revert as an incident action item and have CI fail when any subject's level differs from the expected value.

Alternatives for breaking changes

Approach How it works Cost
Versioned topic or subject (orders.v2) Start a fresh compatibility history Consumers must be migrated
Dual-write or migration job Populate v1 and v2 in parallel, cut consumers over, then retire v1 Temporary double storage and producer complexity
Consumer-side upcasting Readers convert old shapes into the current model The logic lives in every consumer
Contract tests Assert meaning, such as units and value ranges, between teams Ongoing test maintenance

Decision checklist

  • Is the change structurally breaking? Use a new topic and migrate.
  • Is the change semantic only? Add a contract test, and rename the field if you can.
  • Is the topic ephemeral with atomic deploys? NONE is acceptable.
  • Are you about to set NONE on a live subject? Ticket the revert first.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to avro schema compatibility modes, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.