How does Avro schema resolution work when a Kafka consumer reads a message?

Avro binary data carries no field names or type tags. A decoder can only read the bytes if it knows the exact schema used to write them, called the writer schema. The consuming application has its own expectation, the reader schema, which is the schema its code was generated from or configured with. Avro's schema resolution rules reconcile the two at read time, matching fields by name rather than by position:

  • A field in the reader schema but not the writer schema gets the reader's declared default. If the reader declares no default, resolution fails.
  • A field in the writer schema but not the reader schema is decoded and then discarded.
  • Fields present in both must have matching or promotable types, such as int to long.

A schema registry makes the writer schema available to the consumer. Schemas are stored under subjects, which by default are derived from the topic name (for example, orders-value). Each subject holds an ordered list of versions, and each schema has a globally unique ID. With Confluent Schema Registry, the serializer registers or looks up the schema and prefixes every message with a magic byte and that 4-byte ID. The deserializer fetches the writer schema by ID, caches it, and resolves it against the local reader schema.

A compatibility mode is a rule the registry enforces when a new version is registered. It guarantees that resolution will succeed between the new schema and earlier versions in a stated direction. This matters to backend and platform engineers running shared topics where producers and consumers deploy on independent schedules. The registry rejects a breaking change at registration time, so it never reaches a consumer as a deserialization failure in production.

import io
import struct
from fastavro import parse_schema, schemaless_writer, schemaless_reader

writer = parse_schema({
    "type": "record", "name": "Order",
    "fields": [
        {"name": "id", "type": "int"},
        {"name": "coupon", "type": "string"},  # reader doesn't know this field
    ],
})

reader = parse_schema({
    "type": "record", "name": "Order",
    "fields": [
        {"name": "id", "type": "long"},  # int -> long is a legal promotion
        {"name": "currency", "type": "string", "default": "USD"},  # filled from default
    ],
})

payload = io.BytesIO()
schemaless_writer(payload, writer, {"id": 1001, "coupon": "SPRING10"})
message = struct.pack(">bI", 0, 42) + payload.getvalue()

magic, schema_id = struct.unpack(">bI", message[:5])
assert magic == 0
record = schemaless_reader(io.BytesIO(message[5:]), writer, reader)

print(schema_id, record)

Using fastavro, a record written with one schema is resolved against a different reader schema: the int is promoted to long, the unknown field is discarded, and the missing field takes its default, behind the magic-byte-plus-schema-ID header that registry serializers add.

What do BACKWARD, FORWARD, FULL and NONE actually guarantee?

Each mode states which reader can decode which writer's data, and the registry enforces it only at registration time.

Mode New schema is checked against Guarantee
BACKWARD Latest registered version Consumers on the new schema can read data written with the previous schema
FORWARD Latest registered version Consumers on the previous schema can read data written with the new schema
FULL Latest registered version Both directions hold
NONE Nothing No check; any schema is accepted
*_TRANSITIVE Every prior version The same guarantee, against all of history

Why the transitive variants exist

The plain modes compare each new schema only to the version just before it. That can let incompatibility build up across steps. Suppose v2 adds a region field with a default, and v3 removes that default. Under BACKWARD, v3 passes because v2 records always contain region. A v3 consumer that reads v1 records from the start of the topic still fails, because those records have no region field and v3 gives it no default.

This affects any topic where old records stay readable: long retention, compacted topics that keep the oldest surviving value for each key indefinitely, and replays from offset zero to rebuild state. On those topics a consumer can receive data from any version ever registered.

Transitive modes block this drift. The cost is that a schema with many versions becomes harder to change, because every new version must be compatible with all of them.

Choosing

  • Use BACKWARD_TRANSITIVE or FULL_TRANSITIVE for compacted, long-retention, or replayable topics.
  • Plain BACKWARD or FULL is acceptable for short-retention topics where old versions expire quickly.
  • Reserve NONE for development subjects or a deliberate migration to a new topic.

Should consumers or producers upgrade first under each compatibility mode?

A compatibility mode is a contract about deployment order. The registry checks schemas, not deploys. If the pipeline rolls out in the wrong order, a schema that passed the check can still break readers in production.

Mode Upgrade first Adding a field Removing a field
BACKWARD Consumers Needs a default. New consumers fill it in when reading old records. Allowed. New consumers ignore the extra field in old records.
FORWARD Producers Allowed without a default. Old consumers ignore it. Only if the field had a default. Old consumers fill it in.
FULL Either Needs a default. Field must have had a default.

What breaks under each mode

BACKWARD, removing a field. Suppose a producer ships first and stops writing discount_code, which has no default in the old schema. Every lagging consumer that tries to decode the new records fails, because it has no value to fill in. Upgrading consumers first avoids this. They are ready for the field to disappear before any producer stops writing it.

FORWARD, adding a field. Suppose a consumer ships first and expects region, which has no default. Older records and older producers don't carry that field, so the upgraded consumer fails on them. Upgrading producers first works because old readers ignore fields they don't know.

FULL. Every added or removed field carries a default, so any reader can resolve any writer. You can deploy services in whatever order is convenient.

Most schema evolution incidents come from a mismatch between the mode and the real rollout process. A common case is a BACKWARD subject owned by a team whose producer service always deploys before its consumers. Pick the mode that matches how your services already ship. If the process can't guarantee an order, use FULL. Either choice removes the need for coordinated deploys and avoids poison-message recovery during rollouts.

#!/usr/bin/env bash
set -euo pipefail

REGISTRY=""
SUBJECT="orders-value"
SCHEMA_FILE="order-v3.avsc"   # candidate schema, e.g. with discount_code removed
CT="Content-Type: application/vnd.schemaregistry.v1+json"

MODE=$(curl -sf "$REGISTRY/config/$SUBJECT" \
        || curl -sf "$REGISTRY/config")
MODE=$(echo "$MODE" | jq -r '.compatibilityLevel')

RESULT=$(jq -n --arg s "$(cat "$SCHEMA_FILE")" '{schema: $s}' \
  | curl -sf -X POST -H "$CT" --data @- \
      "$REGISTRY/compatibility/subjects/$SUBJECT/versions/latest")

if [ "$(echo "$RESULT" | jq -r '.is_compatible')" != "true" ]; then
  echo "Rejected under $MODE. Do not deploy." >&2
  exit 1
fi

case "$MODE" in
  BACKWARD*) echo "$MODE: deploy ALL consumers first, then producers." ;;
  FORWARD*)  echo "$MODE: deploy producers first, then consumers." ;;
  FULL*)     echo "$MODE: any deploy order is safe." ;;
  NONE)      echo "NONE: no guarantees. Coordinate deploys by hand." >&2; exit 1 ;;
esac

A pre-deploy script that asks the registry whether the candidate schema is compatible and then prints the deploy order that the subject's compatibility mode requires, so the pipeline's actual rollout order matches the mode.

Which Avro schema changes are safe under which mode?

The registry checks one rule: can the reader resolve every field the writer produced? Under BACKWARD the new schema is the reader. Under FORWARD the old schema is the reader. FULL requires both.

Change BACKWARD FORWARD FULL
Add field with default ✅ ✅ ✅
Remove field that had a default ✅ ✅ ✅
Add field without default ❌ ✅ ❌
Rename, no alias ❌ ❌ ❌
Rename with alias ✅ ❌ ❌
int→long/float/double, long→float/double, float→double ✅ ❌ ❌
string↔bytes ✅ ✅ ✅
Add enum symbol ✅ only if old enum has default only if old enum has default
Add union branch ✅ ❌ ❌
Remove union branch ❌ ✅ ❌

A rename without an alias is a removal plus an addition. An alias such as {"name": "customer_id", "type": "string", "aliases": ["client_id"]} lets the new reader match the old writer's client_id. The old reader has no alias for customer_id, so the forward direction still fails.

Promotion only widens. A long reader accepts int data, but an int reader cannot accept a long.

An enum only tolerates unknown symbols if the reader already declared a fallback, as in {"type": "enum", "name": "Status", "symbols": ["ACTIVE", "UNKNOWN"], "default": "UNKNOWN"}. You therefore have to ship the default before the first new symbol.

Unions resolve each datum against the reader's branches. Changing string to {"name": "email", "type": ["null", "string"], "default": null} adds a branch, so it is backward-only. Old readers fail on the first null.

#!/usr/bin/env bash
set -euo pipefail
SR=
SUBJECT=orders-value
CT='Content-Type: application/vnd.schemaregistry.v1+json'

payload() { jq -n --arg s "$(cat "$1")" '{schema: $s}'; }
check() { curl -s -X POST -H "$CT" -d "$(payload "$1")" \
  "$SR/compatibility/subjects/$SUBJECT/versions/latest"; echo; }

curl -s -X PUT -H "$CT" -d '{"compatibility": "FULL"}' "$SR/config/$SUBJECT"; echo

cat > v1.avsc <<'EOF'
{"type":"record","name":"Order","fields":[
 {"name":"quantity","type":"int"},
 {"name":"status","type":{"type":"enum","name":"Status",
   "symbols":["ACTIVE","UNKNOWN"],"default":"UNKNOWN"}}]}
EOF
curl -s -X POST -H "$CT" -d "$(payload v1.avsc)" "$SR/subjects/$SUBJECT/versions"; echo

cat > v2.avsc <<'EOF'
{"type":"record","name":"Order","fields":[
 {"name":"quantity","type":"int"},
 {"name":"status","type":{"type":"enum","name":"Status",
   "symbols":["ACTIVE","UNKNOWN","CANCELLED"],"default":"UNKNOWN"}},
 {"name":"email","type":["null","string"],"default":null}]}
EOF
check v2.avsc   # expect {"is_compatible":true}

sed 's/"quantity","type":"int"/"quantity","type":"long"/' v1.avsc > v3.avsc
check v3.avsc   # expect {"is_compatible":false}

This script uses the Schema Registry REST API to set a subject to FULL and register a v1 that already declares an enum default. It then tests two candidates against the latest version: one that adds an enum symbol and a nullable field passes, and one that promotes int to long fails.

Why are defaults and nullable unions central to FULL compatibility?

Under FULL, every field you add or remove must carry a default. A new reader needs the default to fill a field that old data lacks, and an old reader needs its own default when new data drops a field. If every field has a default from the day it is created, you can add or retire any of them later without a failed check. The common way to supply that default is a union with null as the first branch and "default": null.

Order matters because Avro requires a union's default to match the union's first branch. ["null", "string"] with a default of null is valid. ["string", "null"] with a default of null is not, and depending on the tooling it is either rejected when the schema is parsed or fails later during resolution. If a field needs a non-null default, the matching type has to come first in the union.

This pattern has two costs:

  • Required becomes optional. Once a field such as customer_id is nullable so that it can evolve, the schema no longer guarantees it is present. Producers and consumers have to check for it in application code, and the schema stops documenting that the field is required.
  • Defaults can hide missing data. If retry_count defaults to 0, a consumer cannot tell whether a message reported zero retries or never carried the field. A nullable union keeps that difference visible, because null means the value is absent.

No compatibility mode covers meaning. The registry check only shows that the reader can resolve data written with the writer's schema. If timeout changes from seconds to milliseconds but keeps the type long, it passes BACKWARD, FORWARD, FULL and their transitive variants, and consumers still get wrong values. Changes like that need a new field name.

{
  "type": "record",
  "name": "OrderEvent",
  "namespace": "com.example.orders",
  "fields": [
    {
      "name": "customer_id",
      "type": ["null", "string"],
      "default": null,
      "doc": "Semantically required. Nullable only so it can evolve under FULL, so producers and consumers must check for null in application code."
    },
    {
      "name": "retry_count",
      "type": ["null", "int"],
      "default": null,
      "doc": "null means the field was absent. A plain int with default 0 would make 'absent' look the same as 'zero retries'."
    },
    {
      "name": "priority",
      "type": ["string", "null"],
      "default": "normal",
      "doc": "Non-null default, so the matching branch (string) must come first in the union."
    },
    {
      "name": "timeout_seconds",
      "type": ["null", "long"],
      "default": null,
      "doc": "Original unit. Never reinterpret this field as milliseconds: that change would pass every compatibility mode and still break consumers."
    },
    {
      "name": "timeout_ms",
      "type": ["null", "long"],
      "default": null,
      "doc": "A unit change gets a new field name. It has a default, so adding it now or removing it later passes a FULL check."
    }
  ]
}

An Avro record in which every field has a default that matches the first branch of its union, the nullable unions keep absent values distinguishable from real ones, and a unit change is made by adding a new field instead of reusing the old one.

How do you implement compatibility checks with subject naming, per-subject config and CI gates?

Semantic changes like that need human review, and so does everything else the checks are configured to catch. The checks only protect what they are scoped to.

Subject naming decides what gets compared

With the default topic-based strategy, each topic has two subjects, <topic>-key and <topic>-value. Every new value schema on a topic is checked against that topic's value history, so the guarantee holds per topic. Record-based naming uses the record's fully qualified name as the subject. Topic-record naming uses <topic>-<record name>. Both check compatibility per record type, which lets one topic carry several types. The cost is that the registry no longer constrains the topic as a whole. Any producer can introduce a new type, and every consumer must be ready to handle every type it might receive.

Global versus per-subject config

PUT /config sets the global default. PUT /config/{subject} overrides it for one subject. Keep a strict global default, such as BACKWARD_TRANSITIVE or FULL_TRANSITIVE, and record each relaxed override along with the reason for it.

CI gates and auto-registration

Before merging, CI should POST the candidate schema to /compatibility/subjects/{subject}/versions/latest, or to /versions to test against all versions. The build should fail when the response reports the schema as incompatible. Then set auto.register.schemas=false on producers so that only the pipeline registers schemas. This adds a deploy step for every schema change. Without it, a schema can be registered from someone's laptop and then has to be cleaned up under load.

Operational weight

Clients cache schemas by ID. However, a new schema ID, or a client starting with a cold cache, has to reach the registry. That makes the registry critical infrastructure. Back up its storage, restrict write and config access, and test that restores work.

When should you avoid relying on compatibility modes, and what are the alternatives for breaking changes?

Compatibility checks are structural. They confirm that a reader schema can decode a writer schema. They say nothing about meaning. If you change amount from cents to dollars and keep the type long, every mode passes the change, and every consumer silently computes wrong totals. The registry also cannot help with intentional breaking changes, or with topics read by teams you cannot coordinate with. In those cases the rollout order from section 3 cannot be enforced.

The common shortcut is to set the subject to NONE, register the breaking schema, and restore the original mode. The registry accepts the change, but the topic still holds records written with the old schema ID. Those records stay until retention expires, or indefinitely on a compacted topic. Nothing fails right away. It fails weeks later, when a consumer group resets offsets for a replay, or when a lagging consumer reaches the boundary between old and new data. At that point the consumer's reader schema cannot resolve the old writer schema, and the consumer stops on that record.

Safer options keep the incompatible data apart:

Approach Dev cost Storage cost When it fits
New topic and subject (orders.v2) Low Old topic kept until its retention expires Producers can switch cleanly and history is disposable
Dual-write to old and new topics Medium: producer logic for two formats, plus a risk that the topics diverge Two copies during the overlap Uncontrolled consumers need time to migrate
Migration consumer that translates v1 events into v2 Higher: a translation service to run and monitor Full history in the new format Consumers need complete, replayable history in v2

With any of these, move consumers over one group at a time. Track lag on the old topic, and delete it only after its last consumer group has stopped committing offsets.

Who should use schema registry compatibility modes?

Compatibility modes are worth configuring when the people who change a schema are not the same people who read the data, or when they do not deploy at the same time. In practice that means:

  • Teams sharing Kafka topics across service boundaries. A producer team owns orders-value, and several consumer teams read it on their own release schedules. The registry is the one place where a breaking change can be stopped before it reaches every consumer.
  • Platform teams running a shared Kafka cluster. They own the global default mode, the per-subject overrides, and the CI gate. They also decide which subjects may be relaxed and why.
  • Teams with long-lived or replayable data. Compacted topics, long retention, and event-sourced services that rebuild state from offset zero need transitive modes, because a consumer can receive any version ever written.
  • Data and analytics engineers consuming event streams. Pipelines that land Kafka data in storage for later processing read old and new records side by side. They benefit directly from the guarantee that the current reader schema resolves historical writer schemas.

Compatibility modes add less value for a single service that produces and consumes its own topic and deploys both sides together, or for short-lived development topics. NONE or a plain mode is reasonable there. They also do not replace review for semantic changes such as unit or meaning changes, which no mode detects.

How does it work end to end?

Putting the pieces above together, a schema change moves through these steps:

  1. Author the change. An engineer edits the .avsc file, ideally following the rules in the safe-changes table: add fields with defaults, declare enum defaults before adding symbols, and use a new field name for semantic changes.
  2. Check in CI. The pipeline posts the candidate schema to the registry's compatibility endpoint for the subject. The registry looks up the subject's mode (the per-subject setting if one exists, otherwise the global default). It then runs Avro schema resolution between the candidate and the latest version, or every prior version for transitive modes, in the direction the mode requires. The response reports whether the schema is compatible, and the build fails if it is not.
  3. Register. Once the change merges, the pipeline registers the schema under the subject. The registry stores it as the next version and assigns a schema ID. Registering a schema identical to an existing one returns the existing ID instead of creating a new version. With auto.register.schemas=false, producers only look up schemas and never register them.
  4. Roll out in the order the mode requires. Consumers first for BACKWARD, producers first for FORWARD, either order for FULL.
  5. Produce. The producer's serializer obtains the schema ID, encodes the record with the writer schema, and prepends the magic byte and 4-byte ID.
  6. Consume. The deserializer reads the ID and fetches the writer schema from the registry, or from its cache. It then resolves the writer schema against the consumer's reader schema, filling defaults, dropping unknown fields, and promoting types.

The registry acts only at steps 2 and 3. Everything after that relies on the guarantee those steps established and on the deploy order being followed.

How can it save engineering time?

The main saving is that a breaking change is caught by an automated check before merge, not by consumers failing in production. Without the check, an incompatible schema typically surfaces when a consumer stops on a record it cannot decode. Someone then has to page the owning team, work out which producer change caused it, decide whether to roll back the producer or patch the consumer, and deal with the records already written. On a compacted topic those records may stay indefinitely. With the check, the same mistake costs one failed CI run and a schema edit.

Other time savings come from:

  • Fewer coordinated deploys. A mode that matches how services already ship, or FULL when the order can't be guaranteed, lets producer and consumer teams release independently. They no longer need to schedule joint releases for every schema change.
  • Less review effort on structural changes. Reviewers do not need to work out Avro resolution rules by hand for every added field or enum symbol. The registry answers the structural question, and humans can focus on semantics.
  • Safer replays and backfills. Transitive modes mean a consumer can reset offsets and reprocess history without first auditing every schema version for compatibility.
  • One source of truth. Consumers fetch writer schemas by ID, so teams do not copy schema files between repositories or debug which copy is current.

As an illustrative estimate with assumed inputs: suppose an organisation makes 20 schema changes a quarter. Assume 1 in 10 would be accidentally incompatible, and assume each such incident takes 6 engineer-hours to diagnose, roll back, and clean up across producer and consumer teams. That gives 2 incidents and about 12 engineer-hours per quarter. Under the same assumptions, a CI gate turns those into 2 failed builds that take perhaps 30 minutes each to fix, so roughly 1 hour in total. The inputs here are not measurements. Substitute your own change rate, error rate, and incident cost. The gate's setup and the extra pipeline step for registration are the costs to weigh against the result.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to schema registry compatibility modes, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.