How does Avro schema evolution work at read time?

Avro binary data carries no field names or type tags. A record is just encoded values in the order the writer's schema declares them, so it cannot be decoded without that exact schema. When a producer serializes a Kafka event through a schema registry, it registers (or looks up) its schema and prefixes the payload with a small schema ID. The consumer's deserializer reads that ID, fetches the writer's schema from the registry (and caches it), and then performs schema resolution against its own reader's schema.

Resolution follows a few rules, and every compatibility mode derives from them:

  • Fields are matched by name, not position. Order changes are harmless.
  • A field present only in the writer's schema is decoded and skipped.
  • A field present only in the reader's schema gets the reader's default.
  • If a reader-only field has no default, resolution fails and the event cannot be read.

Optional fields are a common source of confusion. A nullable field is a union such as ["null", "string"]. A union's default must match its first branch, so a null default requires null to be listed first. Defaults live only in the reader's schema. They are never written into the data, so changing a default affects how old records are interpreted, not what they contain.

A registry gives Kafka events three things. Producers and consumers can be deployed independently. Messages stay small because they carry an ID instead of a full schema. Compatibility rules are enforced in one place, when a schema is registered, instead of being discovered at read time.

import io
from fastavro import parse_schema, schemaless_writer, schemaless_reader

writer = parse_schema({"type": "record", "name": "UserCreated", "fields": [
    {"name": "id", "type": "long"},
    {"name": "legacy_flag", "type": "boolean"},  # writer-only: skipped on read
    {"name": "name", "type": "string"},
]})

reader = parse_schema({"type": "record", "name": "UserCreated", "fields": [
    {"name": "name", "type": "string"},  # matched by name, not position
    {"name": "id", "type": "long"},
    # Reader-only optional field: null is the first union branch, so default null is valid.
    {"name": "email", "type": ["null", "string"], "default": None},
]})

buf = io.BytesIO()
schemaless_writer(buf, writer, {"id": 42, "legacy_flag": True, "name": "ada"})

buf.seek(0)
print(schemaless_reader(buf, writer, reader))
strict = parse_schema({"type": "record", "name": "UserCreated", "fields": [
    {"name": "id", "type": "long"},
    {"name": "email", "type": "string"},  # no default
]})
buf.seek(0)
try:
    schemaless_reader(buf, writer, strict)
except Exception as err:
    print("resolution failed:", type(err).__name__)

Decodes one Avro record written with a producer schema using a different consumer schema: a writer-only field is skipped, fields match by name, a nullable reader-only field takes its default, and a reader-only field with no default makes resolution fail.

What do BACKWARD, FORWARD, FULL and NONE actually promise?

Each mode is a promise about which side of the topic can run the new schema first. It is not a quality setting. The registry checks a candidate schema against the previous one, and the mode decides which direction that check runs.

Mode What the registry checks Allowed changes Who deploys first
BACKWARD New schema can read data written with the previous schema Delete fields; add fields that have defaults Consumers
FORWARD Previous schema can read data written with the new schema Add fields; delete fields that had defaults Producers
FULL Both directions hold Add or delete only fields with defaults Either side
NONE Nothing Anything Whoever coordinates by hand

The allowed changes follow from Avro resolution. Under BACKWARD, the new reader may meet old records that lack an added field, so that field needs a default. A field the reader dropped is simply skipped. Under FORWARD the roles swap. Old consumers skip fields they don't know, but a field you remove must have had a default in the old schema, or old readers fail on new records.

Pick the mode from your real deploy order, not from whatever the registry ships as its default. If producers routinely release before consumers, BACKWARD guards the wrong direction.

Every mode except NONE has a coordination cost, and each one assigns that cost to a different team. With BACKWARD, one consumer team that lags on upgrades blocks the producer from writing new data. FORWARD moves the same bottleneck to the producer side. FULL lets either side go first, but in exchange it narrows which changes you can make at all.

How does a registry compatibility check work?

The check runs at registration time, not when messages flow. The sequence for a single registration request looks like this:

  1. A client (a producer with auto-registration enabled, or a CI job) submits a schema to a subject. The subject name comes from the naming strategy covered later in this article.
  2. The registry resolves which compatibility mode applies. It uses the subject-level setting if one exists and falls back to the global default otherwise.
  3. It picks the comparison set. Non-transitive modes compare against the latest registered version only. _TRANSITIVE modes compare against every version in the subject. NONE skips the comparison.
  4. For each schema in that set, it runs Avro's resolution rules in the direction the mode requires. BACKWARD treats the candidate as reader and the existing version as writer. FORWARD swaps the roles. FULL runs both.
  5. If every comparison passes, the registry stores the schema as the next version in the subject and returns its ID. If an identical schema is already registered, it returns the existing ID instead of creating a new version. If any comparison fails, the request is rejected and nothing is stored.

Two consequences follow from this design. First, the check only covers schemas that go through the registry. A producer that serializes with a schema it never registered cannot produce valid framed messages, but a producer that registers under the wrong subject is checked against the wrong history. Second, changing a subject's mode does not re-validate versions already stored. Switching from BACKWARD to BACKWARD_TRANSITIVE protects future registrations, but an existing incompatible pair stays in the history.

The registry does not inspect messages at all. Consumers still perform Avro resolution at read time with the writer's schema fetched by ID. The registration check is a prediction that this resolution will succeed, made once, before any data is written.

When do you need transitive compatibility?

Non-transitive modes (BACKWARD, FORWARD, FULL) compare a new schema only against the latest registered version. The _TRANSITIVE variants compare it against every version in the subject. The gap between the two is easy to fall into.

Take an OrderPlaced record. v1 has orderId and amount. v2 adds currency as a string with default "USD". That passes BACKWARD: a v2 reader resolves v1 data by filling in the default. v3 removes the default from currency, making it required. That also passes BACKWARD, because the check compares v3 only with v2, and every v2 record has currency written into it. But a v3 consumer that reads a v1 record finds no currency in the data and no default in its reader schema, so schema resolution fails. BACKWARD_TRANSITIVE would have rejected v3.

Whether this matters depends on how long old data lives. Compacted topics keep the latest record per key indefinitely, so v1 records can sit next to v3 records for years. Long retention, deliberate replays, and consumers that reset to the earliest offset all read old data.

Non-transitive modes are easier to work with, but they assume nobody reads old data. When that assumption is wrong, nothing warns you. The failure shows up during the first full replay, often in the middle of an incident. FULL_TRANSITIVE is the safest mode. Over time, though, it pushes schemas toward fields that are all optional with defaults. The schema then says less about what a valid event looks like, and that validation moves into application code.

{"type": "record", "name": "OrderPlaced", "fields": [
  {"name": "orderId", "type": "string"},
  {"name": "amount",  "type": "double"}
]}

{"type": "record", "name": "OrderPlaced", "fields": [
  {"name": "orderId",  "type": "string"},
  {"name": "amount",   "type": "double"},
  {"name": "currency", "type": "string", "default": "USD"}
]}

{"type": "record", "name": "OrderPlaced", "fields": [
  {"name": "orderId",  "type": "string"},
  {"name": "amount",   "type": "double"},
  {"name": "currency", "type": "string"}
]}

curl -X PUT \
  -H "Content-Type: application/vnd.schemaregistry.v1+json" \
  --data '{"compatibility": "BACKWARD_TRANSITIVE"}' \
  "$REGISTRY_URL/config/orders-value"

Three OrderPlaced schema versions in which each step passes BACKWARD against the previous version, while v3 fails BACKWARD_TRANSITIVE because it cannot read v1 records, followed by the registry call that sets the subject to the transitive mode.

Which Avro schema changes break under each compatibility mode?

Each verdict below comes from Avro's resolution rules applied in the mode's direction. BACKWARD means a new reader can read old data. FORWARD means an old reader can read new data.

Change BACKWARD FORWARD FULL NONE
Add field with default Pass Pass Pass Pass
Add field without default Fail Pass Fail Pass
Remove field Pass Pass only if the field had a default Pass only if the field had a default Pass
Rename field Fail (an alias in the new reader can fix this) Fail Fail Pass
int→long/float/double, long→float/double, float→double Pass Fail Fail Pass
string↔bytes Pass Pass Pass Pass
Add enum symbol Pass Fail unless the old reader's enum has a default Same as FORWARD Pass
Add union branch Pass Fail Fail Pass

Type promotions run in one direction only. An old int reader cannot hold a long. string and bytes pass in both directions because the spec allows promotion each way.

Widening a union fails FORWARD because an old reader has no branch to resolve a value written to the new branch.

The nullable field is the change people most often get wrong. A field's default must match the first branch of its union. ["null","string"] with "default": null is valid. ["string","null"] with "default": null is rejected. For FULL compatibility, put null first.

An enum default goes on the enum type, not the field: "default": "UNKNOWN", where UNKNOWN is already listed in symbols. The old reader's schema must already carry that default before new symbols ship.

Aliases and enum defaults only work if every Avro library and registry checker in your path implements them. Polyglot teams should test the behavior in each language and in the registry itself rather than trust the spec.

{
  "type": "record",
  "name": "OrderEvent",
  "namespace": "com.example.orders",
  "fields": [
    {"name": "order_id", "type": "string"},
    {
      "name": "coupon_code",
      "doc": "Nullable: null is the FIRST branch, so default null is valid.",
      "type": ["null", "string"],
      "default": null
    },
    {
      "name": "customer_ref",
      "doc": "Renamed from customer_id; the alias lets this reader resolve old data if the library and checker honor aliases.",
      "aliases": ["customer_id"],
      "type": "string"
    },
    {
      "name": "status",
      "type": {
        "type": "enum",
        "name": "OrderStatus",
        "doc": "Enum-level default: unknown symbols written by newer producers resolve to UNKNOWN. Must ship in readers before new symbols are added.",
        "symbols": ["UNKNOWN", "PLACED", "SHIPPED", "DELIVERED"],
        "default": "UNKNOWN"
      },
      "default": "UNKNOWN"
    }
  ]
}

An Avro reader schema showing a null-first nullable union with a null default, a field alias for a rename, and an enum default declared on the enum type itself.

When should you not rely on the registry to catch breaking changes?

Even when every library agrees, a passing check only tells you the new schema can decode the old bytes. The registry compares structure: names, types, defaults, unions. It has no model of what a field means. Change amount from dollars to cents, or timestamp from UTC to local time, and every mode accepts it. The same goes for a field that was always populated and now arrives as an empty string because an upstream service stopped setting it. Consumers deserialize cleanly and then compute wrong totals.

Defaults create a quieter version of the same problem. When a reader fills a missing discount with 0, old records become indistinguishable from records where the discount really was zero. A consumer that applies business rules to that value is acting on data nobody produced. Declaring the field as a union with null is often more honest, because "unknown" stays visible. The cost is that every consumer must handle null explicitly, and some will get that wrong.

Setting a subject to NONE makes sense only when you control every producer and consumer and deploy them together, or when the topic is disposable. Otherwise NONE removes the one automated guard you have. For a single service talking to itself, or for payloads with no long-lived readers, a registry may add operational weight without much benefit.

Treat registry checks as a floor and add three complements. Consumer-driven contract tests catch semantic and population changes. Human schema review asks what a field means. Documented field semantics, including units, nullability intent, and ownership, give that review something concrete to check against.

When should you avoid registry compatibility modes altogether?

Compatibility modes earn their cost when independently deployed producers and consumers share long-lived data. Outside that situation, they can be the wrong tool:

  • One team owns both ends and deploys them together. If producer and consumer ship in the same release, there is no window where old and new schemas coexist on the wire. A compatibility mode then only blocks changes you could have made safely. Note that this stops being true the moment the topic is replayed or a second consumer appears.
  • The data is ephemeral. Request/reply traffic, short-lived work queues, or topics that are drained and discarded on every deploy have no old records to resolve. Strict modes add friction without protecting anything.
  • The schema already travels with the data. Avro object container files store the writer's schema in the file header, so batch files that are written once and read later can be decoded without a registry. Per-message registry lookups add a dependency the format does not need.
  • You need the schema to enforce required fields. Long runs of FULL or FULL_TRANSITIVE push every field toward optional-with-default. If the schema is your main validation contract, and losing required fields would push validation into every consumer, a versioned-topic approach may preserve more meaning than endless compatible evolution.
  • Consumers sit outside your registry. Partners or external clients that cannot reach your registry cannot fetch writer schemas by ID. Registry-framed payloads are unreadable to them, so publish a separate, self-describing contract at that boundary.
  • You cannot operate the registry reliably. It becomes a runtime dependency for producers and consumers that meet new schema IDs. If no one will run it with redundancy and backups, the registry turns into a single point of failure for every topic that depends on it.

In each case the question is the same: will two different schema versions ever need to read each other's data? If the honest answer is no, a compatibility mode is not buying you safety.

What are the alternatives to registry compatibility modes?

None of these is strictly better. Each moves the cost of evolution somewhere else.

Versioned topics or event types. Publish orders.v1 and orders.v2, or OrderPlacedV1 and OrderPlacedV2, as separate streams with independent schemas. Each version can keep strict required fields. The cost is the migration work described in the last section of this article, paid on every breaking change rather than only on rare ones.

A different serialization format with its own evolution rules. Protocol Buffers identifies fields by number, not name. Renames are free on the wire, and adding fields is generally safe, but reusing a removed field number corrupts data, so removed numbers must be reserved. JSON Schema validates documents that carry field names, which makes payloads larger but readable without the writer's schema. Registries often support these formats too, with compatibility rules that differ from Avro's, so the table in this article does not transfer to them.

Schemas versioned in source control, checked in CI. Keep .avsc files in a repository and run compatibility checks with an Avro library during builds. The Java Avro library, for example, includes a SchemaCompatibility class that tests reader/writer pairs. This gives you review and history without a runtime service. You still need a way for consumers to obtain the writer's schema, either by embedding it, by shipping it in a shared artifact, or by agreeing on one schema per topic.

Consumer-driven contract tests. Each consumer publishes the fields and semantics it depends on, and the producer's build fails if a change violates any of them. This catches the semantic changes registries miss, such as units or population, but it only covers consumers that actually wrote contracts.

Schemaless JSON with tolerant readers. Producers emit JSON, and consumers read only the fields they need and ignore the rest. This is the lowest-ceremony option and works well with few, closely coordinated teams. Nothing machine-checks a change before it ships, so breakage surfaces in consumers at runtime.

Many teams combine these: a registry with compatibility modes for internal Avro topics, contract tests for semantics, and versioned topics for the occasional breaking change.

How should you set up subjects, CI checks and the registry in production?

Start with the subject naming strategy, because it decides what the registry compares a new schema against.

Strategy Subject derived from Lineage scope Fits when
TopicNameStrategy Topic name One lineage per topic A topic carries a single event type
RecordNameStrategy Fully qualified record name One lineage per record, across all topics Several event types share topics, and a record means the same thing everywhere
TopicRecordNameStrategy Topic plus record name One lineage per record per topic Several event types share a topic, and each topic evolves independently

Choose deliberately. Switching strategies later changes every subject name, so you would have to re-register history under the new subjects and coordinate the change across all producers and consumers.

Set compatibility explicitly on each subject with PUT /config/{subject}. A global default is easy to change without anyone noticing, and a per-subject setting documents your intent.

In production, set auto.register.schemas=false on producers. If producers can register schemas, an untested schema becomes canonical the first time someone deploys it. Register schemas from CI after merge instead. In the pull request pipeline, call POST /compatibility/subjects/{subject}/versions/latest, or .../versions for transitive checks. A failure then appears in review, where the fix costs only an edit.

Treat the registry as a runtime dependency. Serializers cache schemas, but a freshly started producer or a consumer that meets an unseen schema ID must still reach the registry. Run it with redundancy and back up its storage, since the schema history is the contract.

Illustrative estimate: assume six incompatible changes a year reach production. Assume each costs four engineers three hours to diagnose, roll back and replay. That is 72 hours. The same failures caught in pull requests cost a few minutes each.

#!/usr/bin/env bash
set -euo pipefail

REGISTRY_URL="${REGISTRY_URL:?set REGISTRY_URL to the registry base URL}"
SUBJECT="orders-value"            # TopicNameStrategy: <topic>-value
SCHEMA_FILE="schemas/order.avsc"
CT="Content-Type: application/vnd.schemaregistry.v1+json"

PAYLOAD=$(jq -n --rawfile s "$SCHEMA_FILE" '{schema: $s}')

case "${1:-}" in
  check)
    # Pull request stage: compare against all registered versions (transitive).
    # Use .../versions/latest to compare against the latest version only.
    # A subject with no versions yet returns an error; register it first.
    RESULT=$(curl -sf -X POST -H "$CT" --data "$PAYLOAD" \
      "$REGISTRY_URL/compatibility/subjects/$SUBJECT/versions")
    if [ "$(echo "$RESULT" | jq -r '.is_compatible')" != "true" ]; then
      echo "Incompatible schema for $SUBJECT: $RESULT" >&2
      exit 1
    fi
    echo "Compatible: $SUBJECT"
    ;;
  register)
    # Post-merge stage: pin compatibility on the subject itself, not the global default.
    curl -sf -X PUT -H "$CT" --data '{"compatibility": "BACKWARD_TRANSITIVE"}' \
      "$REGISTRY_URL/config/$SUBJECT" > /dev/null
    # CI is the only writer. Producers run with auto.register.schemas=false.
    curl -sf -X POST -H "$CT" --data "$PAYLOAD" \
      "$REGISTRY_URL/subjects/$SUBJECT/versions"
    echo
    ;;
  *)
    echo "usage: $0 check|register" >&2; exit 2 ;;
esac

A CI script that checks a schema for transitive compatibility during pull requests and, after merge, sets the subject's compatibility mode and registers the schema, so producers never need write access to the registry.

How do you ship a change that no compatibility mode allows?

Some changes fit no mode: renaming a field consumers rely on, changing a type, or splitting one event into two. Don't force these through the registry. Treat them as a planned data migration. Create a new topic, or a new subject or record name, so the new schema starts with no compatibility history. During the migration, either dual-write from the producer or run a translating consumer that reads the old stream and republishes in the new shape. Move consumers over one at a time, then retire the old stream once its last reader is gone.

This costs real engineering time. Dual writes are not atomic, so one stream can get an event the other misses. Ordering is guaranteed only within a partition of one topic, not across two. Consumers need idempotent handling, keyed on a stable event ID, to tolerate duplicates during cutover. Each consumer needs a cutover point, usually an offset or timestamp, recorded so it neither skips nor replays data. You also pay to store both topics until cleanup.

None of this ships a feature, so teams postpone it. They set the subject to NONE "temporarily," push the change, and the setting stays. From then on the registry stops checking anything on that subject.

Deploy order Retention profile Recommended mode Suits
Consumers first Short, no replay BACKWARD Most service-to-service events
Producers first Short, no replay FORWARD Many consumers you don't control
Either order Short FULL Independent team release cycles
Either order Long, compacted or replayed FULL_TRANSITIVE Event sourcing, audit logs

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to schema registry compatibility modes, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.