Who should use Schema Registry compatibility enforcement?

Compatibility enforcement matters most when the people who change a schema are not the people who read the data. The typical cases:

  • Platform teams running shared Kafka clusters. Many producer and consumer teams share topics, and no single team can coordinate every deploy. Registry-enforced compatibility gives the platform one rule that applies to everyone, so it does not depend on each team's discipline.
  • Producer teams with consumers they don't control. If analytics jobs, other product teams or external integrations read your topics, you can't upgrade them on your schedule. A compatibility mode tells you before merge whether a change will break them.
  • Consumer teams that replay history. Services that rebuild state from compacted topics, backfill a data lake, or reprocess from offset zero read records written under many schema versions. Transitive modes are what make that replay predictable.
  • Teams running change-data-capture or sink pipelines. Upstream table changes turn into schema changes on a topic, and downstream sinks fail when a change is incompatible. A check at registration time surfaces the problem at the source.

It is usually not worth the overhead for short-lived topics owned by one team, where producer and consumer deploy together and no data outlives a release. The last section covers that case.

How does it work?

In short, compatibility enforcement combines five mechanisms. The rest of this article covers each one in detail:

  1. Subjects hold version histories. Every schema is registered under a subject. The subject name strategy decides which subject that is, and so which schemas count as versions of one another.
  2. Each subject has a compatibility mode. The mode is set globally or per subject: BACKWARD, FORWARD, FULL, their transitive variants, or NONE. It defines what "compatible" means for that subject.
  3. The registry checks new schemas when they are registered. A candidate schema is compared against the latest version, or against all versions in transitive modes. If the check fails, the registry rejects it, so no producer can serialize with it. CI can run the same check before merge through the compatibility endpoint.
  4. Messages carry a schema ID, not the schema. The serializer writes a magic byte and a 4-byte ID ahead of the Avro payload.
  5. Consumers resolve writer against reader. The deserializer fetches the writer schema by ID, and Avro schema resolution maps it onto the consumer's reader schema. The registry's check exists to predict whether this step will succeed.

How does Kafka Schema Registry compatibility work in the data path?

A schema registry is a service that stores versioned schemas under named subjects and assigns each distinct schema a numeric ID. Teams use it as the contract between producers and consumers. Kafka brokers treat message values as opaque bytes, so without a registry nothing stops a producer from publishing data that downstream services can't decode. With the Confluent Schema Registry, the compatibility check happens when a schema is registered, before any message using it reaches a topic.

The flow of a single message

  1. Producer registers or looks up the schema. The Avro serializer sends its schema to the registry under a subject. If the schema is new, the registry checks it against existing versions using the subject's compatibility mode and rejects it if it fails. If it passes, or already exists, the registry returns its ID.
  2. Serializer embeds the ID. Each message starts with a magic byte, then the 4-byte schema ID, then the Avro-encoded payload. The full schema is not sent with every record. That keeps messages small, but it means a payload can't be decoded without the registry.
  3. Consumer resolves writer against reader. The deserializer reads the ID, fetches the writer schema, and uses Avro schema resolution to map it onto the consumer's own reader schema.

The registry is a runtime dependency

Both clients cache schemas and IDs, so steady-state traffic rarely calls the registry. A producer or consumer that starts cold during a registry outage has an empty cache, though. So does one that meets an ID it hasn't seen before. Either one will fail, which means the registry needs high availability: multiple instances behind a load balancer and durable backing storage.

        +----------------------------------------------------+
        | Schema Registry (HA)                               |
        | multiple instances behind a load balancer          |
        | durable backing storage                            |
        | subject -> versions -> schema ID                   |
        | compat check runs when a new schema registers      |
        +----------------------------------------------------+
(1) lookup  ^       |                           ^       |
or register |       v ID              (3) fetch |       v writer schema
+--------------------------+              +--------------------------+
| Producer                 |              | Consumer                 |
| Avro serializer          |              | Avro deserializer        |
| cache: schema -> ID      |              | cache: ID -> schema      |
|                          |              | resolve writer->reader   |
+--------------------------+              +--------------------------+
            |                                           ^
            | (2) [magic][4-byte ID][Avro payload]      |
            v                                           |
      +--------------------------------------------------------+
      | Kafka broker: topic partitions (values = opaque bytes) |
      +--------------------------------------------------------+

The producer registers or looks up its schema and gets back an ID, writes that ID into each message ahead of the Avro payload, and the consumer fetches the writer schema by ID and resolves it against its reader schema, so both clients depend on a highly available registry.

What does Avro schema resolution actually do?

Every Avro decode involves two schemas: the writer schema the producer serialized with (fetched from the registry by ID) and the reader schema the consumer was compiled or configured with. The Avro library resolves one against the other field by field, and compatibility checks predict whether that resolution will succeed. Three rules do most of the work:

  • Fields are matched by name, not position. Reordering fields is harmless. Renaming one is effectively a delete plus an add, unless the reader declares the old name as an alias.
  • Writer fields missing from the reader schema are skipped.
  • Reader fields missing from the writer schema take the reader's default. If there is no default, decoding fails.

Under Avro backward compatibility, the new schema must be able to read data written with the old one. That makes these changes safe:

  • Adding "discount_code" with "default": null and a ["null","string"] type is safe, because old records get the default.
  • Removing "legacy_flag" is safe, because the new reader skips it.

These changes are unsafe:

  • Adding a required "region" with no default fails, because old records have nothing to fill it with.
  • Changing "amount" from string to int fails. Only specific promotions resolve: int to long, float, or double; long to float or double; float to double; and string to and from bytes.

Enums are the trap. Adding a symbol such as REFUNDED passes a backward check, because the new reader knows every old symbol. An older consumer that receives REFUNDED cannot resolve it and fails, unless its reader schema declares an enum-level default. Support for enum defaults varies across Avro implementations and their versions. Check the library each consumer actually runs before you rely on one.

{
  "type": "record",
  "name": "Order",
  "namespace": "com.example.orders",
  "doc": "v2 reader schema. Backward compatible with v1 (order_id, amount, legacy_flag, status).",
  "fields": [
    {
      "name": "order_id",
      "type": "string",
      "doc": "Unchanged and matched by name, so its position in the list does not matter."
    },
    {
      "name": "amount",
      "type": "string",
      "doc": "Kept as string. Changing it to int would fail because string does not promote to int."
    },
    {
      "name": "discount_code",
      "type": ["null", "string"],
      "default": null,
      "doc": "New field. v1 records do not contain it, so the reader fills in the default null."
    },
    {
      "name": "status",
      "type": {
        "type": "enum",
        "name": "OrderStatus",
        "symbols": ["UNKNOWN", "PLACED", "SHIPPED", "REFUNDED"],
        "default": "UNKNOWN",
        "doc": "Enum-level default. If a later writer sends a symbol this reader lacks, it decodes as UNKNOWN, but only where the consumer's Avro library supports enum defaults."
      }
    }
  ]
}

A v2 Order schema that passes a backward compatibility check against v1: legacy_flag is removed and skipped on read, discount_code is added with a null default, amount keeps its type, and an enum default protects this reader from symbols added later.

Which compatibility mode should you choose: BACKWARD, FORWARD, FULL, or TRANSITIVE?

The registry's default is BACKWARD, but the right mode depends on who reads the topic and when.

Mode Compared against Guarantee Upgrade first
BACKWARD Latest version New schema can read data written with the previous one Consumers
BACKWARD_TRANSITIVE All versions New schema can read every version ever written Consumers
FORWARD Latest version Previous schema can read data written with the new one Producers
FORWARD_TRANSITIVE All versions Every older schema can read new data Producers
FULL / FULL_TRANSITIVE Latest / all Both directions Either
NONE Nothing No check Coordinated deploy

Choose by deployment pattern:

  • Independent consumer teams. If you cannot control when consumers upgrade, producers must ship changes that old readers tolerate. That points to FORWARD or FULL.
  • Producers that lag behind consumers. Many producer instances, edge devices, or slow release trains fit BACKWARD. Consumers deploy the new reader first.
  • Long-retention or compacted topics that get replayed. A new consumer rebuilding state reads data from every era. Use BACKWARD_TRANSITIVE at minimum.

Non-transitive modes check each change only against the latest version, so incompatibilities can build up across versions. Here is an example:

  • v1 has status as a string.
  • v2 drops status. This is backward compatible.
  • v3 re-adds status as an int with a default. This is compatible with v2.
  • A v3 reader replaying v1 records fails, because Avro does not promote string to int.

No individual check failed, but the topic's history is now unreadable to the current schema.

FULL_TRANSITIVE removes that risk: every change is safe for every reader in either deployment order. The cost is slower evolution. You can only add or remove fields that have defaults, and you can never change a type. Teams tend to respond by making every field optional with a null default. That weakens the contract the registry was supposed to enforce.

How do subject naming strategies change what gets compared?

Compatibility is never checked against a topic. It is checked against the latest version registered under a subject, or against every version in transitive modes. The subject name strategy decides which subject a schema lands in, so it decides which schemas the registry treats as versions of one another.

Strategy Subject name Check compares against Good fit
TopicNameStrategy (default) <topic>-value / <topic>-key Every schema written to the topic One event type per topic
RecordNameStrategy Fully qualified record name That record type across all topics One event type reused on many topics
TopicRecordNameStrategy <topic>-<record name> That record type within that topic Topics carrying several event types

The default breaks down on topics that carry several event types. Registering OrderShipped after OrderCreated under orders-value compares two unrelated records, and the check fails. Teams often respond by setting the subject to NONE, which removes the check entirely. TopicRecordNameStrategy gives each event type its own version history on that topic. Consumers must then dispatch on the writer schema rather than assume a single type.

Producers select the strategy with the serializer properties value.subject.name.strategy and key.subject.name.strategy, set to the strategy's fully qualified class name. Every producer writing to a topic must use the same setting. If they don't, one event type's history splits across two subjects, and neither subject sees the full set of versions.

Per-subject mode overrides let a team relax rules for its own events, but they fragment the platform's guarantees. Whether a topic is safe to replay from offset zero then depends on three things: the strategy each producer used, the mode on each subject, and whether that mode ever changed. You can no longer answer it from the topic name. Keep an inventory of subject configs, or the replay question becomes an investigation every time.

bootstrap.servers=kafka-1.internal.example:9092
schema.registry.url=

key.serializer=org.apache.kafka.common.serialization.StringSerializer

value.serializer=io.confluent.kafka.serializers.KafkaAvroSerializer

value.subject.name.strategy=io.confluent.kafka.serializers.subject.TopicRecordNameStrategy

auto.register.schemas=false

Producer serializer configuration that switches a multi-event topic from the default TopicNameStrategy to TopicRecordNameStrategy, with comments showing the subject each record type is registered under and what its compatibility check compares against.

How do you enforce schema compatibility in CI before production?

Treat the .avsc files in version control as the source of truth and the registry as a deployment target. A schema change then becomes a pull request. Reviewers see the diff, and the pipeline can block the merge before any producer serializes a record with the new schema.

Check against the registry, not a local copy

On each pull request, the CI job sends the candidate schema to the registry's compatibility endpoint for the target subject (POST /compatibility/subjects/{subject}/versions/latest, or a specific version). The registry evaluates the candidate under that subject's configured mode, including transitive checks. This catches drift that a local diff against the previous file would miss, such as a version registered by hand or a mode someone changed. Fail the build if the response reports the schema as incompatible.

Register from the pipeline only

After merge, a deploy step registers the schema. In production producers, set auto.register.schemas=false, and typically use.latest.version=true. A producer then cannot introduce an unreviewed schema at runtime. If its schema was never registered, serialization fails at startup instead of silently creating a new version that consumers can't read. Restrict registry write credentials to the pipeline.

Cost

Illustrative estimate, not a measured result. Assume the setup takes about two engineer-days and review adds about 15 minutes to each of 40 schema changes per year, for roughly 26 hours in total. If one breaking change a year would otherwise reach production and cost three engineers a day of incident response, replay, and consumer fixes (24 hours), the controls roughly break even in the first year. They cost less after that, because the setup is a one-time expense.

#!/usr/bin/env bash
set -euo pipefail

MODE="$1"; SUBJECT="$2"; SCHEMA_FILE="$3"
REGISTRY_URL="${REGISTRY_URL:?REGISTRY_URL must be set to the Schema Registry endpoint}"
AUTH="${SR_USER}:${SR_PASSWORD}"
CT='Content-Type: application/vnd.schemaregistry.v1+json'

BODY=$(jq -n --rawfile s "$SCHEMA_FILE" '{schema: $s}')

if [ "$MODE" = "check" ]; then
  # Test against what is actually registered, under the subject's configured mode.
  STATUS=$(curl -s -o resp.json -w '%{http_code}' -u "$AUTH" -H "$CT" -X POST \
    --data "$BODY" \
    "$REGISTRY_URL/compatibility/subjects/$SUBJECT/versions/latest?verbose=true")

  if [ "$STATUS" = "404" ]; then
    # New subject: nothing to be compatible with. Reviewers approve the first version.
    echo "Subject $SUBJECT not registered yet; skipping compatibility check."; exit 0
  fi
  [ "$STATUS" = "200" ] || { echo "Registry error ($STATUS):"; cat resp.json; exit 2; }

  if [ "$(jq -r '.is_compatible' resp.json)" != "true" ]; then
    echo "INCOMPATIBLE schema for $SUBJECT:"; jq -r '.messages[]?' resp.json
    exit 1   # Fail the build so the merge is blocked.
  fi
  echo "$SCHEMA_FILE is compatible with $SUBJECT."

elif [ "$MODE" = "register" ]; then
  # Only the pipeline registers; producers run with auto.register.schemas=false.
  curl -sf -u "$AUTH" -H "$CT" -X POST --data "$BODY" \
    "$REGISTRY_URL/subjects/$SUBJECT/versions" | jq -r '"Registered id \(.id)"'
else
  echo "Unknown mode: $MODE" >&2; exit 2
fi

A CI script that checks a version-controlled .avsc file against the registry's compatibility endpoint on pull requests, failing the build if it is incompatible, and registers the schema from the pipeline after merge.

How can it save engineering time?

Most of the savings come from moving failures earlier, to the point where they are cheapest to fix, and from removing coordination that would otherwise be manual:

  • Breaking changes fail in review, not in production. An incompatible schema fails the CI check on the pull request. The author sees the registry's error message and fixes the change in the same branch. Without the check, the same mistake usually shows up as deserialization errors in a consumer owned by another team. Someone then has to trace the failing records back to a producer deploy, roll it back or patch consumers, and possibly replay or skip the bad records.
  • Safe changes need less cross-team negotiation. Once a subject has a mode, "is this change safe to ship?" has a mechanical answer. A producer team adding an optional field with a default doesn't need sign-off from every consumer team, and the rollout order follows from the mode (see the next section).
  • Replays stop being archaeology. On a subject with a transitive mode applied from its first version, a new consumer can read from offset zero, because every version in history has been checked against the others. Without that guarantee, someone has to work out which historical shapes exist before a rebuild or backfill.
  • Consumers don't hand-write version handling. Avro resolution fills defaults and skips removed fields, so consumer code works with one reader schema instead of branching on payload versions. That code would otherwise need its own tests and maintenance.
  • Poison-pill incidents become rarer. A record a consumer cannot decode can block a partition and require manual offset handling. Blocking incompatible schemas at registration removes a common source of such records. It does not remove all of them, because semantic changes still pass the check.

The time this saves depends on how often schemas change and how many teams share topics. The Cost section above has an illustrative estimate with assumed inputs. Run it with your own change rate and incident history rather than treating it as a typical result.

What is a safe rollout order for a schema change?

Rollout order follows from a simple question: which side must already understand the new schema before the first message using it appears?

Under BACKWARD, the new schema must be able to read data written with the previous one. Roll out in this order:

  1. Register the new schema version, or let the CI check from the previous section do it on merge.
  2. Deploy every consumer with the new reader schema. Those consumers keep reading old-format records correctly.
  3. Deploy producers that write the new format.
  4. Remove consumer code paths for the old shape only after retention or replay windows have expired.

If you reverse steps 2 and 3, an old consumer receives records it was never checked against.

Mode Deploy first Why
BACKWARD Consumers New readers handle old data
FORWARD Producers Old readers handle new data
FULL Either side Both directions are checked

The _TRANSITIVE variants apply the same rule against every prior version, not just the latest. That matters for consumers that replay from the start of a topic.

The checker validates structure, not meaning. Some changes pass and still break consumers:

  • Defaults: A consumer that receives a default discount of 0 for old records computes wrong totals without raising an error.
  • Alias renames: An alias-based rename resolves cleanly, but downstream code may treat the field differently.
  • Unit changes: A long that changes from milliseconds to microseconds is still a long.
  • Enum meanings: An enum symbol that is redefined keeps its name but changes what it means.

Cover these gaps with other practices. Document units and semantics in each field's doc attribute, and review changes to those descriptions with the same care as type changes. Add consumer-driven contract tests that run producer output through real consumer logic and assert on computed values.

When should you avoid compatibility rules, and what are the alternatives?

Some changes no compatibility mode will accept: renaming a field, changing string to int, or splitting one record into two. Don't force these through. Use one of these alternatives instead:

  • Add a new field with a default and deprecate the old one. Readers switch at their own pace.
  • Create a new topic or subject for the new shape. The old history stays valid.
  • Dual-write during migration. Producers emit both shapes until every consumer has moved.
  • Retire the old path deliberately. Confirm there is no remaining lag, wait out retention, then delete the topic.

Setting a subject to NONE for one registration is the tempting shortcut. It is also the worst option. The incompatible version stays in the subject's history permanently. When you restore a transitive mode, every later registration is checked against that version and can fail. Consumers that replay from the start of the topic will also reach records they cannot resolve, often months after the people who made the change have moved on.

Strict enforcement adds little value for short-lived or ephemeral topics owned by a single team, where producer and consumer deploy together and data never outlives a release. NONE is reasonable there, provided you treat it as a property of that topic rather than an exception.

Approach Strength Trade-off
Avro on the registry Mature schema resolution, compact encoding Readers need the writer schema
Protobuf on the registry Field numbers make renames safe Different evolution rules to learn
JSON Schema on the registry Readable payloads Larger messages, subtler compatibility semantics
Versioned event types, no registry No central dependency Every consumer handles versions by hand

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kafka schema registry compatibility, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.