Schema Registry Compatibility Modes: Avro vs Protobuf Events

What a schema registry checks, and why event-driven systems use one

A schema registry is a service that stores schemas and decides whether a new one may be added. The Confluent Schema Registry is the common reference model, and most alternatives copy its concepts.

Schemas are organized under subjects. By default, a subject is derived from the topic name, such as orders-value for the value schema of the orders topic. Each subject holds an ordered list of versions. Each distinct schema also receives a globally unique numeric ID. Registering a schema that already exists under the subject returns the existing ID instead of creating a new version.

Producers do not embed the schema in each message. They prefix the payload with a small header:

• A magic byte (0).
• A 4-byte big-endian schema ID.
• For Protobuf only, a list of message indexes that identifies which message type in the .proto file was serialized.

Consumers read the ID, fetch the schema once, and cache it. Payloads stay small, and every message can be traced back to an exact schema.

When a new schema is registered, the registry checks it against earlier versions in the subject, using the compatibility mode configured for that subject. If the check fails, registration fails. A producer configured to register schemas on first use therefore cannot publish with an incompatible schema.

The guarantee has a hard limit. The registry checks structure, not meaning. Changing amount from cents to dollars passes every check.

The registry is most valuable when producers and consumers are owned and deployed by different teams. In that setup, nobody can upgrade both sides in one release, and the registry is the only shared place where the data contract is enforced.

Copy Team A: owns + deploys producer      Team B: owns + deploys consumer
+------------+            +--------------+            +------------+
|  Producer  |--produce-->| topic orders |--consume-->|  Consumer  |
+------------+            +--------------+            +------------+
   |      ^                                              |      ^
  (1)    (2)                                            (5)    (6)
   v      |                                              v      |
+------------------------------------------------------------------+
|                         Schema Registry                          |
| subject: orders-value   (default name: <topic>-value)            |
|   version 1 -> ID 101    version 2 -> ID 102   (IDs are global)  |
|   existing schema re-registered -> same ID, no new version       |
| on register: compare to earlier versions using subject's mode    |
|   fail -> registration rejected -> producer cannot publish       |
| checks STRUCTURE, not meaning: cents -> dollars still passes     |
+------------------------------------------------------------------+

 Each message on the topic (wire format):
+------+-------------------+--------------------+------------------+
| 0x00 | schema ID, 4 bytes| message indexes    | payload bytes    |
| magic| big-endian        | (Protobuf only)    | (Avro/Protobuf)  |
+------+-------------------+--------------------+------------------+

 (1) register schema      (2) schema ID, or rejection if incompatible
 (5) fetch schema by ID   (6) schema, cached by the consumer

Separately deployed producers and consumers share only the topic and the schema registry: the registry versions schemas per subject and rejects structurally incompatible registrations, and each message carries a magic byte, a 4-byte schema ID, and (for Protobuf) message indexes.

Compatibility modes as deployment contracts

Each mode describes what a new schema version must be able to do relative to earlier versions, which determines which side of the topic can be deployed first.

Transitive vs non-transitive

The plain modes compare a new version only against the latest registered version. Each step can pass while the full history becomes inconsistent. Here is an Avro example under BACKWARD:

• v1 has id.
• v2 adds currency with a default of "USD". v2 can read v1 records because the default fills the gap, so the check passes.
• v3 removes the default from currency. v3 can read v2 records because they all contain currency, so the check passes again.

v3 cannot read v1 records, because they have no currency and v3 no longer provides a default. The *_TRANSITIVE variants check the new version against every registered version, so they would reject v3.

Use a transitive mode whenever old records stay readable:

• Topics with long retention
• Compacted topics
• Consumers that reprocess from offset zero
• New consumers that bootstrap from history

Changing a subject's mode does not revalidate versions that are already registered. If you loosen a subject to NONE, register a breaking version, and then switch to FULL_TRANSITIVE, the history still contains a change that FULL_TRANSITIVE would have rejected.

How Avro and Protobuf reach 'compatible' through different mechanisms

Avro decodes nothing on its own. The reader resolves the writer schema against its own reader schema and matches fields by name, so a consumer needs the exact writer schema at read time. That is why the registry's schema ID travels in every message. Defaults make most evolution work. A field the reader expects but the writer omitted is filled from its default, and fields the reader doesn't know are dropped. Optional fields are a union with null, such as {"name": "currency", "type": ["null", "string"], "default": null}, and a rename survives only if the reader declares "aliases": ["curr"].

Protobuf identifies fields by number. Field names are absent from the wire, so renaming is free. Parsers skip unknown fields, and the generated code in most languages preserves them on re-serialization. proto3 has no required fields, so removal never breaks parsing. The real hazards are reusing a field number with a different meaning and changing a field's wire type, for example int32 to string. Both corrupt data silently instead of failing. The defense is reserved 4; reserved "legacy_id"; on every deleted field.

Enums show the divergence most clearly. An Avro reader that encounters an unknown symbol fails unless its enum declares "default": "UNKNOWN". A proto3 reader keeps the unknown value as its integer, so your code must handle a value it has no name for, plus the zero value STATUS_UNSPECIFIED = 0.

Copy// ---- payment.avsc (Avro reader schema; // lines are annotations, not JSON) ----
{
  "type": "record", "name": "Payment", "namespace": "com.example.payments",
  "fields": [
    {"name": "id", "type": "string"},
// Optional field: union with null plus a default. The alias matches the old writer field "curr".
    {"name": "currency", "type": ["null", "string"], "default": null,
     "aliases": ["curr"]},
// Without the enum-level default, an unknown writer symbol fails resolution.
    {"name": "status", "type": {"type": "enum", "name": "Status",
      "symbols": ["UNKNOWN", "PENDING", "SETTLED"],
      "default": "UNKNOWN"}}
  ]
}

// ---- payment.proto (Protobuf, proto3) ----
syntax = "proto3";
package example.payments;

message Payment {
  // Field 4 (legacy_id) was deleted. Reserve its number and name so they are never reused.
  reserved 4;
  reserved "legacy_id";

  string id = 1;
  string currency = 2;   // Renaming is free because only the number 2 goes on the wire.
  Status status = 3;     // Never change this to a different wire type, such as string.
}

enum Status {
  STATUS_UNSPECIFIED = 0;  // Required zero value, also read when the field is absent.
  STATUS_PENDING = 1;
  STATUS_SETTLED = 2;
  // If a newer writer sends STATUS_REFUNDED = 3, this reader keeps the raw integer 3.
  // Application code must handle a value it has no name for.
}

The same Payment event evolved in both formats: Avro uses a null union, an alias and an enum default, while proto3 uses stable field numbers, reserved tags and an UNSPECIFIED zero value.

Where each format fails, and when to avoid relying on the check

Every compatibility mode checks structure, not meaning. If amount changes from cents to dollars, status gains a new interpretation, or a field that was always populated starts arriving empty, the change passes every mode, including FULL_TRANSITIVE. The registry cannot catch semantic drift. Contract tests and explicit versioning in field names (amount_cents) can.

The two formats also fail differently when a field is removed. In Protobuf, removing a field typically passes the check. A consumer that still expects the field gets the type's zero value, and nothing errors. A missing discount reads as 0, which looks like valid data. In Avro, removing a field that has no default fails the check under FORWARD or FULL. If that change reaches a reader anyway, schema resolution fails and the reader gets a decode error. Avro breaks loudly. Protobuf breaks quietly.

The formats also differ in how much they depend on the registry at runtime. An Avro consumer needs the writer schema to decode anything. That makes the registry a hard runtime dependency, which client-side schema caching softens but does not remove for schema IDs the consumer has not seen yet. A Protobuf message can be parsed without the registry, but you only recover field numbers and wire types, not names or semantic types.

The registry's rules are an approximation of each format's native resolution behavior, not a perfect copy. Test your actual change patterns against it rather than assuming.

FULL_TRANSITIVE tends to push schemas toward fields that are all optional and all defaulted. That keeps the check green, but it moves required-field validation into application code.

NONE or a relaxed mode is the wrong choice for:

• Topics with several independent consumers.
• Topics with long retention or replay.
• Any topic where you cannot coordinate producer and consumer deploys.

Subject naming strategy and multiple event types per topic

The subject is the unit the registry versions and checks. The subject name strategy decides how a serializer derives that subject from a topic and a schema, which determines what a new schema is compared against.

With the default, adding an OrderCancelled event to an orders topic looks like a breaking change to OrderCreated. That is why teams switch strategies. The switch is useful because it keeps related events on one topic, so they share a partition key and stay in order relative to each other. If you split them across topics, per-key ordering across event types is lost.

The flexibility has costs:

• Consumers must dispatch on type. Generic consumers and stream joins become harder because a topic no longer has a single shape.
• Ordering assumptions still depend on the key. Every event type on the topic must use the same partition key, or ordering breaks without any error.
• Governance becomes per type. Each subject has its own compatibility mode and owner. Under RecordNameStrategy, one type's history is shared across topics, so one team's change can block another team's producer.

In the producer serializer config, set value.subject.name.strategy to the chosen class and set auto.register.schemas=false. With auto-registration off, a producer cannot create a subject under the wrong strategy. Registration then has to go through the CI path described in the next section.

Copy# producer.properties: one 'orders' topic carrying OrderCreated and OrderCancelled
bootstrap.servers=broker1.example.internal:9092
schema.registry.url=

# Registry credentials (placeholders; load from a secret store in practice)
basic.auth.credentials.source=USER_INFO
basic.auth.user.info=orders-producer:CHANGE_ME

# Every event type on the topic uses the same key (order ID),
# so all events for one order land on one partition and stay in order.
key.serializer=org.apache.kafka.common.serialization.StringSerializer

# Avro value serializer. For Protobuf, use
# io.confluent.kafka.serializers.protobuf.KafkaProtobufSerializer instead.
value.serializer=io.confluent.kafka.serializers.KafkaAvroSerializer

# Subject = <topic>-<fully qualified record name>, for example
#   orders-com.example.orders.OrderCreated
#   orders-com.example.orders.OrderCancelled
# Each type is checked only against its own history on this topic.
value.subject.name.strategy=io.confluent.kafka.serializers.subject.TopicRecordNameStrategy

# Do not let the producer create subjects or versions at runtime.
# Schemas are registered by CI after a compatibility check, so a
# misconfigured strategy fails at send time instead of silently
# creating a new, unchecked subject.
auto.register.schemas=false

Producer serializer properties that put several event types on one topic using TopicRecordNameStrategy, with auto-registration disabled so only pre-registered, compatibility-checked schemas can be used.

Implementing compatibility checks in CI and handling breaking changes

That CI path has two jobs. First, it pins the mode explicitly per subject rather than inheriting the global default. It does this with curl -X PUT -H "Content-Type: application/vnd.schemaregistry.v1+json" --data '{"compatibility": "BACKWARD"}' $REGISTRY/config/orders-value. Second, on every pull request that touches a schema file, it asks the registry whether the candidate would be accepted. The request is curl -X POST -H "Content-Type: application/vnd.schemaregistry.v1+json" --data @candidate.json $REGISTRY/compatibility/subjects/orders-value/versions/latest. The payload carries the schema as an escaped string plus "schemaType": "PROTOBUF" for Protobuf, since Avro is the default. If the response contains "is_compatible": false, the build fails. The registration call to /subjects/orders-value/versions runs only after merge, from the pipeline's credentials. This catches the break at review time, not at a producer's first send.

When a change has to break

You have three workable options:

• New subject or topic. Create orders.v2, have producers dual-write to both topics, and move consumers over during a fixed migration window. Then retire the old topic. This is the most work, but every subject keeps its guarantee intact.
• Versioned event names. Introduce OrderPlacedV2 as a distinct record type. This works when the topic already carries multiple event types under a record-based subject strategy.
• Temporarily relaxing the mode. Set the subject to NONE, register the change, and restore the original mode.

The third option is the risky one. Compatibility checks only compare new versions against registered ones. They say nothing about records already in the log. After a relaxed registration, a consumer replaying from an early offset can hit data its current schema cannot read. Restoring the mode does not undo that. Leaving the mode relaxed permanently means every later change carries the same risk without any check.

Copy#!/usr/bin/env bash
# CI gate: pin the subject's mode and test a candidate schema before merge.
# Registration (POST /subjects/$SUBJECT/versions) runs only in the post-merge job.
# Producers in production run with auto.register.schemas=false, so this
# pipeline is the only writer to the registry.
set -euo pipefail

REGISTRY="${REGISTRY:-"
SUBJECT="orders-value"
SCHEMA_FILE="schemas/orders.proto"
CT="Content-Type: application/vnd.schemaregistry.v1+json"
AUTH=(-u "${SR_USER}:${SR_PASSWORD}")   # CI-scoped credentials, not producer ones

# 1. Pin the mode on the subject instead of inheriting the global default.
#    Idempotent, so it is safe to run on every build.
curl -sf -X PUT -H "$CT" "${AUTH[@]}" \
  --data '{"compatibility": "BACKWARD"}' \
  "$REGISTRY/config/$SUBJECT"

# 2. Wrap the schema file as an escaped JSON string. schemaType is required
#    for Protobuf; omitting it means Avro.
jq -Rs '{schema: ., schemaType: "PROTOBUF"}' < "$SCHEMA_FILE" > candidate.json

# 3. Ask the registry whether the candidate would be accepted. Nothing is registered.
RESULT=$(curl -sf -X POST -H "$CT" "${AUTH[@]}" \
  --data @candidate.json \
  "$REGISTRY/compatibility/subjects/$SUBJECT/versions/latest")

# 4. Fail the build on an incompatible change so the break surfaces in review.
if [ "$(echo "$RESULT" | jq -r '.is_compatible')" != "true" ]; then
  echo "Schema for $SUBJECT is not BACKWARD compatible:" >&2
  echo "$RESULT" >&2
  echo "Use a new subject/topic or a versioned event type instead of relaxing the mode." >&2
  exit 1
fi

echo "Candidate schema for $SUBJECT is compatible."

A CI script that pins a subject to BACKWARD and builds the Protobuf payload with jq. It checks the candidate against the latest registered version and fails the pull request if the registry reports it as incompatible, so registration is left to the post-merge pipeline.

Choosing a mode per subject: a worked walkthrough and its impact on engineering time

Choose a mode by answering four questions for each subject: which side deploys first, whether one team owns both sides, how far back consumers may replay, and how many consumers read the subject.

Registry checks have alternatives. Consumer-driven contract tests can verify semantic expectations that a structural check cannot see. Versioned topics avoid the compatibility check entirely, but they cost dual-publishing and migrating consumers.

Restrictive modes take engineering time up front. Teams have to review every field removal and wire the check into CI, and some changes become two-step migrations. The question is whether that time is less than the time spent on failures caught late.

Illustrative estimate (assumed numbers, not a benchmark): saved hours/year = breaking changes/year × P(reaches production) × hours per incident − CI and review hours/year

Assume 6 changes, a probability of 0.3, 16 hours per incident, and 10 hours of CI and review work. That gives 6 × 0.3 × 16 − 10 ≈ 19 hours saved per year. A rejected merge request takes minutes to fix. A failed replay can take a full day, which is why the incident-hours input dominates the formula.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to schema registry compatibility modes, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.