What do the Schema Registry compatibility modes actually check?

Schema Registry runs its check when a producer registers a new schema. It compares the candidate against earlier versions in the same subject and rejects it if the subject's compatibility mode is violated. Producer serializers register a schema on first use by default, so an incompatible change typically fails when a producer starts sending with it, before any record with that schema reaches the topic.

Check flow: producer submits schema → registry compares it with prior versions in the subject → on pass, the registry returns a schema ID and the producer writes ID + payload → consumer fetches the writer schema by ID and resolves it against its own reader schema.

Mode Guarantee Compared against Deploy first
BACKWARD New schema reads data written with the previous one Latest version Consumers
BACKWARD_TRANSITIVE New schema reads data from every earlier version All versions Consumers
FORWARD Previous schema reads data written with the new one Latest version Producers
FORWARD_TRANSITIVE Every earlier schema reads new data All versions Producers
FULL / FULL_TRANSITIVE Both directions Latest / all Either order
NONE No check — Coordinated release

BACKWARD is the usual default because consumers can upgrade first, and topics retain old records that new readers must still decode. The non-transitive modes only compare adjacent versions. Without the transitive variant, a long-retention topic can drift until v5 can no longer read v1.

The subject naming strategy decides which schemas are compared at all:

  • Topic name (<topic>-value) means one schema lineage per topic.
  • Record name uses the fully qualified type name, shared across topics.
  • Topic-record name combines the two, so one topic can carry several event types.

Avro named types defined in other files and Protobuf imports must be registered as schema references, or registration fails.

+------------+  1. register schema   +-------------------------+
|  Producer  |---------------------->|     Schema Registry     |
| serializer |                       | subject per naming      |
|            |<----------------------| strategy; check against |
+------------+  2. schema ID (pass)  | latest or all versions  |
      |            or reject         +-------------------------+
      | 3. write ID + payload                    ^
      v                                          | 5. fetch writer
+------------+                                   |    schema by ID
| Kafka topic|  4. poll records    +------------------------------+
| (retains   |-------------------->|  Consumer deserializer       |
|  old data) |                     |  resolve writer vs reader    |
+------------+                     +------------------------------+

Schema references: Avro named types and Protobuf imports are
registered as schema references before the schema that uses them.

Deployment order per mode:
  BACKWARD / BACKWARD_TRANSITIVE : [1] consumers --> [2] producers
  FORWARD  / FORWARD_TRANSITIVE  : [1] producers --> [2] consumers
  FULL     / FULL_TRANSITIVE     : either order
  NONE                           : no check, coordinated release

The registry checks each new schema against prior versions in its subject before issuing an ID, and consumers resolve the writer schema by ID; the compatibility mode determines deployment order.

How does Schema Registry work under the hood?

The registry is a separate service that stores schemas and exposes them over a REST API. Kafka brokers do not inspect payloads or validate them against schemas in this model. All schema handling happens in client-side serializers and deserializers, and the registry is the shared source of truth they consult.

Storage. Each registered schema gets a globally unique numeric ID. Within a subject, each distinct schema also gets an incrementing version number. If you register a schema that is identical to one already stored, the registry returns the existing ID instead of creating a new one. The registry commonly persists its state in a compacted Kafka topic, so it inherits Kafka's durability rather than needing its own database.

Producer side. When the serializer sees a record type for the first time, it computes the subject from the naming strategy, then either registers the schema (with auto-registration on) or looks up its ID (with auto-registration off). Before the ID is issued, the registry runs the compatibility check for that subject's mode. The serializer caches the ID locally, so later records of the same type do not call the registry.

Wire format. The serializer prefixes each payload with a small header: a magic byte, then the 4-byte schema ID. For Protobuf, the header also includes message indexes that identify which message type in the .proto file was used, because one file can define several messages. The record value is the header plus the encoded payload. The schema itself never travels with the record.

Consumer side. The deserializer reads the schema ID from the header, fetches the writer schema from the registry on first sight of that ID, and caches it. It then decodes the payload. For Avro, it resolves the writer schema against the reader schema using the rules described below. For Protobuf, it parses by tag number into the generated or dynamic message type.

Two consequences follow from this design. First, the registry is on the critical path only when a client encounters a schema or ID it has not cached, so short registry outages mostly affect new deploys and new schema versions rather than steady-state traffic. Second, a producer that bypasses the serializer, or uses a different wire header, can write records the registry never checked. Compatibility guarantees only hold for clients that go through the registry.

How does Avro schema evolution work?

Avro never encodes field names or tags in the binary payload. A record is just its field values in schema order, so a consumer can decode it only with the exact schema the producer used (the writer schema). Avro then resolves that data against the consumer's own reader schema, matching fields by name. Field order does not matter. Each change below is legal only if that name-based resolution still succeeds.

  • Adding a field is backward compatible only with a default. A new reader decoding old data finds the field missing and fills in {"name": "currency", "type": "string", "default": "USD"}. Without the default, resolution fails.
  • Removing a field is forward compatible only if the old field had a default. An old reader decoding new data looks for that field, doesn't find it, and needs the default it declared.
  • Renaming is a remove plus an add unless the reader declares the old name: {"name": "customer_id", "type": "string", "aliases": ["cust_id"]}.
  • Type promotions work only in the widening direction: int to long, and int or long to float or double. A reader of type long accepts int data, but a reader of type int rejects long data.
  • Enums: if the writer emits a symbol the reader lacks, decoding fails unless the reader's enum has a fallback such as "default": "UNKNOWN". Declare it before you need to add symbols.
  • Unions resolve by branch type. Removing a member breaks new readers of old data written with that branch. Adding a member breaks old readers when they receive the new branch. For an optional field, null must come first in the union (["null", "string"]) so that "default": null is valid.
{
  "type": "record",
  "name": "Order",
  "namespace": "com.example.orders",
  "doc": "Reader schema v2. Writer v1 had: order_id, cust_id (string), quantity (int), status (enum, no CANCELLED).",
  "fields": [
    {"name": "order_id", "type": "string"},
    {
      "name": "customer_id",
      "type": "string",
      "aliases": ["cust_id"],
      "doc": "Renamed from cust_id. The alias lets this reader resolve v1 data by the old name."
    },
    {
      "name": "quantity",
      "type": "long",
      "doc": "Was int in v1. Widening int to long is an allowed promotion. Narrowing back to int is not."
    },
    {
      "name": "currency",
      "type": "string",
      "default": "USD",
      "doc": "New field. The default fills it in when reading v1 data, which keeps the change backward compatible."
    },
    {
      "name": "status",
      "type": {
        "type": "enum",
        "name": "OrderStatus",
        "symbols": ["UNKNOWN", "PENDING", "SHIPPED", "CANCELLED"],
        "default": "UNKNOWN"
      },
      "doc": "Enum default: a symbol this reader does not know resolves to UNKNOWN instead of failing."
    },
    {
      "name": "note",
      "type": ["null", "string"],
      "default": null,
      "doc": "Optional field. null is listed first so that default null is valid."
    }
  ]
}

A v2 Avro reader schema that resolves v1 data by field name. It uses an alias for a renamed field, an int-to-long promotion, a defaulted new field, an enum default for symbols the reader does not know, and an optional union with null listed first.

How does Protobuf schema evolution work?

Protobuf identifies every field by its tag number. Names never appear on the wire, so renaming a field is invisible to consumers, while changing its number turns it into a different field. When an old reader encounters a tag it doesn't know, it keeps the bytes as unknown fields and preserves them on re-serialization in current runtimes. This is why adding fields is safe in both directions. When you delete a field, declare reserved 3, 7; and reserved "legacy_id"; so nobody reuses the number or name for a field with different meaning.

Type changes are more subtle. int32, uint32, int64, uint64 and bool are wire-compatible with each other, so widening int32 to int64 decodes without error. An old reader still parsing the field as int32 silently truncates values above 2^31. The sint32/sint64 pair and the fixed-width types form separate groups, and moving a field between groups corrupts values.

Proto3 complicates presence. With implicit presence, a field set to 0, "" or false is not serialized, so a consumer cannot distinguish "unset" from "default." Declaring optional int32 retry_count = 4; adds explicit presence and is wire-compatible, so the registry accepts it. The generated code changes, though, and consumer logic that relied on zero-means-absent behaves differently.

Proto3 enums are open, meaning unrecognized values pass through as raw integers. Consumers need a default branch in their handling logic. Moving several existing fields into a oneof is unsafe: if a producer had set more than one of them, a reader keeps only the last, and the other values are lost without any error.

// orders/v2/order.proto: evolved from v1 without breaking existing consumers
syntax = "proto3";

package orders.v2;

enum OrderStatus {
  // Zero value is the implicit default; keep it meaningless so "unset" is detectable.
  ORDER_STATUS_UNSPECIFIED = 0;
  ORDER_STATUS_PLACED = 1;
  ORDER_STATUS_SHIPPED = 2;
  // Added in v2. Old readers see the raw integer 3 (open enum) and need a default branch.
  ORDER_STATUS_REFUNDED = 3;
}

message Order {
  // v1 had: string legacy_id = 3; and string coupon = 7;
  // Reserve both the numbers and the names so neither is reused with a new meaning.
  reserved 3, 7;
  reserved "legacy_id", "coupon";

  string order_id = 1;
  OrderStatus status = 2;

  // Was int32 in v1. Wire-compatible, so the registry accepts it, but a v1 reader
  // still decoding as int32 silently truncates values above 2^31 - 1.
  int64 amount_cents = 5;

  // Was `int32 retry_count = 4;` (implicit presence: 0 is never serialized).
  // Adding `optional` gives explicit presence and generates has_retry_count().
  // Same wire format, but consumers that treated 0 as "absent" must be updated.
  optional int32 retry_count = 4;

  // New field with a new number: old readers keep it as an unknown field.
  string customer_note = 6;

  // Do NOT wrap existing fields (e.g. 1 and 5) in a oneof: if a producer set both,
  // a reader keeps only the last one parsed and drops the other without error.
}

A v2 Order message that reserves removed field numbers and names, widens an integer, adds explicit presence with optional, and extends an open enum, with comments marking where consumers can still observe a change.

Which schema changes pass or fail in Avro vs Protobuf?

Change Avro BACKWARD Avro FULL_TRANSITIVE Protobuf BACKWARD Protobuf FULL_TRANSITIVE
Add field Pass only with a default Pass only with a default Pass Pass
Remove field Pass Pass only if the field had a default Pass Pass
Rename field Fail unless an alias is added Fail Pass (only tags reach the wire) Pass
Change type Pass for promotions such as int→long Fail, because promotion only works one way Pass only for wire-compatible scalars Same, against every version
Add enum value Pass Fail unless the enum declares a default Pass Pass
Add union branch / oneof member Pass Fail Pass for a new field Pass for a new field
Reuse tag or name Checked against latest only Caught if the type conflicts Caught only if latest held the tag Caught if the wire type conflicts

Avro matches fields by name and fills gaps with defaults, so defaults drive most outcomes. Protobuf matches fields by tag and skips unknown fields, so most changes pass. The tag history is where Protobuf fails.

One Order event, three revisions

Avro. v1 has id, amount (int) and status, with no defaults.

  • v2 adds currency with a default. Both modes accept it.
  • v3 removes status. BACKWARD accepts it. FULL_TRANSITIVE rejects it, because v1 readers have no default to fill in.
  • v4 widens amount to long. BACKWARD accepts it. FULL_TRANSITIVE rejects it.

Protobuf. v1 has status as an enum at tag 3.

  • v2 adds currency at tag 4. Both modes accept it.
  • v3 deletes tag 3 without marking it reserved. Both modes accept it.
  • v4 adds a string note at tag 3. BACKWARD compares only against v3, so it accepts the change. FULL_TRANSITIVE also checks v1, where tag 3 is a varint enum and not a length-delimited string, so it rejects the change.
#!/usr/bin/env bash
set -euo pipefail
SR="${SCHEMA_REGISTRY_URL:?set SCHEMA_REGISTRY_URL to your Schema Registry base URL}"
SUBJECT=orders-value
CT='Content-Type: application/vnd.schemaregistry.v1+json'

V4='syntax = "proto3";
message Order {
  string id = 1;
  int64 amount = 2;
  string note = 3;
  string currency = 4;
}'
PAYLOAD=$(jq -n --arg s "$V4" '{schemaType: "PROTOBUF", schema: $s}')

curl -s -X PUT -H "$CT" -d '{"compatibility": "BACKWARD"}' "$SR/config/$SUBJECT"
curl -s -X POST -H "$CT" -d "$PAYLOAD" \
  "$SR/compatibility/subjects/$SUBJECT/versions/latest?verbose=true"
curl -s -X PUT -H "$CT" -d '{"compatibility": "FULL_TRANSITIVE"}' "$SR/config/$SUBJECT"
curl -s -o /dev/null -w '%{http_code}\n' -X POST -H "$CT" -d "$PAYLOAD" \
  "$SR/subjects/$SUBJECT/versions"

This script uses the Schema Registry REST API to show Protobuf Order v4 passing the BACKWARD check against the latest version and failing registration under FULL_TRANSITIVE because tag 3 is reused.

How do you enforce compatibility in CI and production?

Set the mode explicitly on every subject with PUT /config/{subject} and a body such as {"compatibility": "FULL_TRANSITIVE"}. If you rely on the global default, a later change to that default silently changes the rules for every subject that never overrode it. Check what a subject actually enforces with GET /config/{subject}.

In CI, send each candidate schema to POST /compatibility/subjects/{subject}/versions/latest and fail the build when the response contains "is_compatible": false. If your build already uses the registry's Maven or Gradle plugin, run its compatibility-test goal so you don't maintain a second check. Either way, the change from the previous section fails in a pull request, where the fix is to pick a new tag. Otherwise it fails after deploy, when consumers start throwing deserialization errors, someone has to page the topic owner, and bad records may already be sitting in the log for every downstream team to handle.

In production, set auto.register.schemas=false on producers. With auto-registration on, any service holding write credentials can register a schema the first time it sends a record, and that skips review entirely. Register schemas from the pipeline only, and give each subject a named owning team that approves changes.

Match the rollout order to the mode. Under BACKWARD, deploy consumers first, because new readers can handle old data. Under FORWARD, deploy producers first. FULL lets you deploy in either order. Transitive modes don't change the order. They only extend the check to every prior version.

How can Schema Registry save engineering time?

The time savings come from moving schema failures earlier and removing coordination work that would otherwise be manual.

  • Failures move from production to the pull request. A rejected compatibility check costs the author a few minutes to fix the schema. The same change reaching production costs an incident: consumers fail to deserialize, someone pages the topic owner, the producer is rolled back, and records already written with the bad schema must be skipped, replayed, or repaired by every downstream team.
  • Teams stop negotiating every change. With a mode set per subject, a producer team knows which changes are safe to ship without asking consumers first. Additive changes under BACKWARD or FULL need no cross-team meeting or synchronized release.
  • Deployment order becomes a rule, not a judgment call. The mode tells you whether consumers or producers go first, so release planning for schema changes is mechanical.
  • No hand-maintained schema distribution. Consumers fetch writer schemas by ID, so nobody has to copy schema files between repositories or keep a shared library version in lockstep to decode old records. Generic consumers such as connectors and stream processors can decode records without per-topic code.
  • Debugging starts with facts. When a record misbehaves, its schema ID points to the exact writer schema and version, instead of guessing which producer build wrote it.

Illustrative estimate. The inputs below are assumed, not measured. Assume a platform with 30 subjects sees 4 incompatible schema changes per quarter that would otherwise reach production. Assume each production incident takes 6 engineer-hours across the producer team, the topic owner and affected consumers, and that catching the same change in CI takes 0.5 engineer-hours. Under those assumptions, the cost drops from 4 × 6 = 24 engineer-hours to 4 × 0.5 = 2 engineer-hours per quarter, before counting coordination meetings avoided for safe additive changes. Substitute your own incident rate and per-incident cost. The estimate is only as good as those inputs.

The savings depend on the enforcement described above. With auto-registration on and modes left at defaults, the registry records schemas but does not stop bad changes, and most of the benefit disappears.

When should you choose Avro, Protobuf, or neither?

Start by asking how each format behaves when someone makes a mistake. Avro's resolution rules block an incompatible change at registration time, so accidental breakage rarely reaches a topic. The price is ongoing schema clutter: every added field needs a default, and optional fields become ["null", ...] unions with placeholder values that consumers must handle. Protobuf accepts almost any change on the wire, which makes shipping easy. It also means that reusing a deleted tag, or narrowing int64 to int32, can pass the registry check and then silently misread or truncate data. Teams that choose Protobuf need reserved discipline and review, because the format will not catch these errors for them.

Factor Avro Protobuf
Typical failure mode Change rejected loudly Change accepted, data corrupted quietly
Schema hygiene cost Defaults and nullable unions everywhere Reserved tags, reviewer vigilance
Generic/dynamic consumers Strong: writer schema drives decoding Possible, less natural
Cross-language codegen, RPC reuse Adequate Strong, shares types with gRPC

Choose Protobuf when your services already share .proto contracts with gRPC and generate code in several languages. Choose Avro when connectors, stream processors, or data-lake sinks read records generically without compiled classes.

Neither format may be worth the effort for short-lived internal topics that one team owns end to end. For those, NONE compatibility and a coordinated deploy cost less than maintaining schemas. The registry also supports JSON Schema, which suits teams that already validate JSON payloads. Its compatibility checks depend on whether the schema allows additional properties, so make that choice deliberately rather than leaving it at the default.

When should you avoid Schema Registry?

The registry adds a service to operate, a dependency in every client, and a review process for schema changes. That overhead is not always justified.

  • One team owns producer and consumers. If the same team deploys both sides together and the topic has short retention, a compatibility check mostly duplicates what the team's own tests and release process already cover.
  • Payloads are opaque. If the topic carries blobs that consumers pass through without decoding, such as files, compressed archives, or encrypted payloads, there is no field structure for the registry to check.
  • You cannot operate it reliably. Clients cache schemas, but a new producer version, a new consumer instance, or a consumer meeting an uncached ID needs the registry to be reachable. If you cannot run it with the same availability as the brokers, it becomes a source of startup and deploy failures.
  • Not all clients can use the serializers. If some producers or consumers run in languages or environments without registry-aware serializers, they must implement the wire header themselves or bypass the registry. Records from bypassing clients are never checked, which gives a false sense of safety.
  • Schemas change constantly and nobody owns them. If every change would be forced through with NONE because no one reviews schemas, the registry stores versions but enforces nothing. Fix ownership first, or accept that you are using it only as a schema lookup service.
  • Topics are throwaway. Scratch topics for experiments, load tests, or one-off migrations rarely repay the setup and review cost.

In these cases, a schema file versioned alongside the code, contract tests between producer and consumer, or a coordinated release is usually cheaper. Revisit the decision when a second team starts consuming the topic or retention grows long enough that old records outlive the code that wrote them.

What can the registry not protect you from?

The registry compares schemas structurally. It cannot tell whether two schemas mean the same thing, and its guarantees depend on which versions it compares against.

Semantic changes pass every mode. Suppose amount changes from cents to dollars, or a Protobuf field 3 that held a user ID now holds an account ID. Both schemas look identical to the checker, so the change is registered without complaint. Every consumer then misreads the data without raising an error. Only code review and contract tests that assert on values can catch this.

Non-transitive modes compare only against the latest version. BACKWARD checks the new schema against the previous one, not against every schema still present in your data. A consumer replaying a compacted or long-retention topic can hit records written with version 2 while holding version 5. That works only if every step in between happened to stay compatible with version 2. If replay matters, use transitive modes on those subjects.

Transitive modes have a cost. Under transitive modes, a field removed years ago can still block a reasonable change today. Some teams respond by turning compatibility off, which is worse than having no check. Decide per subject, based on how far back consumers actually read.

Mixed formats need separate checklists. Default values, field numbers, and enum handling differ between Avro and Protobuf. A reviewer applying Avro instincts to a .proto diff will miss problems specific to Protobuf.

Format migrations are invisible to the registry. The registry has no rule for comparing an Avro schema with a Protobuf one. To switch formats, dual-write to both formats or move to a new topic. Keep the old format running until every consumer has switched to the new one.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kafka schema registry avro vs protobuf, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.