Schema Registry Compatibility in CI for Avro and Protobuf

How schema registry compatibility modes work

A schema registry stores versioned schemas under named subjects and, before accepting a new version, checks it against earlier versions using that subject's compatibility mode. Teams use it so that a producer cannot publish data that existing consumers fail to deserialize. The check runs before registration, which makes it a natural CI gate.

BACKWARD means consumers on the new schema can read data written with the previous one. It is the common default in Confluent Schema Registry because it fits a typical rollout: consumers deploy first, then producers start emitting the new shape. It also supports replaying a topic from the beginning with current code.

BACKWARD does not protect a consumer that is still on an old schema when producers move ahead. Non-transitive modes compare only against the latest version. A consumer pinned to version 3 can therefore break on version 6, even though every individual step passed. Transitive modes close that gap. On long-lived subjects with many versions, however, they can block reasonable changes because one old version constrains everything that follows.

A passing check proves structural compatibility, not semantic safety. If you change a field from milliseconds to seconds, give an amount a different currency, or reinterpret an enum value, the check still passes and consumers still break.

Subjects, topics and why checking the wrong subject gives false confidence

The registry stores compatibility history per subject, not per topic. Which subject a producer writes to is decided by its serializer's subject-name strategy, and that setting lives in client configuration rather than anywhere the CI job naturally sees.

Suppose the producer uses RecordNameStrategy, but the CI script derives subjects as <topic>-value. It tests the change against orders-value, which has no versions or holds an unrelated history. A subject with no history has nothing to reject, so the check passes. The real subject, com.acme.OrderCreated, would have refused the change, and the first failure appears at deploy time when the producer tries to register.

Make the repository the source of truth

Avoid guessing subjects. Keep every schema in a schemas/ directory and commit an explicit mapping from each file to its subject or subjects, matching the strategy that the producers actually configure. CI reads only that mapping, so a missing entry fails the build and is never silently guessed.

Generate Avro and Protobuf classes from those files. Engineers under deadline will edit a generated class directly, which leaves the schema and the registry unaware of the change. To prevent this:

• Mark generated-code directories as off-limits for edits, for example through code ownership rules or a lint rule.
• In CI, rerun codegen and fail the build if the working tree shows any diff.

Copy#!/usr/bin/env bash
# CI gate: check schemas against their *real* subjects, then verify that generated code matches the schemas.
set -euo pipefail
REGISTRY_URL="${REGISTRY_URL:-"
MAPPING="schemas/subjects.txt"   # lines: <file> <subject> <AVRO|PROTOBUF>, e.g.
# schemas/order_created.avsc com.acme.OrderCreated AVRO   (RecordNameStrategy, as producers configure)

# 1. Every schema file must have an explicit mapping. Never derive <topic>-value.
find schemas \( -name '*.avsc' -o -name '*.proto' \) -print | while read -r f; do
  awk -v f="$f" '$1==f{found=1} END{exit !found}' "$MAPPING" \
    || { echo "FAIL: no subject mapping for $f"; exit 1; }
done

# 2. Test each file against the latest version of its mapped subject.
while read -r file subject type; do
  [[ -z "${file:-}" || "$file" == \#* ]] && continue
  payload=$(jq -n --rawfile s "$file" --arg t "$type" '{schema: $s, schemaType: $t}')
  code=$(curl -sS -o resp.json -w '%{http_code}' -X POST \
    -H 'Content-Type: application/vnd.schemaregistry.v1+json' --data "$payload" \
    "$REGISTRY_URL/compatibility/subjects/$subject/versions/latest")
  # A missing subject means there is no history to check against. Treat it as an error,
  # because it usually means the mapping does not match the producer's strategy.
  # Register genuinely new subjects through a separate, reviewed step.
  [[ "$code" == 404 ]] && { echo "FAIL $subject: subject not in registry"; exit 1; }
  [[ "$code" == 200 ]] || { echo "FAIL $subject: HTTP $code"; cat resp.json; exit 1; }
  jq -e '.is_compatible == true' resp.json >/dev/null \
    || { echo "FAIL $subject: incompatible"; cat resp.json; exit 1; }
  echo "ok  $file -> $subject"
done < "$MAPPING"

# 3. Codegen guard. Regenerate the classes, then fail if any generated file
#    changed or appeared. That would mean someone edited generated code instead of the schema.
make codegen   # placeholder for your Avro/Protobuf codegen target
if [[ -n "$(git status --porcelain -- src/generated)" ]]; then
  echo "FAIL: generated code is out of sync with schemas/. Edit the schema and rerun codegen."
  git status --porcelain -- src/generated
  exit 1
fi

A CI script that requires an explicit file-to-subject mapping, runs each schema through the registry's compatibility endpoint against the real subject, treats a missing subject as a failure rather than a pass, and fails the build if regenerating code produces any diff.

Safe and unsafe Avro schema evolution

Avro resolves a record by matching field names between the writer's schema and the reader's schema, not by position. Under BACKWARD compatibility the new schema is the reader and the old data is the input. Any field the reader expects but the writer never wrote must be filled from the reader's default. If there is no default, resolution fails. That is why a new field without a default is the most common BACKWARD break: the change looks additive in review, but every record already on the topic lacks that name.

Some registry compatibility checkers handle aliases differently, so confirm that your checker honours them before relying on a rename.

Defaults have a cost. A consumer that reads region = "unknown" cannot tell whether the producer sent that value or was built before the field existed. If that difference matters, use a nullable union with a null default, and treat null explicitly as "not provided".

Copy# Requires: pip install fastavro
import io
from fastavro import parse_schema, schemaless_writer, schemaless_reader

# v1: the writer schema for records already on the topic.
v1 = parse_schema({"type": "record", "name": "Order", "namespace": "com.example",
    "fields": [{"name": "order_id", "type": "string"},
               {"name": "zip", "type": "string"},
               {"name": "quantity", "type": "int"}]})

# v2 safe: rename via alias, int -> long promotion,
# and a nullable region where null explicitly means "producer predates this field".
v2_safe = parse_schema({"type": "record", "name": "Order", "namespace": "com.example",
    "fields": [{"name": "order_id", "type": "string"},
               {"name": "postal_code", "type": "string", "aliases": ["zip"]},
               {"name": "quantity", "type": "long"},
               {"name": "region", "type": ["null", "string"], "default": None}]})

# v2 unsafe: region added with no default. Looks additive, breaks BACKWARD.
v2_bad = parse_schema({"type": "record", "name": "Order", "namespace": "com.example",
    "fields": [{"name": "order_id", "type": "string"},
               {"name": "zip", "type": "string"},
               {"name": "quantity", "type": "int"},
               {"name": "region", "type": "string"}]})

# Encode one "old" record with v1, as a producer did before the change.
buf = io.BytesIO()
schemaless_writer(buf, v1, {"order_id": "o-1001", "zip": "94107", "quantity": 3})
old_bytes = buf.getvalue()

# New reader (v2) resolves old data by field name; missing fields need a default.
for label, reader in [("v2_safe", v2_safe), ("v2_bad", v2_bad)]:
    try:
        rec = schemaless_reader(io.BytesIO(old_bytes), v1, reader)
        print(label, "OK:", rec)  # region is None: not provided, not a real value
    except Exception as exc:
        # Expected for v2_bad: no value for 'region' in old data and no default.
        print(label, "FAILED:", type(exc).__name__, exc)

Reading a record written with v1 through two candidate v2 reader schemas shows that an aliased rename, an int-to-long promotion and a nullable field with a null default resolve, while a new field without a default fails resolution.

Safe and unsafe Protobuf changes and registry references

Protobuf does not resolve fields by name at all. Avro matches reader and writer fields by name. Protobuf identifies every field by its tag number, and the name exists only in generated code and the JSON mapping. Renaming a field is therefore invisible on the wire. Reusing a tag is a silent corruption: an old producer's bytes for tag 7 will be decoded as whatever new field now claims tag 7. Never renumber or recycle tags. When you delete a field, add both its number and its name to a reserved statement so the compiler blocks future reuse, including reuse through JSON, where the name matters.

Proto3 has no required, so every field is optional on the wire and absent scalars decode as zero values. A missing field cannot break a reader. Your application code is where "required" gets enforced, and it should be enforced there.

Type changes need caution. The wire format tolerates some swaps, such as int32 to int64, but values can be truncated or sign-extended when read back at the narrower type. Registry rules and runtime behavior are separate things. The registry's checker may reject a type change that decodes fine. It may also accept a change, such as adding enum values, that languages handle differently: some keep the unrecognized number, others move it to unknown fields.

Imported .proto files are registered as their own subjects, and dependents point to them as references by subject and version. When a shared message changes, every subject that imports it inherits the change the next time that subject re-registers. Run the compatibility check against all dependents, not just the subject for the shared file.

Copy// orders/order.proto (v2): field 7 'discount_code' was deleted in this version
syntax = "proto3";

package shop.orders.v1;

// Shared import: registered as its own subject and referenced by subject+version.
// A change to Money reaches every subject that imports it on its next registration.
import "common/money.proto";

message Order {
  // Block the deleted field's tag so an old producer's tag-7 bytes
  // can never decode as a different, newer field.
  reserved 7;
  // Block the name too, so the JSON mapping cannot silently reuse it.
  reserved "discount_code";

  string order_id = 1;
  string customer_id = 2;

  // Renaming a field (formerly 'total') is invisible on the wire: tag 3 is the identity.
  shop.common.v1.Money order_total = 3;

  // proto3 has no 'required'. An absent status decodes as 0 (ORDER_STATUS_UNSPECIFIED),
  // so the application must validate it instead of trusting the decoder.
  OrderStatus status = 4;

  // Do NOT change this to int32. The wire format accepts the swap, but
  // large values are truncated when read as the narrower type.
  int64 quantity = 5;

  // New fields always take a fresh tag, never 7 and never a renumbered old one.
  string sales_channel = 8;
}

enum OrderStatus {
  ORDER_STATUS_UNSPECIFIED = 0;
  ORDER_STATUS_PLACED = 1;
  ORDER_STATUS_SHIPPED = 2;
  // Adding values may pass the registry check, but older readers handle the
  // unknown number differently depending on language runtime and syntax.
  ORDER_STATUS_RETURNED = 3;
}

A proto3 schema change that reserves both the deleted field's tag and its name. It also keeps existing tags stable through a rename, avoids a narrowing type change, and imports a shared message that the registry tracks as a reference.

Implementing the CI gate: check before merge, register after

The gate has three layers, and each catches something the others cannot.

Local check. Run a Protobuf breaking-change linter against the main branch, or an Avro unit test that loads the previous schema from Git and asserts compatibility with the new one. This gives feedback in seconds. It does not replace the registry check, because Git history is not registry history. Only the registry knows every version actually registered under each subject, including versions registered from other repositories.

Pull request job. Trigger it on changes to schema files. For each affected subject, the job POSTs the candidate schema to /compatibility/subjects/{subject}/versions/latest using read-only credentials. A minimal script reads is_compatible from the JSON response with curl -s and jq, then exits non-zero when the value is false. Response fields and verbose error options differ between registry versions, so confirm them against the documentation for the version you run. If the job tests against versions/latest but the subject is set to a transitive mode, the registry still applies the subject's configured mode.

Post-merge job. Only one trusted pipeline, running on the main branch, holds write credentials and registers schemas.

Disable producer auto-registration. If you leave it on, any service starting with an unreviewed schema can register it, which bypasses the gate. Registration must never happen at application startup, because a rolling deploy or a stray local run would then change the shared contract. This adds a deployment ordering rule: the registration job must complete before any producer that uses the new schema is deployed.

Copy#!/usr/bin/env bash
# PR gate: ask the registry whether a candidate schema is compatible with the
# subject's registered history. Uses READ-ONLY credentials; never registers.
# Usage: ./check-compat.sh <subject> <schema-file> [AVRO|PROTOBUF]
set -euo pipefail

SUBJECT="$1"            # e.g. orders-value
SCHEMA_FILE="$2"        # e.g. schemas/orders.avsc
SCHEMA_TYPE="${3:-AVRO}"
REGISTRY_URL="${REGISTRY_URL:?set REGISTRY_URL, e.g. "

# Embed the schema file as a JSON string. jq handles the escaping.
if [ "$SCHEMA_TYPE" = "AVRO" ]; then
  BODY=$(jq -Rs '{schema: .}' "$SCHEMA_FILE")   # Avro is the default type
else
  BODY=$(jq -Rs --arg t "$SCHEMA_TYPE" '{schema: ., schemaType: $t}' "$SCHEMA_FILE")
fi

# Capture the body and HTTP status separately so registry errors are not
# mistaken for a compatibility result.
RESP=$(curl -s -w '\n%{http_code}' -u "$REGISTRY_RO_USER:$REGISTRY_RO_PASSWORD" \
  -X POST -H 'Content-Type: application/vnd.schemaregistry.v1+json' \
  --data "$BODY" \
  "$REGISTRY_URL/compatibility/subjects/$SUBJECT/versions/latest")
STATUS=$(printf '%s\n' "$RESP" | tail -n1)
JSON=$(printf '%s\n' "$RESP" | sed '$d')

if [ "$STATUS" != "200" ]; then
  # e.g. 404 for a subject that does not exist yet; decide that policy explicitly.
  echo "Registry returned HTTP $STATUS for $SUBJECT: $JSON" >&2
  exit 2
fi

# Confirm the response field name against the docs for your registry version.
if [ "$(printf '%s' "$JSON" | jq -r '.is_compatible')" != "true" ]; then
  echo "INCOMPATIBLE: $SCHEMA_FILE vs $SUBJECT: $JSON" >&2
  exit 1
fi
echo "OK: $SCHEMA_FILE is compatible with $SUBJECT"

A pull request gate script that POSTs a candidate Avro or Protobuf schema to the registry's compatibility endpoint using read-only credentials and fails the job on any non-200 response or when is_compatible is not true.

Handling an intentional breaking change

When the check fails because a change really is incompatible, don't push the schema through the existing subject. Give it a new home. The cleanest option is a new topic, such as orders.v2, which gets its own subject under the default topic-based naming. If topology must stay fixed, a record-based naming strategy can give the new message type a separate subject on the same topic. In that case, consumers must handle mixed types, which many deserializers do not handle well by default.

Producers then dual-write to both paths, or consumers migrate one at a time. The old path is retired only when its lag reaches zero and no consumer group still reads it.

The tempting shortcut is to set the production subject to NONE, register the schema, and restore the setting afterward. This is risky for two reasons:

• Consumers still on the old schema fail at deserialization time, which is too late for a safe rollback.
• Every other change registered while the gate is open skips the check, including ones nobody meant to break.

Per-subject overrides cause a quieter version of the same problem. Each override looks justified when it is set, but together they become policy that nobody reviews. A scheduled job that lists subject-level configuration, compares each entry against the global default, and reports deviations with an owner keeps that policy visible.

Breaking cleanly has real costs:

• Dual-writing doubles produce traffic and adds failure modes where the two topics disagree.
• Consumer coordination spans teams and release schedules.
• Retention and backfill require a decision on whether historical data is replayed into the new topic or left to expire.

The alternative is to keep evolving compatibly and accumulate deprecated fields. That has a cost too, but it is paid in readability rather than downtime. For most teams, that trade is acceptable until the deprecated fields start misleading new consumers.

Who should adopt this, when to avoid it, and the impact on engineering time

The gate pays off when producers and consumers belong to different teams, deploy on independent schedules, and share subjects that live for years. In that setup no single reviewer knows every reader of a topic. The registry's compatibility check is then the only place where the full version history is evaluated against a proposed change.

It is overkill when one team owns both ends, when topics are short-lived or internal-only, or when the work is a prototype. In those cases, a coordinated deploy costs less than maintaining the check.

The costs are real:

• Credentials: CI needs read-only credentials, and the deploy pipeline needs write credentials.
• Availability coupling: CI now depends on the registry being up. A mirror removes that dependency but can drift from the real registry.
• Extra deploy step: registration must happen after merge.

Illustrative estimate (assumed numbers, not a benchmark)

Net hours per year = breaks × (incident hours − review-fix hours) − maintenance hours

Assumed inputs:

• 6 breaks per year
• 8 hours per incident, summed across teams
• 1 hour to fix the same break in review
• 12 hours per year of maintenance

Result: 6 × (8 − 1) − 12 = 30 hours per year, before a one-time setup cost. With fewer breaks or cheaper incidents, the check may not pay for itself.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to schema registry compatibility, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.