Schema Registry Avro vs Protobuf vs JSON Schema for Event Streams

How does a schema registry work on the wire for event streams?

A schema registry is a service that stores versioned schemas under named subjects and enforces compatibility rules when a new version is registered. Teams use it so producers and consumers can evolve independently without shipping full schemas inside every event or coordinating releases by hand.

How it works, step by step

1. A schema is registered under a subject. An engineer (ideally through CI) submits a schema for a subject such as orders-value. The registry checks it against the subject's compatibility mode and existing versions. If the check passes, the registry stores it as a new version and returns a schema ID. Submitting a schema that is already registered returns the existing ID.
2. The producer tags every message with the ID. The serializer looks up the ID for the schema it was built with, caches it, and writes the ID at the front of each message before the encoded payload.
3. The consumer resolves the ID at read time. The deserializer reads the ID from the message, fetches the matching schema from the registry (or its cache), and decodes the payload. With Avro, it also resolves the writer's schema against the consumer's own reader schema.
4. The next change repeats the check. Every new version passes through the same compatibility gate. That gate is what lets producers and consumers deploy independently, in a known order.

The rest of this section covers the details of each step.

The wire format

With Confluent Schema Registry serializers, each message value starts with a magic byte and a four-byte schema ID, followed by the encoded payload. Protobuf payloads also carry message indexes that identify which message type in the schema file was used. On read, the deserializer extracts the ID, fetches that schema from the registry, and decodes the bytes. The registry is therefore on the runtime read path, not just a build-time tool.

Caching and outages

Clients cache schemas by ID, so a consumer that has already seen every ID in a topic can keep running during a registry outage. A restarted consumer, or one that meets an unfamiliar ID, will fail to deserialize until the registry returns. On the producer side, a serializer that has not yet registered or looked up its schema must reach the registry before it can send. Deploying a new schema version during an outage will block.

Subject naming strategies

• Topic name (the default): one subject per topic. Compatibility checks treat every event on the topic as one evolving type, which effectively limits a topic to a single record type.
• Record name: the subject is the fully qualified record name. This allows many types per topic, and each type evolves on its own across every topic that uses it.
• Topic plus record name: this also allows many types per topic, but compatibility is scoped per topic and type, so the same record can evolve differently on different topics.

Copy+----------------+   register / look up   +----------------------+
|    Producer    |----------------------->|   Schema Registry    |
|   serializer   |<-----------------------|  subject -> versions |
|   [ID cache]   |       schema ID        |  version -> ID       |
+----------------+                        |  compatibility rules |
        |                                 +----------------------+
        | produce                             ^             |
        v                                     | get by ID   | schema
+------------------------------------+        | (cache miss)v
| topic: orders                      |    +----------------------+
|  +-------+-----------+----------+  |    |       Consumer       |
|  | magic | schema ID | payload  |  |    |     deserializer     |
|  | byte  |  4 bytes  | + proto  |  |    |      [ID cache]      |
|  | 0x00  |           | msg idx  |  |    +----------------------+
|  +-------+-----------+----------+  |               ^
+------------------------------------+               |
        |                                            |
        +------------------ consume -----------------+

Subject naming (key used for compatibility checks):
+-------------------+----------------------------+-------------------+
| strategy          | subject                    | types per topic   |
+-------------------+----------------------------+-------------------+
| topic (default)   | orders-value               | one               |
| record            | shop.OrderPlaced           | many, shared rules|
| topic + record    | orders-shop.OrderPlaced    | many, per topic   |
+-------------------+----------------------------+-------------------+

Producers register or look up a schema to get its ID and write that ID into every message. Consumers resolve the ID through the registry at read time, using their cache when they can. The subject naming strategy decides how many event types one topic can carry.

Who should use a schema registry?

A registry pays off when the people who write events are not the people who read them:

• Organizations with several teams sharing topics. When one team produces OrderPlaced and four others consume it, the registry is the agreed contract. Without it, every change needs a meeting or a chat thread.
• Topics with long retention or replay. If consumers rebuild state from offset 0, or a data lake ingests history, every old version must remain decodable. Transitive compatibility checks make that a rule instead of a hope.
• Data platform pipelines. Connectors, stream processors and lake ingestion jobs can read the schema from the registry instead of hard-coding it per topic.
• Polyglot environments. One registered schema can drive generated code in several languages, so services don't each keep their own drifting copy of the event shape.

Who it suits less well:

• A single team that owns both ends of a topic and deploys producer and consumer together. Shared code and ordinary tests may be enough.
• Consumers that cannot reach the registry, such as external partners. See the section on when to avoid each format.
• Teams without anyone to operate it. The registry sits on the runtime read path, so it needs the same availability, backup and access control as any other production dependency.

For the people involved, producer engineers own schema changes, consumer engineers review them and own their deploy order, and a platform team usually runs the registry and the CI checks.

What do backward, forward and full compatibility modes mean for deploy order?

A compatibility mode is a rule the registry enforces when a new schema version is registered under a subject. It compares the candidate against existing versions and rejects it if the rule fails. The mode tells you which side of the stream can safely deploy first.

The plain modes compare only against the latest registered version. The transitive variants (backward transitive, forward transitive, full transitive) compare against every prior version. That matters when a topic retains old data or a consumer replays from the beginning. Under non-transitive rules, each step can pass its own check while version 5 still cannot read version 1. Transitive modes close that gap but make some legitimate cleanups harder.

These checks are structural only. If amount changes from cents to dollars, or a timestamp moves from UTC to local time, the schema is identical and the registry accepts it. Contract tests between producer and consumer, plus code review of schema changes, are what catch semantic breaks.

Governance is a separate decision. Letting producer clients auto-register schemas at runtime removes friction, but it also means any deploy can push a new version without central review. Registering through CI keeps an explicit approval step, which adds friction but stops accidental breaking changes before they reach production. Pair that with clients that refuse to register on their own.

Copy#!/usr/bin/env bash
set -euo pipefail

REGISTRY="${SCHEMA_REGISTRY_URL:?SCHEMA_REGISTRY_URL must be set}"
SUBJECT="orders-value"
SCHEMA_FILE="schemas/order.avsc"
CT="Content-Type: application/vnd.schemaregistry.v1+json"

curl -sf -X PUT -H "$CT" \
  --data '{"compatibility": "BACKWARD_TRANSITIVE"}' \
  "$REGISTRY/config/$SUBJECT"

PAYLOAD=$(jq -n --rawfile s "$SCHEMA_FILE" '{schema: $s, schemaType: "AVRO"}')

OK=$(curl -sf -X POST -H "$CT" --data "$PAYLOAD" \
  "$REGISTRY/compatibility/subjects/$SUBJECT/versions/latest" | jq -r '.is_compatible')

if [ "$OK" != "true" ]; then
  echo "Schema change breaks $SUBJECT compatibility; rejecting." >&2
  exit 1
fi

curl -sf -X POST -H "$CT" --data "$PAYLOAD" \
  "$REGISTRY/subjects/$SUBJECT/versions"

A CI script sets a subject to BACKWARD_TRANSITIVE, checks a candidate Avro schema against the registry's REST API, and registers it only if the check passes, so producer clients never need to auto-register.

How do Avro, Protobuf and JSON Schema each handle schema evolution?

Avro

Avro bytes carry no field names or tags, so a consumer cannot decode anything without the exact writer schema. The registry ID in the message header provides it. The decoder then resolves the writer schema against the consumer's reader schema by field name. A field present in the reader but missing from the writer is filled from the reader's default. If there is no default, resolution fails. This is why backward-compatible additions need defaults. Optional fields are modelled as a union with null listed first and a default of null. Renames work through aliases on the reader schema, which map the old name to the new one.

Protobuf

Field numbers are the contract. Names never appear on the wire, so renaming is free, but reusing a number breaks things. If an old producer writes number 7 as a string and a new consumer reads 7 as an int, the data is silently misread. Mark removed numbers and names as reserved so nobody can reuse them. Proto3 has no required fields, and parsers keep unknown fields instead of rejecting them, so most additions and removals are safe. The pitfall is presence. A scalar without the optional keyword cannot tell "not set" from zero, false or an empty string. Use optional or wrapper types wherever absence carries meaning.

JSON Schema

Evolution depends on additionalProperties:

• Open model: old readers ignore new fields.
• Closed model: a new field fails validation for anyone using the old schema.

JSON Schema is a validation language, not a data definition. Changes to keywords such as oneOf, pattern or numeric bounds can make compatibility verdicts surprising, so compatibility results are harder to predict and need closer review.

Copy// order_placed.proto, v2 of a message registered under the topic's value subject
syntax = "proto3";

package orders.v2;

message OrderPlaced {
  // v1 had `string coupon_code = 7;`. It is removed in v2.
  // Reserve the number and the name so no later edit can reuse 7 with another type.
  // Reusing it would make old bytes silently misread.
  reserved 7;
  reserved "coupon_code";

  string order_id = 1;

  // Renamed from `customer_ref` in v1. This is wire-safe because only the number 2 is encoded.
  // Code that refers to the old name in source or in JSON mappings still breaks.
  string customer_id = 2;

  int64 amount_cents = 3;

  // Implicit presence: a value of 0 is not written to the wire,
  // so consumers cannot tell "producer set 0" from "producer never set it".
  int32 retry_count = 4;

  // Explicit presence: generated code gets a has-check,
  // so "no discount supplied" is distinct from "discount of 0".
  optional int64 discount_cents = 5;

  // Added in v2 with a fresh number.
  // v1 consumers keep it as an unknown field instead of rejecting the message.
  string sales_channel = 8;
}

A proto3 message evolved from v1 to v2 that reserves a removed field number and name, renames a field without changing the wire format, adds a new field, and uses optional where absence must be distinguishable from zero.

Worked migration: adding an optional field and removing a field in all three formats

Take one event, OrderPlaced, through two changes: add an optional couponCode, then remove legacyChannel. All rows assume the field being removed was optional when it was introduced.

Avro. The default is what makes either direction work. When a reader's field is missing from the writer's data, the reader fills it from its own schema's default. Declare new fields as a union with null first and "default": null. Removal is safe only if old readers can supply a default for the missing field.

Protobuf. Compatibility depends on field numbers, not names. Readers skip unknown fields, and missing fields take their type's zero value. Use proto3 optional when consumers need to tell "unset" apart from "". When you delete a field, add both reserved 7; and reserved "legacy_channel";. Without them, a later edit can reuse number 7 with a different type and silently corrupt old payloads.

JSON Schema. The verdict depends on the content model, not on the change. A closed schema gives you only one direction, so deploy in matching order: consumers first for additions, producers first for removals. Leaving additionalProperties open reverses each verdict. Old data could have used any property name with any type, so a checker cannot prove the reverse direction safe. Neither model makes these changes fully compatible.

Copy=== Avro (OrderPlaced.avsc) after both changes ===
{"type": "record", "name": "OrderPlaced", "namespace": "com.example.orders",
 "fields": [
   {"name": "orderId", "type": "string"},
   {"name": "couponCode", "type": ["null", "string"], "default": null}
 ]}

=== Protobuf (order_placed.proto) after both changes ===
syntax = "proto3";
package orders.v1;

message OrderPlaced {
  // legacy_channel was field 7. Reserve the number and the name
  // so no later edit can reuse them with a different type.
  reserved 7;
  reserved "legacy_channel";

  string order_id = 1;
  // New, never-used number. Old readers skip it as an unknown field.
  // `optional` lets consumers tell unset apart from "".
  optional string coupon_code = 8;
}

=== JSON Schema (closed content model) after both changes ===
{
  "title": "OrderPlaced",
  "type": "object",
  "properties": {
    "orderId":    {"type": "string"},
    "couponCode": {"type": "string"}
  },
  "required": ["orderId"],
  "additionalProperties": false
}

The OrderPlaced schema in Avro, Protobuf and JSON Schema after adding couponCode and removing legacyChannel, showing the null default, the reserved field number and name, and the closed additionalProperties setting that make each change safe.

How do code generation and developer workflow affect engineering cost?

Format choice changes daily work for the engineers who produce and consume events, and that cost tends to exceed any difference in bytes on the wire.

Avro offers two modes. Generic records read fields by name at runtime, so a consumer can process schemas it has never seen. That suits pipelines and connectors, but a mistyped field name only fails when the code runs. Specific records are classes generated from .avsc files, so the same mistake fails at compile time. The cost is ramp-up time: engineers must learn how reader and writer schemas resolve, how union defaults work, and how codegen fits into the build.

Protobuf generates idiomatic types for many languages from one .proto file. If you already use gRPC, the same message definitions and toolchain serve both RPC and events. Field numbers make the evolution rules explicit, and reviewers learn to check them.

JSON Schema has uneven code generators, so teams often write types by hand and validate payloads at runtime. Payloads are readable in any log or consumer tool. Avro and Protobuf payloads are opaque bytes until you fetch the schema and decode them, which slows debugging during an incident.

Payload size and serialization speed depend on the data shape, the library and the language. Benchmark with your own events rather than trusting published comparisons.

How can a schema registry save engineering time?

The savings come from moving failures earlier and removing coordination work:

• Breaking changes fail in CI, not in production. A compatibility check on the pull request rejects an incompatible schema in minutes. Without it, the same mistake surfaces as consumer deserialization errors after deploy. Then someone has to diagnose, roll back, and possibly reprocess or skip poisoned messages.
• Deploy order becomes a rule, not a meeting. Once a subject's mode is fixed, everyone knows which side upgrades first. Backward means consumers first, and forward means producers first. Teams stop scheduling coordinated releases for routine additions.
• Generated code replaces hand-written parsing. With Avro specific records or Protobuf, the schema produces typed classes. Field typos and type mismatches fail at compile time, and nobody maintains parallel parsing code in each consuming service.
• The schema is the documentation. Engineers joining a topic can read the registered versions and their history instead of reverse-engineering payloads from logs or asking the producing team.
• Replays stay safe. With transitive modes, rebuilding a consumer's state from the start of a topic does not hit an undecodable old version halfway through.

The savings have costs to set against them: running and monitoring the registry, adding the CI step, and teaching engineers the evolution rules of the chosen format.

An illustrative estimate shows how to reason about the trade-off. All inputs here are assumed, not measured. Assume your teams make 20 schema changes a month, assume 1 in 20 would have been breaking without a check, and assume each breaking change that reaches production costs 8 engineer-hours across diagnosis, rollback and cleanup. That is roughly 1 avoided incident and 8 engineer-hours a month, before counting coordination meetings that no longer happen. Compare that figure with the time your platform team spends operating the registry. Substitute your own change rate and incident cost; with very few shared topics, the balance can come out the other way.

When should you avoid each format, and what are the alternatives?

Avro

Avro is a poor fit when some consumers cannot reach the registry, such as partners outside your network or archival jobs that run after a registry migration. An Avro payload carries no field names, so a reader without the writer's schema cannot decode it. Avro also causes trouble for teams unfamiliar with defaults and unions. A new field without a default breaks backward compatibility. A nullable field needs a union with null listed first, plus a null default. Teams learn these rules by having deploys rejected.

Protobuf

Protobuf is a poor fit when the difference between "unset" and "zero" matters and nobody models it. A proto3 scalar without optional arrives as 0, "" or false when absent. A consumer that applies a zero-value discount or empty-string status will silently act on missing data.

JSON Schema

JSON Schema is a poor fit when evolution must be strictly enforced. Its open content model, conditional keywords and inconsistent additionalProperties usage make compatibility checks less conclusive than Avro's or Protobuf's.

Alternatives

• Unregistered JSON with consumer-driven contract tests. Breaking changes are caught in CI without running a registry. Coverage extends only to consumers that publish tests, and nothing stops a bad message at runtime.
• A schema embedded in each message. Every event decodes on its own, but each one carries the schema's size, and consumers still need logic to reconcile differing versions.

Switching formats later

Changing the format on a live topic is costly. Either every consumer must detect and decode both formats during the transition, or producers must write to a new topic. A new topic means migrating consumers, handling offsets, and deciding whether to replay history.

Which format should you choose for your event streams?

Because switching formats later costs this much, the real question is which format's compatibility rules your team can enforce and reason about under deadline pressure. Strictness you can't verify in CI only produces surprises in production.

Avro works best when every consumer resolves schemas through the registry, because the format depends on that resolution. Protobuf's field-number rules are simpler to explain to many teams, but they make every field effectively optional, so required-ness moves into application code. JSON Schema compatibility checking is the hardest to reason about because open and closed content models interact with additions and removals. Pick it when reach and readability matter more than rigor.

Fix these settings before the first production event:

• Compatibility mode: Decide between backward, forward or full, and decide whether checks are transitive. This choice sets your deploy order for every future change.
• Subject naming strategy: Topic-based naming allows one schema per topic. Record-based naming allows multiple event types per topic. Changing the strategy later re-keys your schema history.
• Registration policy: Disable auto-registration in production producers and register schemas from CI. That way an incompatible change fails a build, not a deploy.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to schema registry avro vs protobuf, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.