How does a Kafka schema registry work in the produce and consume path?
A schema registry is a separate service that stores versioned schemas under named subjects and assigns each distinct schema a numeric ID. For event contracts it solves two problems. Producers and consumers deploy independently, so they need a shared, authoritative definition of each event. Putting that definition in every message would waste bytes, so the registry lets each message carry only a small reference.
On the produce path, the serializer derives a subject name from the topic and record (by default <topic>-value). It then either registers the schema or looks up an existing match. The registry returns the schema ID, and the serializer caches it so later sends skip the network call. Each payload is prefixed with a magic byte (0) and the 4-byte big-endian schema ID. Protobuf payloads also include message indexes that identify which message type in the .proto file was used.
On the consume path, the deserializer reads the prefix, fetches the schema by ID on first sight, and caches it. A schema ID always maps to the same schema, so that cache never needs invalidation.
In production, set auto.register.schemas=false so a service cannot publish a new contract simply by deploying. Pair it with use.latest.version=true, or pin a specific version, so the serializer resolves the schema your pipeline already registered.
The registry enforces structural compatibility rules, such as whether a field removal breaks readers. It does not validate business meaning. A field renamed from cents to dollars with the same type passes every check.
# producer.properties: production producer that resolves schemas but never registers them
bootstrap.servers=broker-1.internal:9092,broker-2.internal:9092
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=io.confluent.kafka.serializers.KafkaAvroSerializer
# For Protobuf contracts, swap in:
# value.serializer=io.confluent.kafka.serializers.protobuf.KafkaProtobufSerializer
schema.registry.url=
# Deploying a service must not create a new contract version.
# Schemas are registered by the CI pipeline after compatibility checks pass.
auto.register.schemas=false
# Serialize with the latest version already registered under the subject
# (default subject name: <topic>-value). The serializer caches the returned
# schema ID and writes it after the magic byte in every message.
use.latest.version=true
# ---------------------------------------------------------------
# consumer.properties: matching consumer
# The deserializer reads the magic byte (0) and the 4-byte schema ID,
# fetches that schema from the registry once, then serves it from cache.
# key.deserializer=org.apache.kafka.common.serialization.StringDeserializer
# value.deserializer=io.confluent.kafka.serializers.KafkaAvroDeserializer
# schema.registry.url=
# specific.avro.reader=true
Producer and consumer serializer settings that turn off auto-registration in production, so services serialize against the latest schema the pipeline has already registered, and consumers resolve schemas by the ID embedded in each message.
How do Avro and Protobuf decode events differently?
Avro and Protobuf react very differently to that kind of change, because they put different information on the wire.
An Avro binary record contains no field names and no tags. It contains only values, written in the order the writer schema declares them. That is why the consumer must fetch the writer schema by the ID in the message header. Without it, the bytes cannot be parsed at all. The deserializer then resolves the writer schema against the reader schema by field name:
- Fields the reader doesn't know are skipped.
- Fields the reader expects but the writer lacks are filled from the reader's default.
- If no default exists, resolution fails.
So a field like {"name": "amount_cents", "type": "long"} is bound by its name. Rename it without an alias, or add it without a default, and the registry rejects the schema or the consumer throws.
Protobuf writes a tag before each value. The tag is the field number plus the wire type. A field declared as int64 amount_cents = 3; is identified on the wire only as "field 3, varint". Older or newer generated classes still parse the message:
- Unknown numbers are skipped.
- Absent fields read as type defaults.
Names never travel on the wire. If you reuse number 3 for a different varint field, or change its unit, old consumers decode it without complaint and get the wrong meaning.
| Aspect | Avro | Protobuf |
|---|---|---|
| Contract key | Field name | Field number |
| Needs writer schema to decode | Yes | No |
| Missing field | Reader default, or error | Type default |
| Typical failure | Loud: rejection or decode error | Silent: wrong values |
// order_placed.avsc (Avro). Field names are the contract; the bytes carry only values in writer order.
{
"type": "record",
"name": "OrderPlaced",
"namespace": "com.example.orders",
"fields": [
{"name": "order_id", "type": "string"},
{"name": "amount_cents", "type": "long", "aliases": ["amount"]},
{"name": "currency", "type": "string", "default": "USD"}
]
}
// amount_cents: the alias lets a reader schema with this name resolve data written under the old name 'amount'.
// currency: the default lets a reader with this field decode records whose writer schema lacked it.
// Without the default, resolution fails, and a compatibility check in the registry would reject the change.
// order_placed.proto (Protobuf). Field numbers are the contract; each value is prefixed by a tag (number + wire type).
syntax = "proto3";
package com.example.orders;
message OrderPlaced {
string order_id = 1;
string currency = 2; // absent on the wire -> reads as "" (type default), not "USD"
int64 amount_cents = 3; // on the wire this is only "field 3, varint"; the name never travels
// A removed field: reserve its number and name so nobody reuses them.
// If 4 were reused for another varint, old consumers would decode it silently with the wrong meaning.
reserved 4;
reserved "discount_cents";
}
// Changing amount_cents to mean whole units while keeping number 3 raises no error anywhere,
// and every consumer reads values that are off by a factor of 100.
The same OrderPlaced event in Avro and Protobuf. Avro relies on field names, aliases and defaults to resolve schemas safely, while Protobuf identifies fields only by number, so reused numbers must be reserved.
Which schema changes are safe in Avro and Protobuf?
Safe and unsafe changes side by side
| Change | Avro | Protobuf |
|---|---|---|
| Add a field | Safe only with a default: {"name": "currency", "type": "string", "default": "USD"} |
Safe with a new, never-used field number: string currency = 7; |
| Remove a field | Safe only if the field had a default, so readers still expecting it can fill it in | Safe if you add reserved 4; reserved "discount"; so nobody reuses the number or name |
| Rename a field | Add "aliases": ["sku"] to the renamed product_sku field in the reader schema |
The name never goes on the wire, but JSON mappings and code generation break |
| Change a type | Only the promotions the spec allows, such as int to long |
Treat as unsafe and use a new field number instead |
| Add an enum value | Old readers fail unless the enum declares a default symbol |
Old proto3 readers keep the raw integer, which code then has to handle |
The enum row hides the most risk in Protobuf. Decoding succeeds, but a switch statement written before the new value existed sends it to whatever fallback branch it has. Sometimes that branch is nothing at all.
Nullability
In Avro, a nullable field is a union, and the default must match the union's first branch. ["null", "string"] with "default": null is valid. ["string", "null"] with a null default is rejected.
Proto3 scalars use implicit presence. A price of 0 is not serialized, so "free" and "unknown" decode to the same thing. To keep them distinct, declare optional double price = 3; or use a google.protobuf.DoubleValue wrapper from the start.
Adding presence later is awkward. Switching a field to optional is wire-compatible, but events already written never encoded their zeros, so the ambiguity stays in the stored history. Moving to a wrapper type changes a scalar into a message, which means a new field number and a dual-write migration.
// v1 (before):
// message OrderPlaced {
// string order_id = 1;
// string product_sku = 2;
// double price = 3; // implicit presence: 0 and unset look the same
// string discount = 4;
// }
// v2 (after)
syntax = "proto3";
message OrderPlaced {
// Removed 'discount': reserve its number and name so they are never reused.
reserved 4;
reserved "discount";
string order_id = 1;
string product_sku = 2;
// Left as-is. Old events never encoded zeros, so marking it optional
// would not fix the ambiguity in stored history.
double price = 3 [deprecated = true];
// New field with explicit presence: has_unit_price() tells free from unknown.
optional double unit_price = 8;
// Added with a fresh, never-used number.
string currency = 7;
OrderStatus status = 9;
}
enum OrderStatus {
ORDER_STATUS_UNSPECIFIED = 0; // proto3 requires a zero value; use it as the fallback
ORDER_STATUS_PLACED = 1;
ORDER_STATUS_SHIPPED = 2; // added in v2: old readers see the raw value 2
}
A before/after Protobuf schema that reserves a removed field's number and name, adds new fields under fresh numbers, adds an optional field instead of retrofitting presence onto an existing one, and appends an enum value that older readers will only see as an unknown integer.
What do backward, forward, and full compatibility modes mean for deployment order?
The registry checks each new schema version against earlier ones under a per-subject compatibility mode. The mode defines which side can read which data, and that determines who deploys first.
| Mode | Guarantee | Deploy first |
|---|---|---|
| Backward | New schema reads data written with the previous one | Consumers |
| Forward | Previous schema reads data written with the new one | Producers |
| Full | Both directions | Either |
*_TRANSITIVE |
Same rule, checked against every registered version | Same as base mode |
| None | No check | Coordinated release |
Under backward compatibility, upgraded consumers can still read records from old producers, so consumers ship first. Under forward compatibility, old consumers can tolerate records from new producers, so producers ship first. Full compatibility lets teams release independently, but it usually limits you to adding or removing optional fields with defaults.
Non-transitive modes compare only against the latest version. A sequence of v1→v2→v3 can pass every step while v3 still cannot read v1. If v1 records remain within retention, a consumer that resets its offsets will fail partway through the topic. Transitive modes prevent this, but each new version must fit the whole history. This constraint becomes more restrictive over time.
Use these defaults:
- Compacted, long-retention, or replayable topics: backward transitive.
- Short-retention operational streams: backward is usually enough.
- Topics with many independently released consumers: full transitive.
- None: reserve for scratch topics.
A passing check only proves the schemas are structurally compatible. A renamed field that keeps its type, or a unit changed from milliseconds to seconds, passes every mode. You still need consumer contract tests and a named owner for each subject who approves semantic changes.
How should you name subjects and model multiple event types per topic?
The subject name decides which earlier schemas a new one is compared against, so it also decides where compatibility is enforced.
Topic-based vs record-based subjects
With the default topic-based strategy, the subject is <topic>-value. Every schema written to that topic is checked against the same version history. This fits a topic that carries one event type, because the topic itself becomes the contract.
The record-based strategy names the subject after the fully qualified record or message name, such as com.acme.orders.OrderPlaced. The topic-record strategy combines both, giving <topic>-com.acme.orders.OrderPlaced. Under either one, each event type evolves independently, and the registry never asks whether a single topic holds a coherent set of types.
You select the strategy on the serializer with value.subject.name.strategy, and key.subject.name.strategy for keys. Producers and consumers must agree on it.
Several event types on one topic
You have two ways to keep a topic-based subject while publishing several types:
- A single wrapper schema. In Avro, the top-level schema is a union of records. In Protobuf, it is an envelope message with a
oneof. - Schema references. Each event type is registered under its own subject, and the topic subject's union or envelope references them. In practice, producers then run with
auto.register.schemas=falseanduse.latest.version=true, so they serialize against the registered wrapper rather than the concrete type.
When the strategy is wrong
Each mismatch fails in a predictable direction:
- Topic-based subjects on a mixed topic reject legitimate changes. Publishing
OrderCancelledlooks like an incompatible replacement ofOrderPlaced. - Record-based subjects on a single-type topic fail open. A producer can publish an unrelated record, and that record starts a new, unchecked subject on the same topic.
# producer.properties: one event type per topic (orders.placed).
# The default strategy maps the subject to <topic>-value,
# e.g. orders.placed-value, so the topic is the contract.
bootstrap.servers=broker-1.example.internal:9092
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=io.confluent.kafka.serializers.KafkaAvroSerializer
schema.registry.url=
value.subject.name.strategy=io.confluent.kafka.serializers.subject.TopicNameStrategy
# ---------------------------------------------------------------
# Alternative A: mixed topic (orders), each type evolves on its own.
# The subject becomes orders-com.acme.orders.OrderPlaced,
# orders-com.acme.orders.OrderCancelled, and so on.
# Nothing checks that the topic holds a coherent set of types.
# value.subject.name.strategy=io.confluent.kafka.serializers.subject.TopicRecordNameStrategy
# ---------------------------------------------------------------
# Alternative B: mixed topic (orders) that keeps the topic subject.
# orders-value holds an Avro union, or a Protobuf envelope with a
# oneof. That wrapper references the per-type subjects.
# value.serializer=io.confluent.kafka.serializers.protobuf.KafkaProtobufSerializer
# value.subject.name.strategy=io.confluent.kafka.serializers.subject.TopicNameStrategy
# Keep producers from registering the concrete type as a new
# version of orders-value. Register the wrapper through CI instead.
# auto.register.schemas=false
# Serialize against the latest registered wrapper in orders-value.
# use.latest.version=true
# Keys follow the same rules. Choose their strategy separately.
key.subject.name.strategy=io.confluent.kafka.serializers.subject.TopicNameStrategy
These producer properties show how value.subject.name.strategy determines which subject a schema is checked against, in three cases: a single-type topic, a mixed topic with per-record subjects, and a mixed topic that keeps one topic subject through a wrapper schema with references.
How do you implement schema evolution checks in CI to save engineering time?
Where contracts live and how code is generated
Keep .avsc, .avdl, and .proto files in a shared contracts repository, with an owning team assigned per subject in a CODEOWNERS-style file. Producers and consumers then review the same diff, and generated code comes from one source of truth instead of copies that drift apart. Avro gives you a choice. Generic records skip code generation, but field-name typos only show up at runtime. Generated specific classes catch those mistakes at compile time, at the cost of a build step. Protobuf always generates code, and it does so consistently across languages, which makes it cheaper when teams use several language stacks.
Check compatibility before deploy
In CI, submit each changed schema to the registry's compatibility check against the subject's latest version (or all versions under transitive modes), and fail the merge if the check fails. Register the schema in a controlled release step. Do not let it happen implicitly. A rejection found in review costs one pull request. The same rejection found at produce time means a failed rollout and a rollback.
Operational rules
- Turn off auto-registration in production producers so the CI-approved schema is the only path into the registry.
- Treat the registry as critical infrastructure and rely on client-side caching. During an outage, cached schemas keep working, but new schemas cannot be registered and unseen schema IDs cannot be resolved.
- For a genuinely breaking change, create a new topic or a versioned event type. Dual-publish to the old and new targets until every consumer has migrated, then retire the old one.
When should you choose Avro, Protobuf, or an alternative?
The choice depends mostly on where your schemas already live and who reads the data after it leaves Kafka.
| Situation | Lean toward | Why |
|---|---|---|
| Data platform, lakehouse ingestion, analytics consumers | Avro | Name-based field resolution and container files that carry their writer schema suit long-retention, batch-read data |
Organization already defines RPC contracts in .proto |
Protobuf | One IDL and one code generator across sync and async contracts; field numbers are already a familiar discipline |
| Teams want human-readable payloads and existing JSON tooling | JSON Schema with the registry | You keep registry compatibility checks but lose compact encoding and some evolution guarantees |
| A few tightly coupled services owned by one team | Schemaless JSON with contract tests | The registry becomes a runtime dependency and an operational component that costs more than it protects |
A registry pays off once producers and consumers are owned by different teams and deploy on different schedules. That is when an unreviewed field change can break a consumer nobody on the producing team knows about. If one team ships all sides together, consumer-driven contract tests in the shared build catch the same mistakes. They also avoid a network call on the serialization path and another service to keep available.
Whatever you pick, pick one format for the organization. Two formats mean two serializer configurations, two sets of CI compatibility checks, and two code-generation pipelines. They also mean consumers that join streams from several teams must handle both. That consumer-side cost falls on the teams least able to push back on it.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to kafka schema registry avro vs protobuf, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.