Protobuf Backward Compatibility: Reserved Fields and Safe gRPC Changes

How protobuf identifies fields on the wire

Protobuf backward compatibility means that code built against one version of a .proto file can still exchange messages with code built against another version, without errors or silent misreads. This works because the binary format identifies fields by number and never by name.

Take message User { int64 id = 1; string email = 3; }. Each encoded field starts with a tag, computed as (field_number << 3) | wire_type. For email, that is (3 << 3) | 2 = 26, written as the single byte 0x1A:

• 0x1A = 0001 1010
• Upper five bits 00011: field number 3.
• Lower three bits 010: wire type 2 (length-delimited).
• Next byte: a varint giving the string length, followed by the UTF-8 bytes.

The name email appears nowhere in the payload. You can rename the field and the wire format does not change. If you renumber it, every existing reader looks for the wrong tag.

When a parser meets a field number it does not recognize, it uses the wire type to work out how many bytes to skip. Current proto3 parsers keep those bytes as unknown fields, so they are written back out on re-serialization.

This is why adding a field with a new number is the basic safe change:

• Old client, new server: the client skips the new field.
• New client, old server: the new field is absent, so the client reads its default value.

The trade-off is that unknown-field retention depends on the language and runtime, and on how a message is copied. A proxy that decodes a message into its own types, or copies data field by field, can drop fields it does not know about.

Copy# Hand-decode a User message to show that only field numbers and wire types are on the wire.
# Schema known to this reader: message User { int64 id = 1; string email = 3; }

def read_varint(buf, i):
    result, shift = 0, 0
    while True:
        b = buf[i]; i += 1
        result |= (b & 0x7F) << shift
        if not b & 0x80:          # high bit clear marks the last byte
            return result, i
        shift += 7

# id=42 (tag 0x08), email (tag 0x1A), plus field 4 added by a newer schema (tag 0x22)
payload = (bytes([0x08, 42, 0x1A, 15]) + b"ann@example.com"
           + bytes([0x22, 2]) + b"US")

KNOWN = {1: "id", 3: "email"}     # this "old reader" has never heard of field 4
i = 0
while i < len(payload):
    tag, i = read_varint(payload, i)
    field_number, wire_type = tag >> 3, tag & 0x07
    if wire_type == 0:            # varint: value follows directly
        value, i = read_varint(payload, i)
    elif wire_type == 2:          # length-delimited: length varint, then that many bytes
        length, i = read_varint(payload, i)
        value, i = payload[i:i + length], i + length
    else:
        raise ValueError(f"wire type {wire_type} not handled in this sketch")
    # The wire type alone tells us how far to skip, so an unknown number is harmless
    name = KNOWN.get(field_number, "<unknown, skipped>")
    print(f"tag=0x{tag:02X} field={field_number} wire_type={wire_type} {name}: {value!r}")

# Output:
# tag=0x08 field=1 wire_type=0 id: 42
# tag=0x1A field=3 wire_type=2 email: b'ann@example.com'
# tag=0x22 field=4 wire_type=2 <unknown, skipped>: b'US'

A dependency-free Python decoder splits each tag into field number and wire type. It shows that no field names appear in the payload, and that an old reader can skip a field number it does not recognize by using only the wire type.

Removing fields safely with reserved numbers and names

Removing a field is where those wire-level rules start to matter. The correct removal is a single change: delete the field declaration and, in the same commit, add both its number and its name to the message's reserved statements. Doing only half of that leaves a gap someone will eventually step into.

Reserving the number is the critical part. Old binaries, queued messages, caches and stored blobs still contain bytes tagged with that number. If a later change reuses it for a new field, those bytes decode into the new field with no error. A removed int64 account_id = 4 reborn as int64 retry_count = 4 will parse cleanly and return nonsense. If the wire types differ, the result is a parse failure or a value silently treated as unknown. With reserved in place, protoc rejects any attempt to declare that number again.

Reserving the name protects consumers that never see field numbers. The JSON mapping and text format key on names, so a reintroduced account_id with different semantics would quietly match old JSON documents and config files. It also stops a well-meaning engineer from reviving the name later.

The syntax takes single numbers and ranges, such as reserved 4, 9 to 11;, and names as quoted strings, such as reserved "account_id";. Numbers and names go in separate statements. Enums follow the same pattern: when you remove a value, reserve its number and its name inside the enum block.

On long-lived messages, the reserved lists grow and become noise. Tidy them by grouping adjacent numbers into ranges and adding comments. Never shorten them by un-reserving a number, because that reopens the exact path to corruption the reservation exists to block.

Copysyntax = "proto3";

package billing.v1;

message Invoice {
  // Removed in this change: int64 account_id = 4;
  // Earlier removals: discount_code = 9, legacy_total = 10, notes = 11.
  // Never un-reserve these. Old payloads and stored blobs still carry them.
  reserved 4, 9 to 11;
  // Names are reserved in a separate statement. This protects JSON and
  // text-format consumers and blocks reviving a name with new semantics.
  reserved "account_id", "discount_code", "legacy_total", "notes";

  string invoice_id = 1;
  string customer_id = 2;
  int64 amount_cents = 3;
  Status status = 5;

  // New fields take fresh numbers. They never reuse a reserved one.
  int64 retry_count = 12;

  enum Status {
    STATUS_UNSPECIFIED = 0;
    STATUS_DRAFT = 1;
    STATUS_ISSUED = 2;
    STATUS_PAID = 4;
    // Removed value: STATUS_DISPUTED = 3;
    // Reserve its number and its name inside the enum block.
    reserved 3;
    reserved "STATUS_DISPUTED";
  }
}

// If someone later adds `int64 retry_count = 4;` or `string account_id = 13;`,
// protoc rejects the file because both the number and the name are reserved.

This example removes a message field and an enum value, then reserves each one's number and name in separate statements, using both single numbers and a range.

Which schema changes are safe, unsafe or conditional

Compatibility questions fall into a few categories. Learning the categories works better than memorising a full matrix.

The silent cases are the most dangerous ones. The parser raises no error, so corrupted values reach business logic.

Enums

Proto3 requires the first enum value to be zero. Make it an explicit sentinel such as STATUS_UNSPECIFIED, so that "not set" never looks like a real state. Adding new values is safe on the wire. It can still break consumers:

• Exhaustive switches that have no default case.
• Languages that handle enums as closed sets.
• Validators that reject unknown values.

No schema check will flag these failures. Review new enum values with the consumers in mind.

Field presence

A proto3 scalar declared without optional cannot tell unset from zero. Decide up front whether a field needs optional or a wrapper type. Presence adds nil checks to every caller. Skipping it saves that work now, but if you later need to express "not provided," you have to add a new field and migrate callers to it.

Copysyntax = "proto3";

package orders.v1;

import "google/protobuf/wrappers.proto";

enum OrderStatus {
  // Zero is the proto3 default. Make it an explicit "not set" sentinel
  // so an unset field never reads as a real state.
  ORDER_STATUS_UNSPECIFIED = 0;
  ORDER_STATUS_PENDING = 1;
  ORDER_STATUS_SHIPPED = 2;
  // Added later. This is safe on the wire, but review consumers with
  // exhaustive switches, closed-enum handling or strict validators first.
  ORDER_STATUS_CANCELLED = 3;
}

message Order {
  string id = 1;
  OrderStatus status = 2;

  // Keep this as int32. Switching to sint32 keeps the varint wire type
  // but changes how negatives decode (two's-complement vs zigzag).
  int32 quantity_delta = 3;

  // No presence: an unset field and 0 look identical to the reader.
  int32 priority = 4;

  // Explicit presence: callers can tell "not provided" apart from 0.
  optional int32 discount_percent = 5;

  // A wrapper type gives the same distinction as a nullable message.
  google.protobuf.Int64Value max_price_cents = 6;
}

An order schema with an UNSPECIFIED enum sentinel, an enum value added later, an int32 field that must not change to sint32, and scalar fields declared with and without explicit presence.

gRPC service-level changes and levels of compatibility

gRPC routes every call by a path built from the package, service and method names, in the form /package.Service/Method. Field numbers play no part in it. Renaming a service or RPC, or moving it to another package, changes that path. Existing clients then get an UNIMPLEMENTED error even though every message is byte-for-byte identical. Adding a new RPC is safe, because old clients never call it. Replacing an RPC's request or response with a different message type is unsafe. The method path stays the same, but each side decodes the bytes against a schema the other side never sent, and the mismatch can go unnoticed.

Three levels of compatibility

Choose your promise based on who consumes the API. For internal services that are rebuilt together from one repository, wire compatibility is usually enough, and a rename only needs one coordinated commit. Mobile apps stay installed for months and cannot be forced to update, so wire compatibility has to be strict. Third parties and JSON gateways need all three levels, because you cannot fix their code for them. A stricter promise means names you chose early are effectively permanent. That cost is worth paying once people outside your build regularly regenerate code from your schema.

Enforcing the rules in CI with buf breaking

Review alone will not hold these promises. A reviewer reading a diff sees a renamed field or a changed number as one plausible edit among many, and missing it costs nothing until a client deserializes garbage. A mechanical check compares the proposed schema against a baseline and fails the build, so the rule holds even when everyone is busy.

With buf breaking, the setup is small. In buf.yaml, set breaking.use to the one category that matches your published promise:

• WIRE if you only guarantee binary compatibility.
• WIRE_JSON if clients use the JSON mapping, where field names matter.
• FILE or PACKAGE if consumers regenerate code from your files and need source compatibility.

In CI, run buf breaking --against '.git#branch=main' on every pull request. For a published module, compare against the last released version instead. Main tells you what you are about to merge. The release tells you what consumers already depend on.

When the check fails, treat it as a design question: deprecate and add a new field, or follow the multi-step migration below. Do not add a rule exception or suppression comment to make the build pass. If you must take a break deliberately, record why in the pull request.

The check is only as good as its inputs:

• Baseline: a stale or wrong baseline silently approves breaks.
• Scattered schemas: copies spread across repositories or vendored into clients drift beyond the check's reach.
• Semantics: no tool notices that timeout changed from seconds to milliseconds while staying int64.

Copy# buf.yaml (repository root)
version: v2
modules:
  - path: proto
breaking:
  use:
    # Pick ONE category matching the published promise:
    # WIRE = binary only, WIRE_JSON = binary + JSON names,
    # FILE/PACKAGE = source compatibility for regenerated code.
    - WIRE_JSON
  # Deliberately no `except` or `ignore` entries: a failure is
  # a design discussion, not something to silence here.
---
# .github/workflows/proto-breaking.yaml
name: proto-breaking
on: pull_request
jobs:
  breaking:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0   # full history so the baseline branch is available
      # Assumes a pinned buf binary is already on the runner image.
      - name: Create local main as the baseline
        run: git fetch origin main:main
      - name: Compare proposed schema against main
        run: buf breaking --against '.git#branch=main'
      # For a published module, compare against the last release instead,
      # e.g. a release tag: buf breaking --against '.git#tag=v1.4.0'

A minimal buf.yaml that selects a single breaking-change category with no exceptions, plus a pull-request CI job that fetches main as the baseline and fails the build on any detected break.

A multi-step migration for changes that really are breaking

A change like that seconds-to-milliseconds switch can't be made in place. It needs a new field and a sequence of releases, each of which is compatible with the one before it:

1. Add the new field (timeout_ms under a fresh number) alongside the old one. Nothing reads it yet.
2. Dual-write and dual-read. Servers populate both fields and accept either, preferring the new one when present.
3. Migrate clients to send and read only the new field.
4. Stop using the old field. Servers stop writing it and stop falling back to it.
5. Remove and reserve the old number and name.

Each step is its own deploy because clients and servers don't upgrade together. At any moment, some running binaries are one step behind. If you combine steps 2 and 4, an older client that still sends only timeout gets its value silently ignored and falls back to a default. If you combine steps 1 and 3, clients start sending a field that some servers don't read yet. The failure is rarely an error. It is usually a zero value that looks legitimate.

The process is worth it when one field changes inside a widely shared message. When many fields change, or an RPC's contract changes, defining CreateJobV2 or a new versioned package is often simpler. Old and new run side by side, and the old one retires as a whole.

Mobile apps and third-party integrators can keep step 3 open for years. With no forcing deadline, steps 4 and 5 get postponed indefinitely. Put an owner and a removal date on the old field when you add the new one.

Alternatives, costs and where engineering time is saved

None of the alternatives replaces an automated check outright. They work alongside it.

What enforcement costs

Each removed field adds an entry to the reserved lists, and that clutter is permanent. Strict rule sets slow down early-stage APIs where nothing is deployed yet. Some teams start with a looser rule category and tighten it after the first external release. CI setup is a one-time job: choose a baseline branch, then fail the build on violations. Multi-step migrations remain the largest ongoing cost because every team that owns a client has to be scheduled.

Illustrative estimate (assumed numbers, not a benchmark)

Cost = engineers × hours × hourly rate

• CI catch: 1 engineer × 0.25 h × $100 = $25
• Production corruption: 3 engineers × 16 h (detection, bisecting deploys, repairing records) × $100 = $4,800, before you count any customer impact.

Plug in your own staffing, debugging time and rate. The gap stays wide because corrupted data decodes without errors, so nothing alerts until someone notices bad values.

Who benefits most

The benefit is largest for platform teams that own shared schemas used by many services. It is also large for APIs whose clients deploy on their own schedule, such as mobile apps and partner integrations, because old binaries keep running and you cannot force them to upgrade.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to protobuf backward compatibility, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.