gRPC Error Handling: Status Codes, Rich Errors and Retry Policy

Status codes are an API contract: choosing by what the caller does next

A gRPC status is the code, message and optional details a server returns in the trailers to end every call. The code carries the most weight because clients, proxies and retry logic branch on it without reading the message. Choose it by what the caller should do next: fix the request, wait and retry, re-authenticate, restart a read-modify-write transaction, or give up.

UNKNOWN usually means a handler threw an exception that nobody converted to a status. Treat it as a missing mapping and fix the handler.

The library also generates codes itself:

• DEADLINE_EXCEEDED when the deadline expires.
• CANCELLED when the call is cancelled.
• UNAVAILABLE when the connection fails.
• UNIMPLEMENTED when the method is unknown.
• RESOURCE_EXHAUSTED when a message exceeds the size limit.

If your handlers return these codes for unrelated business failures, the caller can't tell a network fault from application logic. Retry policies keyed on UNAVAILABLE will then also replay requests that should not be replayed.

How the rich error model works: google.rpc.Status and the details trailer

A status code alone carries a few bits of meaning. The rich error model adds structure without changing the wire protocol. google.rpc.Status is a protobuf message with three fields: code (the numeric status code), message (a developer-facing string), and details, a repeated google.protobuf.Any. Because Any wraps arbitrary typed messages, a server can attach several machine-readable payloads to a single failure, and clients unpack only the types they recognize.

On the wire, the error travels as HTTP/2 trailers. grpc-status holds the code as a decimal string. grpc-message holds the percent-encoded message. grpc-status-details-bin holds the entire serialized google.rpc.Status, base64-encoded because of the -bin suffix. The code inside the details should match grpc-status. Older clients that don't understand the trailer still get a correct code.

Each language exposes helpers for attaching and reading details. Go's status package provides WithDetails and Details(). Java uses StatusProto to convert to and from exceptions. Python's grpcio-status package provides the same conversion.

The rule that keeps this model useful is that the message is for developers and the details are for programs. Clients branch on the status code and ErrorInfo.reason, never on message strings. Message strings get reworded, localized, and truncated.

Copy  CLIENT                     PROXY / GATEWAY            SERVER
+------------------+       +------------------+       +------------------+
| Application      |       | L7 proxy or      |       | Handler builds   |
| branch on code + |       | JSON transcoder  |       | google.rpc.Status|
| ErrorInfo.reason |       | [D2] strips      |       | [D1] plain error |
| never on message |       |  unknown trailers|       |  = no details    |
+------------------+       |  or header limit |       +------------------+
| gRPC runtime     |       | [D3] JSON error  |       | gRPC runtime     |
| unpacks known Any|       |  body drops      |       | encodes Status   |
| [D4] skips others|       |  details         |       | into trailers    |
+---------+--------+       +---------+--------+       +---------+--------+
          |   request (HEADERS+DATA) |                          |
          |------------------------->|------------------------->|
          |                          |                          |
          |  trailers (END_STREAM)   |                          |
          |<-------------------------|<-------------------------|
          | grpc-status: 5           | grpc-status: 5           |
          | grpc-message: pct-enc    | grpc-message: pct-enc    |
          | grpc-status-details-bin: | grpc-status-details-bin: |
          |   base64(Status bytes)   |   base64(Status bytes)   |

  google.rpc.Status = { code, message, details: repeated Any }
  code inside details-bin must match grpc-status; older clients read code
  message -> developers     details (e.g. ErrorInfo.reason) -> programs
  [Dn] = point where details can be lost

A failed RPC sends grpc-status, grpc-message and the base64-encoded google.rpc.Status in grpc-status-details-bin as HTTP/2 trailers; details can be lost at the handler (D1), in the proxy (D2), in a transcoding gateway (D3) or in the client runtime (D4).

Using the standard detail types from error_details.proto

The detail types in error_details.proto exist so that every service doesn't invent its own shape for "which field was wrong" or "when can I retry". Clients and shared middleware can then handle them generically. Pick the type by what the caller needs to act on:

ErrorInfo is the one to attach almost everywhere. domain scopes the identifier to your service, so two teams can both use QUOTA_EXCEEDED without colliding. reason is a short, constant, UPPER_SNAKE_CASE value that code matches on. metadata carries the specifics, such as the limit or resource ID, as string pairs. Keep these values out of reason itself, so the set of reasons stays small and enumerable.

Strip DebugInfo at any boundary that faces external callers. Stack traces leak implementation details and give attackers a map of your internals.

In the example below, the server packs ErrorInfo and BadRequest into the status. The client unpacks them and switches on reason.

Once a client matches on a reason, that string is part of your API. Renaming it is a breaking change, just like renaming an RPC. Version it with the same care.

Copyimport grpc
from grpc_status import rpc_status
from google.protobuf import any_pb2
from google.rpc import code_pb2, error_details_pb2, status_pb2

def _pack(msg):
    a = any_pb2.Any()
    a.Pack(msg)
    return a

# Server: reject a bad request with a stable reason plus per-field detail.
def CreateOrder(self, request, context):
    info = error_details_pb2.ErrorInfo(
        reason="QUANTITY_OUT_OF_RANGE",       # constant, clients match on this
        domain="orders.example.com",          # scopes the reason to this service
        metadata={"max_quantity": "100"})    # specifics live here, not in reason
    bad = error_details_pb2.BadRequest(field_violations=[
        error_details_pb2.BadRequest.FieldViolation(
            field="items[0].quantity", description="must be between 1 and 100")])
    st = status_pb2.Status(code=code_pb2.INVALID_ARGUMENT,
                           message="invalid order",
                           details=[_pack(info), _pack(bad)])
    context.abort_with_status(rpc_status.to_status(st))

# Client: decode the rich status and branch on domain + reason, never on message text.
def place_order(stub, req):
    try:
        return stub.CreateOrder(req)
    except grpc.RpcError as err:
        st = rpc_status.from_call(err)  # None if the server sent no rich status
        if st is None:
            raise
        for d in st.details:
            if d.Is(error_details_pb2.ErrorInfo.DESCRIPTOR):
                info = error_details_pb2.ErrorInfo()
                d.Unpack(info)
                if (info.domain, info.reason) == ("orders.example.com", "QUANTITY_OUT_OF_RANGE"):
                    print("cap quantity at", info.metadata["max_quantity"])
            elif d.Is(error_details_pb2.BadRequest.DESCRIPTOR):
                bad = error_details_pb2.BadRequest()
                d.Unpack(bad)
                for v in bad.field_violations:
                    print(f"{v.field}: {v.description}")
        raise

A Python server packs ErrorInfo and BadRequest into a google.rpc.Status and aborts with it. The client uses grpc_status.rpc_status.from_call to unpack the details and branch on the ErrorInfo domain and reason, which makes that pair part of the API contract.

Mapping errors at service boundaries

A downstream status describes the downstream's view of the world, not yours. If your order service calls an inventory service and gets NOT_FOUND, the caller asked about an order, not a SKU. Passing that code straight through tells them their order doesn't exist, which may be false. Depending on the cause, it could be FAILED_PRECONDITION (the order references a discontinued item) or INTERNAL (your data is inconsistent). UNAVAILABLE needs the same care. It tells the client that retrying is safe. Forward it from a dependency you have already retried and exhausted, and every client multiplies your retry load. Translate at the point where the call returns, while you still know what the operation meant.

Apply the same rule to internal exceptions. A database constraint violation, a timeout in a connection pool, or a null dereference should each map to a code and reason defined in your own domain. Unmapped exceptions should fall through to INTERNAL with a generic message.

At a trust boundary, remove anything that exposes how the service is built. That covers DebugInfo details, stack traces, SQL fragments, hostnames, and the internal identifiers of downstream systems. Log them server-side with a correlation ID, and return only that ID to the caller, for example in RequestInfo. Support staff can then find the full context without exposing it.

A server interceptor is the right place to enforce this. It catches whatever the handler raised, applies the mapping table, filters the details against an allowlist of types, and logs everything it removed. The example below shows one way to structure it.

Copyimport functools, logging, uuid
import grpc
from google.protobuf import any_pb2
from google.rpc import code_pb2, error_details_pb2, status_pb2
from grpc_status import rpc_status

log = logging.getLogger("orders")
DOMAIN = "orders.example.com"

class DuplicateOrderError(Exception): pass  # raised by our repository layer

def map_error(exc):
    """Translate into OUR codes and reasons; never forward a dependency's status as-is."""
    downstream = exc.code() if isinstance(exc, grpc.RpcError) else None
    if downstream == grpc.StatusCode.NOT_FOUND:
        # Inventory has no such SKU; to our caller the order exists but can't proceed.
        return code_pb2.FAILED_PRECONDITION, "ITEM_DISCONTINUED", "Order references an unavailable item."
    if downstream == grpc.StatusCode.UNAVAILABLE:
        # Retries already exhausted inside this service; don't invite client retries on top.
        return code_pb2.INTERNAL, "INVENTORY_DEPENDENCY_FAILED", "Order could not be processed."
    if isinstance(exc, DuplicateOrderError):
        return code_pb2.ALREADY_EXISTS, "ORDER_ALREADY_EXISTS", "An order with this key already exists."
    return code_pb2.INTERNAL, "INTERNAL_ERROR", "Internal error."  # unmapped: generic only

def boundary(handler):
    """Wrap every public handler; handlers raise exceptions instead of calling context.abort."""
    @functools.wraps(handler)
    def wrapper(self, request, context):
        try:
            return handler(self, request, context)
        except Exception as exc:
            request_id = str(uuid.uuid4())
            code, reason, message = map_error(exc)
            # Full detail (stack trace, downstream message, hosts, SQL) stays in server logs.
            log.exception("rpc failed request_id=%s reason=%s", request_id, reason)
            # Only allowlisted detail types leave the service: ErrorInfo and RequestInfo.
            info, req = any_pb2.Any(), any_pb2.Any()
            info.Pack(error_details_pb2.ErrorInfo(reason=reason, domain=DOMAIN))
            req.Pack(error_details_pb2.RequestInfo(request_id=request_id))
            status = status_pb2.Status(code=code, message=message, details=[info, req])
            context.abort_with_status(rpc_status.to_status(status))
    return wrapper

A Python boundary decorator that maps downstream gRPC statuses and internal exceptions to the service's own codes and reasons, logs the full error server-side under a request ID, and returns only ErrorInfo and RequestInfo to the caller.

Configuring safe retries: retryPolicy, hedging, throttling and deadlines

Retries are configured per method in the client's service config, not in handler code. A retryPolicy names the methods it covers and sets maxAttempts (implementations cap this), initialBackoff and maxBackoff as duration strings, backoffMultiplier, and retryableStatusCodes. The delay grows by the multiplier up to the maximum and is randomized so clients do not retry in lockstep. A conservative policy lists only UNAVAILABLE, because that code usually means the request never reached application logic.

Even UNAVAILABLE is not proof of that. A connection can drop after the server committed a write. Retrying a non-idempotent call is only safe if the request carries a request ID or idempotency key that the server deduplicates. ABORTED is different: the client should not resend the same request but re-read the state and redo the whole read-modify-write sequence, which transparent retries cannot do.

A hedgingPolicy sends extra copies after hedgingDelay without waiting for the first to fail. This reduces tail latency but runs the method more than once by design, so use it only for methods that are truly idempotent.

retryThrottling uses a token bucket. Each failure removes a token, each success adds tokenRatio, and retries stop when the bucket falls below half of maxTokens. This keeps retries from amplifying load during an outage.

All attempts share the caller's deadline, and that deadline should propagate across hops. DEADLINE_EXCEEDED is a poor retry code because the budget is already spent.

Retry and service config support varies by language, and config supplied by the name resolver can override local settings. Test what your stack actually honors.

Copy{
  "methodConfig": [
    {
      "name": [
        { "service": "example.orders.v1.OrderService", "method": "GetOrder" },
        { "service": "example.orders.v1.OrderService", "method": "CreateOrder" }
      ],
      "timeout": "2s",
      "retryPolicy": {
        "maxAttempts": 3,
        "initialBackoff": "0.1s",
        "maxBackoff": "1s",
        "backoffMultiplier": 2,
        "retryableStatusCodes": ["UNAVAILABLE"]
      }
    },
    {
      "name": [
        { "service": "example.catalog.v1.CatalogService", "method": "GetProduct" }
      ],
      "timeout": "0.5s",
      "hedgingPolicy": {
        "maxAttempts": 2,
        "hedgingDelay": "0.05s",
        "nonFatalStatusCodes": ["UNAVAILABLE"]
      }
    }
  ],
  "retryThrottling": {
    "maxTokens": 10,
    "tokenRatio": 0.1
  }
}

A service config that retries only UNAVAILABLE for order methods (CreateOrder must carry a deduplicated idempotency key), hedges an idempotent read, and throttles retries with a token bucket; verify which fields your gRPC implementation honors.

When to avoid rich details and retries, and the alternatives

Rich details only help if they reach the caller. Some browser clients, proxies and HTTP/JSON transcoding gateways drop the grpc-status-details-bin trailer or reduce it to a plain HTTP status. For that reason, the code and message must make sense even when no details arrive. Details also travel in metadata, and implementations cap total header size. Large stack traces or long lists of field violations can push a response over that limit, which turns a useful error into a transport failure. A very fine-grained taxonomy of reasons creates its own problem: every client that switches on a reason string becomes coupled to it, so renaming or merging reasons breaks those clients.

Retries need the same restraint. Never retry non-idempotent methods unless they carry an idempotency key. Retry at only one layer, because retries at several layers multiply. Illustrative estimate, with assumed inputs: suppose a gateway, a mid-tier service and its client library each allow 3 attempts. One user request can then produce up to 3 × 3 × 3 = 27 calls to the bottom service, and that happens exactly when the bottom service is already failing.

Impact on development cost and engineering time

The up-front cost is mostly design and plumbing. Teams have to agree on an error taxonomy and document each reason and domain so the values stay stable. They also write boundary mappers and interceptors, and ship error_details.proto stubs to every client language. That work keeps going after launch. Each new reason is an API change that needs review, and clients built against older protos must tolerate details they don't recognize.

The return comes from removing guesswork. Clients branch on codes and ErrorInfo.reason instead of parsing message strings, which break as soon as someone rewords them. Triage gets faster because RetryInfo or a QuotaFailure violation tells on-call what happened without digging through server logs. Retry behavior moves out of hand-written loops scattered across call sites and into one reviewed service config.

Budget for observability as part of the cost. Retries hide flakiness, so a service can report a healthy success rate while many calls succeed only on a later attempt. Record metrics and trace spans per attempt, not only per call.

Illustrative estimate (assumed numbers, not a benchmark). Assume each attempt fails independently with probability p, and the policy allows up to n attempts. Expected attempts per call = 1 + p + p² + … + pⁿ⁻¹. With p = 0.2 and n = 3, that is 1 + 0.2 + 0.04 = 1.24. The server takes about 24% more load, and the per-call success rate may still look fine.

Incremental adoption checklist

• Fix obvious code misuse first, such as validation failures returned as UNKNOWN or INTERNAL.
• Add ErrorInfo with documented reasons to your highest-traffic methods.
• Add a boundary interceptor that maps errors and strips internal details.
• Ship per-attempt metrics and tracing before enabling any retries.
• Enable retryPolicy only for idempotent methods, with deadlines and throttling configured.
• Delete hand-written retry loops once the config covers those methods.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to grpc error handling status codes, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.