gRPC Error Handling: Status Codes, Rich Details and Safe Retries

How the gRPC status model works

Every gRPC call ends with a status, whether it succeeded or not. The status has three parts: a code from a fixed set of canonical values (OK is the only success), a human-readable message, and optional binary details. On the wire, the server sends these in HTTP/2 trailers after any response messages. Because the status arrives in trailers, a streaming call can deliver several messages and still end in failure.

The code and details are the machine-readable contract. The message is not. Messages get reworded, localized, or wrapped by middleware. A client that matches on "user not found" will break the first time someone fixes a typo. Branch on the code, and when you need more precision, branch on typed details such as a field violation or a retry delay. Section 3 covers those details.

Not every status comes from your handler. The gRPC library and the transport produce some codes on their own:

• UNAVAILABLE when a connection can't be established or drops.
• DEADLINE_EXCEEDED when the client's deadline expires, sometimes before the request reaches the server.
• CANCELLED when either side abandons the call.
• UNKNOWN when a handler throws an exception the framework can't map to a code.

So a client can't assume a given code reflects a deliberate decision by service code. That ambiguity is why errors belong in the API contract, and it affects two groups:

• Backend engineers who own services decide which codes and details their handlers emit.
• Platform engineers who own shared client configuration decide which codes trigger retries, backoff, or failover across every caller.

Copy +----------------+        +---------------------+        +----------------+
 |  Client code   |  call  |  gRPC library and   |  call  | Service handler|
 |                |------->|  HTTP/2 transport   |------->|                |
 |                |<-------|                     |<-------|                |
 +----------------+ status +---------------------+ status +----------------+
         ^                  Can produce on its own:       Emits chosen codes
         |                  UNAVAILABLE                   and details
         |                  DEADLINE_EXCEEDED             (owned by backend
         |                  CANCELLED                     engineers)
         |                  UNKNOWN (unmapped exception)
         |
 +-------+------------------------------------------------------------+
 | Status in HTTP/2 trailers (after any response messages)            |
 |  code    : OK, NOT_FOUND, UNAVAILABLE, ...     -> branch on this   |
 |  message : human-readable, can change          -> never parse it   |
 |  details : typed binary, e.g. field            -> branch on this   |
 |            violation, retry delay                                  |
 +-------+------------------------------------------------------------+
         |
         v
 +--------------------------------------------------------------------+
 | Shared client config (owned by platform engineers):                |
 |  which codes trigger retries, backoff or failover for every caller |
 +--------------------------------------------------------------------+

A gRPC status (code, message, optional details) reaches the client in HTTP/2 trailers and may come from the service handler or from the library and transport, so clients and shared retry config should act on the code and typed details and never on the message text.

Choosing grpc status codes by what the caller should do next

Start from the caller's next move, not from what went wrong inside the server. If two failures need the same reaction from the caller, they can share a code. If the caller must react differently, they need different codes.

Return NOT_FOUND instead of PERMISSION_DENIED when confirming that a resource exists would leak information, such as another tenant's IDs. The cost is that legitimate callers and on-call engineers can't tell a typo from a missing grant. Record the real reason in server logs.

Fine-grained codes help clients branch, but every code you return becomes part of the contract. Once a client checks for FAILED_PRECONDITION, switching that path to ABORTED breaks the client without warning. For that reason, a small set of codes chosen on purpose is usually safer than full precision.

Rich error details with google.rpc.Status

A status code tells the caller what kind of failure happened. google.rpc.Status carries the specifics. It has three fields: an integer code, a developer-facing message, and details, a repeated google.protobuf.Any. Because each detail is packed as Any, a client can unpack only the types it recognizes and safely ignore the rest.

The standard detail types live in google/rpc/error_details.proto:

Clients should branch on ErrorInfo.reason plus domain, not on message. Message text changes. A reason such as INSUFFICIENT_CREDITS is a contract.

On the wire, the server serializes the full Status into the binary grpc-status-details-bin trailer. Each language exposes it through its own helper. Go uses the status package with WithDetails, Java uses StatusProto, and Python uses grpcio-status. The language-agnostic pattern works like this:

• Server: build a Status with INVALID_ARGUMENT, then attach an ErrorInfo and a BadRequest.
• Client: convert the error to a Status, iterate over details, and type-switch on each one.

Rich details come with costs:

• Client dependencies: clients need the proto definitions and the matching helper library.
• Size limits: payloads count against header and trailer size limits, which proxies may enforce.
• Intermediaries: some gateways strip or rewrite trailers.
• Exposure: never send DebugInfo across a trust boundary.

For simple internal services, a code-only contract with a small set of stable messages is often enough.

Copyimport grpc
from grpc_status import rpc_status
from google.protobuf import any_pb2
from google.rpc import code_pb2, error_details_pb2, status_pb2

# Server side: inside a servicer method, attach ErrorInfo and BadRequest.
def reject_order(context):
    info = error_details_pb2.ErrorInfo(
        reason="INSUFFICIENT_CREDITS",   # stable machine key, part of the contract
        domain="billing.example.com",
        metadata={"account_id": "acct-1234"},
    )
    bad = error_details_pb2.BadRequest(field_violations=[
        error_details_pb2.BadRequest.FieldViolation(
            field="order.quantity", description="must be greater than 0"),
    ])
    detail_info, detail_bad = any_pb2.Any(), any_pb2.Any()
    detail_info.Pack(info)
    detail_bad.Pack(bad)
    status = status_pb2.Status(
        code=code_pb2.INVALID_ARGUMENT,
        message="order rejected",          # for developers, not for branching
        details=[detail_info, detail_bad],
    )
    # Serialized into the grpc-status-details-bin trailer.
    context.abort_with_status(rpc_status.to_status(status))

# Client side: recover the Status and unpack only the types we recognize.
def handle(err: grpc.RpcError):
    status = rpc_status.from_call(err)
    if status is None:                   # no rich details: fall back to the code
        return err.code()
    for detail in status.details:
        if detail.Is(error_details_pb2.ErrorInfo.DESCRIPTOR):
            info = error_details_pb2.ErrorInfo()
            detail.Unpack(info)
            if (info.domain, info.reason) == ("billing.example.com", "INSUFFICIENT_CREDITS"):
                print("prompt user to top up", info.metadata.get("account_id"))
        elif detail.Is(error_details_pb2.BadRequest.DESCRIPTOR):
            bad = error_details_pb2.BadRequest()
            detail.Unpack(bad)
            for v in bad.field_violations:
                print(f"{v.field}: {v.description}")
        # Unknown detail types are ignored.

A Python sketch using grpcio-status in which a server attaches ErrorInfo and BadRequest to an INVALID_ARGUMENT status, and the client unpacks those details and branches on domain plus reason rather than on the message text.

Implementing server-side error mapping and deadlines

Handlers should not choose status codes ad hoc. If each one decides independently, NotFound from one service and FailedPrecondition from another will end up meaning the same thing. Instead, handlers return domain errors, and a single server interceptor translates them. The example below maps known error types to codes and sends everything else to a catch-all.

Rules for the mapping layer

• Never put raw exception text or stack traces in the status message. They leak table names, file paths and library versions. They also tempt clients to parse prose.
• Keep DebugInfo internal. Attach it only when the caller sits inside your trust boundary, and strip it at the edge.
• Log the full error with a correlation ID and return that ID to the client, either in the message or in a RequestInfo detail. Support can then find the log line without the client ever seeing internals.

The catch-all that turns unknown errors into INTERNAL keeps clients safe, but it also hides mistakes. A validation error you forgot to map will show up as a server fault, and nobody will notice. To catch this, emit a metric for every catch-all hit, labeled by the original error type, and alert when it grows.

Deadlines are error handling

• Every client call sets a deadline. Without one, a stuck backend holds connections and threads indefinitely.
• Propagate the remaining deadline downstream. Do not start a fresh timeout for each hop.
• Preserve DEADLINE_EXCEEDED from dependencies. Rewrapping it as INTERNAL or UNAVAILABLE misleads both retry logic and dashboards.
• Expect deadline expiry after the work is done. The server may have committed the write just before the client gave up. The client sees a failure for an operation that succeeded, which is why idempotency matters before any retry.

Copyimport logging, uuid
import grpc

log = logging.getLogger("rpc")
class NotFoundError(Exception): pass
class ValidationError(Exception): pass

# The only place that decides what a domain error means on the wire.
ERROR_CODES = {NotFoundError: grpc.StatusCode.NOT_FOUND,
               ValidationError: grpc.StatusCode.INVALID_ARGUMENT}

def record_catchall(method, error_type):
    pass  # increment a counter labeled by method and error_type; alert on growth

class ErrorMappingInterceptor(grpc.ServerInterceptor):
    def intercept_service(self, continuation, handler_call_details):
        handler = continuation(handler_call_details)
        if handler is None or handler.unary_unary is None:
            return handler  # sketch covers unary-unary only
        method = handler_call_details.method

        def wrapped(request, context):
            try:  # handlers raise domain errors; they do not call context.abort
                return handler.unary_unary(request, context)
            except Exception as err:
                cid = str(uuid.uuid4())
                # Full error and stack trace stay in server logs, keyed by cid.
                log.exception("rpc failed method=%s cid=%s", method, cid)
                code = ERROR_CODES.get(type(err))
                if isinstance(err, grpc.RpcError) and err.code() == grpc.StatusCode.DEADLINE_EXCEEDED:
                    code = grpc.StatusCode.DEADLINE_EXCEEDED  # keep downstream deadline expiry honest
                if code is None:  # catch-all: safe for clients, so make it visible to operators
                    record_catchall(method, type(err).__name__)
                    code = grpc.StatusCode.INTERNAL
                # Fixed message plus correlation ID; never str(err) or a traceback.
                context.abort(code, f"request failed (correlation_id={cid})")

        return grpc.unary_unary_rpc_method_handler(
            wrapped,
            request_deserializer=handler.request_deserializer,
            response_serializer=handler.response_serializer)

A Python gRPC server interceptor that maps domain errors to status codes from one table, keeps downstream DEADLINE_EXCEEDED unchanged, logs the full error under a correlation ID, and counts every catch-all INTERNAL so misclassified errors show up in metrics.

Configuring a grpc retry policy, throttling and hedging

Client retries in gRPC are declared in the service config, not in application code. Each methodConfig entry is scoped by name: a service alone applies to every method, and a service plus method targets one RPC. Its retryPolicy sets maxAttempts, which includes the original call, and the backoff fields initialBackoff, maxBackoff and backoffMultiplier. It also sets retryableStatusCodes. For an idempotent read, UNAVAILABLE is the usual candidate. Every retry still has to fit inside the caller's deadline.

The top-level retryThrottling block is a client-side brake. It is a token bucket defined by maxTokens and tokenRatio. Failures drain the bucket and successes refill it. When the level drops low enough, the client stops retrying, so a struggling backend does not get hit with even more traffic.

Transparent retries are separate from this. The library may resend a request that never reached the server application, such as a stream the server refused. It can do this because no handler ran.

hedgingPolicy is an alternative to retryPolicy for latency-sensitive idempotent reads. You configure one or the other on a method. Hedging sends extra copies after hedgingDelay and keeps the first response. This cuts tail latency, but every hedge is real server work, so treat it as a capacity decision.

Support differs across language implementations. A config that one client honors may be silently ignored by another. Check your language's documentation and test retry behavior in each language you ship.

Copy{
  "methodConfig": [
    {
      "name": [
        { "service": "catalog.v1.CatalogService", "method": "GetItem" }
      ],
      "timeout": "2s",
      "retryPolicy": {
        "maxAttempts": 4,
        "initialBackoff": "0.1s",
        "maxBackoff": "1s",
        "backoffMultiplier": 2,
        "retryableStatusCodes": ["UNAVAILABLE"]
      }
    },
    {
      "name": [
        { "service": "catalog.v1.CatalogService" }
      ],
      "timeout": "5s"
    }
  ],
  "retryThrottling": {
    "maxTokens": 10,
    "tokenRatio": 0.1
  }
}

A service config that retries only the idempotent GetItem read on UNAVAILABLE (up to 4 attempts including the original) within a 2s timeout. Other methods on the service get a timeout and no retries, and retryThrottling caps retries client-wide.

When retries are unsafe: idempotency, overload and streaming

Idempotency is the gate for every retry decision. Retrying UNAVAILABLE broadly is safe for reads and naturally idempotent writes such as "set X to 5", because the request either never reached a handler or repeating it changes nothing. Mutations like "charge card" or "append event" need an idempotency key or request ID that the server records and checks before acting.

DEADLINE_EXCEEDED and INTERNAL are different from UNAVAILABLE. They tell you nothing about whether the server committed the work. A timeout can fire after the database write succeeded, so a retry duplicates the side effect. For non-idempotent methods, keep both out of retryableStatusCodes. Idempotency keys fix the duplicate, but they create their own design work: where keys are stored, how long they live, and what a replay returns when the original request is still in flight.

Retries also multiply across layers. If the client, the sidecar or mesh, and the load balancer each make three attempts, one failing call can become 27 requests at the backend. That is an illustrative estimate, not a measurement. Pick one layer to own the retry policy and enable throttling there. Throttling limits retries when the failure rate rises, so it matters more than tuning backoff values. Retrying RESOURCE_EXHAUSTED without honoring the server's RetryInfo delay sends more load to a server that has already said it is overloaded.

Streaming adds another constraint. The status arrives in trailers, often after the client has already received and processed some messages. Built-in retries stop once response headers or messages have been received. To resume a stream after a failure, the application needs its own checkpoints or resume tokens so the client can ask to continue from the last message it processed.

Impact on cost and engineering time, plus a team error convention checklist

Most of the cost comes at the start. Someone has to write the mapping table from domain errors to status codes. The team has to agree on a shared proto package for error details. Each language you ship needs a thin helper library so engineers attach and read ErrorInfo the same way everywhere. Idempotency keys need a store with a retention window at least as long as your longest retry chain, and that store has its own capacity limits and failure modes. Retry behavior also needs testing in each client language, because retry support and service-config handling vary between gRPC implementations. A fault-injection test that returns UNAVAILABLE on the first attempt and succeeds on the second catches most misconfigurations.

The payback comes from removing guesswork. Clients branch on the status code and ErrorInfo reason instead of parsing message strings, so rewording an error message no longer breaks a caller. On-call engineers can filter by status code and reason to separate a quota problem from a dependency outage without reading logs line by line. Retry behavior lives in configuration rather than in hand-written loops, so it is predictable under load. Retry storms and duplicate writes become rarer.

Team error convention checklist

• A code-mapping table from each domain error to a status code, reviewed whenever the API changes.
• Required ErrorInfo reasons for each domain, with a stable domain string.
• A list of which methods are idempotent and which require an idempotency key.
• Retry config stored and versioned alongside the service definition.
• Metrics broken down by method and status code, on both client and server.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to grpc error handling, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.