How the gRPC error model works
Every gRPC call ends with a status, whether it succeeded or not. The status has three parts: a code from a fixed set of 17 canonical values (OK, INVALID_ARGUMENT, NOT_FOUND, UNAVAILABLE, and so on), a message string meant for developers, and an optional details payload. The details are an encoded google.rpc.Status protobuf, which repeats the code and message and adds a list of google.protobuf.Any entries holding typed error information.
The split exists because the parts serve different readers. The code is a contract that clients branch on to decide whether to retry, re-authenticate, fix the request, or give up. That only works if the set stays small and its meanings stay stable across languages and services. The message is for humans reading logs. The details carry structured context that a single enum cannot express, such as which field failed validation or how long to wait before retrying. The model matters to anyone who builds or consumes gRPC APIs, and especially to platform engineers who own proxies, gateways, and shared client libraries.
How does it work? The server attaches the status to the end of the response stream as HTTP/2 trailers. The client library decodes those trailers into a language-native error. The caller reads the code to decide what to do next and unpacks the typed details only when it needs specifics. The steps below trace that path.
How a status travels
- The server handler returns an error object built from a code, a message, and optionally details.
- The gRPC library ends the HTTP/2 stream with trailers. It writes
grpc-statuswith the numeric code andgrpc-messagewith the percent-encoded message. If details are present, it serializes thegoogle.rpc.Statusinto the binarygrpc-status-details-bintrailer. - If the call fails before the server sends any response message, the status can arrive in a single trailers-only response.
- The client library reads the trailers and surfaces an error or exception. Callers inspect the code first, then unpack the details if they need more.
At the HTTP layer the response usually still shows a 200 status. The outcome of the call lives in the trailers, not in the HTTP status line.
+---------------------------+ +---------------------------+
| SERVER HANDLER | | CLIENT CALLER |
| returns error: | | 1. check code -> branch |
| code = NOT_FOUND | | retry/auth/fix/stop |
| message = 'user 42 ...' | | 2. log message |
| details = [Any, ...] | | 3. unpack details if any |
+-------------+-------------+ +-------------^-------------+
| |
v |
+-------------+-------------+ +-------------+-------------+
| gRPC SERVER LIBRARY | | gRPC CLIENT LIBRARY |
| packs details into a | | reads trailers, decodes |
| google.rpc.Status, ends | | grpc-status-details-bin, |
| stream with trailers | | raises error/exception |
+-------------+-------------+ +-------------^-------------+
| |
v |
+-------------+---------------------------------------------+-------------+
| HTTP/2 RESPONSE STREAM (one per call) |
| HEADERS :status 200, content-type: application/grpc |
| DATA zero or more length-prefixed response messages |
| HEADERS trailers, END_STREAM flag set |
| grpc-status: 5 |
| grpc-message: user%2042%20not%20found |
| grpc-status-details-bin: <base64 google.rpc.Status> |
| |
| Trailers-only: if no message was sent, a single HEADERS frame carries |
| :status, content-type and the grpc-* fields, with END_STREAM set. |
+-------------------------------------------------------------------------+
A gRPC error flows from the server handler through the server library into HTTP/2 trailers (grpc-status, grpc-message, grpc-status-details-bin), and the client library decodes those trailers into an error whose code the caller checks before it unpacks any details.
Who should use this approach
Status codes are always present, so every gRPC user is already using part of this model. The question is how deliberately to use it, and how much of the rest to adopt.
- Service owners exposing gRPC APIs to other teams. When you don't control your callers, a documented set of codes and
ErrorInforeasons is the only reliable way to tell them what went wrong without them parsing your messages. - Teams building client SDKs and shared libraries. Retry, backoff, and re-authentication logic all key off status codes. Consistent codes let one library serve many services.
- Platform and infrastructure engineers who run gateways, proxies, and service meshes. They translate gRPC codes to HTTP statuses, apply retry policies, and build dashboards and alerts on code distributions.
- Teams with user-facing validation flows, such as form submissions passing through a backend-for-frontend.
BadRequestfield violations map directly onto per-field UI errors. - On-call and support engineers, who benefit from request IDs and stable reasons when tracing a client report back to server logs.
Rich details are less useful when a single team owns both client and server and the only consumer is a log line, or when traffic crosses intermediaries that drop binary trailers. In those cases, correct status codes and clear messages may be enough. The section on when to avoid rich details covers the alternatives.
Choosing status codes by what the client can do next
Before picking a code, ask what the caller should do next: retry, fix the request, or stop. The code is the only part of the error that every client, proxy, and retry policy can read, so it should answer that question without anyone parsing the message.
| Code | Meaning | Caller's next step |
|---|---|---|
INVALID_ARGUMENT |
Request is bad no matter what the system state is | Fix the request. Do not retry it unchanged. |
FAILED_PRECONDITION |
Request is valid, but the system is not in the required state | Change the state (e.g. delete children first), then retry |
OUT_OF_RANGE |
Value is past a valid range that can move | Stop iterating, or retry after the range changes |
UNAUTHENTICATED |
No valid credentials | Refresh or obtain credentials |
PERMISSION_DENIED |
Caller is known but not allowed | Give up, or request access |
Use OUT_OF_RANGE for something like reading past the end of a file. A client paging through data can treat it as a clean stop signal. INVALID_ARGUMENT cannot give that signal because it covers every kind of malformed input.
NOT_FOUND also overlaps with permission checks. If you return NOT_FOUND to a caller who lacks access, you hide whether the resource exists, and attackers cannot enumerate IDs. The cost falls on legitimate users with misconfigured roles: they see "not found" and start debugging the wrong problem. Use this only for resources whose existence is itself sensitive.
Reserve UNKNOWN and INTERNAL for server bugs and broken invariants. If they become catch-alls, alerting on them turns into noise and clients cannot tell a defect from bad input.
The code set is small, and teams are tempted to stretch it by overloading codes or putting meaning in message strings. Don't. Message text changes without notice, and any client that parses it breaks silently. Put machine-readable specifics in error details instead.
Retry semantics, idempotency, and backoff
The status code a server returns is effectively an instruction to the client's retry policy, so choose it knowing what the client will do with it.
| Code | Retry? | Condition |
|---|---|---|
UNAVAILABLE |
Yes | Transient by definition; retry with backoff |
RESOURCE_EXHAUSTED |
Sometimes | Only with backoff, and only if the quota resets |
ABORTED |
Yes, at a higher level | Re-read state, then retry the whole transaction |
DEADLINE_EXCEEDED |
Only if idempotent | The server may have completed the work |
INVALID_ARGUMENT, FAILED_PRECONDITION |
No | Retrying sends the same bad request again |
DEADLINE_EXCEEDED needs the most care. The client stopped waiting, but the server may have committed the write. Retrying a non-idempotent call such as "charge card" can apply it twice. The usual fix is an idempotency key: the client generates a unique ID per logical operation and sends it in metadata. The server stores the result under that key and returns the stored result when the key repeats.
Backoff and jitter matter because naive retries multiply load. If a backend drops requests and each client retries three times immediately, the backend receives up to four times its normal traffic at the moment it can least absorb it. Exponential backoff spreads attempts out over time. Jitter stops clients from retrying in synchronized waves.
When the server knows when capacity will return, it can attach google.rpc.RetryInfo with a retry_delay. The client should use that delay instead of its own estimate. In gRPC's service config, a retry policy sets maxAttempts, initialBackoff, maxBackoff, backoffMultiplier, and retryableStatusCodes. Keep that list short.
import json
import uuid
import grpc
import payments_pb2, payments_pb2_grpc # generated stubs (placeholder service)
service_config = {
"methodConfig": [{
"name": [
{"service": "payments.v1.PaymentService", "method": "GetPayment"},
{"service": "payments.v1.PaymentService", "method": "Charge"},
],
"retryPolicy": {
"maxAttempts": 4, # 1 original call + up to 3 retries
"initialBackoff": "0.2s",
"maxBackoff": "5s",
"backoffMultiplier": 2,
"retryableStatusCodes": ["UNAVAILABLE"], # keep this list short
},
}]
}
channel = grpc.insecure_channel(
"payments.internal:50051",
options=[
("grpc.enable_retries", 1),
("grpc.service_config", json.dumps(service_config)),
],
)
stub = payments_pb2_grpc.PaymentServiceStub(channel)
key = str(uuid.uuid4())
reply = stub.Charge(
payments_pb2.ChargeRequest(order_id="order-1234", amount_cents=4999),
metadata=[("idempotency-key", key)],
timeout=3.0,
)
A Python gRPC client sets a retry policy in the service config so only UNAVAILABLE is retried with exponential backoff, and it sends an idempotency key in metadata so a retried Charge call cannot apply twice.
Rich error details with google.rpc.Status
The status code tells a client which category of failure occurred. It does not say which field was invalid, which quota ran out, or how long to wait. google.rpc.Status adds that context. It has three fields: code, message, and details. The details field is a repeated google.protobuf.Any, so each entry is a typed protobuf message. gRPC serializes the whole Status into the grpc-status-details-bin trailer, and clients that don't understand it still see the plain code and message.
google/rpc/error_details.proto defines the standard detail types:
| Type | Key fields | Typical code | Client uses it to |
|---|---|---|---|
ErrorInfo |
reason, domain, metadata |
Any | Branch on a stable identifier |
BadRequest |
field_violations (field, description) |
INVALID_ARGUMENT |
Highlight specific form fields |
RetryInfo |
retry_delay |
UNAVAILABLE, RESOURCE_EXHAUSTED |
Replace its own backoff estimate |
QuotaFailure |
violations (subject, description) |
RESOURCE_EXHAUSTED |
Identify which limit was hit |
PreconditionFailure |
violations (type, subject, description) |
FAILED_PRECONDITION |
Fix system state first |
ResourceInfo |
resource_type, resource_name, owner |
NOT_FOUND, PERMISSION_DENIED |
Name the affected resource |
LocalizedMessage |
locale, message |
Any | Show text to end users |
Include ErrorInfo on every error. Together, reason (for example CART_ITEM_LIMIT) and domain (for example orders.example.com) form an identifier that clients can switch on. Message strings get reworded and translated, so code that parses them breaks when the wording changes. A reason stays the same.
On the server, wrap each detail in Any, add it to details, and convert the result into the language's status error. On the client, unpack the trailer and check each entry's type URL against the types you handle. Skip any entry you don't recognize, because servers add new detail types over time. If details is absent or empty, fall back to the status code alone.
from google.rpc import code_pb2, error_details_pb2, status_pb2
from grpc_status import rpc_status # pip package: grpcio-status
def reject_add_item(context):
status = status_pb2.Status(
code=code_pb2.INVALID_ARGUMENT, message="Cart item limit reached")
# reason + domain is the stable identifier clients switch on
status.details.add().Pack(error_details_pb2.ErrorInfo(
reason="CART_ITEM_LIMIT", domain="orders.example.com",
metadata={"max_items": "50"}))
status.details.add().Pack(error_details_pb2.BadRequest(field_violations=[
error_details_pb2.BadRequest.FieldViolation(
field="quantity", description="Cart cannot exceed 50 items")]))
# Sets code, message and the grpc-status-details-bin trailer
context.abort_with_status(rpc_status.to_status(status))
def classify(err):
status = rpc_status.from_call(err) # None if no details trailer was sent
reason, fields = None, []
for detail in (status.details if status else []): # empty list is fine
if detail.Is(error_details_pb2.ErrorInfo.DESCRIPTOR):
info = error_details_pb2.ErrorInfo()
detail.Unpack(info)
reason = (info.domain, info.reason)
elif detail.Is(error_details_pb2.BadRequest.DESCRIPTOR):
bad = error_details_pb2.BadRequest()
detail.Unpack(bad)
fields = [(v.field, v.description) for v in bad.field_violations]
# Unknown detail types are skipped, not treated as errors
if reason == ("orders.example.com", "CART_ITEM_LIMIT"):
return ("show_limit_banner", fields)
# No ErrorInfo: fall back to the status code alone, never parse the message
return ("generic_error:" + err.code().name, fields)
A Python server packs ErrorInfo and BadRequest into google.rpc.Status details, and the client unpacks known types, skips unknown ones, handles a missing or empty details list, and branches on domain plus reason instead of the message text.
Mapping errors at service boundaries and gateways
A status code describes the service that returned it, so never forward a dependency's status unchanged. Suppose your order service calls an inventory service that returns UNAVAILABLE. If you pass that through, the caller concludes your service is down and retries it, even though your service is healthy and the next attempt will probably hit the same broken dependency. Decide what the failure means for your own contract. If a retry against you could succeed, UNAVAILABLE may still be correct, but choose it on purpose. If the dependency rejected the request as malformed, that is a bug in your service, so return INTERNAL. A mapping function at each outbound call site makes these decisions explicit. It takes the downstream status and returns one of your public codes, with a default branch that falls back to INTERNAL.
gRPC-to-HTTP gateways apply a fixed translation that follows the guidance in google.rpc.Code:
| gRPC status | HTTP status |
|---|---|
INVALID_ARGUMENT |
400 |
UNAUTHENTICATED |
401 |
PERMISSION_DENIED |
403 |
NOT_FOUND |
404 |
ALREADY_EXISTS |
409 |
RESOURCE_EXHAUSTED |
429 |
INTERNAL |
500 |
UNAVAILABLE |
503 |
DEADLINE_EXCEEDED |
504 |
HTTP clients and proxies make retry and caching decisions based on these HTTP statuses, so a wrong gRPC code becomes a wrong HTTP status too.
Keep operator details and client-facing text separate. Put stack traces, SQL, hostnames, and the raw downstream errors in your logs. Send the client only a short, safe message and a request ID. If you use RequestInfo, put the ID in its request_id field. Write the same ID to your log entries and trace spans, so a support engineer can start from what the client reports and find the full context.
import logging
import uuid
import grpc
log = logging.getLogger("orders")
_INVENTORY_TO_PUBLIC = {
# Chosen on purpose: a retry against the order service may succeed.
grpc.StatusCode.UNAVAILABLE: grpc.StatusCode.UNAVAILABLE,
# The dependency is throttling us. That is not the caller's quota.
grpc.StatusCode.RESOURCE_EXHAUSTED: grpc.StatusCode.UNAVAILABLE,
# We sent a malformed request downstream, so the bug is ours.
grpc.StatusCode.INVALID_ARGUMENT: grpc.StatusCode.INTERNAL,
}
def map_inventory_error(err: grpc.RpcError) -> grpc.StatusCode:
# Anything not mapped explicitly falls back to INTERNAL.
return _INVENTORY_TO_PUBLIC.get(err.code(), grpc.StatusCode.INTERNAL)
def reserve_stock(inventory_stub, request, context):
metadata = dict(context.invocation_metadata())
request_id = metadata.get("x-request-id") or str(uuid.uuid4())
try:
return inventory_stub.Reserve(request, timeout=2.0)
except grpc.RpcError as err:
# Operator view: the full downstream detail goes to the log only.
log.error(
"inventory.Reserve failed request_id=%s downstream_code=%s details=%s",
request_id, err.code().name, err.details(),
)
# Client view: a public code, a safe message and the request ID.
context.abort(
map_inventory_error(err),
f"Could not reserve stock. Request ID: {request_id}",
)
A Python gRPC handler translates an inventory-service error into one of the order service's public codes, falling back to INTERNAL for anything unmapped, and logs the raw downstream details while sending the client only a safe message and a request ID.
Error handling in streaming RPCs
In a streaming RPC the status arrives in the trailers, after the last message. A server-streaming call can deliver hundreds of valid messages and then end with UNAVAILABLE or DEADLINE_EXCEEDED. The client cannot treat each received message as a sign that the call is succeeding. It has to process messages as they arrive and still be ready for a failure that turns them into a partial result. Decide up front what partial means for each method. Either the client discards everything or it keeps what it received and continues from there.
Built-in retry policies do not help much here. Once the client has received response data, the call is committed and will not be retried automatically, because the library cannot know which messages the application has already acted on. Resumption has to be part of your API design.
Resumption tokens and checkpoints
Include an opaque resume token or offset in each response message. On failure, the client opens a new stream and passes the last token it processed, and the server continues from that point. This only works if tokens stay valid across server restarts and replicas. A token that points to in-memory state on a single instance stops working as soon as that instance goes away.
Per-item errors versus stream status
A non-OK status ends the stream, so use it only for conditions that make continuing pointless, such as an expired credential or a dropped backend. In bulk or bidirectional workloads, one malformed item should not cancel thousands of valid ones. Put a google.rpc.Status field in the per-item response message and keep the stream open. The client then gets a result for every item, and the stream status reports only on the call as a whole.
When to avoid rich details, and the alternatives
Rich details travel in the grpc-status-details-bin trailer, so they depend on every hop preserving binary trailing metadata. Browsers cannot read HTTP/2 trailers directly. gRPC-Web works around this by encoding trailers into the response body, but an intermediary that doesn't understand that encoding can drop the details without any error. Some proxies and older language libraries also strip unknown -bin headers. The client still receives a code and a message, and nobody notices the missing detail until a user reports a vague error.
Size is the second problem. Implementations cap total metadata size, and a BadRequest carrying hundreds of field violations can exceed that cap. When it does, the peer may reject the response or replace your status with a generic INTERNAL or RESOURCE_EXHAUSTED. The client then loses the original code as well as the details.
Alternatives
| Approach | Works through browsers and proxies | Machine-readable | Main cost |
|---|---|---|---|
| Code plus message only | Yes | Code only | Clients parse strings or guess |
| Error fields in the response message | Yes | Yes, schema-typed | RPC returns OK on failure, so generic retry and metrics logic must inspect the payload |
| Custom metadata keys (ASCII) | Usually | Partly | No shared schema, still subject to size limits |
Rich details also make leaks more likely. DebugInfo stack traces, internal hostnames, and raw validation input can reach external callers. Strip or allowlist detail types at the edge, and do not send debug payloads outside your trust boundary.
How it saves engineering time
A consistent error contract removes several recurring categories of work.
- Less client-side guesswork. When codes reliably answer "retry, fix, or stop", client teams don't have to reverse-engineer each service's failure behavior or write per-service special cases. One shared retry and error-handling library can cover every service that follows the contract.
- No breakage from message rewording. Clients that branch on
ErrorInforeasons don't break when someone edits or translates a message. That removes a class of bugs that is hard to catch in review, because the server change looks harmless. - Faster incident triage. Clean separation between
INTERNALand caller errors likeINVALID_ARGUMENTkeeps alerts meaningful, so on-call engineers spend less time on pages caused by bad input. Correct boundary mapping points them at the failing dependency instead of the healthy service that forwarded its status. - Shorter support loops. A request ID in the client-visible error lets a support engineer go straight from a user report to the matching log entries and trace spans. Without it, they reconstruct the request from timestamps and guesses.
- Less UI glue code.
BadRequestfield violations map directly onto form fields, so frontend and backend teams don't negotiate a custom error format for every endpoint. - Fewer self-inflicted incidents. Retrying only transient codes, with backoff and idempotency keys, avoids retry storms and duplicate writes, both of which are expensive to diagnose and clean up.
To give a sense of scale, here is an illustrative estimate with assumed inputs, not a measured result. Assume a team handles 20 error-related support tickets a month. Without a request ID, each one takes an assumed 45 minutes to locate in the logs, and with a request ID it takes an assumed 10 minutes. Under those assumed numbers the team saves about 35 minutes per ticket, or roughly 11–12 engineer-hours a month, before counting avoided client breakages or false alerts. Plug in your own ticket volume and lookup times. The direction of the saving holds even if the numbers differ.
Testing, governance, and impact on engineering cost
Treat the error contract as part of the API surface. For each method, document the status codes it can return and, for each code, the ErrorInfo reasons and domain it may attach. Keep this list next to the proto definitions so reviewers see contract changes in the same diff.
Tests that pin the contract
Assert on the code and the detail types, not on message strings. A good test calls the method with a known-bad input. It then checks that the status code is FAILED_PRECONDITION, unpacks the details, and confirms there is exactly one ErrorInfo with reason ACCOUNT_SUSPENDED and the expected domain. It also checks that no DebugInfo is present when the call goes through the external path. Message text can change freely. Codes, reasons, and detail types cannot.
Reasons are API
Renaming a reason, or splitting QUOTA_EXCEEDED into per-resource variants, breaks every client that branches on the old value. Version reasons the way you version fields. Add the new reason alongside the old one, migrate callers, then remove the old one.
Cost and payoff
This takes upfront work. Someone has to define the reasons. After that, the docs, the contract tests, and the gateway mappings all have to stay in sync across services, and they drift unless CI checks them. The return is concrete:
- Clients stop parsing error text and breaking when wording changes.
- Request IDs in
RequestInfoconnect a client report to server logs in one lookup. - Correct retryability signals keep clients from retrying non-transient failures, which would otherwise multiply load during an incident.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to grpc error handling status codes, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.