gRPC Bidirectional Streaming in Production: Pitfalls and Fixes

What is a gRPC stream, and how does gRPC bidirectional streaming work on HTTP/2?

Every gRPC call, unary or streaming, maps to exactly one HTTP/2 stream. That stream has its own lifecycle, its own flow-control window (layered under the connection-wide window), a single deadline (sent as the grpc-timeout header), and exactly one final status. The status travels in trailers, a closing HEADERS frame that carries grpc-status and grpc-message. Streaming does not change this model. It only changes how many length-prefixed messages each side writes into DATA frames before it sets END_STREAM, which is the HTTP/2 flag gRPC uses as half-close.

CopyOne HTTP/2 connection (client-initiated streams use odd IDs)

stream 1  C->S  HEADERS(:path, grpc-timeout)  DATA  DATA  DATA[END_STREAM]
          S->C  HEADERS  DATA  HEADERS(grpc-status=0)[END_STREAM]
stream 3  C->S  HEADERS  DATA  DATA ...                (still open)
          S->C  HEADERS  DATA  DATA ...                (bidi, interleaved)
stream 5  C->S  HEADERS  DATA[END_STREAM]
          S->C  HEADERS  DATA DATA DATA  HEADERS(grpc-status=14)[END_STREAM]

Half-close matters because readers block on it. In a client-streaming RPC, the server handler typically reads until end-of-stream and only then computes its response. If the caller never closes its send side, the server never sees END_STREAM, so the read never returns. With no deadline set, nothing ends the call. The stream stays open, counts against the peer's concurrent-stream limit, and holds handler memory indefinitely.

Copyimport itertools
import grpc
import upload_pb2, upload_pb2_grpc

class UploadService(upload_pb2_grpc.UploaderServicer):
    def Upload(self, request_iterator, context):
        total = 0
        for chunk in request_iterator:  # blocks until the next DATA frame or END_STREAM
            total += len(chunk.data)
        return upload_pb2.Summary(bytes=total)  # sent with trailers (grpc-status=0)

def finite_chunks():
    for i in range(3):
        yield upload_pb2.Chunk(data=b"part-%d" % i)
    # Generator returns -> grpcio sends END_STREAM (client half-close)

def endless_chunks():
    # Never returns, so the client never half-closes and the server loop never ends.
    for i in itertools.count():
        yield upload_pb2.Chunk(data=b"tick")

channel = grpc.insecure_channel("localhost:50051")
stub = upload_pb2_grpc.UploaderStub(channel)

print(stub.Upload(finite_chunks(), timeout=5.0))

try:
    stub.Upload(endless_chunks(), timeout=2.0)
except grpc.RpcError as err:
    print(err.code())  # StatusCode.DEADLINE_EXCEEDED

A Python client-streaming RPC where finishing the request iterator is the half-close the server waits for, and an iterator that never ends is only stopped by the call deadline.

How does a bidirectional streaming call work, step by step?

The table and frame diagram above show the wire format. Here is the sequence of a single bidirectional call:

1. Definition. The .proto file marks both sides as streams, for example rpc Chat(stream ChatRequest) returns (stream ChatResponse);. The code generator produces a client stub and a server interface. In both, each side gets a send operation and a receive operation instead of a single request and response.
2. Call setup. The client opens a new HTTP/2 stream on an existing connection and sends a HEADERS frame. It carries the method path (:path set to /package.Service/Method), content-type: application/grpc, any metadata such as auth tokens, and grpc-timeout if the caller set a deadline. The server routes on the path and invokes the handler once for the whole call, not once per message.
3. Message exchange. Each message is serialized and framed with a 5-byte prefix: a 1-byte compressed flag and a 4-byte big-endian length. The framed messages are carried in DATA frames. A message can span several DATA frames, and one DATA frame can hold several messages. The two directions are independent. The server can send before it has read anything, and the client can keep writing while it reads. Ordering is guaranteed only within one direction of one stream.
4. Flow control. Every DATA frame consumes stream-level and connection-level window credit, which the receiver returns as it consumes messages. This is covered in detail below.
5. Half-close. When the client has nothing more to send, it sets END_STREAM on its last frame. The server's receive loop then sees end-of-stream, but the server can keep sending.
6. Completion. The server ends the call by sending trailers with grpc-status and grpc-message, flagged END_STREAM. The call is over only when the client has read that status.
7. Abnormal termination. If either side cancels, or the deadline expires, the stream is reset with an HTTP/2 RST_STREAM frame. The other side observes CANCELLED or DEADLINE_EXCEEDED. If the connection itself dies, every stream on it fails, typically with UNAVAILABLE.

Everything in the rest of this article follows from these steps. Flow control explains backpressure. The single deadline explains why long-lived streams need a different timeout strategy. The connection-scoped lifecycle explains why load balancing and deploys need special handling.

When should you use gRPC streaming, and when should you avoid it?

A stream pays off when the per-call cost of unary RPCs dominates. Each unary call carries headers, metadata, auth checks and its own deadline. A stream sets up that context once and then sends only message frames. Server push matters too. With a stream, the server sends an event when it happens, and the client does not have to ask. Price ticks, config subscriptions, job progress and telemetry ingestion fit this pattern. Backend and platform teams with high-frequency or push-style traffic benefit most.

Streams have costs. A stream is stateful, so a dropped connection is a recovery problem you must handle yourself. Connection-level load balancing pins a long-lived stream to one backend, so new replicas receive no traffic until clients reconnect. Built-in retries cannot replay a stream that has already exchanged messages. A failed stream is harder to debug than a failed request. Some proxies and gateways buffer or time out long-lived HTTP/2 streams, and gRPC-Web supports neither client streaming nor bidirectional streaming.

Start with unary calls. Move to streams when measured overhead or push latency justifies the extra state.

What are the alternatives to gRPC bidirectional streaming?

The table above compares options at a high level. Each alternative has a distinct use, and several are simpler than a full bidirectional stream:

• Server-streaming only. Many designs labelled "bidirectional" only need the server to push. The client sends one subscription request and the server streams events back. If the client occasionally needs to change its subscription, a separate unary call is often enough. You keep push delivery and drop half of the concurrency and resume problem.
• Client-streaming only. Bulk uploads and telemetry ingestion often need only the client to stream, with a single summary at the end. Resume is still needed, but the server never has to manage an outbound write path.
• Unary calls, batched. If the motivation for streaming is per-call overhead, batching several items into one unary request recovers much of the saving. You keep stateless calls, built-in retries and per-request load balancing.
• Long polling or paginated unary calls. The client asks "anything since cursor X?" and the server holds the request briefly or returns immediately. This adds latency and idle requests, but each call is independent and survives deploys and proxies cleanly.
• WebSockets. For browser clients, WebSockets provide full-duplex messaging, which gRPC-Web does not. You give up the generated stubs, the status model and gRPC deadlines, and you must define your own message framing and error semantics.
• Server-Sent Events (SSE). For browser clients that only need server push, SSE runs over plain HTTP and reconnects natively, with a last-event ID that maps to the resume offsets described later. It is one-directional and text-based.
• A message broker. When producers and consumers should be decoupled, when messages must survive consumer outages, or when one event fans out to many consumers, a broker provides durability and replay. Building those on top of gRPC streams means writing much of a broker yourself. The cost is operating another system.

A useful test: if you cannot name a message that both sides need to send while the other side is also sending, you probably do not need a bidirectional stream.

How do flow control and backpressure work in gRPC streams?

HTTP/2 gives every stream a receive window, and every connection gets a second window shared by all of its streams. A sender may only transmit DATA frames while both windows have credit. The receiver restores credit with WINDOW_UPDATE frames after the application actually consumes the messages, not when bytes reach the socket. A slow reader therefore stops returning credit. The windows drain to zero and the writer stalls. Backpressure travels end to end without any application protocol.

What the writer experiences depends on the runtime:

• Blocking. In grpc-go, Send blocks until there is window space.
• Not-ready signal. grpc-java exposes isReady() and an on-ready callback.
• Silent buffering. In Node, write() returns false but keeps accepting data. Application code that ignores the return value can buffer without limit.

The safe pattern is a single writer that owns the stream. Producers hand it work through a bounded source, or the writer pulls on demand. It sends only while the runtime reports readiness and resumes on the readiness hook. Avoid an unbounded queue in front of send, because it moves the backlog into your heap, where flow control cannot see it.

The window size is a trade-off. A large window keeps a high-latency link full, since throughput is capped at roughly window ÷ RTT. It also lets each stuck consumer pin that much buffered memory, and that cost multiplies across thousands of slow streams. Sharing a connection has costs too. Streams that share one TCP connection compete for its connection window, and a single lost packet stalls all of them at the TCP layer.

Concurrency rules are strict. Never call send from two threads at once, and never call receive from two threads at once. One sending thread and one receiving thread working concurrently is the normal arrangement.

Copyimport io.grpc.stub.ServerCallStreamObserver;
import io.grpc.stub.StreamObserver;
import java.util.Iterator;
import java.util.concurrent.atomic.AtomicBoolean;

// Inside a grpc-java service implementation.
@Override
public void export(ExportRequest req, StreamObserver<Row> responseObserver) {
  ServerCallStreamObserver<Row> call = (ServerCallStreamObserver<Row>) responseObserver;

  // Pull-based source: a row is produced only when the writer asks for it,
  // so a slow client never causes rows to pile up in our heap.
  Iterator<Row> rows = repository.scan(req.getTable());
  AtomicBoolean done = new AtomicBoolean(false);

  // Stop producing as soon as the client goes away.
  call.setOnCancelHandler(() -> done.set(true));

  // The onReady handler is the ONLY place that calls onNext, so there is exactly
  // one writer. gRPC runs it on the call's serialized executor, never concurrently.
  call.setOnReadyHandler(() -> {
    // Send only while HTTP/2 flow control reports window credit.
    while (!done.get() && call.isReady()) {
      if (!rows.hasNext()) {
        done.set(true);
        call.onCompleted();
        return;
      }
      call.onNext(rows.next());
    }
    // isReady() is false: the window is full. Return without blocking.
    // gRPC calls this handler again after the client's WINDOW_UPDATE restores credit.
  });
  // Return immediately; all sending happens in the onReady handler.
}

A grpc-java server handler that uses isReady() and the on-ready callback as a single serialized writer, pulling rows from the source on demand instead of pushing them into an unbounded queue.

How should you handle deadlines, cancellation and stream errors?

A gRPC deadline applies to the entire call, not to each message. A bidirectional stream with a 30-second deadline is terminated with DEADLINE_EXCEEDED at 30 seconds, even if messages are flowing normally. For a long-lived stream, pick one of these strategies:

Cancellation must reach the server's handler. When the client disconnects or cancels, the server's call context is cancelled. Every blocking operation in the handler should observe that context, including database queries, channel reads and downstream RPCs. If it does not, the handler's goroutine, thread or task keeps running for a client that is gone. With thousands of streams, these leaked handlers accumulate.

A stream ends with exactly one status, delivered after the last message. Every receive loop must read that status and branch on it. A loop that treats any end of stream as success will hide failures.

Treat transport failures differently from application errors. UNAVAILABLE and connection resets mean the stream broke, so reconnect with backoff. Codes such as INVALID_ARGUMENT or PERMISSION_DENIED will fail again on retry, so surface them instead.

Do not count on built-in retry policies to cover streams. A call becomes committed once the client receives response headers or a message, or once it exceeds the retry buffer. After that, the built-in policy no longer retries the call. Recovery from a broken stream is your application's job.

Copyimport random, time
import grpc

RETRYABLE = {grpc.StatusCode.UNAVAILABLE}  # stream broke; a reconnect may succeed

def consume(stub, make_requests, handle):
    backoff = 0.5
    while True:
        # No deadline: a deadline would end the whole stream, not one message.
        call = stub.Chat(make_requests())  # make_requests resumes from last ack
        try:
            for msg in call:          # raises grpc.RpcError if final status != OK
                backoff = 0.5         # data is flowing again, reset backoff
                handle(msg)
            return                    # iterator ended: final status was OK
        except grpc.RpcError as err:
            if err.code() in RETRYABLE:
                # Transport failure. Built-in retry won't help after messages
                # were exchanged, so reconnect here with jittered backoff.
                time.sleep(backoff + random.uniform(0, backoff))
                backoff = min(backoff * 2, 30)
                continue
            # Application error (INVALID_ARGUMENT, PERMISSION_DENIED, ...):
            # retrying sends the same request and fails the same way.
            raise
        finally:
            # If we exit early (exception, shutdown), cancel so the
            # server's call context is cancelled and its handler is released.
            call.cancel()


class ChatServicer:
    def Chat(self, request_iterator, context):
        for req in request_iterator:
            if not context.is_active():   # client cancelled or disconnected
                return
            yield self.process(req)

A Python bidirectional-stream client that reads the final status on every receive path, reconnects with backoff only on UNAVAILABLE, surfaces application errors, and cancels the call so the server handler, which checks context.is_active(), stops working for a departed client.

How do you resume a bidirectional stream without losing or duplicating messages?

Recovery from a broken stream is therefore an application concern, because HTTP/2 and gRPC give no exactly-once guarantee across connections. When a stream dies mid-flight, the client knows which messages it wrote, not which ones the server processed. A message may have been processed with its acknowledgement lost, or it may have been sitting in a buffer when the connection dropped. Resending everything risks duplicates. Resending nothing risks loss.

The fix is to make progress explicit in the protocol:

• Sequence numbers per session. The client tags each message with a session ID and a monotonically increasing sequence number.
• Cumulative acknowledgements. The server periodically sends back the highest sequence it has durably processed. It must commit that offset in the same transaction as the side effect. Otherwise a crash between the two reintroduces the ambiguity.
• A bounded replay buffer. The client keeps unacknowledged messages and drops them once they are acknowledged. When the buffer fills, the client stops sending. This ties resumption to the backpressure covered earlier.
• A resume handshake. After reconnecting with exponential backoff and jitter, the client's first message carries its session ID. The server replies with its last committed sequence, and the client replays only what follows it.
• Idempotent handling. The server discards any sequence at or below its committed offset, so overlapping replays are harmless.

Server-to-client traffic needs the same mechanism in reverse, with the client reporting its offset.

This adds real code: buffering, acknowledgement timers, an offset store and handshake states. If each team writes its own version, each version gets the edge cases wrong in its own way. A shared resume library, tested once against forced disconnects, lets services reuse that work.

Copyimport random, sqlite3
from collections import OrderedDict

db = sqlite3.connect("resume.db")
db.execute("CREATE TABLE IF NOT EXISTS offsets (session_id TEXT PRIMARY KEY, seq INTEGER NOT NULL)")
db.execute("CREATE TABLE IF NOT EXISTS events (session_id TEXT, seq INTEGER, payload BLOB)")

def committed_offset(session_id):
    # Sent back in the resume handshake so the client knows where to restart.
    row = db.execute("SELECT seq FROM offsets WHERE session_id = ?", (session_id,)).fetchone()
    return row[0] if row else 0

def handle(session_id, seq, payload):
    with db:  # one transaction: the event and the offset commit together or not at all
        if seq <= committed_offset(session_id):
            return False  # replayed duplicate, already applied
        db.execute("INSERT INTO events VALUES (?, ?, ?)", (session_id, seq, payload))
        db.execute("INSERT OR REPLACE INTO offsets VALUES (?, ?)", (session_id, seq))
    return True

class ReplayBuffer:
    def __init__(self, limit=1000):
        self.limit, self.pending = limit, OrderedDict()
    def full(self):
        return len(self.pending) >= self.limit  # stop sending: backpressure
    def add(self, seq, msg):
        self.pending[seq] = msg
    def ack(self, upto):
        # Cumulative ack: drop everything at or below the server's offset.
        while self.pending and next(iter(self.pending)) <= upto:
            self.pending.popitem(last=False)
    def replay_after(self, committed):
        return [(s, m) for s, m in self.pending.items() if s > committed]

def backoff(attempt, base=0.2, cap=30.0):
    # Full jitter: random wait up to an exponentially growing ceiling.
    return random.uniform(0, min(cap, base * 2 ** attempt))

Server applies each message and commits its sequence offset in one transaction, discarding replays; the client keeps a bounded buffer of unacknowledged messages, trims it on cumulative acks, and replays only what follows the server's offset after a jittered reconnect.

How should you configure keepalive and load balancing for long-lived streams?

A long-lived stream can sit idle for minutes. NAT gateways, cloud load balancers and firewalls often drop idle flows without notifying either end. Each side then waits on a socket that is already dead, and the failure surfaces only on the next write or when a deadline expires. Client keepalive prevents this by sending HTTP/2 PINGs at a fixed interval. If no ACK arrives within a timeout, the client closes the connection. Set the interval below the shortest idle timeout on the network path.

The server must agree to that interval. gRPC servers enforce a minimum ping interval and usually reject pings on connections with no active calls unless configured to allow them. A client that pings too often gets a GOAWAY, typically with ENHANCE_YOUR_CALM and "too_many_pings", and the connection closes. Configure both sides together: client interval, client timeout and whether to ping without active calls, plus the server's minimum interval and permit-without-calls setting. Option names and defaults vary by runtime, so check your runtime's documentation. Keepalive also has costs: extra traffic, and on mobile devices each ping can wake the radio.

L4 balancers assign a TCP connection to one backend. Every stream multiplexed on that connection stays there, so new pods receive no traffic.

Graceful shutdown, deploys and stream-level observability

A graceful shutdown for a streaming server has four steps, in order. First, fail readiness checks and stop accepting new streams so traffic shifts away. Second, send HTTP/2 GOAWAY. Clients then open new streams on other backends while existing streams keep running. Third, drain active streams for a fixed window, sized to your typical stream length. Fourth, cancel anything still open. The hard cutoff matters because a single idle stream can otherwise block a deploy indefinitely.

Clients must reconnect with exponential backoff and random jitter. When a pod drains, all of its streams drop at about the same moment. Without jitter, those clients retry in lockstep and overload the next pod in the rollout.

Per-stream metrics

For logging, generate a session ID on the client that persists across reconnects, and add a separate per-attempt stream ID. Log both on every message. One grep then reconstructs a session's full history across pods.

Production readiness checklist

• Drain window and hard cancel deadline are configured.
• Readiness probe fails before GOAWAY is sent.
• Client backoff includes jitter and a maximum delay.
• Resume uses acknowledged offsets.
• All four stream metrics are on a dashboard.
• Session and stream IDs appear in every log line.

How can gRPC streaming save engineering time?

Streaming costs engineering time to get right, as the sections above show. It saves time in two ways: when it replaces code you would otherwise write by hand, and when its hard parts are solved once and shared.

Where streaming removes work:

• No hand-built push or polling machinery. Without a stream, server push usually means a polling loop with cursors, tuned intervals and handling for empty responses. Alternatively, it means a side channel such as a webhook receiver. A server-streaming or bidirectional RPC replaces that with a receive loop over generated stubs.
• One contract for both directions. The .proto file defines request and response message types, and the generators produce typed clients and servers for each supported language. Teams do not write and maintain a custom framing format, a serializer, or a separate client library per language, as they would with a raw socket or WebSocket protocol.
• Flow control included. HTTP/2 windows provide backpressure without an application-level credit protocol. Your code only has to respect the runtime's readiness signal, as shown in the grpc-java example.
• Standard status and cancellation semantics. Deadlines, cancellation propagation and status codes behave the same way across languages, so error handling and on-call runbooks carry over between services.

Where shared infrastructure saves the time back:

• A shared resume library. Sequence numbers, the replay buffer, the resume handshake and offset storage are the most error-prone parts. Writing them once and testing them against forced disconnects means each new streaming service inherits correct behaviour instead of rediscovering the edge cases.
• Interceptors for observability. gRPC supports client and server interceptors for streaming calls. One interceptor can count messages per stream, record duration and final status, and attach session and stream IDs to logs. Every service then gets the per-stream metrics table above without per-service code.
• Shared configuration defaults. Keepalive values that match on client and server, maximum connection age, drain windows and backoff parameters can live in one internal configuration module. That prevents the mismatched keepalive and lockstep-reconnect problems from recurring in each team.
• A reusable disconnect test. A test harness that kills connections mid-stream, restarts servers during a drain and checks for lost or duplicated messages catches regressions before they reach a deploy.

To decide whether the trade is worth it, compare the work honestly. List the code you would write without streaming: the polling loop, cursor handling, and any custom protocol and client libraries. List the code streaming requires: the single writer, the receive loop with status handling, resume, and the shutdown sequence. If most of the second list already exists as shared libraries and interceptors, a new streaming endpoint is mostly schema and business logic. If it does not exist, the first streaming service carries that cost, and plain unary calls may remain the faster choice until a second or third use case justifies the shared investment.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to gRPC bidirectional streaming, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.