Why does gRPC streaming load balancing break request-level assumptions?

A unary gRPC call is one request and one response. A load balancer can treat it the way it treats an HTTP request: pick a backend, wait for the response, and pick again for the next call. A streaming RPC is also a single HTTP/2 stream, but it can stay open for minutes or days and carry thousands of messages in either direction. Server streaming pushes many responses for one request. Bidirectional streaming lets both sides send at any time over the same stream.

Teams choose streaming for good reasons. The server can push updates without polling. They avoid per-call setup latency. They can move high message volumes without paying header and routing overhead on every message. Long lifetime is what produces those benefits, and it is also what breaks balancing.

An L4 balancer routes TCP connections, and HTTP/2 multiplexes every stream onto one connection. All streams from that client therefore land on the same backend for as long as the connection lives. An L7 proxy or client-side balancer does better because it chooses per stream. It still chooses only once, when the stream opens. When the Kubernetes autoscaler adds replicas, existing streams stay on the old pods. The new pods sit idle until clients reconnect.

Request-rate dashboards hide this. A stream opened yesterday counts as one request, even if it now carries most of a pod's traffic. To see the imbalance, measure open streams and messages per backend.

This article is for backend and platform engineers who run server or bidirectional streams behind proxies or in Kubernetes.

(a) L4 balancer: picks a backend once per TCP connection
+-------------------+     +--------------+     +----------------------+
| Client A          |     | L4 balancer  |     | Pod 1 (old)          |
| one HTTP/2 conn   |---->| conn pinned  |---->| streams s1, s2, s3   |
|  s1  s2  s3       |     | to Pod 1     |     | all messages         |
+-------------------+     +--------------+     +----------------------+
                                               +----------------------+
                                               | Pod 3 (new): idle    |
                                               +----------------------+

(b) L7 proxy or client-side LB: picks a backend once, when a stream opens
+-------------------+     +--------------+     +----------------------+
| Client A          |     | L7 proxy     |-s1->| Pod 1 (old)  s1      |
| s1, s2 opened     |---->| or client LB |     +----------------------+
| before scale-up   |     | chooses once |     +----------------------+
+-------------------+     | per stream   |-s2->| Pod 2 (old)  s2      |
                          +--------------+     +----------------------+
                                               +----------------------+
                autoscaler adds Pod 3 ------>  | Pod 3 (new): idle    |
                                               | until a new stream   |
                                               +----------------------+

Request-rate view: each long-lived stream counts as 1 request, so Pod 1
looks light even when it carries most messages. Track open streams and
messages per backend instead.

An L4 balancer pins every stream on a connection to one pod, and an L7 balancer places each stream only when it opens, so replicas added later stay idle until clients open new streams.

Who should use these techniques?

The techniques in this article apply if you run long-lived gRPC streams and any of these are true:

  • Your backends scale horizontally. If an autoscaler or a deploy pipeline adds and removes replicas, long-lived streams will not follow those changes on their own. You need connection recycling and resumable streams to make new capacity useful.
  • Traffic crosses NAT, cloud load balancers or sidecar proxies. Every middlebox with an idle timeout can silently drop a quiet stream. You need keepalives aligned with those timeouts.
  • Consumers run at different speeds. Mobile clients, batch consumers or downstream services that pause occasionally will push back through HTTP/2 flow control. Your server code needs bounded buffers that respect that backpressure.
  • Losing or duplicating a message matters. Feeds, order events, telemetry pipelines and collaborative sessions need sequence numbers and resume handshakes so recycling a connection does not corrupt state.

If you run a single server instance, use only unary calls, or have streams that last a few seconds, most of this is unnecessary. Ordinary request-level balancing already works for you. The last section covers when to avoid streaming altogether.

How does it work?

Balancing long-lived streams is not a single setting. It is several mechanisms working together, each covering a different failure:

  1. Placement. A balancer picks a backend when a connection opens (L4) or when a stream opens (L7 or client-side). That choice is never revisited while the stream lives.
  2. Backpressure. HTTP/2 flow control limits how much data a sender can have in flight per stream and per connection. gRPC exposes this to application code as blocking sends or a readiness signal, so a slow consumer slows the producer instead of filling memory.
  3. Liveness. Keepalive PINGs keep middlebox state warm and detect dead peers. Server enforcement policy limits how often clients may ping.
  4. Rebalancing. A maximum connection age makes the server send GOAWAY on a schedule. Clients reconnect, and the balancer places the new connection using current backend membership. A grace period lets in-flight streams finish first.
  5. Continuity. Sequence numbers, checkpoints and a resume handshake let a client reopen a stream on a different backend without losing or duplicating messages. Jittered backoff keeps reconnect waves from overloading whichever backend receives them.

Placement decides where a stream starts. Connection age decides how long it stays. Resumption makes moving it safe. Keepalives and flow control keep it healthy in between. The following sections cover each mechanism in order.

How does HTTP/2 flow control apply backpressure to gRPC streams?

HTTP/2 gives every stream its own flow-control window and wraps all streams on a connection in a second, shared connection-level window. A sender may only transmit DATA frames while both windows have room. As the receiver consumes bytes, it returns credit with WINDOW_UPDATE frames, either for a single stream or for the whole connection. The protocol's default initial window is 65,535 bytes, and gRPC implementations typically raise or tune it.

When one consumer stalls, it stops sending WINDOW_UPDATE for its stream. That stream's window drains to zero and the sender pauses only that stream, so other streams keep flowing. The isolation has a limit. Bytes the receiver has accepted but the application has not read still count against the connection window. If enough stalled streams fill that shared window, every stream on the connection stops.

gRPC passes this backpressure up to application code. Blocking APIs make Send wait for window space. Async APIs expose a readiness signal, such as an isReady flag with an on-ready callback. If your code ignores that signal and pushes messages into an unbounded in-memory queue, flow control stops protecting you. The transport slows down, but your queue keeps growing until the process runs out of memory.

Window size is a trade-off. Over a high-latency link, a window smaller than the bandwidth-delay product caps throughput. A larger window fixes that, but it raises the worst-case memory buffered per stream and lets a slow consumer fall further behind before the sender feels any backpressure.

The following example writes only while the stream reports ready. It holds pending messages in a fixed-capacity queue, and when that queue is full it blocks the producer instead of growing.

// grpc-java server-streaming handler. `producers` (ExecutorService) and `eventSource` are fields.
public void subscribe(SubscribeRequest req, StreamObserver<Event> observer) {
  ServerCallStreamObserver<Event> call = (ServerCallStreamObserver<Event>) observer;
  // Fixed capacity: a slow client blocks the producer instead of growing the heap.
  BlockingQueue<Event> pending = new ArrayBlockingQueue<>(256);
  AtomicBoolean producerDone = new AtomicBoolean(false);
  AtomicBoolean completed = new AtomicBoolean(false);

  // Send only while the transport reports flow-control credit; serialize onNext calls.
  Runnable drain = () -> {
    synchronized (call) {
      while (call.isReady() && !call.isCancelled()) {
        Event e = pending.poll();
        if (e == null) break;
        call.onNext(e);
      }
      if (producerDone.get() && pending.isEmpty() && !call.isCancelled()
          && completed.compareAndSet(false, true)) {
        call.onCompleted();
      }
    }
  };

  // gRPC runs this when WINDOW_UPDATE credit makes the stream writable again.
  call.setOnReadyHandler(drain);
  // Free queue space so a producer blocked in put() wakes and sees the cancellation.
  call.setOnCancelHandler(pending::clear);

  producers.execute(() -> {
    try {
      for (Event e : eventSource.stream(req)) {
        if (call.isCancelled()) return;
        pending.put(e);   // blocks when 256 messages are already waiting
        drain.run();      // push what the window allows right now
      }
    } catch (InterruptedException ie) {
      Thread.currentThread().interrupt();
    }
    producerDone.set(true);
    drain.run();          // finish the stream once the queue empties
  });
}

A grpc-java server-streaming handler that calls onNext only while isReady() is true, resumes sending from the on-ready callback, and holds pending messages in a 256-slot ArrayBlockingQueue so a slow client blocks the producer instead of growing memory.

How should you configure gRPC keepalives without getting disconnected?

A long-lived stream can go quiet for minutes, and quiet connections are what NAT gateways, cloud load balancers and L7 proxies evict. They track flows and discard state after an idle timeout, usually without notifying either endpoint. The client still believes its stream is open, and it finds out otherwise only when the next write stalls or a reset arrives much later.

gRPC keepalives prevent this by sending HTTP/2 PING frames on the connection. The sender expects a PING ACK within a timeout and closes the transport if none arrives. This keeps middlebox state warm and detects dead peers.

The rule is to set the keepalive interval below the shortest idle timeout anywhere in the path. That path includes every NAT, load balancer and sidecar, not only the one you configured.

Servers enforce a minimum ping interval. A client that pings more often, or pings while no stream is active when the server forbids it, receives a GOAWAY with ENHANCE_YOUR_CALM and too_many_pings debug data, and the connection drops. Client keepalive settings must match server policy.

Illustrative example. Assume the shortest idle timeout in the path is 60 seconds. You could configure:

Side Option Value
Client grpc.keepalive_time_ms 30000
Client grpc.keepalive_timeout_ms 10000
Client grpc.keepalive_permit_without_calls Only if the server allows it
Server Minimum permitted ping interval (grpc.http2.min_recv_ping_interval_without_data_ms in C-core, EnforcementPolicy.MinTime in Go) 20000

Each value has a trade-off. Shorter intervals detect failures sooner. They also add traffic, multiply server ping handling across many connections, wake mobile radios and drain batteries, and risk enforcement disconnects.

from concurrent import futures
import grpc

CLIENT_OPTIONS = [
    ("grpc.keepalive_time_ms", 30_000),        # send a PING after 30s without activity
    ("grpc.keepalive_timeout_ms", 10_000),     # close the transport if no ACK within 10s
    ("grpc.keepalive_permit_without_calls", 1),  # ping with no active RPC; server must allow it
    ("grpc.http2.max_pings_without_data", 0),  # 0 = no cap on pings while a stream is quiet
]

channel = grpc.insecure_channel("orders.internal:50051", options=CLIENT_OPTIONS)

SERVER_OPTIONS = [
    # Accept pings as often as every 20s. The client pings every 30s,
    # so it stays inside the policy and avoids GOAWAY(ENHANCE_YOUR_CALM).
    ("grpc.http2.min_recv_ping_interval_without_data_ms", 20_000),
    # Allow pings when no RPC is active, matching permit_without_calls on the client.
    ("grpc.keepalive_permit_without_calls", 1),
    # Number of policy violations tolerated before the server sends GOAWAY.
    ("grpc.http2.max_ping_strikes", 2),
    # The server also pings, so it can detect clients that disappeared behind a NAT.
    ("grpc.keepalive_time_ms", 60_000),
    ("grpc.keepalive_timeout_ms", 10_000),
]

server = grpc.server(
    futures.ThreadPoolExecutor(max_workers=16),
    options=SERVER_OPTIONS,
)
server.add_insecure_port("[::]:50051")

Illustrative Python grpcio configuration in which the client's 30-second keepalive stays below an assumed 60-second idle timeout and inside the server's 20-second minimum ping interval, so the server does not disconnect the client with too_many_pings.

Bounding stream lifetime with max connection age and graceful GOAWAY

Keepalives confirm a connection is alive. They do not tell you whether it is on the right server. A client that connected before a scale-up will keep using its original backend for as long as the connection survives, so new replicas sit idle and drained replicas never fully drain. The fix is to have the server recycle connections on purpose.

Set a maximum connection age on the server, plus a grace period. When a connection reaches its age, the server sends an HTTP/2 GOAWAY. The client stops opening new streams on that connection and opens a fresh one, which the balancer can place anywhere. Existing streams keep running until the grace period ends. Then the server closes them, and the client must resume them (see the next section). Add jitter to the age so connections opened together do not all expire together.

Shorter ages rebalance faster, but each recycle costs a TLS handshake, a reconnect, and a stream resume carrying in-flight state. Also, even a perfect connection count does not mean even load. A server holding ten quiet streams is lightly loaded. A server holding ten firehose streams is not.

Illustrative estimate (assumed numbers, not a benchmark)

  • Assumed: 10 servers, 5 added, 60,000 client connections, max age A = 30 min, grace G = 5 min, random placement.
  • Share of load on new servers at time t ≤ A: (t / A) × 5/15. At 15 min that is 17%, against a target of 33%.
  • Time to reach full balance: about A + G = 35 min.
  • Reconnect rate: connections / A = 60,000 / 1,800 s ≈ 33 handshakes/s. Halving A doubles this rate.
from concurrent import futures
import grpc

MIN = 60 * 1000  # milliseconds per minute

server = grpc.server(
    futures.ThreadPoolExecutor(max_workers=32),
    options=[
        # After ~30 min the server sends GOAWAY. The client stops opening
        # streams on this connection and reconnects, so the balancer can
        # place the new connection on any backend, including new replicas.
        # gRPC core adds a small random jitter to this age, so connections
        # opened together do not all expire at the same moment.
        ("grpc.max_connection_age_ms", 30 * MIN),
        # Streams already open get 5 more minutes to finish. Anything still
        # running after that is closed, and the client must resume it.
        ("grpc.max_connection_age_grace_ms", 5 * MIN),
    ],
)

server.add_insecure_port("[::]:50051")  # use add_secure_port with TLS credentials in production
server.start()

try:
    server.wait_for_termination()
except KeyboardInterrupt:
    # On deploy, drain the same way: stop accepting new RPCs and give
    # in-flight streams the same grace window before forcing them closed.
    server.stop(grace=5 * 60).wait()

A Python gRPC server that recycles connections with a 30-minute maximum connection age and a 5-minute grace period, and uses the same grace window to drain during a deploy.

How do you implement resumable bidirectional streaming and stream deadlines?

Every forced reconnect is a stream that dies mid-flight. Unless the protocol on top of gRPC expects this, rebalancing will lose or duplicate messages. The transport cannot help here. When the stream breaks, neither side knows which in-flight messages the peer actually processed.

Resume handshake

Give every message a monotonically increasing sequence number per logical session. The receiver persists the highest sequence it has fully processed, which serves as its checkpoint. When the client reconnects, its first message carries the session ID and its last acknowledged sequence. The server replays from that point and then resumes live delivery. The handlers must also be idempotent, because a message can be processed just before the stream drops but never acknowledged. Dedupe on (session, sequence), or make the operation itself safe to repeat.

Reconnect with backoff

When a node drains, thousands of clients reconnect at once. Fixed retry intervals keep them synchronized, and the new backend absorbs a spike. Use exponential backoff with full jitter: pick the delay uniformly between zero and the current cap, then double the cap up to a ceiling. Reset the cap only after a stream has stayed healthy for a while, not merely after it connected.

Deadlines

A single overall deadline on a stream meant to run for hours either kills healthy streams or is set so high it never catches anything. Use an application-level liveness timeout instead. If neither data nor heartbeat arrives within N seconds, cancel the stream. Propagate that cancellation promptly. Server handlers should watch the stream context and release buffers, subscriptions and database cursors as soon as it is cancelled. If they don't, abandoned streams keep consuming memory until the process restarts.

What are the alternatives for gRPC streaming load balancing?

A Kubernetes ClusterIP Service balances at the connection level. kube-proxy picks a backend when the TCP connection opens. gRPC multiplexes every call and stream over one long-lived HTTP/2 connection, so each client stays pinned to a single pod. New replicas receive traffic only from clients that connect after they start. Three approaches fix this.

L4 proxy. An L4 proxy makes the same per-connection choice as a ClusterIP Service. It is cheap and protocol-agnostic. It only spreads load when there are many clients or when connections are recycled, for example by max connection age.

L7 HTTP/2-aware proxy or sidecar. This proxy terminates HTTP/2 and routes each stream independently, so balancing works per call. The costs are an extra hop, added latency and another component to deploy, upgrade and debug. The proxy also has its own idle, keepalive and stream-duration timeouts. If any of them is shorter than your stream lifetime, the proxy will cut streams that the client and server consider healthy.

Client-side balancing. The client resolves every backend address and picks one per call. With a headless Service, DNS returns pod IPs instead of a single virtual IP, so the client can open a subchannel to each pod. Every client, in every language runtime, then has to handle discovery, re-resolution, health checking and balancing policy.

Approach Balancing granularity Extra hop Main cost
ClusterIP / L4 proxy Per connection No (L4 only) Uneven load, slow rebalancing
L7 proxy or sidecar Per stream Yes Latency, operations, timeout alignment
Client-side via headless Service Per call No Per-language client logic, DNS staleness

How can it save engineering time?

Most of the time lost to long-lived streams goes to incidents that are hard to reproduce: a pod that runs hot while its neighbours idle, streams that die "randomly" every few minutes, or a server that slowly runs out of memory. Each mechanism in this article turns one of those open-ended investigations into a known setting with a known symptom.

  • Fewer mystery disconnects. When keepalive intervals are written down next to every idle timeout on the path, a dropped stream can be traced to a specific timeout or to a too_many_pings GOAWAY. Engineers stop chasing network flakiness.
  • Scaling that works without manual intervention. With a maximum connection age in place, new replicas pick up load on a predictable schedule. Nobody has to restart clients or roll old pods by hand to force rebalancing after a scale-up.
  • Deploys that don't need babysitting. Graceful GOAWAY, a grace period and resumable streams mean a rollout closes streams in a controlled way, and clients continue from their last checkpoint. There is no message reconciliation to do afterwards.
  • Memory problems caught in design, not production. A bounded queue that respects isReady makes a slow consumer show up as producer backpressure, which is visible and local. The alternative is an out-of-memory crash on a server that looked healthy.
  • Faster diagnosis. Per-backend stream counts, stream duration and reconnect rate point directly at the misconfigured layer. Request-rate dashboards would hide the problem.
  • Less code where you don't need it. The decision table in the next section helps teams keep unary calls for traffic that doesn't benefit from streaming. That avoids building resume protocols and timeout alignment for workloads that never needed them.

The work is front-loaded. You configure keepalives, connection age and backpressure once per service and add the resume protocol once per stream type. After that, these settings remove a recurring class of on-call investigations rather than adding new ones.

When should you avoid gRPC streaming, and what should you monitor if you use it?

Most of the operational cost above disappears if the traffic doesn't need a stream in the first place. A unary call is balanced per request, retried independently, and visible in ordinary request metrics. Streaming is worth it only when latency, server push, or message volume make per-request overhead the real bottleneck.

Workload Better fit Why
Request/response, even at high QPS Unary Balances per call with no connection pinning
Status checks every few seconds Polling (unary) Staleness is tolerable, and each poll rebalances
Sub-second push of server events Server streaming Polling at that rate wastes calls and still adds latency
Sustained high-volume telemetry or chat Bidirectional streaming Per-message framing costs far less than per-call setup

If you keep streaming, these metrics save the most debugging time:

  • Active streams per backend. Skew here is the first sign that long-lived streams are defeating the balancer.
  • Stream duration distribution. It should cluster near max connection age. A long tail means age limits or GOAWAY aren't being honored.
  • Flow-control stalls or send-buffer depth. Growth points to slow consumers, not slow servers.
  • Reconnect rate. Spikes that line up with deploys or age expiry indicate synchronized reconnects.
  • GOAWAY and keepalive errors. Separate "too many pings" from idle-timeout resets so each can be traced to its own setting.

Alignment checklist

  • Client keepalive interval is at least the server's minimum permitted ping interval.
  • Keepalive interval is shorter than the smallest idle timeout of every proxy and load balancer on the path.
  • Max connection age is jittered so connections don't all expire together.
  • The grace period exceeds your longest expected in-flight message exchange and fits within the proxy drain timeouts.
  • Reconnect backoff uses jitter and starts above zero, so a wave of GOAWAYs doesn't become a reconnect storm.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to grpc streaming load balancing, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.