gRPC Load Balancing on Kubernetes: Why HTTP/2 Breaks Round Robin

How a ClusterIP Service balances connections, not requests

A Kubernetes ClusterIP Service is a virtual IP. No process listens on it. kube-proxy programs the node's kernel so that traffic sent to that IP is rewritten to a real pod IP. The key detail is when that choice happens: once, at connection setup.

In iptables mode, the first packet of a new TCP connection (the SYN) walks a chain of rules. Each rule matches with a calculated probability, so one endpoint is chosen at random. The kernel then performs DNAT and records the mapping in conntrack. Every later packet on that connection matches the conntrack entry and skips the rules entirely, so it reaches the same pod. IPVS mode uses a kernel virtual server with schedulers such as round robin or least connections. It still schedules per connection and keeps a connection table, so the effect is the same.

Copyclient pod
    │  TCP connect to 10.96.0.20:50051 (ClusterIP)
    ▼
node kernel: iptables/IPVS picks ONE endpoint at SYN time
    │  conntrack: client:port → 10.244.1.7:50051
    ▼
pod-b (10.244.1.7)   ← every later byte on this connection
pod-a, pod-c         ← receive nothing from this connection

HTTP/1.1 traffic usually hides this behavior. A connection carries one request at a time, so clients open many connections in parallel through pools. Pooled connections also get closed and replaced on idle timeouts. With many connections opening over time, per-connection random choice averages out across pods, and the result looks like request-level balancing even though it is not.

Copy (a) gRPC over HTTP/2: one long-lived connection
 ┌─────────────┐  SYN  ┌──────────────────────────────┐
 │ client pod  │──────►│ ClusterIP 10.96.0.20:50051   │
 │ 1 TCP conn, │       │ iptables/IPVS picks ONE pod  │
 │ many streams│       │ conntrack pins the mapping   │
 └─────────────┘       └──────────────┬───────────────┘
                                      │ every request, every byte
                                      ▼
         ┌──────────┐            ┌──────────┐            ┌──────────┐
         │  pod-a   │            │  pod-b   │            │  pod-c   │
         │   idle   │            │ all load │            │   idle   │
         └──────────┘            └──────────┘            └──────────┘

 (b) HTTP/1.1: pooled, short-lived connections
 ┌─────────────┐  SYNs ┌──────────────────────────────┐
 │ client pod  │──────►│ ClusterIP (virtual IP)       │
 │ pool of N   │──────►│ new random pick per SYN      │
 │ TCP conns   │──────►│ pools recycle idle conns     │
 └─────────────┘       └──────────────┬───────────────┘
                                      │ connections spread over time
              ┌───────────────────────┼───────────────────────┐
              ▼                       ▼                       ▼
         ┌──────────┐            ┌──────────┐            ┌──────────┐
         │  pod-a   │            │  pod-b   │            │  pod-c   │
         │  ~1/3    │            │  ~1/3    │            │  ~1/3    │
         └──────────┘            └──────────┘            └──────────┘

kube-proxy picks a backend once per TCP connection, so a single long-lived HTTP/2 connection pins all gRPC traffic to one pod, while many pooled HTTP/1.1 connections average out across pods.

Why HTTP/2 long-lived connections pin gRPC clients to one pod

gRPC removes the churn that makes that averaging work. A gRPC channel opens an HTTP/2 connection to the address it resolved and multiplexes every RPC over it as a separate stream. Concurrent calls do not need extra connections, so the client never opens more, and a healthy channel can stay connected for the life of the process. kube-proxy chose a backend once, when the TCP connection was established. Every RPC sent after that goes to the same pod.

The skew follows directly. Illustrative estimate, assumed inputs: three client pods, each holding one channel, call a service with ten replicas. At most three replicas receive traffic and seven sit idle. If two clients' connections land on the same pod, only two replicas do all the work.

Scaling out does not help. Existing connections stay where they are, so a new pod only gets traffic from connections opened after it becomes ready. For long-running clients, that may never happen. An autoscaler reacting to CPU on the hot pods can keep adding replicas that stay idle while the original pods remain saturated.

The usual workarounds reduce the skew but do not fix it. Opening several channels per client spreads load across more pods, but each connection is still a random, fixed pick, so collisions and idle replicas remain. Forcing reconnects on a timer or at random intervals rebalances gradually, but load then depends on connection age and timing. That makes periodic latency spikes and uneven per-pod graphs hard to explain.

How to diagnose uneven gRPC load across pods

Before changing Service types or client code, confirm that connection pinning is actually the cause. Three signals together make the case.

Compare per-pod request rate and CPU. Run kubectl top pods with a label selector for the backend and look for a few replicas near their CPU limit while others sit idle. CPU alone can mislead, so pair it with a per-pod request-rate query from your metrics system: the rate of the server's handled-RPC counter, grouped by pod. If a few pods carry most of the requests, load is skewed.

Count connections per client. Check the server's open-connection gauge, or count established TCP connections on the gRPC port inside each pod. If each client instance holds one or two connections, and the busy pods are the ones those connections land on, the client is pinned. Compare the distinct client count against the replica count. Ten clients spread across thirty pods cannot cover every pod.

Watch a scale-up or rolling restart. Scale the Deployment and confirm with kubectl get endpointslices that the new pods are listed and ready. Ready pods that receive almost no traffic until clients reconnect are the signature of this problem. The skew then reshuffles at each reconnect instead of evening out.

Rule out other causes. Hot keys cause skew that follows the data. The same pods stay busy no matter which clients connect, and the request count per connection is uneven. A slow pod shows higher latency per request at a similar request rate, which usually points to a noisy node or a bad replica. Connection pinning shows something different: latency per request looks normal, and load shifts whenever connections are re-established.

Copy#!/usr/bin/env bash
# Diagnose connection pinning for a gRPC backend. Placeholder names below.
set -euo pipefail
NS=payments
APP=ledger-grpc        # pod label app=<APP>
SVC=ledger-grpc        # Service name in front of the pods
PORT=50051             # gRPC server port

# 1. CPU per pod, busiest first. A few hot pods next to idle ones suggests skew.
kubectl top pods -n "$NS" -l app="$APP" --sort-by=cpu

# 2. Established connections per pod and distinct client IPs per pod.
#    Requires `ss` in the container image. Few clients holding long-lived
#    connections to only the busy pods points to pinning.
for pod in $(kubectl get pods -n "$NS" -l app="$APP" -o name); do
  conns=$(kubectl exec -n "$NS" "$pod" -- \
    ss -tn state established "( sport = :$PORT )" | tail -n +2)
  total=$(printf '%s\n' "$conns" | grep -c . || true)
  # Column 4 is the peer address when a state filter is used; strip the port.
  clients=$(printf '%s\n' "$conns" | awk 'NF {print $4}' \
    | sed 's/:[0-9]*$//' | sort -u | grep -c . || true)
  echo "$pod connections=$total distinct_clients=$clients"
done

# 3. Scale up, then confirm the new pods are registered and ready.
kubectl scale deployment/"$APP" -n "$NS" --replicas=6
kubectl rollout status deployment/"$APP" -n "$NS"
kubectl get endpointslices -n "$NS" -l kubernetes.io/service-name="$SVC" -o wide

# Re-run steps 1 and 2 after a few minutes. Ready pods showing near-zero
# connections and CPU until clients reconnect confirm pinning. Pair this
# with a per-pod request rate from your metrics system, for example the
# rate of the server's handled-RPC counter grouped by pod.

A shell walkthrough that compares per-pod CPU, counts established gRPC connections and distinct client IPs in each pod, then scales the Deployment and checks EndpointSlices so you can see whether new ready pods stay idle until clients reconnect.

Fix 1: headless Service grpc with client-side load balancing

The most direct fix moves balancing into the client. Set clusterIP: None on the Service so cluster DNS returns one A record per ready pod instead of a single virtual IP. Point the client at a dns:/// target such as dns:///orders.prod.svc.cluster.local:50051 and select round_robin through the service config's loadBalancingConfig. The client then holds one subchannel per pod and rotates RPCs across them, so a single long-lived connection no longer carries all traffic.

Where DNS discovery falls short

The gRPC DNS resolver does not watch for changes. It re-resolves on events such as a subchannel failing, so new pods can sit idle while existing connections stay healthy. DNS caching and record TTLs add more delay. Readiness probes decide which pods appear in the records, so a probe that passes too early sends traffic to pods that cannot yet serve it.

To force periodic re-resolution, set a maximum connection age on the server, with a grace period that lets in-flight RPCs finish. If the age is too short, clients spend CPU on TLS handshakes and see latency spikes. If it is too long, scale-ups take a long time to receive traffic. Pair it with graceful shutdown: the pod should fail readiness, send GOAWAY, and drain before it exits, so clients holding stale entries do not send RPCs to a terminating pod.

Connection count grows roughly as clients × pods. That affects file descriptor limits and keepalive traffic. Subsetting limits the count, but you then have to manage subset assignment and uneven load across subsets.

Copyimport json
import signal
from concurrent import futures

import grpc

# --- Client: resolve every ready pod IP through the headless Service ---
# The Service must set clusterIP: None so DNS returns one A record per ready pod.
SERVICE_CONFIG = json.dumps({"loadBalancingConfig": [{"round_robin": {}}]})

def make_channel():
    # dns:/// selects the DNS resolver; round_robin opens a subchannel per
    # resolved address and rotates RPCs across them.
    return grpc.insecure_channel(
        "dns:///orders.prod.svc.cluster.local:50051",
        options=[("grpc.service_config", SERVICE_CONFIG)],
    )

# --- Server: cap connection lifetime so clients reconnect and re-resolve ---
def serve():
    server = grpc.server(
        futures.ThreadPoolExecutor(max_workers=16),
        options=[
            # After ~5 min the server sends GOAWAY; clients reconnect and
            # re-resolve DNS, which lets them discover newly scaled-up pods.
            ("grpc.max_connection_age_ms", 300_000),
            # Let in-flight RPCs finish before the connection is closed.
            ("grpc.max_connection_age_grace_ms", 30_000),
        ],
    )
    # register your servicers here, e.g. add_OrdersServicer_to_server(...)
    server.add_insecure_port("[::]:50051")
    server.start()

    # On SIGTERM, stop accepting new RPCs and drain existing ones. Keep the
    # grace below terminationGracePeriodSeconds and let readiness fail first.
    def on_sigterm(signum, frame):
        server.stop(grace=25)
    signal.signal(signal.SIGTERM, on_sigterm)

    server.wait_for_termination()

A Python gRPC client uses a dns:/// target against a headless Service with round_robin set in the service config, and the server caps connection age with a grace period and drains on SIGTERM, which forces clients to reconnect and re-resolve.

Alternatives: L7 proxies, service meshes and xDS-style balancing

If you would rather not change application clients at all, move the balancing decision into infrastructure that understands HTTP/2. An L7 proxy terminates the client's connection, decodes HTTP/2 frames, and sends each RPC to an upstream pod of its choosing. The client still holds one long-lived connection, but now it points at the proxy, and the proxy spreads streams across pods.

You can deploy that proxy in two ways. A sidecar mesh runs a proxy next to every pod. A dedicated proxy tier is a shared Deployment that sits in front of the backend Service. Either way, the costs are similar:

Each proxy is another component that can fail or be misconfigured. When latency spikes, debugging moves into proxy configuration, proxy metrics and proxy logs, so your team has to learn those.

Lookaside, or xDS-style, balancing keeps the decision in the client but takes the logic out of application code. Clients subscribe to a control plane, and it pushes them endpoint lists and balancing policy. There is no extra hop, and policy is managed centrally. This tends to pay off for large fleets, where per-service DNS tuning does not scale.

Do not stack these approaches. If you run client-side balancing behind a mesh proxy, the client balances across the local proxy rather than across pods. At best that work is redundant. At worst, the client's retries and connection churn work against the proxy's own decisions. Pick one layer to own load balancing.

Choosing a policy beyond round robin, and the streaming problem

Round robin assumes every request costs roughly the same. That holds for uniform unary calls, but it breaks down when one method returns in a millisecond and another runs a ten-second report. Each pod gets the same count of requests, yet one pod can hold most of the expensive ones. A least-request policy picks among endpoints by how many requests are currently outstanding on each. A common form samples two endpoints at random and picks the less busy one. Outstanding-request count is a rough but cheap stand-in for load. It reacts to slow pods automatically, and sampling avoids every client stampeding to the same "idlest" pod. Some gRPC implementations and most L7 proxies offer this, so check what your stack supports.

Streams are balanced once

Server-streaming and bidirectional RPCs are balanced only at the moment they open. After that, a stream stays bound to its pod for its whole life. No connection-level or request-level policy can move it. If clients open streams that run for hours, pods that were busy during a rollout keep those streams, and newly scaled pods sit idle.

Two mitigations help:

• Bound stream lifetime. The server ends streams, or whole connections, after a maximum age, ideally with jitter so they don't all expire at once. The client then reopens the stream and gets re-balanced. The cost is reconnect overhead and a brief gap in delivery.
• Design clients to resume. Treat stream termination as normal, not as an error. Reconnect with backoff, and resume from a cursor or sequence number so no messages are lost or duplicated. This is real protocol work, but without it, bounded lifetimes become outages.

When to use each approach, when to avoid it, and the cost in engineering time

The right choice depends less on which mechanism balances best and more on who owns the decision and how many places it has to be configured.

Client-side LB has no extra hop, but its cost grows with each client. Every language's gRPC library has its own resolver and policy settings, so behavior can drift between services. Each fix also needs a coordinated upgrade in every application. A proxy or mesh puts that work in one place, but it needs ongoing operations: upgrades, capacity planning and on-call.

Illustrative estimate (assumed numbers, not a benchmark):

• Client-side cost = services × hours per change. With 30 services × 4 hours, that is 120 hours per policy change.
• Proxy cost = fixed operations hours plus 1 central change. With 40 + 2, that is 42 hours.

With few clients, the result flips in favor of client-side LB.

The bigger saving is in debugging. When one layer is clearly responsible for balancing, "why is pod-3 hot?" has a single place to look.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to grpc load balancing kubernetes, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.