gRPC Load Balancing on Kubernetes: Why Connections Stick and Fixes

How a ClusterIP Service balances traffic, and why gRPC defeats it

A ClusterIP Service has a virtual IP that no process listens on. kube-proxy programs each node so traffic sent to that IP is rewritten to a backend pod. In iptables mode, a rule picks a pod at random when the first packet of a new connection arrives. In IPVS mode, the kernel's IPVS scheduler picks one, round-robin by default. In both modes, conntrack records the choice, and every later packet on that TCP connection goes to the same pod. Balancing is L4 and happens once, at connect time. kube-proxy never sees requests, only connections.

That works for HTTP/1.1 clients that open many short connections. It fails for gRPC. gRPC runs on HTTP/2, which multiplexes many concurrent RPCs as independent streams over a single TCP connection. A gRPC channel pointed at a ClusterIP resolves one address, opens one connection, and keeps it open for the process lifetime. This connection reuse is a deliberate design choice that avoids handshake cost, but it means every RPC from that client lands on whichever pod the connection reached first. Scaling the Deployment from 3 to 10 pods changes nothing for existing clients, because they never reconnect.

CopyL4 (kube-proxy): pod chosen per connection
  client A ══ one TCP conn ══> pod-1   (RPC 1..N all here)
  client B ══ one TCP conn ══> pod-1   (random pick, same pod)
                               pod-2   idle
                               pod-3   idle

L7 (per-request): pod chosen per RPC
  client A ──> balancer ──> RPC1 pod-1
                        ──> RPC2 pod-2
                        ──> RPC3 pod-3

CopyL4: kube-proxy picks a pod once per TCP connection (conntrack pins it)

+----------+  one TCP conn, HTTP/2     +------------+          +-------+
| client A |==========================>|  ClusterIP |=========>| pod-1 |
+----------+  RPC1 ... RPCn as streams |  (VIP)     |    ^     +-------+
+----------+                           | iptables / |    |     +-------+
| client B |==========================>|   IPVS +   |====+     | pod-2 | idle
+----------+  one TCP conn, HTTP/2     | conntrack  |          +-------+
                                       +------------+          +-------+
                                                               | pod-3 | idle
                                                               +-------+

L7: a gRPC-aware balancer picks a pod for every RPC

                                                        +-------+
                               +---------------+   +--->| pod-1 | RPC1, RPC4
+----------+                   | L7 balancer   |   |    +-------+
| client A |==================>| gRPC-aware    |   |    +-------+
+----------+  one TCP conn     | picks per RPC |---+--->| pod-2 | RPC2, RPC5
                               +---------------+   |    +-------+
                                                   |    +-------+
                                                   +--->| pod-3 | RPC3, RPC6
                                                        +-------+

===>  long-lived HTTP/2 connection     --->  per-RPC routing decision

kube-proxy chooses a backend only when a TCP connection opens, so every multiplexed gRPC stream on that connection lands on the same pod, while an L7 balancer makes a new choice for each RPC.

Symptoms: how to tell your gRPC traffic is pinned

The problem rarely shows up as an error. Requests succeed and latency looks acceptable on average, but the load across replicas is uneven. Three patterns are typical.

Hot pods next to idle ones. A few replicas run near their CPU limit while others in the same Deployment sit mostly unused. The distribution follows which pods the clients happened to connect to, not how much capacity each pod has.

New replicas get little or no traffic. The HPA adds pods during a spike, and they pass readiness checks, but existing clients keep sending RPCs over connections they already hold. Only connections opened after the scale-up can reach the new pods. The pods you added for the spike stay idle while the overloaded ones keep scaling the metric up.

Imbalance lasts until something breaks the connection. The skew stays until a client restarts, a pod is deleted, or a network event drops the connection. After a rolling deploy of the clients, the load can briefly look balanced and then drift back.

To confirm it, measure requests per second per pod, either from server-side gRPC metrics or from a dashboard grouped by pod. Do not rely on connection counts. Each pod may show a similar number of connections while one connection carries most of the traffic, because HTTP/2 multiplexes many RPCs over a single connection.

Not every service needs a fix. If your clients are short-lived (batch jobs, CLI tools, serverless functions) or send little traffic, connections turn over often enough that the load evens out without intervention. The problem matters most for long-running, high-throughput callers such as API gateways and other backend services.

Client-side load balancing with a gRPC headless Service

The fix is to stop asking the kernel to balance traffic and let the gRPC client do it. That starts with a headless Service: the same manifest you already have, with the same selector and port, plus clusterIP: None. Kubernetes then allocates no virtual IP and kube-proxy programs no forwarding rules. A DNS query for orders.payments.svc.cluster.local returns one A record per ready pod. Pods that fail their readiness probe are left out of the answer, so the client only receives addresses that should be able to serve.

Next, point the client at the dns resolver, for example dns:///orders.payments.svc.cluster.local:50051. The resolver passes the full address list to the load-balancing policy. The default policy is pick_first, which connects to the first reachable address and sends every request there. With pick_first, the headless Service gives you exactly the pinning you had before. Override it through service config with {"loadBalancingConfig": [{"round_robin": {}}]}. In Go, you pass this to grpc.WithDefaultServiceConfig. Other languages expose an equivalent default-service-config option on the channel builder. With round_robin, the client opens one subchannel per pod and rotates requests across the ready ones. Because HTTP/2 multiplexes many requests on each connection, the spread holds even for a single long-lived channel.

This approach fits best when:

• You support one or two client languages.
• Your team owns and ships the client code.
• The path is latency-sensitive, and adding a proxy hop or sidecar would cost you.

Section 4 covers its main weakness: the client only refreshes the pod list at certain times.

Copyimport json

import grpc

# Generated from your .proto; placeholder module and message names.
import orders_pb2
import orders_pb2_grpc

# Target the headless Service (clusterIP: None) through the dns resolver.
# DNS returns one A record per ready pod, and the client sees every address.
TARGET = "dns:///orders.payments.svc.cluster.local:50051"

# Replace the default pick_first policy with round_robin.
# The client opens one subchannel per pod and rotates requests across
# the ready subchannels.
SERVICE_CONFIG = json.dumps({
    "loadBalancingConfig": [{"round_robin": {}}]
})


def make_channel() -> grpc.Channel:
    # "grpc.service_config" sets the default service config for this channel.
    # It is the Python counterpart of Go's grpc.WithDefaultServiceConfig.
    return grpc.insecure_channel(
        TARGET,
        options=[("grpc.service_config", SERVICE_CONFIG)],
    )


def main() -> None:
    # Use one long-lived channel. Calls are spread across pods even
    # though the application never opens a second channel.
    with make_channel() as channel:
        stub = orders_pb2_grpc.OrdersStub(channel)
        for i in range(10):
            reply = stub.GetOrder(orders_pb2.GetOrderRequest(id=f"order-{i}"))
            print(reply)


if __name__ == "__main__":
    main()

A Python gRPC client resolves a headless Service through the dns resolver and sets round_robin in the channel's service config, so one long-lived channel spreads its calls across all ready pods.

Fixing DNS staleness with MAX_CONNECTION_AGE

…list when something forces it to. The gRPC DNS resolver does not poll on a timer. It re-resolves when a connection drops or the server tells it to go away. In a healthy cluster connections rarely drop, so a scaled-up Deployment can add pods that never appear in any client's address list.

The fix is to make connections expire on the server side. gRPC servers expose two keepalive settings for this. Max connection age sets how long a connection may live before the server sends an HTTP/2 GOAWAY. Max connection age grace sets how long in-flight RPCs get to finish before the server closes the connection. When a client receives GOAWAY, it re-resolves DNS, picks up the current pod IPs, and connects to any new pods. Most implementations add jitter to the age so that clients do not all reconnect at once.

The tuning is a trade-off. Each reconnect costs a TCP handshake, a TLS handshake, and a new HTTP/2 setup, and it can cause a brief p99 spike. If the age is too short, you spend CPU on handshakes. If it is too long, the imbalance stays in place.

Illustrative estimate. Assume a max age of 5 minutes, ±10% jitter, a 30-second DNS cache, and connections opened at random times. A new pod's first traffic arrives within seconds to a few minutes, as soon as the first client connection expires. A full share of traffic takes up to about 5.5 minutes plus 30 seconds. That is roughly 6 minutes in the worst case.

Copyfrom concurrent import futures

import grpc

# Values match the illustrative estimate: 5-minute max age, 30-second grace.
MAX_CONNECTION_AGE_MS = 5 * 60 * 1000
MAX_CONNECTION_AGE_GRACE_MS = 30 * 1000

server_options = [
    # After this age the server sends HTTP/2 GOAWAY. The client then
    # re-resolves DNS and can connect to pods added since the last resolve.
    # gRPC core applies jitter to this value so clients do not all
    # reconnect at the same moment.
    ("grpc.max_connection_age_ms", MAX_CONNECTION_AGE_MS),
    # After GOAWAY, in-flight RPCs get this long to finish before the
    # server closes the connection. Set it above your longest normal RPC.
    ("grpc.max_connection_age_grace_ms", MAX_CONNECTION_AGE_GRACE_MS),
]


def serve() -> None:
    server = grpc.server(
        futures.ThreadPoolExecutor(max_workers=16),
        options=server_options,
    )
    # Register your service implementations here, for example:
    # orders_pb2_grpc.add_OrdersServicer_to_server(OrdersServicer(), server)
    server.add_insecure_port("[::]:50051")
    server.start()
    server.wait_for_termination()


if __name__ == "__main__":
    serve()

A Python gRPC server that sets max connection age and max connection age grace, so clients receive GOAWAY every few minutes, re-resolve DNS, and start sending traffic to newly scaled pods.

Graceful shutdown during rollouts

Scale-ups are only half of a rollout. Scale-downs and pod replacements are where RPCs actually fail.

When Kubernetes deletes a pod, two things start in parallel. The control plane marks the endpoint as not ready, so it drops out of headless DNS answers and proxy endpoint lists. The kubelet runs the pod's preStop hook and then sends SIGTERM. Neither waits for the other. Clients that already hold an HTTP/2 connection to the pod don't consult DNS again. They keep multiplexing new RPCs onto that connection until the server tells them to stop or the connection breaks.

Coordinating the shutdown

1. Add a preStop hook that sleeps a few seconds. This delays SIGTERM long enough for endpoint removal to reach resolvers, proxies, and kube-proxy, so new connections stop arriving before the server begins closing.
2. On SIGTERM, set the gRPC health service to NOT_SERVING and begin a graceful stop. In most implementations, graceful stop sends GOAWAY, refuses new streams, and waits for in-flight streams to finish. A client that receives GOAWAY re-resolves and opens connections to other backends without failing the RPCs it already sent.
3. Bound the drain. Long-lived streams can block a graceful stop indefinitely. Run it against a deadline, then force-close whatever remains.
4. Size terminationGracePeriodSeconds to cover the whole sequence: the preStop sleep plus the drain deadline plus some margin. When the grace period expires, the kubelet sends SIGKILL regardless of what is still running.

Why this applies to every approach

This sequence is needed whichever balancing approach you choose. An L7 proxy or a sidecar holds pooled connections to the terminating pod in the same way a client does. A proxy only drains cleanly if the backend signals with GOAWAY and stays alive long enough for in-flight RPCs to complete.

Copy# Pod spec this server is paired with (YAML shown as comments):
#   terminationGracePeriodSeconds: 35          # preStop 5s + drain 20s + 10s margin
#   lifecycle:
#     preStop:
#       exec: {command: ["sleep", "5"]}       # let endpoint removal propagate first
import signal
import threading
from concurrent import futures

import grpc
from grpc_health.v1 import health, health_pb2, health_pb2_grpc

DRAIN_SECONDS = 20  # upper bound on waiting for in-flight RPCs and streams


def serve() -> None:
    server = grpc.server(futures.ThreadPoolExecutor(max_workers=16))
    health_servicer = health.HealthServicer()
    health_pb2_grpc.add_HealthServicer_to_server(health_servicer, server)
    # add_MyServiceServicer_to_server(MyService(), server)  # app services here
    server.add_insecure_port("[::]:50051")
    server.start()
    health_servicer.set("", health_pb2.HealthCheckResponse.SERVING)

    # SIGTERM arrives only after the preStop sleep has finished.
    terminating = threading.Event()
    signal.signal(signal.SIGTERM, lambda *_: terminating.set())
    terminating.wait()

    # Tell health-checking clients and proxies to stop picking this backend.
    health_servicer.set("", health_pb2.HealthCheckResponse.NOT_SERVING)

    # Graceful stop: reject new RPCs, let in-flight ones run for up to
    # DRAIN_SECONDS, then cancel whatever is left. Exiting before SIGKILL.
    server.stop(grace=DRAIN_SECONDS).wait()


if __name__ == "__main__":
    serve()

A Python gRPC server that, after a preStop sleep, flips its health status to NOT_SERVING on SIGTERM and runs a bounded graceful stop sized to fit inside terminationGracePeriodSeconds.

Alternatives: L7 proxies, service meshes, and crude workarounds

An HTTP/2-aware proxy moves the balancing decision off the client. The client opens one long-lived connection to the proxy. The proxy terminates it, decodes HTTP/2 frames, and routes each request to a backend pod over its own connection pool. A sidecar mesh applies the same idea per pod, intercepting outbound traffic locally. Clients stay simple, and every language gets identical behaviour.

That simplicity has costs. Each request takes an extra hop, or two with sidecars on both ends. You pay CPU and memory for every proxy instance, either per pod or per node. You also add a component that can be misconfigured, upgraded badly, or become the place where a timeout actually happens. That makes it harder to see where failures originate.

The crude workarounds are opening N connections per client or rebuilding the client on a timer. Both spread load statistically but do not balance it. With few clients and many pods, the distribution stays lumpy. New pods still wait for the next reconnect cycle. Recreating clients can also mask a slow pod or a broken resolver, because the symptom fades without the cause ever being found.

When to avoid each approach, and the impact on cost and engineering time

Client-side balancing

Avoid it when many languages and teams must keep resolver, policy, and keepalive settings consistent. Each runtime has its own defaults, and one misconfigured client quietly reverts to pinning. Development cost recurs: every new service and library upgrade needs the configuration re-checked.

Watch connection growth in fan-out topologies, because every client opens a connection to every pod.

Illustrative estimate (assumed numbers, not a benchmark): connections = client pods × server pods × channels per client. With 200 client pods, 60 server pods, and 1 channel each, that is 200 × 60 × 1 = 12,000 connections. Doubling both tiers gives 48,000, so the count grows with the product of the two tiers. Each connection costs memory and keepalive traffic on both ends.

round_robin balances by request count, not by cost. If some requests are much heavier than others, or pods have different CPU limits, a pod can be overloaded while its request count looks even.

Service mesh

Avoid it when the latency budget is tight, since each sidecar adds a hop. Also avoid it when no team can own upgrades, certificate rotation, and proxy debugging. You trade per-client configuration for a platform component that must stay healthy, and when it fails, every service is affected.

Choosing deliberately

Pinning wastes engineering time on incidents that look like capacity problems. It also leads to over-provisioning, because teams add replicas that never receive traffic.

To verify a change:

1. Compare per-pod request rates before and after the change. The spread between the busiest and the idlest pod should narrow.
2. Run a scale-up test. Confirm that new pods receive traffic within the expected reconnect window: MAX_CONNECTION_AGE plus the grace period plus DNS re-resolution time.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to grpc load balancing kubernetes, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.