Why does gRPC load balancing break on Kubernetes Services?
A Kubernetes ClusterIP Service is a virtual IP. When a client opens a TCP connection to it, kube-proxy (in iptables or IPVS mode) picks one backend pod and connection tracking pins the flow to that pod. That choice happens once per connection, at L4. kube-proxy never sees individual requests.
This works for HTTP/1.1 clients that open many short connections, because each new connection is a new chance to land on a different pod. gRPC does the opposite. A gRPC channel normally opens one HTTP/2 connection and multiplexes every RPC over it as separate streams. HTTP/2 long-lived connections can stay open for hours or days. So every request from that client goes to whichever pod kube-proxy chose when the connection was first made.
How the skew shows up
Take an illustrative case: 4 client pods and 4 server pods. Each client picks a backend at random, and the picks often collide. Two clients may land on the same server while another server gets nothing. The load stays that uneven for as long as those connections live.
Scaling makes this worse. When an autoscaler adds pods to relieve the busy ones, existing connections stay where they are. The new pods receive traffic only from clients that reconnect, which may not happen until a client restarts. Rolling deployments cause a similar imbalance: clients that reconnect after a pod is terminated tend to bunch up on whichever pods are ready at that moment.
Adding replicas does not fix this. Balancing has to happen per request, not per connection. The real decision is where that request-level balancing lives: inside the client, or in a proxy that understands HTTP/2.
+----------+ +----------+ +----------+ +----------+
| client A | | client B | | client C | | client D |
+----------+ +----------+ +----------+ +----------+
| | | | one long-lived
| | | | HTTP/2 conn each,
v v v v RPCs = streams
+------------------------------------------------------------------+
| ClusterIP Service (virtual IP), kube-proxy iptables/IPVS, L4 |
| picks a backend pod ONCE per TCP connection; conntrack pins it |
| never sees RPCs: every stream follows its connection's pod |
+------------------------------------------------------------------+
A B C D
| | | |
v v v v
+----------+ +----------+ +----------+ +----------+ +----------+
| pod 1 | | pod 2 | | pod 3 | | pod 4 | | pod 5 |
| HOT | | busy | | busy | | idle | | NEW |
| 2 conns | | 1 conn | | 1 conn | | 0 conns | | 0 conns |
| ######## | | #### | | #### | | | | |
+----------+ +----------+ +----------+ +----------+ +----------+
pod 4: Ready, but no connection happened to land on it
pod 5: added by autoscaler; idle until some client reconnects
rollouts: reconnecting clients bunch onto pods Ready at that moment
Illustrative example: kube-proxy picks a pod once per TCP connection, so every RPC on a client's long-lived HTTP/2 connection goes to the same pod. Pod 1 gets two clients, pod 4 gets none, and pod 5, added by the autoscaler, stays idle until some client reconnects.
How does request-level gRPC load balancing work?
Both approaches in this article work on the same principle. Something that understands HTTP/2 has to see every backend address and choose a backend for each RPC, not for each TCP connection. The two models differ only in where that component runs.
- Discovery. The balancer needs the full list of pod IPs, not the single ClusterIP. In both models this usually comes from a headless Service, whose DNS name returns one A record per ready pod. A service mesh control plane can provide the same list through an endpoint API instead.
- Connection pool. The balancer opens an HTTP/2 connection to each backend pod. In gRPC terms each of these is a subchannel. In Envoy it is a connection in the upstream cluster's pool.
- Per-RPC pick. Every new RPC is a new HTTP/2 stream. The balancer runs its policy, such as round-robin or least-request, and sends the stream over the connection to the chosen pod.
- Updates. When pods are added or removed, the balancer refreshes its endpoint list through DNS re-resolution, a pushed endpoint update, or a reconnect forced by the server. It then opens or closes connections to match.
- Health. Health checks and error tracking remove failing backends from the pick set before discovery catches up.
With client-side balancing, steps 1 through 5 run inside the gRPC library in each client process. With Envoy, they run in the proxy, and the client keeps a single connection to the proxy. The sections below cover each step in detail for both models.
How does client-side gRPC load balancing work with a headless Service?
A normal ClusterIP Service gives clients one virtual IP and hides the pods behind it. A headless Service removes that layer. When you set clusterIP: None, Kubernetes allocates no virtual IP and kube-proxy does not program forwarding rules for the Service. The cluster DNS name my-svc.my-ns.svc.cluster.local then returns one A record per ready pod. The client sees every backend address and can open a connection to each one.
The manifest is an ordinary Service with clusterIP: None and a selector matching your pods. No kube-proxy translation happens, so clients connect straight to the pod IP. The port in your dial target must therefore be the port the container actually listens on.
The gRPC client has to be told to use those addresses. Two settings matter:
- Resolver: Use a target such as
dns:///my-svc.my-ns.svc.cluster.local:50051. The explicitdnsscheme makes the client fetch all A records instead of passing the hostname through as a single address. Some implementations default to a different resolver when no scheme is given. - Balancing policy: The default policy is
pick_first, which connects to one address and sends everything there. That recreates the original problem. Setround_robinthrough the service config, for example{"loadBalancingConfig": [{"round_robin": {}}]}. The client then keeps a subchannel to every resolved pod and rotates requests across them.
How you supply that config differs by language. Some implementations take a JSON default service config. Others expose a builder method for the default policy. Defaults for resolver scheme and DNS re-resolution also vary between implementations, so check the documentation for each language you ship.
import json
import grpc
TARGET = "dns:///my-svc.my-ns.svc.cluster.local:50051"
SERVICE_CONFIG = json.dumps({"loadBalancingConfig": [{"round_robin": {}}]})
channel = grpc.insecure_channel(
TARGET,
options=[("grpc.service_config", SERVICE_CONFIG)],
)
A headless Service manifest, shown as a comment, and a Python gRPC channel that uses the dns resolver with a round_robin service config so calls spread across every pod IP.
How do clients discover new and removed pods, and why set a max connection age?
Most gRPC DNS resolvers do not poll. They resolve on startup and again when the channel signals trouble, such as a subchannel disconnecting or a connection attempt failing. As long as every existing connection stays healthy, the client never asks DNS again. Pods added by a scale-up or a rollout stay invisible, possibly for the lifetime of the process, and all traffic keeps flowing to the original set.
Removed pods follow a different path. When a pod terminates, its connections close, and that triggers re-resolution. The answer can still be stale, though. DNS caches in CoreDNS, in node-local caching, or in the client runtime may keep returning the old IP for a short window. During that window the client can try to connect to an endpoint that no longer exists, which costs failed attempts and backoff before the endpoint list converges.
Forcing periodic reconnects
A server-side max connection age fixes the scale-up blind spot. When a connection reaches that age, the server sends GOAWAY. In-flight RPCs are allowed to finish, new RPCs move to other connections, and the client re-resolves. The max connection age grace setting is a hard deadline after GOAWAY, at which the server forcibly closes any streams still open. Adding jitter to the age prevents all clients from reconnecting at the same moment.
This approach has costs:
- Reconnect churn. Each reconnect means a new TCP connection and, in most deployments, a full TLS handshake. A short age multiplies CPU and latency overhead across every client.
- Interrupted streams. Long-lived server or bidirectional streams that outlast the grace period are cut off. Set the grace period longer than your longest expected stream, or make clients resume streams.
Pick an age of minutes, not seconds. Rebalancing after a scale-up then completes within one age interval, and handshake overhead stays negligible.
from concurrent import futures
import grpc
SERVER_OPTIONS = [
# Send GOAWAY after ~5 minutes so clients reconnect, re-resolve DNS,
# and pick up pods added since the connection was opened.
# C-core applies a small random jitter so clients don't reconnect in lockstep.
("grpc.max_connection_age_ms", 5 * 60 * 1000),
# After GOAWAY, give in-flight RPCs and streams up to 10 minutes to finish
# before the server force-closes them. Size this above your longest stream.
("grpc.max_connection_age_grace_ms", 10 * 60 * 1000),
# Close connections that have carried no RPCs for 15 minutes.
("grpc.max_connection_idle_ms", 15 * 60 * 1000),
# Ping clients every 60s and drop the connection if no ack within 20s,
# so dead peers (e.g. a vanished node) are detected promptly.
("grpc.keepalive_time_ms", 60 * 1000),
("grpc.keepalive_timeout_ms", 20 * 1000),
# Accept client keepalive pings even when no RPC is active, but
# no more often than every 30s.
("grpc.keepalive_permit_without_calls", 1),
("grpc.http2.min_ping_interval_without_data_ms", 30 * 1000),
]
def serve() -> None:
server = grpc.server(
futures.ThreadPoolExecutor(max_workers=16),
options=SERVER_OPTIONS,
)
# Register your generated servicers here, e.g.:
# orders_pb2_grpc.add_OrdersServicer_to_server(OrdersService(), server)
server.add_insecure_port("[::]:50051") # use TLS credentials in production
server.start()
server.wait_for_termination()
if __name__ == "__main__":
serve()
A Python gRPC server that caps connection age with a grace period to force periodic client reconnects and DNS re-resolution, alongside keepalive settings that detect dead peers.
How does Envoy balance gRPC requests over HTTP/2?
A proxy removes the need for every client to get this right. Envoy terminates the client's HTTP/2 connection and keeps its own pool of HTTP/2 connections to the upstream pods. Each gRPC call is an HTTP/2 stream, so Envoy picks an upstream for every request instead of once per connection. A client holding a single long-lived connection still has its calls spread across pods.
Deployment shapes
- Sidecar: Envoy runs in each client pod. Latency stays local, but every pod pays the proxy's CPU and memory cost.
- Shared gateway: a pool of Envoy instances sits behind a regular Service. It costs less in total, but it is a central dependency. Run several replicas, spread them across nodes and zones, and autoscale them on CPU so they do not become the bottleneck.
- Service mesh: sidecars or node proxies configured by a control plane. You get balancing, retries, and mTLS together, along with a whole system to operate.
Endpoint discovery
The cluster definition matters most. With type: STRICT_DNS pointed at a headless Service, Envoy resolves every pod IP and re-resolves on an interval. New pods only receive traffic after the next refresh. Under a control plane (EDS), endpoint changes are pushed as soon as the control plane sees them, which handles rapid pod churn better. Upstreams must also be marked as HTTP/2 through http2_protocol_options. Without it, Envoy speaks HTTP/1.1 to the pods and gRPC fails.
Costs
Each call takes an extra network hop, which adds latency. Envoy uses CPU and memory per pod or per node. It is also one more component to upgrade, monitor, and debug when calls fail.
clusters:
- name: orders_grpc
connect_timeout: 1s
# Resolve the headless Service to every pod IP, not a single ClusterIP.
type: STRICT_DNS
# Re-resolve often so new pods receive traffic soon after they become Ready.
dns_refresh_rate: 5s
dns_lookup_family: V4_ONLY
# Envoy picks an endpoint per request (per HTTP/2 stream), not per connection.
lb_policy: ROUND_ROBIN
typed_extension_protocol_options:
envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
"@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
explicit_http_config:
# Required for gRPC: without this Envoy speaks HTTP/1.1 to the pods.
http2_protocol_options: {}
# Remove unhealthy pods between DNS refreshes using the gRPC health protocol.
health_checks:
- timeout: 1s
interval: 5s
unhealthy_threshold: 2
healthy_threshold: 1
grpc_health_check: {}
load_assignment:
cluster_name: orders_grpc
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
# Headless Service (clusterIP: None) in the "shop" namespace.
address: orders-headless.shop.svc.cluster.local
port_value: 50051
A minimal Envoy v3 cluster that discovers gRPC pods through STRICT_DNS against a headless Service, enables HTTP/2 to the upstreams, and balances each request round-robin across pods.
How should health checking and failing endpoints be handled in each model?
The gRPC health checking protocol
gRPC defines a standard service, grpc.health.v1.Health, with a unary Check method and a streaming Watch method. A server reports SERVING or NOT_SERVING, either for the whole server or per service name. Both balancing models can use it, which lets one server-side implementation serve both.
Kubernetes readiness probes
Kubernetes can probe the health service directly with its native gRPC probe type. When a readiness probe fails failureThreshold times in a row, the kubelet marks the pod not ready. The endpoint is then marked not ready in its EndpointSlice, and a headless Service stops returning that pod's IP in DNS. Reaction time is about periodSeconds × failureThreshold, plus the time the change takes to propagate.
Readiness has a gap: it does not close existing connections. A client that already holds an HTTP/2 connection to the pod keeps sending requests on it.
Client-side health checking
gRPC clients can enable health checking through the service config. The client then opens a Watch stream on each subchannel. When a backend reports NOT_SERVING, the client stops picking it almost immediately, without waiting for DNS. This only applies to balancing policies that track multiple subchannels, such as round_robin. It does not apply to pick_first.
Envoy active and passive checks
Envoy's active gRPC health checks poll each endpoint on an interval. An endpoint is removed after a configured number of consecutive failures. Outlier detection is passive. It ejects hosts that return consecutive errors on real traffic, for a backoff period. This catches pods that pass their health check but fail actual requests, such as a backend with a broken database connection. Because it reacts to live traffic, outlier detection is often the fastest of these signals.
Who should use client-side balancing, an Envoy proxy, or a mesh?
You need one of these approaches if you run gRPC servers with more than one replica behind a Kubernetes Service and your clients hold long-lived connections. Signs that this applies to you include uneven CPU across replicas of the same Deployment, new pods that stay idle after a scale-up, or latency that improves when clients restart. Teams that only call gRPC services through short-lived processes, such as batch jobs that open a channel, make a few calls, and exit, see less skew. Each run gets a fresh connection.
Which approach fits depends on who you are:
- Application teams that own both client and server, in one or two languages, with internal callers only. Client-side balancing is usually the best fit. You change a dial target and a service config, add server-side connection age settings, and deploy nothing new.
- Platform teams serving many consumers in several languages, or consumers they cannot redeploy, such as other teams' services, partner systems, or browser clients through a gRPC-Web gateway. A shared Envoy tier or sidecars let you fix balancing once, in one place, without coordinating changes across every caller.
- Organizations that already need mTLS, traffic shifting, and uniform telemetry across many services. A service mesh delivers request-level gRPC balancing as part of that package. It is a poor choice if balancing is the only goal.
- Teams with very latency-sensitive paths or tight per-pod resource budgets. Client-side balancing avoids the extra hop and the proxy's CPU and memory, as long as you accept owning the configuration in each client.
Client-side vs proxy vs service mesh: when should you use or avoid each?
The right choice depends less on traffic volume than on who owns the callers and what else you need from the network layer.
| Option | Development cost | Operational cost | Where complexity lands |
|---|---|---|---|
Client-side (headless Service + round_robin) |
Repeated per language: resolver config, keepalives, testing | Low: no extra hop or extra fleet | In every client and its config |
| Envoy proxy | Low for application teams | Medium: deploy, configure and upgrade the proxy tier | In proxy config, usually owned by a platform team |
| Service mesh | Near zero per service | High: control plane, sidecars, certificate lifecycle | In the platform and its upgrade cycle |
Client-side balancing saves the most time when you have one or two languages, internal traffic, and clients you can redeploy. You configure it once per language and avoid running new infrastructure. It stops paying off when you cannot change the callers, such as browsers, partner systems, or a dozen services in different languages. Each of those would need its own correct configuration.
A proxy makes sense once you also need centralized retries, outlier detection, or request-level metrics. A mesh is worth its cost when you need mTLS, traffic shifting, and consistent telemetry across many teams. Avoid a mesh if load balancing is your only problem. You would be taking on a control plane to fix something a resolver setting already solves.
Some limits apply to every option. A long-lived streaming RPC stays on one backend for its entire lifetime. Also, round_robin spreads requests evenly by count and ignores how expensive each request is. If request cost varies a lot, consider a least-request policy, which sends new requests to the backend with the fewest active requests.
How can fixing gRPC load balancing save engineering time?
Connection pinning rarely looks like a load balancing bug at first. It shows up as one slow replica, an autoscaler that adds pods without lowering latency, or a node that seems to have noisy neighbours. Engineers can spend a long time profiling application code or tuning resource limits for a problem that a resolver setting or a proxy fixes. Getting balancing right saves time in several concrete ways:
- Less incident investigation. When load is spread per request, a hot pod points to a real problem in that pod, such as a bad node, a stuck dependency, or a slow code path, rather than to a random connection pattern. Per-pod dashboards become trustworthy, and on-call engineers stop chasing skew that comes and goes with client restarts.
- No manual rebalancing. Without a fix, teams often work around skew by restarting client Deployments after a scale-up or rollout to force new connections. Max connection age on the server, or a proxy that balances per stream, removes that manual step.
- Capacity planning that matches reality. If a few pods carry most of the traffic, replica counts get sized for the hottest pod. Even distribution lets you size for average load plus headroom, and autoscaling decisions start to have the effect you expect.
- One fix instead of many. With a proxy or mesh, a platform team configures discovery, health checks, retries, and outlier detection once. Application teams do not each have to learn resolver schemes, service config JSON, and keepalive tuning in their language. With client-side balancing, wrapping the channel setup in a shared internal library gives a similar benefit for the languages you support.
- Fewer rollout errors. Health-aware balancing plus graceful draining reduces the
UNAVAILABLEspikes during deployments that otherwise lead to rollbacks and follow-up debugging.
Illustrative estimate, with assumed inputs: suppose a team hits skew-related incidents twice a month, and each one takes two engineers about three hours to diagnose before someone restarts the clients. That is roughly 12 engineer-hours a month spent on a symptom. If the fix is a service config change, server connection age settings, and the verification steps below, the one-time effort is likely to cost less than a single month of those incidents. Substitute your own incident rate and time spent to judge whether that holds for you.
How do you verify that gRPC traffic is balanced across pods?
Measure before you change anything. Record per-pod request rate and CPU for the server Deployment under steady load, then repeat after switching to client-side balancing or a proxy. kubectl top pods -l app=<server> gives a quick CPU and memory snapshot. Per-pod request rate from your metrics system is more reliable, because CPU also reflects garbage collection, background jobs, and uneven request cost. Query the request counter as a rate, grouped by pod label, and compare each pod's share against the mean. A connection-pinning problem looks distinctive: one or two pods carry most of the traffic while the rest sit near idle.
Illustrative estimate: with 6 replicas and 600 requests per second, a balanced setup puts roughly 100 requests per second on each pod. If one pod shows 400 and three show close to zero, clients are still pinned to long-lived connections.
Test churn, not only steady state
Static balance can look fine even when discovery is broken, so also test these cases:
- Scale-up. Increase replicas and watch the per-pod rate graph. Record how long the new pods take to reach their share. That delay should be about your DNS refresh interval or max connection age, not "never."
- Rolling update. Trigger a rollout and follow it with
kubectl rollout status deployment/<server>. Check that terminating pods drop to zero requests before they exit. Check that client error rates, especiallyUNAVAILABLE, do not spike. A spike usually means the server is not draining connections gracefully or the client is not retrying.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to grpc load balancing kubernetes, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.