What service discovery and load balancing actually solve

In a microservices system, instances come and go constantly. Autoscalers add replicas, deploys replace them, and nodes fail. Each change gives an instance a new address. A caller cannot hardcode where orders lives, because the answer changes several times a day. Service discovery and load balancing solve this problem together. Discovery answers "which instances exist and are usable?" Load balancing answers "which one gets this request?"

The system has four parts:

  • Registry: the source of truth for which instances exist and at what addresses.
  • Health signals: checks or heartbeats that mark an instance as usable or not.
  • Propagation: how registry changes reach the callers or proxies that route traffic.
  • Selection: the algorithm that picks one instance for each request or connection.
Server-side discovery              Client-side discovery

caller                             registry
  |                                   |  (pushes or is polled)
  v                                   v
[ LB / proxy ] <-- registry        caller [endpoint list]
  |    |    |                        |       |       |
  v    v    v                        v       v       v
 A     B    C                       A       B       C

Each pattern makes the same trade-off. It decides how stale the endpoint list may become and where the balancing decision happens. That can be one central hop or every caller. Staleness cannot be avoided. Propagation always lags reality, so callers will sometimes send traffic to an instance that has just died or miss one that has just started. A robust design assumes this window exists and handles it with retries, timeouts, and graceful draining.

SERVER-SIDE DISCOVERY                   CLIENT-SIDE DISCOVERY

  +----------+                          +----------------+
  |  caller  |                          |    registry    |<---+
  +----------+                          +----------------+    |
       |                                        |             |
       | one hop                                | propagation |
       v                                        | (push/poll) |
+----------------+    +----------+              v             |
|   LB / proxy   |<---| registry |      +----------------+    |
|  [selection]   |    +----------+      |     caller     |    |
+----------------+          ^           | [endpoint list]|    |
   |    |    |              |           |   [selection]  |    |
   v    v    v              |           +----------------+    |
  +-+  +-+  +-+             |              |    |    |        |
  |A|  |B|  |C|-------------+              v    v    v        |
  +-+  +-+  +-+ health                    +-+  +-+  +-+       |
                                          |A|  |B|  |C|-------+
                                          +-+  +-+  +-+ health

decision: one central hop               decision: inside every caller

In server-side discovery, health signals feed a registry that updates a central load balancer or proxy, which picks the instance. In client-side discovery, the registry propagates the endpoint list to each caller, which picks the instance itself.

How it works: DNS, server-side proxies, and client-side load balancing

DNS service discovery

The simplest mechanism maps a service name to instance addresses. A/AAAA records return IPs only. SRV records add port, priority and weight, but many HTTP clients and language runtimes ignore them. Staleness compounds in layers. Resolvers cache for the TTL, some runtimes cache longer than the TTL, and many clients resolve once at connect time and then reuse that connection for hours. A 5-second TTL therefore does not mean a 5-second view. DNS also carries no load or health data, so it cannot steer traffic away from a slow instance.

Server-side discovery

A central load balancer or proxy holds the instance list, and clients only need one stable address. This hides the topology and keeps clients simple. The cost is an extra network hop, another tier to scale, patch and monitor, and a component whose failure takes out every caller behind it.

Client-side load balancing

Each client fetches instances from a registry and picks a target itself. This removes the hop and allows algorithms that use per-request latency or in-flight counts. The cost is that balancing logic must exist in every language runtime you run. The registry becomes critical infrastructure and must be highly available. If it is unreachable, clients should keep serving from a last-known-good list.

Aspect DNS Server-side proxy Client-side
Staleness TTL plus runtime and connection caching Proxy's refresh interval Registry push or poll interval
Latency No extra hop Extra hop No extra hop
Operational surface Low Proxy tier Registry plus client libraries
Language coupling None None Per runtime
Health information None Proxy health checks Registry health plus client-observed signals

How Kubernetes Services implement discovery

Kubernetes builds discovery from two decoupled pieces: a selector that defines membership and a control loop that publishes it. When a Service declares a label selector such as app: orders, the EndpointSlice controller watches for pods carrying those labels and writes their IPs and ports into EndpointSlice objects. It splits large services across multiple slices so that one pod change doesn't rewrite a single huge object that every node is watching.

Readiness gates membership. A pod whose readiness probe fails stays in the slice, but its ready condition is set to false, and consumers skip it. This is why a readiness probe should check whether the process can serve traffic right now, such as whether it has warmed caches and opened its database connections. Liveness is a separate question. Without a readiness probe, a pod counts as ready as soon as its containers start.

For a regular Service, the cluster assigns a ClusterIP, which is a virtual address that no network interface owns. Cluster DNS resolves the Service name to that single IP. On each node, kube-proxy (or a replacement data plane) programs forwarding rules that rewrite the destination to one ready pod IP when a connection is opened. The choice happens once per connection, not once per request.

A headless Service (clusterIP: None) skips the virtual IP. DNS returns an A record for each ready pod, and the client picks which one to call. This is the hook that client-side load balancing relies on.

To inspect a Service, run kubectl get service orders -o wide and kubectl describe service orders. To see its endpoints, run kubectl get endpointslices -l kubernetes.io/service-name=orders -o yaml.

# Regular Service: DNS resolves "orders" to one virtual ClusterIP
apiVersion: v1
kind: Service
metadata:
  name: orders
spec:
  selector:
    app: orders            # membership is defined by labels, not by a list of IPs
  ports:
    - port: 80             # port clients use on the ClusterIP
      targetPort: 8080     # port node forwarding rules send traffic to on the pod
---
# Headless Service: no virtual IP; DNS returns one A record per ready pod
apiVersion: v1
kind: Service
metadata:
  name: orders-headless
spec:
  clusterIP: None          # the client chooses which pod IP to call
  selector:
    app: orders
  ports:
    - port: 8080
---
# Pod matching both selectors; readiness sets its ready condition in EndpointSlices
apiVersion: v1
kind: Pod
metadata:
  name: orders-0
  labels:
    app: orders
spec:
  containers:
    - name: orders
      image: registry.example.com/orders:1.0.0
      readinessProbe:      # fails until caches are warm and DB connections are open
        httpGet:
          path: /ready
          port: 8080
        periodSeconds: 5

A regular Service and a headless Service select the same pods by label, and the pod's readiness probe decides whether its EndpointSlice entry is marked ready and receives traffic.

Why long-lived connections break connection-level balancing

kube-proxy picks a backend once per TCP connection, when the connection opens. That works for HTTP/1.1 clients that open and close connections often. HTTP/2 and gRPC behave differently. They multiplex many concurrent requests as streams over a single TCP connection and keep that connection open for as long as possible. A gRPC client talking to a ClusterIP typically opens one connection, lands on one pod, and sends every request there. With ten client replicas and thirty server pods, at most ten pods receive traffic. Pods added during a scale-up get nothing until some client reconnects. The Service is doing exactly what it was designed to do. It balances connections, and you need to balance requests.

There are three fixes:

  • Per-request L7 balancing. A proxy or sidecar terminates HTTP/2 and spreads individual streams across endpoints. Clients need no changes. The cost is an extra hop, added latency, CPU and memory for the proxies, and another component to operate.
  • Client-side balancing against a headless Service. Set clusterIP: None so DNS returns pod IPs instead of a virtual IP. The client then opens a connection to each pod and rotates requests across them. Conceptual example: in gRPC, target dns:///orders-headless.default.svc.cluster.local:50051 and set the service config to {"loadBalancingConfig": [{"round_robin": {}}]}. The cost is that every client language needs correct configuration. DNS caching also delays discovery of new pods.
  • Periodic connection recycling. Set a maximum connection age on the server so clients reconnect and get rebalanced. This is cheap, but balancing is only approximate and depends on how often connections are recycled. It also pairs well with the headless approach, because each reconnect triggers a fresh DNS lookup.
import json
from concurrent import futures

import grpc

# --- Client: round-robin across pods behind a headless Service ---
# Assumes a Service named orders-headless in namespace default with
# spec.clusterIP: None, so DNS returns one A record per ready pod.
TARGET = "dns:///orders-headless.default.svc.cluster.local:50051"

# The default policy is pick_first: one connection, one pod.
# round_robin opens a subchannel to every resolved pod IP and
# rotates RPCs across them.
SERVICE_CONFIG = json.dumps({"loadBalancingConfig": [{"round_robin": {}}]})

# Insecure channel only to keep the example short; use TLS credentials in production.
channel = grpc.insecure_channel(
    TARGET,
    options=[("grpc.service_config", SERVICE_CONFIG)],
)
# stub = orders_pb2_grpc.OrdersStub(channel)  # each RPC goes to the next pod

# --- Server: cap connection age so clients reconnect and re-resolve DNS ---
server = grpc.server(
    futures.ThreadPoolExecutor(max_workers=16),
    options=[
        # After about 5 minutes the server sends GOAWAY. The client then
        # reconnects and runs a fresh DNS lookup, which picks up new pods.
        ("grpc.max_connection_age_ms", 300_000),
        # Give in-flight RPCs 30 seconds to finish before the connection closes.
        ("grpc.max_connection_age_grace_ms", 30_000),
    ],
)
# orders_pb2_grpc.add_OrdersServicer_to_server(OrdersService(), server)
# server.add_insecure_port("[::]:50051")
# server.start()

Conceptual Python gRPC example: the client uses round-robin balancing against a headless Service DNS name, and the server sets a maximum connection age so clients periodically reconnect and pick up new pods.

Choosing a load-balancing algorithm

Once traffic reaches a set of endpoints, the algorithm decides which instance gets each request. The choice matters most when request costs are uneven.

Round robin and random assume every request costs the same. When one request takes 5 ms and the next takes 2 s, a slow instance keeps receiving its share while its queue grows. Least-requests (least-outstanding) sends each request to the instance with the fewest in-flight requests. Slow instances are skipped because their queues are visibly longer. Scanning every endpoint on every request is costly, though. Power-of-two-choices gets most of the benefit for little work: pick two endpoints at random and send the request to the less busy one. That small comparison is enough to avoid the worst choices.

Consistent hashing maps a key, such as a user ID, to an instance. The same key lands on the same instance, which keeps its cache warm. The trade-off is that a hot key concentrates load on one instance. Adding or removing an instance also reassigns a fraction of keys, and those keys arrive at instances with cold caches.

All of these run per caller, with no global view. With few instances, many clients can independently pick the same "least loaded" target. After a scale-up, new cold instances can be flooded before they warm up.

Algorithm Good fit Failure mode State needed
Round robin Uniform, cheap requests Overloads slow instances Counter
Random Large fleets, stateless clients Uneven at small scale None
Least-requests Variable request cost O(n) selection; herding In-flight count per endpoint
Power-of-two-choices Variable cost, large fleets Stale counts under bursts In-flight count per endpoint
Consistent hashing Cache or session affinity Hot keys; reshuffle on membership change Hash ring

Health checks, graceful rollouts, retries, and timeouts

Active health checks probe each endpoint on a schedule. Passive outlier detection watches real traffic and ejects endpoints that return consecutive errors or time out. Passive detection reacts faster to failures that a synthetic probe misses. It also has a failure mode: during a broad incident, every endpoint looks unhealthy. Cap the maximum ejection percentage so the balancer never routes the full load onto a few survivors and knocks them over too.

Keep probes narrow. Readiness answers "should I get traffic?" Liveness answers "should I be restarted?" If liveness checks a shared database, one database blip restarts every pod at once, and a partial outage becomes a full one.

On deploy, Kubernetes marks a pod terminating and removes its endpoint asynchronously. Proxies and clients keep sending traffic until the change propagates. The pod spec that follows uses a preStop sleep to cover that window, a terminationGracePeriodSeconds longer than the sleep plus the time to drain, and separate probes. After SIGTERM, the pseudocode handler fails readiness, stops accepting new connections, and lets in-flight requests finish. With HTTP/2 or gRPC it also sends GOAWAY so clients reconnect elsewhere. Without these steps, every deploy produces connection-refused errors and reset bursts.

Retry on a different endpoint, but bound the retries. Illustrative estimate (assumed numbers, not a benchmark): amplification = attemptsᵈᵉᵖᵗʰ. Three attempts at each of three layers gives 3³ = 27× load on the deepest service. A retry budget limits that load, for example by allowing retries for at most 10% of requests. Deadline propagation limits it further: each hop passes along the remaining time and drops work that can no longer finish before the deadline.

apiVersion: v1
kind: Pod
metadata:
  name: orders-api
spec:
  # Must exceed preStop sleep (10s) + worst-case drain time (~20s).
  terminationGracePeriodSeconds: 45
  containers:
    - name: app
      image: registry.example.com/orders-api:1.4.2
      ports:
        - containerPort: 8080
      lifecycle:
        preStop:
          exec:
            # Keep serving while endpoint removal propagates to proxies.
            # SIGTERM is sent only after this hook returns.
            command: ["sleep", "10"]
      readinessProbe:
        # "Should I get traffic?" May fail during shutdown or warm-up.
        httpGet:
          path: /ready
          port: 8080
        periodSeconds: 5
        failureThreshold: 2
      livenessProbe:
        # "Should I be restarted?" Checks only this process, never the
        # shared database, so a DB blip cannot restart every pod at once.
        httpGet:
          path: /healthz
          port: 8080
        periodSeconds: 10
        failureThreshold: 3
# SIGTERM handler (pseudocode):
#   on SIGTERM:
#     ready = false              # /ready now returns 503
#     server.stopAccepting()     # close listener; no new connections
#     http2.sendGoAway()         # HTTP/2 / gRPC clients reconnect elsewhere
#     server.awaitInFlight(30s)  # let in-flight requests finish
#     exit(0)

A pod spec that uses a preStop sleep to cover endpoint-removal propagation, sets a grace period longer than the sleep plus drain time, and keeps readiness and liveness probes separate, followed by a pseudocode SIGTERM handler that drains connections before exiting.

When to use each approach, when to avoid it, and the cost impact

Start with the traffic pattern you actually have and pick the simplest mechanism that handles it.

Approach Fits Avoid when Main cost
Kubernetes Service / DNS Short-lived connections, uniform request cost gRPC or other long-lived, multiplexed connections Uneven load once connections persist
Client-side library One team, one language stack, needs per-request balancing Polyglot organizations Keeping each language implementation consistent
Proxy or service mesh Language-agnostic L7 policy, mTLS, traffic shifting Small fleets with uniform traffic Per-pod resources, extra hop latency, harder debugging

Each step down the table adds machinery. Client libraries put balancing logic in every language you ship, and each version can drift in retry, timeout, and health-check behavior. Sidecars avoid that, but they add a container to every pod and a network hop to every call. When a request fails, you also have to work out whether the cause was the application, the local proxy, or the remote one.

Illustrative estimate (assumed numbers, not a benchmark): reserved sidecar CPU = pods × sidecar CPU request = 200 pods × 0.1 vCPU = 20 vCPU that is idle when the mesh is doing little.

If the only problem is gRPC connections sticking to one backend, a headless Service with client-side balancing fixes it without that overhead or the operational work of running a mesh.

Whatever you choose, set connection draining, readiness gating, and retry budgets as defaults at the platform layer, not in each service. Teams that inherit sane defaults stop rediscovering the same dropped-request incidents during deploys.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to service discovery microservices, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.