Kubernetes Service Discovery for Microservices: DNS and Headless

How Kubernetes service discovery works: DNS names, ClusterIP and kube-proxy

Service discovery lets a client find a healthy instance of another service by name, without knowing which Pods exist or what their IPs are. Microservices need it because Pod IPs change every time a Pod is rescheduled, scaled or replaced.

For a normal ClusterIP Service, the cluster DNS server answers the lookup. That server is CoreDNS or kube-dns, and in both cases it sits behind a Service named kube-dns. It returns a single virtual IP that stays the same for the life of the Service. No process listens on that VIP. Instead, kube-proxy on each node programs packet rules that rewrite traffic sent to the VIP toward one backend Pod. The rewrite happens when a connection opens, so every request on that connection goes to the same Pod.

CopyClusterIP:  client -> DNS -> VIP -> kube-proxy rules -> one Pod (chosen per connection)
Headless:   client -> DNS -> [Pod IP, Pod IP, Pod IP] -> client picks a Pod

The full name is <service>.<namespace>.svc.<cluster-domain>, and the cluster domain is usually cluster.local. Short names work because each Pod's /etc/resolv.conf lists search domains such as <namespace>.svc.cluster.local, svc.cluster.local and cluster.local. With these, orders resolves within the Pod's own namespace, and orders.payments reaches the payments namespace.

The same search list adds cost for external names. Kubernetes sets ndots:5 by default. Any name with fewer than five dots, such as api.example.com, is tried against every search domain before it is queried as written, and each of those attempts is a wasted query. To skip the search list, write external hosts with a trailing dot (api.example.com.) or lower ndots in the Pod's dnsConfig.

Copy# Throwaway client Pod in the 'payments' namespace to see DNS the way an app does
kubectl run dnsprobe -n payments --image=busybox:1.36 --restart=Never -- sleep 3600
kubectl wait -n payments --for=condition=Ready pod/dnsprobe --timeout=60s

# Search domains and ndots that kubelet wrote into the Pod
kubectl exec -n payments dnsprobe -- cat /etc/resolv.conf
# Typical shape (IP and domain vary per cluster):
#   nameserver 10.96.0.10
#   search payments.svc.cluster.local svc.cluster.local cluster.local
#   options ndots:5

# The nameserver is the ClusterIP of the kube-dns Service (CoreDNS or kube-dns)
kubectl get svc kube-dns -n kube-system -o jsonpath='{.spec.clusterIP}{"\n"}'

# Short name: expanded via the first search domain (own namespace)
kubectl exec -n payments dnsprobe -- nslookup orders
# namespace-qualified name: expanded via svc.cluster.local
kubectl exec -n payments dnsprobe -- nslookup orders.payments
# Fully qualified with trailing dot: queried as written, no search list
kubectl exec -n payments dnsprobe -- nslookup orders.payments.svc.cluster.local.

# All three should return this one stable virtual IP, not Pod IPs
kubectl get svc orders -n payments -o jsonpath='{.spec.clusterIP}{"\n"}'

# External name with trailing dot skips the ndots:5 search expansion
kubectl exec -n payments dnsprobe -- nslookup api.example.com.

# Clean up
kubectl delete pod -n payments dnsprobe

Inspecting a Pod's /etc/resolv.conf and resolving a ClusterIP Service by short, namespace-qualified and fully qualified names, which all return the same virtual IP, plus a trailing-dot external lookup that bypasses the search list.

Headless Services, SRV records and stable Pod identities

Setting clusterIP: None on a Service removes the virtual IP. kube-proxy programs no rules for it, and the DNS name stops resolving to a single stable address. A query for orders.shop.svc.cluster.local instead returns one A or AAAA record per ready Pod. The client gets the full list and decides which endpoint to call, so connection spreading, retries and failover become the application's job.

Suppose the manifest declares a named port, for example name: grpc with protocol: TCP. The cluster DNS then publishes SRV records at _grpc._tcp.orders.shop.svc.cluster.local. Each SRV record carries the port number and a target hostname for one Pod. A client can therefore discover the port as well as the addresses, so the port number does not have to be hard-coded.

When a StatefulSet sets its serviceName to a headless Service, each Pod also gets its own resolvable name, such as db-0.db.shop.svc.cluster.local and db-1.db.shop.svc.cluster.local. The name follows the ordinal, not the Pod's IP. If db-1 is rescheduled and comes back at a new address, its name still points to it. This is what databases, consensus systems and sharded workloads rely on. A replica set needs to know which member is the primary. A Raft or etcd cluster lists its peers by name in its configuration. A shard map routes key ranges to specific owners. A load-balanced VIP that sends each request to an arbitrary backend cannot provide any of this.

Large headless Services have a cost. A response containing hundreds of records can exceed the UDP payload size. The server then sets the truncation bit, and the client must retry over TCP. Some resolvers handle that retry poorly or silently use the partial answer.

Copy# Headless Service: no virtual IP, so DNS returns one record per ready Pod
apiVersion: v1
kind: Service
metadata:
  name: db
  namespace: shop
spec:
  clusterIP: None          # headless: kube-proxy programs no rules
  selector:
    app: db
  ports:
    - name: grpc           # named port -> SRV at _grpc._tcp.db.shop.svc.cluster.local
      protocol: TCP
      port: 7000
      targetPort: 7000
---
# StatefulSet bound to the headless Service gives each Pod a stable DNS name
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: db
  namespace: shop
spec:
  serviceName: db          # must match the headless Service name
  replicas: 3
  selector:
    matchLabels:
      app: db
  template:
    metadata:
      labels:
        app: db
    spec:
      containers:
        - name: db
          image: registry.example.com/db:1.0
          ports:
            - name: grpc
              containerPort: 7000
# Resulting DNS (shape, not literal output):
#   db.shop.svc.cluster.local             -> A/AAAA per ready Pod IP
#   _grpc._tcp.db.shop.svc.cluster.local  -> SRV per Pod: port 7000, target db-N.db.shop.svc.cluster.local
#   db-0.db.shop.svc.cluster.local        -> only Pod db-0, follows it across reschedules

A headless Service with a named gRPC port and a StatefulSet that uses it, annotated with the A, SRV and per-Pod DNS names the cluster DNS publishes.

Why long-lived connections break per-connection balancing

A ClusterIP Service balances connections, not requests. kube-proxy programs rules that pick a backend Pod when a TCP connection is opened. After that, every byte on the connection goes to the same Pod. With HTTP/1.1 clients that open short connections or keep small pools, new connections are opened often enough that traffic spreads out reasonably well.

HTTP/2 and gRPC behave differently. A client opens one connection and multiplexes many concurrent requests over it as independent streams, often for hours. A gRPC channel to a ClusterIP address typically resolves to a single connection, so all requests from that client land on one Pod. Consider a few large clients calling many backends: some Pods saturate while others sit idle. When you scale up, the new Pods receive almost nothing, because existing clients have no reason to reconnect. The extra capacity stays unused until connections break or restart.

On the ClusterIP side, the usual fix is to bound connection lifetime. Many HTTP/2 and gRPC servers support a maximum connection age. When it expires, the server sends a GOAWAY frame, lets in-flight streams finish within a grace period, and the client reconnects, which may select a different Pod. Add jitter so that clients do not all reconnect at the same moment. The trade-off is rebalancing speed against handshake and TLS overhead.

The alternative is client-side load balancing: the client learns every Pod address and spreads requests across them.

Check whether this affects you. Look for per-Pod request rates that stay uneven after a scale-up. Then compare your connection lifetime with how quickly you need new capacity to start taking traffic.

Implementing client-side load balancing on a headless Service

The fix is to make the client aware of every Pod. With a headless Service, a DNS lookup returns one A record per ready Pod instead of a single virtual IP. The client opens a connection to each address and spreads requests across them.

In gRPC, point the channel at a target such as dns:///orders.shop.svc.cluster.local:50051. Then set the policy in the service config with {"loadBalancingConfig": [{"round_robin": {}}]}. This matters because gRPC's default policy is pick_first, which connects to one address and sends everything there. With that default you are back to the uneven distribution described above. Under round_robin, the channel keeps one subchannel per resolved address and rotates requests across those that are ready.

Resolution is the weak point. gRPC's DNS resolver does not poll on the record TTL. It typically re-resolves only when a subchannel fails or disconnects, and the exact triggers differ between language implementations. A new Pod can therefore go unused until something forces a lookup. A common remedy is to set a maximum connection age on the server, so connections recycle and clients re-resolve on a known schedule.

Caching can make results stale in the other direction. Language runtimes, OS resolver libraries and connection pools may hold DNS answers past the TTL, or indefinitely. A client in that state keeps sending traffic to Pods that have already terminated. Check every caching layer in your stack.

Bypassing the virtual IP also skips kube-proxy. That means features it provides, such as sessionAffinity: ClientIP, no longer apply. You must reimplement them in the client or drop them.

Copyimport json
from concurrent import futures

import grpc

# Headless Service (clusterIP: None): DNS returns one A record per ready Pod.
TARGET = "dns:///orders.shop.svc.cluster.local:50051"

# Override the default pick_first policy. round_robin keeps one subchannel
# per resolved address and rotates RPCs across the READY ones.
SERVICE_CONFIG = json.dumps({"loadBalancingConfig": [{"round_robin": {}}]})


def make_client_channel() -> grpc.Channel:
    return grpc.insecure_channel(
        TARGET,
        options=[("grpc.service_config", SERVICE_CONFIG)],
    )


def make_server() -> grpc.Server:
    # The DNS resolver does not poll on TTL. Capping connection age makes the
    # server close connections gracefully on a known schedule, which prompts
    # clients to re-resolve and pick up new Pods.
    return grpc.server(
        futures.ThreadPoolExecutor(max_workers=16),
        options=[
            ("grpc.max_connection_age_ms", 5 * 60 * 1000),  # recycle every 5 minutes
            ("grpc.max_connection_age_grace_ms", 30 * 1000),  # let in-flight RPCs finish
        ],
    )


if __name__ == "__main__":
    channel = make_client_channel()
    # Use generated stubs as usual, e.g. orders_pb2_grpc.OrdersStub(channel).
    # Session affinity from kube-proxy does not apply on this path; if you
    # need stickiness, implement it in the client.
    grpc.channel_ready_future(channel).result(timeout=10)
    print("channel ready")

A Python gRPC client that targets a headless Service through the dns:/// resolver with round_robin set in the service config, and a server that caps connection age so clients re-resolve on a known schedule.

Readiness, endpoint propagation and graceful shutdown

Every Service, ClusterIP or headless, is fed from its EndpointSlices, and a Pod's address is listed there as ready only while its readiness probe passes. That one condition controls DNS answers, kube-proxy rules and anything that watches EndpointSlices directly.

Termination is not instantaneous

When a Pod is deleted, the control plane marks it terminating and the EndpointSlice controller flips its endpoint to not ready. That change then has to reach every node's kube-proxy, CoreDNS's cache and every client's resolver or connection pool. None of these happen at the same moment. Meanwhile the kubelet runs the preStop hook and then sends SIGTERM. If the process exits immediately, peers that haven't caught up yet send requests to a closed port.

The fix is to delay shutdown, as in the Pod spec fragment that follows:

• Readiness probe: hits a lightweight local endpoint.
• preStop hook: sleeps for several seconds so endpoint removal can propagate.
• terminationGracePeriodSeconds: large enough to cover the preStop sleep plus the time to drain. The grace period includes the hook.

On SIGTERM, the application should:

1. Stop accepting new connections.
2. Finish in-flight requests.
3. Close idle keep-alive connections so clients reconnect elsewhere.
4. Exit.

Probe failure modes

A probe that returns 200 as soon as the HTTP listener binds is too lenient. It advertises Pods that haven't warmed caches or opened database pools.

The opposite mistake is worse. If readiness checks a shared downstream dependency, an outage in that dependency fails every replica's probe at once. The Service then empties, and callers get connection errors instead of useful degraded responses. Readiness should answer one question: can this process serve traffic? It should not report on the health of everything behind it.

Copy# Deployment Pod template fragment (spec.template.spec)
spec:
  # Total budget for preStop + SIGTERM handling. The preStop sleep counts
  # against this, so it must exceed sleep (10s) + worst-case drain time.
  terminationGracePeriodSeconds: 45
  containers:
    - name: orders-api
      image: registry.example.com/orders-api:1.4.2
      ports:
        - name: http
          containerPort: 8080
      readinessProbe:
        httpGet:
          # Local-only check: caches warm, DB pool open in THIS process.
          # Do not call shared downstream services from here.
          path: /readyz
          port: http
        periodSeconds: 5
        failureThreshold: 3
      livenessProbe:
        httpGet:
          path: /livez
          port: http
        periodSeconds: 10
        failureThreshold: 3
      lifecycle:
        preStop:
          exec:
            # Keep serving while the endpoint is removed from EndpointSlices
            # and that change reaches kube-proxy, DNS caches and clients.
            # Requires a sleep binary in the image.
            command: ["sleep", "10"]
      # After preStop, the kubelet sends SIGTERM: the app stops accepting
      # connections, finishes in-flight requests, closes idle keep-alives
      # and exits. SIGKILL follows if the grace period runs out.

A Pod spec fragment that combines a local-only readiness probe, a preStop sleep to let endpoint removal propagate, and a terminationGracePeriodSeconds large enough to cover both the sleep and the SIGTERM drain.

Choosing an approach: ClusterIP, headless DNS or watching EndpointSlices

A client or sidecar watching EndpointSlices learns about readiness changes as soon as the control plane publishes them. It does not wait for a TTL to expire. Each watcher holds a long-lived API server connection and needs a ServiceAccount with read access. During an API server outage, clients route on a stale cache. This adds an operational dependency.

ClusterIP is the default for a polyglot fleet with short requests, and it is the cheapest to adopt in every language. Avoid relying on it alone for long-lived gRPC unless servers recycle connections.

Headless DNS with client-side balancing suits gRPC in a few languages that have maintained libraries. Each language still needs library work and testing. For short-lived HTTP/1.1 in a polyglot fleet, that cost buys little. Low TTLs and aggressive re-resolution also raise cluster DNS query load, which shows up as intermittent latency or timeouts that look like application bugs.

API watching fits latency-sensitive rollouts where you control the client or sidecar. Avoid it where portability or RBAC simplicity matters.

Choosing on purpose costs less engineering time than chasing uneven load and rollout errors later.

Debugging service discovery from inside the cluster

Work from the Service outward, so each step rules out one layer before you move to the next.

1. Confirm the Service exists and selects Pods. kubectl get svc <name> -n <ns> shows whether it has a ClusterIP or None (headless). Then kubectl get endpointslices -n <ns> -l kubernetes.io/service-name=<name> lists the actual endpoints. If none appear, the selector does not match the Pod labels. If endpoints appear but are not ready, the readiness probe is failing.
2. Resolve from a client's point of view. Start a throwaway Pod with kubectl run using --rm -it --restart=Never and an image that includes nslookup or dig. A ClusterIP Service should return one virtual IP. A headless Service should return one A record per ready Pod. dig SRV _<port-name>._tcp.<svc>.<ns>.svc.cluster.local confirms that named ports are published.
3. Read /etc/resolv.conf inside the client Pod. It shows the search domains and ndots value the resolver actually uses. With the usual ndots:5, a name such as api.example.com is tried against every search domain before it is queried as written. This multiplies queries and can turn into timeouts when DNS is under load.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes service discovery microservices, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.