What is service discovery, and which of its three jobs do you actually need?
Service discovery answers one question: which network address should this request go to right now? Static config files can answer that only when instances never move. Autoscaling adds and removes instances, schedulers reschedule workloads onto new hosts, and rolling deploys replace every address. The answer has to be computed at runtime, and the work splits into three jobs:
- Registration: recording which instances exist and whether they are healthy.
- Resolution: turning a service name into one or more addresses.
- Load balancing: choosing one address for a given connection or request.
The three common approaches put these jobs in different places:
Plain DNS: client --> resolver (cached, TTL) --> A records --> instances
[client picks a record]
Consul: instances --register+checks--> Consul catalog
client --DNS or HTTP API--> Consul --> healthy instances
[client picks an instance]
Kubernetes: pods --readiness--> EndpointSlices
client --cluster DNS--> ClusterIP --kube-proxy--> pod
[node picks a pod per connection]
| Job | Plain DNS | Consul | Kubernetes Services |
|---|---|---|---|
| Registration | Manual or scripted record changes | Agent or API registration with health checks | Automatic, from label selectors and pod readiness |
| Resolution | Recursive resolvers with TTL caching | DNS interface or HTTP API | Cluster DNS returns a virtual IP |
| Load balancing | Client picks among returned records | Client, or record shuffling | kube-proxy, once per connection |
The rest of the article compares these approaches on three questions. Where does a name resolve? How quickly does a failed instance stop receiving traffic? Where does the balancing decision happen? You may need only one or two of the three jobs. If so, a full mesh is solving problems you don't have.
[Reg] = registration [Res] = resolution [LB] = load balancing
PLAIN DNS [Reg] manual/scripted edits
+--------+ [Res] +-----------------+ +-------------+
| client |-------->| recursive |--------->| zone with |
| [LB] |<--------| resolver, cached|<---------| A records |
+---+----+ records +-----------------+ +-------------+
|
+---------- picks one returned A record ------------> instance
CONSUL [Reg] register + health checks
+--------+ [Res] +-----------------+ +-------------+
| client |-------->| Consul catalog |<---------| instance A |
| [LB] |<--------| healthy only |<---------| instance B |
+---+----+ DNS or +-----------------+ checks +------+------+
| HTTP API ^
+---------- picks one healthy instance ------------+
KUBERNETES [Reg] pod readiness
+--------+ [Res] +-------------+ +----------------+ +-------+
| client |-------->| cluster DNS | | EndpointSlices |<--| pods |
+---+----+ +-------------+ +-------+--------+ +---+---+
| ClusterIP | ^
v v watched |
+-------------------------------------------------------+ |
| kube-proxy on each node [LB] one pod per connection |-----------+
+-------------------------------------------------------+
Where registration, resolution and load balancing happen in plain DNS, Consul and Kubernetes Services.
How does DNS-based service discovery work, and why is it slow to notice failure?
DNS-based discovery answers only the first question: how does a name resolve? A registry, such as a DNS server backed by an inventory, or a controller watching instances, publishes one A (IPv4) or AAAA (IPv6) record per healthy instance under a service name. Clients resolve the name and connect to one of the returned addresses. SRV records add a port, a priority and a weight for each target. Clients that read them can find services on non-standard ports and apply simple preference rules. Many HTTP clients ignore SRV entirely.
The failure-detection problem comes from caching at three layers. The registry removes a terminated instance, but recursive resolvers keep serving the old answer until its TTL expires. Language runtimes and libraries may then cache it longer. Java, for example, follows networkaddress.cache.ttl rather than the record's TTL. The largest gap is connection reuse: a client holding open connections from a pool does not re-resolve at all, so it keeps sending traffic until a connection errors.
To see what a resolver is serving, run dig +noall +answer orders.internal A and dig +noall +answer _http._tcp.orders.internal SRV. The second column of each answer line is the remaining TTL in seconds. When the answer comes from a caching resolver, that number counts down between queries.
DNS also carries no health or load data. An address in the answer means only that the registry published it, so any health checking must happen in the registry before publication.
#!/usr/bin/env bash
# Inspect what DNS-based discovery is handing to clients for one service.
# Resolver and registry addresses are placeholders for your environment.
CACHING_RESOLVER=10.0.0.2 # the recursive resolver clients actually use
REGISTRY_DNS=10.0.0.53 # the registry's authoritative DNS server
# A records: one address per instance the registry has published.
dig +noall +answer @"$CACHING_RESOLVER" orders.internal A
# Example output (illustrative):
# orders.internal. 27 IN A 10.0.1.12
# orders.internal. 27 IN A 10.0.1.13
# ^^ remaining TTL in seconds
# SRV records: priority, weight, port and target host for each instance.
dig +noall +answer @"$CACHING_RESOLVER" _http._tcp.orders.internal SRV
# Example output (illustrative):
# _http._tcp.orders.internal. 27 IN SRV 10 50 8080 orders-1.internal.
# _http._tcp.orders.internal. 27 IN SRV 10 50 8080 orders-2.internal.
# TTL prio wt port target
# Query again a few seconds later. If the answer is cached, the TTL
# counts down instead of resetting to the full value.
sleep 5
dig +noall +answer @"$CACHING_RESOLVER" orders.internal A
# Compare with the registry directly. If an instance was just terminated,
# the registry may already omit it while the cache still returns it
# until its TTL expires.
dig +noall +answer @"$REGISTRY_DNS" orders.internal A
# Note: none of this shows instance health or load. An address appears
# only because the registry published it. Clients that cache lookups in
# the runtime or reuse pooled connections can keep using an address
# after it disappears from both answers.
Querying A and SRV records with dig shows the remaining TTL and how a caching resolver can keep returning an instance the registry has already removed.
How do Kubernetes Services, EndpointSlices and cluster DNS handle discovery?
Kubernetes moves that checking server-side. A ClusterIP Service gets a stable virtual IP that no network interface owns. kube-proxy on each node watches the API server and programs packet rules (iptables or IPVS, depending on mode) that rewrite connections to that IP toward a backend pod. Cluster DNS resolves orders.payments.svc.cluster.local to the virtual IP. Because the IP never changes as pods come and go, DNS caching stops being a staleness problem.
The backend list lives in EndpointSlices. The control plane adds a pod's address when the pod's readiness probe passes and marks it not ready when the probe fails. A probe is only as good as what it checks. An HTTP handler that returns 200 without touching the database pool or a required downstream will keep a broken pod in rotation. Make readiness reflect whether the pod can actually serve requests. Keep liveness separate and shallow, so that a dependency outage does not trigger restarts across the whole Deployment.
Termination has a race. When a pod is deleted, the kubelet sends SIGTERM, and in parallel the endpoint removal propagates to every node's kube-proxy. Neither waits for the other, so for a short window new connections can still reach a process that is shutting down. The fix has two parts:
- A
preStophook that sleeps a few seconds. This delays SIGTERM until the endpoint change has had time to spread. - Graceful shutdown in the app. On SIGTERM, stop accepting new connections and finish in-flight requests.
Set terminationGracePeriodSeconds to cover the sleep plus the drain time.
apiVersion: v1
kind: Service
metadata:
name: orders # resolves as orders.payments.svc.cluster.local
namespace: payments
spec:
type: ClusterIP # stable virtual IP; kube-proxy routes it to ready pods
selector: {app: orders}
ports:
- port: 80
targetPort: 8080
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: orders
namespace: payments
spec:
replicas: 3
selector:
matchLabels: {app: orders}
template:
metadata:
labels: {app: orders}
spec:
terminationGracePeriodSeconds: 45 # covers preStop sleep (10s) + app drain time
containers:
- name: orders
image: registry.example.com/orders:1.4.2
ports: [{containerPort: 8080}]
readinessProbe: # deep check: handler verifies DB pool and required downstreams
httpGet: {path: /readyz, port: 8080}
periodSeconds: 5
failureThreshold: 2
livenessProbe: # shallow check: process responsive only, so dependency outages do not cause restarts
httpGet: {path: /livez, port: 8080}
periodSeconds: 10
lifecycle:
preStop: # delay SIGTERM so endpoint removal reaches every node first
exec: {command: ["sleep", "10"]}
A ClusterIP Service for orders in the payments namespace, plus a Deployment with a deep readiness probe, a shallow liveness probe, a preStop sleep that covers endpoint propagation, and a termination grace period long enough for the sleep and connection draining.
Why do gRPC and connection pools break ClusterIP load balancing, and when should you use a headless Service?
A ClusterIP Service balances connections, not requests. kube-proxy picks a backend pod when a connection is established, and every request on that connection then goes to the same pod. With short HTTP/1.1 requests this evens out. With gRPC it does not. HTTP/2 multiplexes all requests over one long-lived connection, so a client that opened its channel when only three pods existed keeps hitting those three after you scale to ten. Connection pools in database drivers and HTTP clients behave the same way: the pool is filled once and reused, so load stays skewed until the connections are recycled.
A headless Service (clusterIP: None) skips the virtual IP. A cluster DNS lookup returns the IPs of the ready pods directly, and the client chooses among them. It fits three cases:
- Client-side load balancing, where the client needs the full backend list.
- Stateful workloads, where each replica must be addressable on its own. Paired with a StatefulSet, every pod also gets a stable DNS name.
- gRPC clients that should spread requests across every backend instead of pinning to one.
The usual fixes, roughly in order of effort:
- Client-side balancing over a headless Service. Use a
dns:///target with theround_robinpolicy. gRPC's default,pick_first, sends everything to a single address. - Periodic connection recycling. Set a maximum connection age on the server so clients reconnect and re-resolve DNS. Without this, a gRPC client may not see new pods until a connection fails.
- An L7 proxy that balances individual requests.
The first two push the logic into every client stack. Each language runtime needs equivalent resolver, policy and keepalive settings, and you have to keep them consistent across teams.
# Headless Service. Apply it with kubectl before running the client.
#
# apiVersion: v1
# kind: Service
# metadata:
# name: orders
# namespace: shop
# spec:
# clusterIP: None # no virtual IP: DNS returns each ready pod's IP
# selector:
# app: orders
# ports:
# - name: grpc
# port: 50051
# targetPort: 50051
import json
import grpc
# The dns:/// scheme makes gRPC resolve every A record behind the name,
# so the channel can open a subchannel to each pod.
TARGET = "dns:///orders.shop.svc.cluster.local:50051"
SERVICE_CONFIG = json.dumps({
# The default policy is pick_first, which sends every RPC to one address.
# round_robin spreads RPCs across all resolved pods.
"loadBalancingConfig": [{"round_robin": {}}],
})
channel = grpc.insecure_channel(
TARGET,
options=[("grpc.service_config", SERVICE_CONFIG)],
)
# The client does not pick up new pods on its own. Pair this with a
# server-side max connection age, e.g. ("grpc.max_connection_age_ms", 300000).
# Clients then reconnect periodically and re-resolve DNS.
A headless Service manifest and a Python gRPC channel that resolves it through a dns:/// target and spreads RPCs across all pod IPs with the round_robin policy.
How do you implement client-side load balancing without a mesh?
The pattern has two parts. A resolver periodically re-resolves the service name (every 15–30 seconds is a common starting point, or on connection errors) and produces an address list. A picker chooses one address per request. Round robin works when requests cost roughly the same. Least-request, which samples two endpoints and takes the one with fewer in-flight calls, handles uneven latency better because slow instances naturally receive less traffic.
Around the picker sit per-attempt timeouts, a total deadline, and retries with exponential backoff plus full jitter. Without jitter, clients that failed together retry together. Budget retries as a fraction of total traffic, or a partial outage multiplies load on the instances that are still healthy.
Outlier ejection removes endpoints that fail repeatedly. It also has a dangerous failure mode. Under overload, every instance starts erroring, ejection shrinks the pool, the remaining instances receive more traffic and fail, and the whole pool cascades to zero. A minimum-healthy floor, often called a panic threshold, stops this. A picker sketch:
- On each failure, increment
consecutive_errors[endpoint]. If it crosses N, eject the endpoint untilnow + base_ejection * times_ejected. - When picking, set
healthy = endpoints not currently ejected. - If
len(healthy) < floor * len(endpoints), ignore ejection and pick from all endpoints. - Otherwise, apply round robin or least-request to
healthy.
The cost is ongoing maintenance. Every language needs an equivalent implementation, the details drift between stacks, and debugging means knowing which stack you are dealing with. A shared internal library with the same defaults in every language is the main way to keep that cost manageable.
When is Consul service discovery worth adding, and when should you avoid it?
Consul moves registration into a local agent on every node. Each service ships a definition naming the service, its port and an HTTP check, for example a check block with http pointing at /healthz and an interval of a few seconds. The agent runs the check locally and reports state changes to the servers. Failing instances drop out of query results, which reacts faster than waiting for DNS TTLs to expire.
Clients can query in two ways:
- DNS:
dig @127.0.0.1 -p 8600 web.service.consul SRVreturns healthy instances with their ports, so legacy software needs no changes. - HTTP API:
/v1/health/service/web?passingreturns the same set with metadata. Adding?index=turns it into a blocking query that returns only when the result changes.consul watchuses the same mechanism to give near-real-time updates without polling.
When it earns its place
Use Consul when workloads span several Kubernetes clusters, VMs, bare metal or datacenters and need one catalog. It is not worth adding to a single Kubernetes cluster, where EndpointSlices already provide discovery. It also needs a team that can operate it: server quorum, gossip encryption, ACL bootstrapping and rolling upgrades are ongoing work.
Failure behavior
If the servers lose quorum, registrations stop. Reads can continue with ?stale, but the data ages. Clients should cache the last known-good instance list and keep using it rather than failing closed.
Running Consul alongside Kubernetes creates a second source of truth. Sync tooling lags, so pick one system as authoritative for each workload and monitor any drift between them.
Which approach should you choose, and when do you actually need a service mesh?
Start from the traffic you actually run, not from the most capable tool:
- Kubernetes Services alone cover a single cluster serving mostly short-request HTTP. Connections churn often enough that per-connection balancing evens out, and you write no discovery code.
- Headless Services plus client-side balancing fit long-lived RPC such as gRPC, where a single connection would otherwise pin a client to one pod.
- Consul earns its place when workloads span clusters, VMs and data centres that need one registry.
| Option | Failure convergence | Balancing granularity | Operational cost | Development cost |
|---|---|---|---|---|
| ClusterIP Service | Fast; readiness failures remove endpoints | Per connection | Minimal, built in | None |
| Headless + client-side | Bounded by client re-resolution or watch interval | Per request | Low | Moderate; every client stack needs configuring |
| Consul | Fast with health checks and watches | Per connection (DNS) or per request (client integration) | High; servers, agents, ACLs, upgrades | Low to moderate |
A mesh becomes the right alternative when you need guarantees no single library can enforce across teams and languages:
- mTLS on every hop
- uniform L7 authorization policy
- consistent retries, timeouts and telemetry
- weighted traffic shifting for canaries
Without those requirements, a mesh adds sidecar or node-proxy lifecycle, certificate rotation, control-plane upgrades and a new layer to debug during every incident. Choosing the simplest sufficient option keeps that work off your roadmap. Moving later is still feasible because Service names and health checks carry over to a mesh. Adopting one by default means paying its operational cost before you have a requirement it solves.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to service discovery without service mesh, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.