Microservice Service Discovery: Kubernetes DNS vs Consul vs Client-Side

Where routing decisions live: three models of microservice service discovery

Service discovery answers one question at runtime: which network address should this request go to right now? In a static deployment you could hard-code hostnames. With microservices you can't, because Pods get new IPs when they are rescheduled, autoscalers add and remove replicas, and VMs are replaced during rolling deploys. The set of healthy endpoints behind a service name changes all the time. Something has to track that set and pick one member for each connection or request.

The three models covered here differ mainly in where that selection happens:

• Kubernetes DNS: the platform dataplane selects. A service name resolves to a stable virtual IP, and kube-proxy (or an equivalent dataplane) chooses a backend Pod when a connection is opened.
• Consul: an external registry holds the endpoint list, which it fills from service registrations and health checks. Selection happens in DNS answers or, with the service mesh, in a sidecar proxy beside each workload.
• Client-side lookup: the application selects. A library fetches the endpoint list and picks a target for every request.

Our position: most teams should start with built-in Kubernetes DNS. Add a registry, a mesh, or client-side balancing only when you hit a concrete limit, such as skewed load on long-lived HTTP/2 connections or workloads that run outside the cluster.

How Kubernetes DNS service discovery works end to end

A client calling orders never learns where a Pod runs. When the Service is created, Kubernetes assigns it a stable ClusterIP, and the cluster DNS server (usually CoreDNS) publishes orders.<namespace>.svc.cluster.local pointing at that IP. The endpoint controller watches Pods that match the Service selector and writes their IPs into EndpointSlices. A Pod is marked ready in the slice only while its readiness probe passes. That rule is what pulls a failing backend out of rotation without anyone editing DNS.

The request path looks like this:

client → DNS lookup → ClusterIP (virtual) → kube-proxy / dataplane rules → EndpointSlice → ready Pod

The ClusterIP is not bound to any interface. kube-proxy, or a replacement dataplane, programs rules on each node that rewrite traffic sent to the virtual IP into traffic for one ready Pod IP. The backend is chosen when the connection opens. Every request that later travels over that connection goes to the same Pod.

A headless Service (clusterIP: None) skips the virtual IP. DNS returns one A record per ready Pod, and the client chooses which one to use. The client also has to re-resolve on its own schedule.

The difference shows up directly when you inspect it. kubectl get endpointslices -l kubernetes.io/service-name=orders lists the Pod IPs in both cases. From inside a Pod, dig orders returns a single address for a ClusterIP Service and several addresses for a headless one. The client sees one stable target in the first case and the full backend set in the second.

Copy# Two Services selecting the same Pods: one ClusterIP, one headless
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: orders
spec:
  selector:
    app: orders
  ports:
    - port: 80
      targetPort: 8080
---
apiVersion: v1
kind: Service
metadata:
  name: orders-headless
spec:
  clusterIP: None   # no virtual IP; DNS answers with ready Pod IPs
  selector:
    app: orders
  ports:
    - port: 80
      targetPort: 8080
EOF

# Both Services have EndpointSlices listing the same ready Pod IPs
kubectl get endpointslices -l kubernetes.io/service-name=orders
kubectl get endpointslices -l kubernetes.io/service-name=orders-headless

# Resolve from inside the cluster using a throwaway Pod in the same namespace
kubectl run dns-test --rm -it --restart=Never --image=busybox:1.36 -- nslookup orders
# expect one address: the Service ClusterIP

kubectl run dns-test --rm -it --restart=Never --image=busybox:1.36 -- nslookup orders-headless
# expect one A record per ready Pod; the client picks among them

Creates a ClusterIP Service and a headless Service over the same Pods, then compares their EndpointSlices and DNS answers from inside the cluster.

Why per-connection balancing breaks down for gRPC and HTTP/2

A ClusterIP Service balances connections, not requests. kube-proxy programs the node so that each new TCP connection to the virtual IP is sent to one backend Pod, and every packet on that connection goes to the same Pod afterward. With HTTP/1.1 clients that open many short connections, this spreads load reasonably well.

gRPC and other HTTP/2 clients behave differently. They open one long-lived connection and multiplex thousands of concurrent requests over it. The backend chosen when that connection opened receives all of that client's traffic for as long as the connection lives, which can be hours or days. The effect is clearest after a scale-up. Suppose you go from three replicas to six. Existing clients stay pinned to the original three, and the new Pods sit nearly idle until something forces a reconnect. Rolling deployments make it worse, because clients that reconnect at the same moment can pile onto whichever Pods are ready first.

There are two usual fixes:

• Client-side load balancing against a headless Service. The client resolves every Pod address, holds a connection to each, and spreads requests across them. gRPC supports this with its DNS resolver and a round-robin policy.
• A proxy that balances per request. A sidecar or node-level L7 proxy terminates HTTP/2 and distributes individual requests across backends.

This mismatch is the most common concrete reason teams move beyond plain ClusterIP.

Two partial mitigations are capping connection lifetime on the server, for example with gRPC's maximum connection age setting, and having clients reconnect periodically. Both make clients re-resolve and redistribute over time. Load is still uneven between reconnects, and the cap trades rebalancing speed against reconnection overhead.

How Consul works: an explicit registry, health checks, and an optional service mesh

Consul starts from a different premise: services are recorded in an explicit catalog rather than derived from a scheduler's view of running pods. A process on Kubernetes, a VM, or bare metal registers itself, so a single registry can describe workloads that no single cluster's DNS can see. Agents run each service's health checks locally and report results to the servers. Only instances that pass are returned to callers. Clients can query that state three ways: DNS, the HTTP API, or a local sidecar proxy that resolves upstreams for them.

A conservative service definition gives a name, a port, and an HTTP check such as GET /healthz on a fixed interval with a timeout. A caller can then resolve web.service.consul through Consul's DNS interface and receive only the passing instances. The code example below shows both.

Discovery and service mesh are separate decisions. Consul service mesh runs Envoy-based sidecars that add mutual TLS, identity-based authorization through intentions, and traffic policies such as splitting and retries. You can use the catalog without any of this. Choosing the mesh also means accepting extra latency, memory, and CPU on every hop.

Consul has real operating costs. The servers need a Raft quorum, typically three or five nodes. You have to manage agents or dataplanes, coordinate upgrades, and get ACLs right from the start. A degraded Consul cluster degrades discovery for every service registered with it. Running Consul next to Kubernetes DNS also gives you two sources of truth that can disagree during rollouts. Decide which one callers actually use.

Copy# 1. Describe the service: a name, a port, and an HTTP health check.
#    The local agent runs the check and only passing instances are returned.
cat > web.hcl <<'EOF'
service {
  name = "web"
  port = 8080

  check {
    id       = "web-http"
    name     = "HTTP GET /healthz"
    http     = ""
    method   = "GET"
    interval = "10s"   # how often the agent probes
    timeout  = "2s"    # a slow response counts as a failure
  }
}
EOF

# 2. Register with the local agent. If ACLs are enabled, the token
#    needs write access to this service (placeholder value shown).
export CONSUL_HTTP_TOKEN="00000000-0000-0000-0000-000000000000"
consul services register web.hcl

# 3. Discover through DNS. The agent serves the .consul domain on port 8600.
#    An SRV query returns the port as well as the address of each healthy instance.
dig @127.0.0.1 -p 8600 web.service.consul SRV

# 4. Run the same lookup through the HTTP API, keeping only instances
#    whose checks are passing.
curl -s ""

Registers a service named web with an HTTP /healthz check against the local Consul agent, then looks up its passing instances through Consul DNS (web.service.consul) and through the HTTP health API.

How client-side lookup works and what it costs every team

In client-side lookup there is no proxy in the request path. The calling service owns discovery. A library inside each process asks DNS or a registry for the current endpoints, keeps them in memory, and decides where every request goes. Because the decision is made per request instead of per connection, it can spread load from long-lived gRPC and HTTP/2 channels and react to failures without waiting for a central health check.

In language-neutral pseudocode, the loop looks like this:

1. Resolve: if the cached list is older than its refresh interval, query DNS or the registry and replace the list.
2. Pick: choose an endpoint that is not ejected, using round robin or least outstanding requests.
3. Send: issue the call with a per-attempt timeout and an overall deadline.
4. Record: on success, clear the endpoint's failure count. On failure, increment it, and eject the endpoint for a backoff period once it crosses a threshold.
5. Retry: if the error is retryable and the deadline and retry budget allow, return to step 2 and exclude the endpoint that just failed.

The cost is that every team runs this loop. Each language your organization uses needs a library that implements it correctly. Timeouts, retry budgets, and ejection thresholds must match across those libraries, or one service will amplify an outage that another service absorbs. A bug fix or policy change ships only when every consuming service rebuilds and redeploys.

DNS adds a further problem. Some language runtimes cache resolved addresses longer than the record's TTL, sometimes for the life of the process, unless you configure them otherwise. Intermediate resolvers can also stretch or clamp TTLs. The library may therefore be balancing across endpoints that no longer exist.

Copy# Language-neutral client-side discovery loop (pseudocode, not a real SDK)
config: REFRESH = 30s, ATTEMPT_TIMEOUT = 300ms, DEADLINE = 1s
        MAX_RETRIES = 2, EJECT_AFTER = 5 failures, EJECT_FOR = 30s

function call(service, request):
    deadline = now() + DEADLINE
    tried = {}
    for attempt in 0 .. MAX_RETRIES:
        # Resolve: refresh from DNS or registry when the cache is stale
        if now() - cache[service].fetched_at > REFRESH:
            cache[service].endpoints = resolve(service)   # may be stale if runtime ignores TTL
            cache[service].fetched_at = now()

        # Pick: skip ejected hosts and hosts already tried for this request
        candidates = [e for e in cache[service].endpoints
                      if e.ejected_until < now() and e not in tried]
        if candidates is empty: return error("no healthy endpoints")
        ep = least_outstanding(candidates)                # or round_robin(candidates)

        # Send: per-attempt timeout, never past the overall deadline
        timeout = min(ATTEMPT_TIMEOUT, deadline - now())
        if timeout <= 0: return error("deadline exceeded")
        result = send(ep, request, timeout)

        # Record: outlier detection on consecutive failures
        if result.ok:
            ep.failures = 0
            return result
        ep.failures += 1
        if ep.failures >= EJECT_AFTER:
            ep.ejected_until = now() + EJECT_FOR

        # Retry: only retryable errors, and only within the retry budget
        if not retryable(result) or not retry_budget.allow():
            return result
        tried.add(ep)
    return error("retries exhausted")

Pseudocode for the resolve, cache, pick, send, record, and retry loop that every service's discovery library has to implement the same way.

How stale health data turns into errors: a worked recovery-time example

…that the registry has already removed. The rest of the delay comes from how health signals reach each model.

In Kubernetes, a failing readiness probe removes the pod from its EndpointSlice, and kube-proxy or the DNS server for headless Services picks up the change. In Consul, the local agent runs checks and marks the instance critical, which removes it from DNS and catalog answers. In client-side setups, outlier detection ejects an endpoint after it returns repeated errors, so detection depends on traffic rather than on a probe schedule.

Check depth matters in both directions:

• Too shallow. A check that only opens a TCP port passes while the application is deadlocked.
• Too aggressive. A short interval with a threshold of 1 ejects instances during GC pauses. The instances then flap in and out of rotation, and the remaining ones get overloaded.

Illustrative estimate (assumed numbers, not a benchmark). Assume a 5 s check interval, a failure threshold of 3, 2 s for the registry change to propagate, a 30 s DNS TTL, and a 30 s client cache. In the worst case these delays add up because each layer refreshes independently:

Window = (interval × threshold) + propagation + DNS TTL + client cache

The ClusterIP row covers only new connections. Existing connections continue until they fail.

Each way to shorten the window has a cost:

• Lower TTLs and shorter caches increase DNS query load.
• Shorter check intervals increase probe traffic and the risk of flapping.
• Outlier detection needs a capped ejection percentage so that one bad deploy does not eject every instance.

Choosing a model: when to use each, when to avoid it, and the impact on engineering time

The right model depends less on feature lists than on where your workloads run and which protocols they speak. Start from your current conditions, not your projected ones.

Other options fit the same rows. Alternative mesh dataplanes compete with Consul's mesh. Platform-native multi-cluster Services can cover the multi-cluster row with less custom routing.

Starting with DNS saves engineering time because nothing new has to be operated, upgraded or explained during incidents. Add a registry, mesh or client library only when you see a concrete signal:

• Uneven load from connection pinning. Long-lived connections stick to a few backends while others sit idle.
• Workloads Kubernetes cannot see. VMs or other runtimes need to be discovered alongside cluster services.
• Cross-region failover requirements. You need controlled failover between clusters or datacenters.
• Recurring errors from stale endpoints. Your recovery-time measurements show traffic still reaching endpoints that have already failed.

Alternatives to consider

Several other approaches solve parts of the same problem. They are worth evaluating before you commit to a registry or a library.

A central Layer 7 load balancer

Route all east-west traffic through a shared HTTP-aware proxy tier. The proxy balances per request, so long-lived HTTP/2 connections do not pin traffic to one backend, and only one component needs to understand health and retries. The cost is an extra network hop on every call. The proxy tier also becomes a capacity and availability dependency that you must scale and make redundant yourself. This approach fits small service counts or teams that already run such a tier at the edge.

Headless Services with an in-process resolver

A Kubernetes headless Service returns pod IPs directly instead of a single virtual IP. A gRPC client configured for round-robin over those addresses gets request-level balancing without a registry. It still depends on DNS refresh behavior, so it inherits the staleness window you measured earlier.

Proxyless control-plane clients

Some RPC frameworks can subscribe to a control plane over xDS-style APIs. The client receives endpoint and weight updates as a stream and does not poll for them. This gives you mesh-like routing without sidecars. It requires framework support in every language you use.

Static configuration

For a handful of stable, rarely redeployed backends, an address list managed through configuration deployment is predictable and easy to audit. It fails as soon as instances change faster than you deploy.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to microservice service discovery, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.