What is the difference between liveness and readiness in an application health check endpoint?

An application health check endpoint answers one of two questions, and they need different answers. Liveness asks whether the process can still make progress. If it cannot, because it is deadlocked, has exhausted its thread pool, or is stuck in an event loop that never yields, the right response is to kill it and start a new one. Readiness asks whether this instance should receive traffic right now. An instance can be alive but not ready. It may still be warming caches, it may have lost its database connection pool, or it may be draining before shutdown. The right response is to stop routing requests to it and leave the process running.

Liveness Readiness
Question Can the process make progress? Should it get traffic now?
Action on failure Restart the process Remove from load balancer rotation
Checks In-process only In-process state plus critical dependencies
Expected failure duration Permanent until restart Often temporary

Merging the two into one endpoint fails in both directions. If the combined check includes a database ping, a database blip makes every instance fail liveness at once, and the orchestrator restarts healthy processes that will hit the same unavailable database on boot. That is a restart loop. If the combined check is shallow, the load balancer keeps sending requests to instances that cannot serve them.

This checklist is for backend and platform engineers running one large application behind a load balancer or orchestrator, where every instance shares the same dependencies and fails the same way at the same time.

Why does a single deep health check turn one slow dependency into a fleet-wide outage?

A load balancer treats each instance's health check as an independent verdict: pass and you receive traffic, fail enough consecutive probes and you are taken out of rotation. That model assumes failures are independent. A deep readiness check that queries the shared primary database or cache breaks the assumption, because every instance is now reporting on the same dependency.

Suppose the database starts answering slowly, perhaps because of lock contention or a long-running migration. Every instance's readiness query exceeds its timeout within the same few seconds. Every instance fails its probes on the same schedule, and the load balancer removes all of them. This is correlated mass removal. The database is no healthier for it, and you have turned a partial degradation, where some requests are slow, into a total one, where none succeed. When the database recovers, instances return together and hit it with a burst of reconnects and cold-cache traffic, which can knock it over again.

Readiness should answer one question: is this instance worse than its peers? Failures that are local to an instance qualify:

  • Its connection pool is exhausted while others have capacity.
  • Warm-up hasn't finished.
  • Required in-memory caches or config haven't loaded.

Removing that instance shifts load to healthy peers. Removing every instance for a shared fault shifts load to nowhere.

When every target is unhealthy, some load balancers fail open and route to all targets anyway. Others drop all traffic. Check which behavior your setup has before you rely on it.

 SHARED FAULT: deep check, same DB    | LOCAL FAULT: instance-only check
                                      |
   +------------------+               |   +------------------+
   |  Load Balancer   |               |   |  Load Balancer   |
   |  0 of 3 healthy  |               |   |  2 of 3 healthy  |
   +------------------+               |   +------------------+
     |FAIL  |FAIL  |FAIL              |     |OK    |OK    |FAIL
     v      v      v                  |     v      v      v
   +---+  +---+  +---+                |   +---+  +---+  +---+
   | A |  | B |  | C |                |   | A |  | B |  | C |
   +---+  +---+  +---+                |   +---+  +---+  +---+
     |      |      |                  |         C only: pool exhausted,
     +------+------+                  |         warm-up unfinished, or
            | /ready query times out  |         caches not loaded
            v                         |
   +------------------+               |   +------------------+
   |  Shared primary  |               |   |  Shared primary  |
   |  DB: slow, locks |               |   |  DB (not probed) |
   +------------------+               |   +------------------+
------------------------------------------------------------------------------
 Left: all 3 fail together -> 0 capacity; the DB is no healthier, and
       recovery returns them at once in a burst of reconnects.
 Right: only C is removed -> its load shifts to healthy peers A and B.
 All targets unhealthy: some LBs fail open, others drop all traffic.

When readiness probes a shared database, one slow dependency fails every instance at once and the load balancer removes all capacity, while an instance-local readiness check removes only the instance that is actually worse than its peers.

Which dependency checks should gate readiness?

Readiness answers one question: would sending this request to a different instance help? If the answer is no, because every instance shares the same broken dependency, failing readiness only removes capacity. That rules out most dependency checks before you start.

Sort each dependency into one of two groups. A hard dependency is one without which the instance cannot serve meaningful requests at all. A soft dependency degrades a feature but leaves the core request path working. Gate readiness only on hard conditions that are local to the instance. "My connection pool is initialized and can lend a connection" is instance-local. "The database cluster is healthy" is not.

Dependency Class Gates readiness? Where failure shows up
Primary DB connection pool initialized Hard Yes Readiness
Config and secrets loaded at startup Hard Yes Readiness
Local cache warmed (if required to serve) Hard Yes Readiness
Recommendations service Soft No Feature error rate, fallback counter
Email/SMTP relay Soft No Queue depth, send failures
Search index Soft No Search endpoint latency and errors

Monoliths blur this line. The same process serves checkout, admin pages and reporting, and each depends on a different subset of backends. If any one of those backends can fail readiness, the whole app goes dark for a fault that affected one page. Scope readiness to the critical path only, meaning the requests the business cannot run without. Make partial outages visible through per-route error rates, fallback counters and feature-level alerts instead. A broken recommendations widget should page someone. It should not remove the instance from the load balancer.

How do you implement fast, bounded liveness and readiness checks?

Liveness should touch nothing but the process itself. The handler runs on the same thread pool or event loop that serves real traffic and returns 200. If that loop is wedged, the probe times out, which is the signal you want. A database call here would let one database blip restart every instance at once.

Readiness needs more care:

  • Bound every check. Give each dependency check its own timeout, well below the probe timeout. If the probe gives up at 2 seconds, a check capped at 500 ms still lets the handler return a definite answer instead of timing out.
  • Never check serially on the request path. Run checks concurrently, or run them from a background loop every few seconds and cache each result with a timestamp. The handler only reads the cache: if cache.age > max_staleness or any(gating_check.failed) then 503 else 200. Probe traffic never reaches the dependency, so a slow database cannot pile up blocked handler threads.
  • Separate signal from detail. The load balancer acts only on the status code. Serve a JSON body with per-check status and latency on an internal-only path or port, because it reveals internal topology. For example: {"status":"not_ready","checks":{"db":{"ok":true,"ms":12},"cache":{"ok":false,"ms":500,"error":"timeout"}}}.
  • Keep the route lean. Exempt health paths from auth, rate limiting and heavy middleware. Exclude them from latency SLO metrics, because frequent fast probes pull percentiles down.
  • Log transitions, not probes. Record ready→not-ready changes along with the failing check. Logging every probe only adds noise.
# Pseudocode: liveness is dependency-free; readiness reads a cache filled by a background loop
CHECK_TIMEOUT  = 500ms   # per check, well below the 2s probe timeout
INTERVAL       = 5s      # how often the background loop refreshes results
MAX_STALENESS  = 15s     # if the loop itself stalls, report not ready
GATING         = {"db", "cache"}   # checks that decide readiness

cache = { results: {}, updated_at: never }
last_ready = unknown

loop every INTERVAL:                      # runs off the request path
    # Start all checks at once so total time is about one CHECK_TIMEOUT, not the sum
    tasks = { name: start_async(check, timeout=CHECK_TIMEOUT) for name, check in checks }
    results = {}
    for name, task in tasks:
        start = now()
        try:   task.wait();  results[name] = {ok: true,  ms: elapsed(start)}
        catch Timeout:      results[name] = {ok: false, ms: CHECK_TIMEOUT, error: "timeout"}
        catch err:          results[name] = {ok: false, ms: elapsed(start), error: short(err)}
    cache = { results: results, updated_at: now() }   # swap the whole object in one step

    ready = all(results[n].ok for n in GATING)
    if ready != last_ready:               # log transitions only, never each probe
        log.warn("readiness", from=last_ready, to=ready,
                 failing=[n for n in GATING if not results[n].ok])
        last_ready = ready

GET /livez:                               # same pool or event loop as real traffic
    return 200                            # no I/O. If the loop is wedged, the probe times out

GET /readyz:                              # public path: status code only
    snap = cache
    stale = now() - snap.updated_at > MAX_STALENESS
    failed = any(not snap.results[n].ok for n in GATING)
    return 503 if stale or failed else 200

GET /internal/readyz (internal port only): # detail reveals topology, so keep it private
    return json({status: "ready" or "not_ready", checks: cache.results})
# e.g. {"status":"not_ready","checks":{"db":{"ok":true,"ms":12},"cache":{"ok":false,"ms":500,"error":"timeout"}}}

# Router setup: register health routes before auth, rate limiting and heavy middleware,
# and tag them so the latency SLO histogram skips them.

Language-neutral pseudocode for a dependency-free liveness handler and a cached readiness check. A background loop runs bounded, concurrent dependency checks, the public probe returns only a status code, internal detail is served on a separate port, and the loop logs only state transitions.

How should readiness behave during startup, shutdown, and deploys?

  • Report not-ready until warm-up finishes. A monolith often needs to load configuration, fill connection pools, prime caches, or wait for a JIT to settle before it can serve at normal latency. Liveness can pass early. Readiness should flip only after these steps complete, so the first real requests don't land on a cold process and time out.
  • Make shutdown a sequence. On SIGTERM, flip readiness to failing first while continuing to serve traffic. Wait until the load balancer has had time to observe the failure and stop routing new requests. Then drain in-flight requests and exit. If the process exits immediately, the load balancer keeps sending traffic to a closed port until its own checks catch up, and clients see connection errors.
  • Size the wait from your probe settings. As an illustrative estimate, assume a 5-second interval and a failure threshold of 3. Removal takes roughly 15 seconds. Add the longest request you expect to drain, for example 10 seconds, and the shutdown delay comes to about 25 seconds. Set your orchestrator's termination grace period above that number.
  • Tune interval, timeout, and both thresholds together. Aggressive settings remove broken instances quickly, but a GC pause or network blip can push a healthy instance out of rotation, and it then flaps back in. Lenient settings tolerate noise, but a broken instance keeps receiving traffic for longer. A higher success threshold also slows recovery after a brief failure.
  • Account for deploy and scale-in cost. Every second of shutdown delay is paid once per instance. In a rolling deploy with a small batch size, those delays add up across the whole rollout. During scale-in, capacity you are paying for stays on until draining finishes.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: monolith
spec:
  replicas: 6
  strategy:
    rollingUpdate:
      maxSurge: 1        # small batches: each shutdown delay is paid ~6 times per rollout
      maxUnavailable: 0
  selector:
    matchLabels: { app: monolith }
  template:
    metadata:
      labels: { app: monolith }
    spec:
      # Illustrative estimate: 5s x 3 failures = ~15s to removal, plus ~10s to drain
      # = ~25s. Keep the grace period above that so the process isn't killed mid-drain.
      terminationGracePeriodSeconds: 35
      containers:
        - name: app
          image: registry.example.com/monolith:1.4.2
          ports:
            - containerPort: 8080
          # Liveness only asks "is the process stuck?" It can pass during warm-up.
          livenessProbe:
            httpGet: { path: /live, port: 8080 }
            periodSeconds: 10
            failureThreshold: 3
          # The app returns 503 on /ready until pools, caches and config are loaded,
          # and again as soon as it receives SIGTERM, while it keeps serving.
          readinessProbe:
            httpGet: { path: /ready, port: 8080 }
            periodSeconds: 5     # lower = faster removal, more probe traffic
            timeoutSeconds: 2    # must exceed a normal GC pause, or healthy pods flap
            failureThreshold: 3  # 3 misses in a row before traffic stops
            successThreshold: 1  # higher values slow recovery after a blip

A Kubernetes Deployment that ties readiness probe interval, timeout and thresholds to the termination grace period and rolling-update batch size, using the illustrative 5-second, 3-failure estimate from the text.

When should you avoid deep dependency checks, and what are the alternatives?

Deep checks have their own cost, and it grows with fleet size. Every instance probes every dependency on a fixed interval. With many instances, that becomes a constant query load that no user asked for. During an incident the timing is worst: the database is already saturated, and the probes add load and time out together. Caching probe results cuts the volume, but readiness then reflects state that may be several intervals old. A detailed response body can also expose hostnames, driver versions, and internal topology to anyone who can reach the port. Keep detailed output on an internal-only path, or log it instead of returning it.

Alternatives and complements

Approach What it catches Weakness
Shallow readiness + error-rate alerting Broken dependencies, seen through real traffic A human or automation must act on the alert
Synthetic end-to-end checks Full user paths, run from one place Doesn't remove bad instances by itself
Circuit breakers, feature-level degradation A failing dependency, isolated per feature Needs fallback behavior designed per call site
Passive/outlier detection at the load balancer One instance returning errors when its peers don't Only works if the load balancer supports it, and it needs tuning

All of these accept the same trade-off. Readiness will sometimes say "ready" while a dependency is broken. That false negative costs some failed requests on an instance that stays in rotation. The alternative failure is a shared dependency failing every instance's probe at once, which removes the whole fleet and turns a partial outage into a total one. A few errors you can alert on are preferable to an empty backend pool.

How much engineering effort does this take, and how does it save time? A checklist

The build cost is modest but not zero. You write two endpoints instead of one, a shutdown handler that marks the instance unready before the server stops accepting work, and a short document that classifies each dependency as hard or soft. The readiness endpoint usually returns per-check detail, which makes it a second internal surface. You have to keep it off the public network, restrict who can call it, and update it whenever someone adds a dependency.

Cost you pay Time it saves
Two endpoints and their tests Fewer self-inflicted outages from probes pulling healthy instances
Hard/soft dependency document Faster triage, because per-check output names the failing dependency
Shutdown handler Deploys that drain in-flight requests instead of dropping them
Securing the internal endpoint Transition logs that show when and why an instance left rotation

Illustrative estimate (assumed numbers, not a benchmark): Hours saved per year = (outages avoided × engineers involved × hours per outage) + (incidents × triage hours saved per incident).

Assume 2 avoided outages × 3 engineers × 2 hours = 12 hours. Add 10 incidents × 0.5 hours = 5 hours. That gives 17 engineer-hours per year. Compare it with your own estimate of the build and maintenance time.

Checklist

  • One liveness endpoint that checks only the process itself
  • One readiness endpoint with per-check output
  • A documented list of hard and soft dependencies
  • Check timeouts set below the probe timeout
  • A graceful shutdown sequence: mark unready, drain, then exit
  • Known load balancer behavior when every target is unhealthy
  • Alerting on error rate, independent of probe results
  • Alerting on readiness flapping

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to application health check endpoint, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.