What are liveness, readiness and dependency checks, and what does each one trigger?

Each health check is defined by what happens when it fails. Design every endpoint backward from that consequence: decide which action you want to trigger, then include only conditions that justify it.

  • Liveness: a failure makes the kubelet kill and restart the container. Only report failure when a restart would actually fix the problem.
  • Readiness: a failure removes the pod from Service endpoints, and from any load balancer that follows them. The process keeps running and returns to rotation once the check passes again.
  • Startup: while this probe runs, liveness and readiness checks are held off, so slow initialization isn't mistaken for a hang. If it never succeeds within its threshold, the container is restarted.
  • Dependency/diagnostic: no orchestrator action at all. It exists for humans, dashboards and alerting.
Check Question it answers Consumer Action on failure Include Exclude Typical mistake
Liveness Is this process stuck beyond self-recovery? kubelet Restart container Event loop or worker responsiveness, deadlock detection Databases, downstream services, caches Checking the database, so a DB blip restarts every pod
Readiness Should this instance get traffic now? Endpoints controller, load balancers Stop routing traffic Warm-up done, local capacity, shutdown in progress Shared dependencies every replica uses Failing all replicas together and emptying the Service
Startup Has initialization finished? kubelet Restart after threshold Config loaded, migrations or cache warm-up done Ongoing runtime state Omitting it and stretching liveness delays instead
Dependency Which downstreams are degraded? Humans, dashboards, alerting None Per-dependency status and latency Anything wired to a probe Exposing it on a probe path

The rule is to keep consumers separate. If a signal meant for humans is routed to the kubelet, the kubelet will act on it.

Why does putting dependency checks in probes cause cascading outages?

The failure mode is correlation. A slow shared database is one fault, but if every replica of every service tests that database in its readiness probe, they all fail at the same moment for the same reason.

Here is how it plays out. The database's p99 latency rises past the probe timeout. Each readiness check fails, and Kubernetes removes every pod from the Service endpoints. The load balancer now has nowhere to send traffic, so clients get connection errors on every route. That includes routes that never touch the database, such as cached reads, static config and health pages. A partial degradation has become a full outage, and the orchestrator caused it.

The liveness variant is worse. Failed liveness checks restart containers, and restarting does not repair a remote database. Each restart drops in-flight requests, empties local caches and pools, and then runs startup work again, often against the same struggling database. Capacity falls while the dependency is still down. When the database recovers, the pods are still in crash-loop backoff and come back slowly.

Dependency in probe              Dependency outside probe
-------------------              ------------------------
DB slow                          DB slow
  -> readiness fails (all)         -> probes still pass
  -> endpoints = 0                 -> pods stay in rotation
  -> LB: no backends               -> DB calls: fast 503 / timeout
  -> 100% of routes fail           -> other routes: 200
  -> (liveness) restarts           -> circuit breaker sheds load
     lower capacity further        -> recovery is immediate

In the alternative design, pods stay ready. Calls to the database fail fast through short timeouts and a circuit breaker. Requests that need the database return explicit errors or degraded responses, and every other route keeps serving normally.

 A) Dependency in probe                B) Dependency outside probe
 +-------------------+                 +-------------------+
 |      Clients      |                 |      Clients      |
 +---------+---------+                 +---------+---------+
           | every route fails                   |  requests flow
           v                                     v
 +-------------------+                 +-------------------+
 | LB / Service      |                 | LB / Service      |
 | endpoints = 0     |                 | endpoints = all   |
 +---------+---------+                 +---------+---------+
           x  no backends                        |
 +-------------------+                 +---------v---------+
 | Pods (all)        |                 | Pods (all)        |
 | readiness: FAIL   |                 | readiness: OK     |
 | liveness: FAIL    |                 | liveness:  OK     |
 |  -> restart loop  |                 | breaker: OPEN     |
 +---------+---------+                 +----+-----------+--+
           | probe queries DB               | db call   | cache/static
           v                                | short     | routes
 +-------------------+                      | timeout   |
 | Shared DB (slow)  |                      v           v
 +-------------------+                 +---------+ +---------+
                                       | DB slow | | 200 OK  |
                                       | -> 503  | | served  |
                                       +---------+ +---------+

When a slow shared database is checked in every pod's probes, all endpoints are removed and every route fails (A). When the dependency is kept out of the probes, pods stay in rotation, database calls fail fast behind a circuit breaker, and other routes keep serving (B).

What should a liveness probe check, and when should you skip it?

A liveness probe answers one question: is this process stuck in a state that only a restart will clear? Good candidates are a deadlocked event loop, a worker pool where every thread is blocked forever, or an internal error the code has marked as unrecoverable. A database outage does not belong on that list. Restarting the pod cannot fix the database, and every replica would fail the same check.

Depth is the hard trade-off. A handler that returns a static 200 from the HTTP server shows only that the accept loop is running. Request workers can be wedged while the probe stays green. A deep probe that runs real queries has the opposite problem: it turns slowness into restarts, and a restart under load usually adds to that load.

Locks and thread pools need deliberate choices. If the probe runs on a dedicated thread that shares nothing with request handlers, it will report healthy through a deadlock in those handlers. Taking the same lock or submitting work to the same pool is correct when detecting that deadlock is the point. It is a mistake when the probe only needs to report that the process exists.

A cheap middle ground is a progress timestamp. Each worker records the time whenever it finishes a unit of work or an idle poll. The liveness handler compares the oldest timestamp with the current time and fails if it exceeds a threshold. The handler makes no downstream calls, takes no shared locks, and uses no I/O.

If you cannot name a failure that restarting would fix, skip the liveness probe. No probe is safer than one that restarts healthy pods.

import queue, threading, time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

STALL_LIMIT_S = 30   # must exceed the slowest legitimate job plus the idle poll interval
NUM_WORKERS = 4
jobs = queue.Queue()
# One slot per worker. Each worker writes only its own slot and the probe only reads,
# so no lock is shared with the request path.
last_progress = [time.monotonic()] * NUM_WORKERS

def process(job):
    pass  # real work goes here; if it hangs or kills the thread, the slot stops updating

def worker(i):
    while True:
        try:
            job = jobs.get(timeout=5)
        except queue.Empty:
            last_progress[i] = time.monotonic()  # an idle poll still proves the loop is alive
            continue
        process(job)
        last_progress[i] = time.monotonic()      # finished a unit of work

class Liveness(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/livez":
            self.send_response(404); self.end_headers(); return
        # No downstream calls, no shared locks, no I/O: compare the oldest timestamp to now.
        stalled = time.monotonic() - min(last_progress)
        self.send_response(200 if stalled < STALL_LIMIT_S else 503)
        self.end_headers()
        self.wfile.write(f"oldest worker progress {stalled:.1f}s ago\n".encode())

    def log_message(self, *args):
        pass  # keep probe traffic out of the logs

if __name__ == "__main__":
    for i in range(NUM_WORKERS):
        threading.Thread(target=worker, args=(i,), daemon=True).start()
    ThreadingHTTPServer(("0.0.0.0", 8081), Liveness).serve_forever()

A liveness endpoint that fails only when some worker has stopped recording progress for longer than a threshold, without calling dependencies or taking the workers' locks.

How should readiness work during startup, overload and graceful shutdown?

Readiness answers a narrower question: can this instance serve a request right now? Return "not ready" until startup work is finished. That means config is loaded and validated, caches are warmed, and the schema version has been checked against what the code expects. Traffic sent earlier only produces slow or failed responses.

A hard dependency belongs in readiness only when no request could succeed without it. Before adding one, consider what happens when that dependency degrades. Every replica fails readiness at the same moment, the service loses all its endpoints, and clients see connection errors instead of a clear 503 they could handle.

Local saturation, such as a full request queue or an exhausted connection pool, has the same risk. Under a traffic spike all replicas saturate together, and saturation-based readiness pulls them out together. Load shedding inside the request path is usually safer. Reject excess work with a fast 503 or 429 and keep the instance in rotation so it continues serving what it can.

Shutdown is where readiness earns its keep. On a termination signal:

  1. Flip readiness to failing first.
  2. Keep the listener open and keep serving in-flight and newly arriving requests. Endpoint removal takes time to propagate to load balancers and proxies, so traffic keeps arriving for a short while.
  3. After a short delay, stop accepting connections, wait for in-flight requests to complete up to a deadline, then exit.

Skipping the delay is a common source of dropped connections during rollouts. The code example accompanying this section shows a readiness flag and a signal handler that drains in this order.

const http = require('node:http');

let ready = false;      // false until startup work completes, false again once draining
const DRAIN_DELAY_MS = 10000;  // time for endpoint removal to reach load balancers
const SHUTDOWN_DEADLINE_MS = 25000; // keep below the orchestrator's kill grace period

const server = http.createServer((req, res) => {
  if (req.url === '/healthz/ready') {
    res.writeHead(ready ? 200 : 503).end(ready ? 'ready' : 'not ready');
    return;
  }
  // Normal request handling continues during the drain delay.
  res.writeHead(200).end('ok');
});

async function startup() {
  // Placeholder: load and validate config, warm caches, check schema version.
  ready = true;
}

process.on('SIGTERM', () => {
  ready = false; // 1. fail readiness first; keep the listener open
  setTimeout(() => {
    // 3. stop accepting new connections; the callback runs once in-flight requests finish
    server.close(() => process.exit(0));
    server.closeIdleConnections(); // drop idle keep-alive sockets so close() can complete
  }, DRAIN_DELAY_MS); // 2. meanwhile, keep serving traffic that still arrives

  // Hard deadline: exit even if some requests never complete.
  setTimeout(() => process.exit(1), SHUTDOWN_DEADLINE_MS).unref();
});

server.listen(8080, () => {
  startup().catch((err) => { console.error(err); process.exit(1); });
});

A Node.js server keeps a readiness flag that stays false until startup finishes. On SIGTERM it fails readiness, keeps serving through a drain delay, then closes the listener and exits once in-flight requests finish or a hard deadline expires.

How do you tune probe timing without false restarts?

Every probe has the same handful of knobs. In Kubernetes these are initialDelaySeconds or a separate startupProbe for slow boots, periodSeconds, timeoutSeconds, failureThreshold and successThreshold. The last one must be 1 for liveness and startup probes. Together, period, timeout and failure threshold set detection time. The worst case is roughly period × failureThreshold + timeout.

The timeout is the most dangerous knob. A check fails if it doesn't respond in time, whether the handler is broken or not. A stop-the-world GC pause, CFS CPU throttling against a tight limit, or a cold JIT or cache path can all exceed a one-second timeout while the process is healthy. If that happens on enough consecutive probes, liveness kills the pod. Under load this is self-reinforcing: the restarted pod drops its warm caches and connections, its traffic shifts to peers that are already slow, and they start failing probes too.

A startup probe suspends liveness until the first success, so a slow boot isn't killed partway through. The cost is that a pod that will never boot is only declared failed after the full startup budget has run out. Rollouts that wait on each pod slow down by the same amount.

Illustrative estimate. The values below are assumed, not measured. Readiness uses period 5s, timeout 2s, threshold 3. Liveness uses period 10s, timeout 2s, threshold 3.

Scenario Readiness outcome Liveness outcome
Instance hangs completely Receives traffic for up to ~17s Restarted after up to ~32s
3s GC pause, once Possibly one failed probe, stays in service One failure, no restart
Throttled 2.5s responses for 30s Removed after ~15s, returns when recovered Wrongly restarted after ~30s

Readiness needs to be faster than liveness and more forgiving of slowness.

# Pod spec fragment; values match the illustrative estimate above
containers:
  - name: orders-api
    image: registry.example.com/orders-api:1.4.2
    ports:
      - name: http
        containerPort: 8080
    # Liveness and readiness are held off until this succeeds once.
    # Boot budget is about 5s x 24 = 120s. A pod that never boots is
    # only declared failed after that, and rollouts wait just as long.
    startupProbe:
      httpGet: { path: /livez, port: http }
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 24
    # Fast: a hung pod stops getting traffic in up to ~5x3+2 = 17s.
    # Failing here only removes the pod from endpoints; it is not restarted.
    readinessProbe:
      httpGet: { path: /readyz, port: http }
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 3
      successThreshold: 1
    # Slow and tolerant: restart only after up to ~10x3+2 = 32s.
    # /livez checks only the process itself, never downstream dependencies.
    # A 2s timeout is still shorter than a long GC pause or CPU throttling,
    # so check it against observed pause and latency data before relying on it.
    livenessProbe:
      httpGet: { path: /livez, port: http }
      periodSeconds: 10
      timeoutSeconds: 2
      failureThreshold: 3
      successThreshold: 1   # must be 1 for liveness and startup probes
    resources:
      requests: { cpu: 500m, memory: 512Mi }
      limits: { memory: 512Mi }  # no CPU limit here, which avoids CFS throttling stalls

A Kubernetes container spec with three probes: a startup probe that covers slow boots, a fast readiness probe that stops traffic to a hung pod, and a slower, more tolerant liveness probe that avoids restarts during brief pauses; the timings match the illustrative estimate.

What are the alternatives to probe-based dependency health?

Dependency health still matters, but its signals belong in observability, where humans and alerting rules can interpret them. The orchestrator's restart and routing machinery is the wrong consumer.

Instrument every outbound call with per-dependency latency histograms, error rates and timeout counts. If you use circuit breakers, export their state (closed, open, half-open) as a metric. An open breaker on the payments client tells an on-call engineer far more than a failing probe, and it does so without pulling instances out of rotation. Alert on these metrics with thresholds and durations chosen for humans, not for a kubelet.

For interactive diagnosis, expose a separate endpoint such as /health/deps that reports each dependency's reachability and recent error rate. No probe should point at it. Because it reveals hostnames, connection pools and internal topology, put it behind authentication or restrict it to an internal network. Your liveness and readiness endpoints must stay unauthenticated, so the two cannot share an access policy.

At runtime, the correct response to a sick dependency is graceful degradation: serve cached data, disable the affected feature, and return fast errors through an open breaker instead of letting requests queue behind timeouts.

Per-instance checks have a blind spot. They only see what the instance can observe about itself. A bad DNS record, a misrouted load balancer or a broken authentication flow can leave every probe green while users fail. End-to-end synthetic transactions and SLO burn-rate alerts measure what users actually experience, so they catch the failures that every probe reports as healthy.

Implementation rules and how good health check design saves engineering time

Rules for the handlers themselves

  • Keep handlers cheap. A probe runs every few seconds on every replica, so any work it does is multiplied across the fleet. Liveness should touch only in-process state.
  • Cache expensive results with a short TTL. If a diagnostic endpoint checks a database, cache the result for a few seconds. You will detect failures more slowly and sometimes report stale status, but probes will no longer add load to a dependency that is already struggling.
  • Never require auth from the prober. If credential rotation or an identity-provider outage makes probes fail, the orchestrator will restart healthy pods. Restrict health endpoints at the network level instead.
  • Return clear status codes. Use 200 for healthy and 503 for not ready. You can add an optional body that names the reason, such as "draining" or "warming cache", so a human can read it without searching logs.

Who should adopt this, and when

Any service that runs under an orchestrator should adopt this design from its first deployment. Adding it later means changing probe semantics on a service that is already in production, which is riskier than getting it right at the start.

Why it pays off

Separating the endpoints and exporting dependency metrics takes a few hours of work. A single restart storm can take far longer to diagnose, and a probe that pages on every dependency blip keeps costing on-call time.

The split also speeds up triage. If liveness fails, the process is broken. If readiness fails, the pod is not accepting traffic. If a diagnostic check fails, a dependency is degraded. Each signal points to a different responder. Rollouts become more predictable for the same reason: a pod receives traffic only when it can actually serve it.

Review checklist

  • Liveness checks only the process itself and has no dependency calls.
  • Readiness turns false during startup and while draining.
  • No probe requires authentication.
  • Expensive checks are cached.
  • Dependency health is reported through metrics and alerts, not through restarts.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to microservice health check design, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.