Kubernetes Liveness vs Readiness Probes (and Startup): Getting Them Right

Three probes, three different questions

Kubernetes gives you three probes, and each one asks the kubelet to answer a different question about a container. Most probe-related outages come from making one probe answer another's question, for example a liveness probe that checks the database. When the database slows down, every replica gets restarted at once, and a dependency blip becomes a full outage.

Readiness failures are reversible and cheap. Liveness failures are destructive: a restart drops in-flight requests, clears in-memory state and puts the container through startup again. That asymmetry should drive every probe decision.

Two groups need to understand this. Backend engineers write the health endpoints, so they decide what each check actually tests. Platform engineers set the defaults in Helm charts, templates and admission policies, and those defaults get copied into dozens of services. A bad default fails across your whole fleet, not just one service.

Who should use probes, and when

Who should use them

Anyone who runs long-lived containers on Kubernetes needs to make probe decisions, even if the decision is to leave a probe out:

• Service owners whose pods sit behind a Service or ingress. They need readiness so traffic only reaches pods that can serve it, and they own the code behind every health endpoint.
• Platform and SRE teams who maintain shared charts, templates and policies. Their defaults decide whether a new service starts with safe probes or with a copied liveness check that queries the database.
• Teams running slow-booting workloads such as JVM services, model servers or apps that load large caches. They are the ones most likely to see restart loops when liveness runs before startup finishes.
• On-call engineers. They don't write probes, but they need to recognise a probe-induced restart storm and tell it apart from an OOM kill or a crash.

Short-lived batch Jobs that run to completion rarely benefit from probes. Exit status already tells Kubernetes whether the work succeeded.

When each probe should be used

• Readiness: almost always, for anything that receives traffic. If a pod is behind a Service, readiness is what keeps new, warming or draining pods out of the endpoint list. It also gates rolling updates, because a Deployment counts a new pod as available only once it is Ready.
• Startup: when boot time is long or unpredictable. Use it when the app can take longer to start than you would want liveness to tolerate during normal operation. Don't use a large initialDelaySeconds instead.
• Liveness: only when you can name the wedged state it detects. Examples are a deadlocked worker pool or a stalled event loop that leaves the process running but useless. If the process exits on fatal errors, the restart policy already covers you, and a liveness probe mostly adds risk.

Revisit probes when something about the workload changes. A new dependency, a heavier boot path, tighter CPU limits or a change in traffic shape can each make settings that used to be safe start causing false failures.

How probes work: mechanisms and timing fields

Probes are executed by the kubelet on the node running the pod. They do not go through the Service, kube-proxy or ingress. A passing probe tells you nothing about DNS, network policy or load balancer configuration between your users and the pod.

Mechanisms

Timing fields

The example that follows uses these fields:

• initialDelaySeconds: the wait before the first probe runs.
• periodSeconds: the interval between checks.
• timeoutSeconds: how long a single check can take before it counts as failed.
• failureThreshold: how many consecutive failures trigger action. For liveness the action is a restart; for readiness the pod is removed from endpoints.
• successThreshold: how many consecutive passes are needed to recover. Liveness and startup probes require this to be 1.

Illustrative estimate. Assume periodSeconds: 10, failureThreshold: 3 and timeoutSeconds: 2. Worst-case detection is roughly 10 × 3 + 2 = 32 seconds after the container stops responding.

Tightening these values detects failures faster but increases the risk of false positives. Under load, a garbage-collection pause or a saturated event loop can push one response past a 1-second timeout. With a low threshold, a few slow responses in a row restart a healthy pod during the moment you need it most. That restart moves its traffic to the other replicas, which are already under the same load.

CopyapiVersion: v1
kind: Pod
metadata:
  name: web-example
spec:
  containers:
    - name: web
      image: registry.example.com/web:1.0.0
      ports:
        - containerPort: 8080
      # Readiness: decides whether the pod receives Service traffic.
      readinessProbe:
        httpGet:
          path: /ready        # should check the app can actually serve requests
          port: 8080
        initialDelaySeconds: 5  # wait before the first check
        periodSeconds: 5        # check every 5s
        timeoutSeconds: 2       # a check slower than 2s counts as a failure
        failureThreshold: 3     # 3 consecutive failures -> removed from endpoints
        successThreshold: 2     # 2 consecutive passes before traffic returns
      # Liveness: restarts the container if it is wedged.
      livenessProbe:
        httpGet:
          path: /healthz      # cheap check that the process is not stuck
          port: 8080
        initialDelaySeconds: 15
        periodSeconds: 10
        timeoutSeconds: 2
        failureThreshold: 3     # illustrative worst case: ~10 x 3 + 2 = 32s to restart
        successThreshold: 1     # must be 1 for liveness probes
      # Avoid for serving checks: an open port does not prove the app can respond.
      # tcpSocket:
      #   port: 8080

An annotated Pod spec showing httpGet readiness and liveness probes with each timing field explained, including the illustrative 32-second worst-case liveness detection window.

Liveness probes: keep them shallow, or leave them out

A liveness probe answers one narrow question: is this process wedged in a way that only a restart will fix? Good candidates are a deadlocked worker pool, a stuck event loop, or corrupted in-memory state the code cannot recover from. The check should exercise only the process itself, for example by confirming the main loop is still turning. If restarting the container would not fix the failure, liveness should not detect it.

Why deep checks cascade

Suppose liveness queries a shared database and the database slows down:

DB slow → every pod's probe times out → kubelet restarts pods → fewer replicas serve traffic → survivors overload → their probes fail → more restarts

1. The database latency rises above the probe timeout.
2. Every replica fails liveness at roughly the same moment, because they all share the dependency.
3. The kubelet kills them. Restarted pods reconnect, warm caches and open connection pools, which adds load to the database that is already struggling.
4. The pods still running absorb the redirected traffic, slow down and fail their own probes.

The database problem was recoverable. The restart loop turns it into an outage. The same pattern applies to caches, queues and downstream APIs.

Aggressive settings punish normal behavior

A short timeout combined with a low failure threshold can turn routine slowness into restarts. A long GC pause, CPU throttling against a container limit, or a burst of traffic can each make a healthy process miss a few probes in a row.

If you cannot name a specific wedged state that the probe detects, leave liveness out. A crashing process exits and gets restarted anyway. A bad liveness probe adds a new way for healthy pods to fail.

Copy+-------------------------------------------------------------+
|      Shared database  (slow: latency > probe timeout)       |
+-------------------------------------------------------------+
     ^              ^              ^              ^
     | deep         | deep         | deep         | reconnect +
     | liveness     | liveness     | liveness     | pool setup +
     | query        | query        | query        | cache warm-up
+---------+    +---------+    +---------+    +---------+
|  pod A  |    |  pod B  |    |  pod C  |    |  pod D  |
| TIMEOUT |    | TIMEOUT |    | TIMEOUT |    | RESTART |
+---------+    +---------+    +---------+    +---------+
     ^              ^              ^              ^   liveness probes
     +--------------+------+-------+--------------+
                           |
                 +-------------------+
                 |      kubelet      |<-------------+
                 | liveness failures |              |
                 +-------------------+              |
                           |  kill + restart pods   |
                           v                        |
                 +-------------------+              |
                 | fewer pods serve  |              |
                 | all the traffic   |              |
                 +-------------------+              |
                           |                        |
                           v                        |
                 +-------------------+              |
                 | survivors overload|              |
                 | and miss probes   |--------------+
                 +-------------------+

A liveness probe that queries a shared database makes every replica fail together when the database slows. Restarts then add reconnect and warm-up load to the database and push traffic onto the remaining pods, which miss their own probes and are restarted in turn.

Readiness and startup probes in practice

A readiness probe answers a narrower question: can this instance usefully serve a request right now? A good check covers local state, such as whether caches are warm, connection pools are open and configuration is loaded. Dependencies need more care. If every replica checks the same database and that database goes down, every pod fails readiness at once and the Service has zero endpoints. Clients then get connection errors instead of a fast, explicit 503. Only check a dependency if this instance can fail it while its peers pass, for example a broken local connection pool. A failing readiness probe removes the pod from endpoints. It never restarts it.

For slow starts such as JVM warm-up, model loading or migrations, use a startup probe instead of a large initialDelaySeconds on liveness. Liveness and readiness checks do not run until the startup probe succeeds, so you can size the startup budget on its own. In the example below, failureThreshold × periodSeconds covers the worst plausible startup. A generous budget tolerates slow nodes, but a pod that is truly hung at boot takes longer to be killed. A tight budget can put healthy but slow pods into restart loops.

Readiness also controls rolling updates. The Deployment counts a new pod as available only once it is Ready, and only then does it remove old pods. Probes do not drain traffic on shutdown, though. Kubernetes sends SIGTERM at roughly the same time it starts removing the pod from endpoints. Your process should catch SIGTERM, keep serving while that removal propagates (a short preStop sleep helps), finish in-flight requests and then exit, all within terminationGracePeriodSeconds.

Copyspec:
  terminationGracePeriodSeconds: 45  # must cover preStop sleep + draining in-flight requests
  containers:
    - name: api
      image: registry.example.com/orders-api:1.4.2
      ports:
        - containerPort: 8080
      # Startup budget: failureThreshold x periodSeconds = 30 x 10s = 300s.
      # Size this to the worst plausible boot (JVM warm-up, cache or model load).
      # Liveness and readiness do not run until this probe succeeds.
      startupProbe:
        httpGet:
          path: /healthz/started
          port: 8080
        periodSeconds: 10
        failureThreshold: 30
      # Readiness checks local state only (caches warm, pool open, config loaded).
      # A failure removes the pod from endpoints. It never restarts the pod.
      readinessProbe:
        httpGet:
          path: /healthz/ready
          port: 8080
        periodSeconds: 5
        failureThreshold: 3
      # Liveness stays cheap and fast. No initialDelaySeconds needed here.
      livenessProbe:
        httpGet:
          path: /healthz/live
          port: 8080
        periodSeconds: 10
        failureThreshold: 3
      lifecycle:
        preStop:
          # Keep serving while endpoint removal propagates. SIGTERM is sent
          # after this hook returns, and the app then drains in-flight requests.
          exec:
            command: ["sleep", "10"]

A container spec that uses a 300-second startup probe budget for slow boots, a local-state readiness probe, a lightweight liveness probe, and a preStop sleep sized within terminationGracePeriodSeconds so traffic drains before shutdown.

Implementing health endpoints in your application

Expose two endpoints, for example /livez and /readyz, backed by two separate handlers. The liveness handler should check only that the process can run code and respond, so it should return 200 without touching anything else. The readiness handler can check what the instance needs before it accepts traffic: that warm-up has finished, that a connection pool has at least one usable connection, and that the process has not started draining. Keep each check cheap and bounded. Cache the result of an expensive dependency check for a few seconds instead of running it on every probe. Every replica is probed on every period, so a costly check multiplies across the fleet.

Register both routes ahead of authentication, rate limiting and request logging middleware. The kubelet sends no credentials, so an auth layer returns 401 and the probe fails. A rate limiter can throttle probes during a traffic spike, which is exactly when a false failure does the most damage. Logging every hit buries useful lines under thousands of identical ones each hour. Log only state changes, such as ready to not-ready.

Then decide where the handlers run:

• Shared request path. Probes queue behind real requests, so readiness reflects actual serving capacity. A saturated worker pool also delays liveness responses, though, which can cause restarts under load.
• Separate thread or port. Probes always answer, but an instance whose main pool is deadlocked can still report healthy.

A common compromise serves readiness on the main path and liveness on a lightweight side listener. The liveness handler then checks a heartbeat that the main loop updates, so a deadlocked main loop still fails the probe.

Copyimport threading, time
from http.server import ThreadingHTTPServer, BaseHTTPRequestHandler

state = {"heartbeat": time.monotonic(), "warmed_up": False, "draining": False}
HEARTBEAT_MAX_AGE = 10  # seconds; must exceed the main loop's slowest normal iteration

class QuietHandler(BaseHTTPRequestHandler):
    def log_message(self, *args):  # suppress per-request access logs for probe hits
        pass
    def reply(self, code):
        self.send_response(code)
        self.send_header("Content-Length", "0")
        self.end_headers()

class AppHandler(QuietHandler):  # main port 8080: real traffic plus readiness
    def do_GET(self):
        if self.path == "/readyz":  # matched before auth, rate limiting and logging
            ok = state["warmed_up"] and not state["draining"]
            return self.reply(200 if ok else 503)
        # auth, rate limiting and request logging would wrap only the code below
        self.reply(200)

class LiveHandler(QuietHandler):  # side port 8081: liveness only
    def do_GET(self):
        if self.path != "/livez":
            return self.reply(404)
        age = time.monotonic() - state["heartbeat"]  # stale heartbeat means the main loop is stuck
        self.reply(200 if age < HEARTBEAT_MAX_AGE else 503)

def main_loop():
    state["warmed_up"] = True  # set once caches are loaded and pools are open
    while True:
        state["heartbeat"] = time.monotonic()  # stops advancing if this loop deadlocks
        time.sleep(1)  # stands in for one iteration of the application's real work

threading.Thread(target=main_loop, daemon=True).start()
live = ThreadingHTTPServer(("", 8081), LiveHandler)
threading.Thread(target=live.serve_forever, daemon=True).start()
ThreadingHTTPServer(("", 8080), AppHandler).serve_forever()

A standard-library Python sketch that serves /readyz on the main port ahead of any middleware and /livez on a side listener, where /livez fails if the main loop stops updating its heartbeat and neither probe endpoint is logged.

Diagnosing probe misconfiguration

When restarts climb or a Service stops receiving traffic, start with kubectl describe pod <name>. Three parts of the output matter. Restart Count shows how often the kubelet has replaced the container. Last State: Terminated shows the reason and exit code of the previous instance. The Events list at the bottom records probe failures as Unhealthy events, with the probe type and the failing response or timeout.

These fields separate an OOM kill from a liveness kill. With an OOM kill, the last state shows Reason: OOMKilled, usually with exit code 137, and no probe failures come before it. With a liveness kill, Liveness probe failed events appear first, followed by a message that the container will be restarted. The last state then usually shows Error with exit code 137 or 143, depending on how the process handled SIGTERM. The two problems need different fixes. Raising memory limits will not fix a probe timeout, and loosening a probe will not fix a leak.

Events are retained only briefly. Run kubectl get events --field-selector involvedObject.name=<pod> soon after an incident, before the evidence ages out.

For missing traffic, run kubectl get endpoints <service>. Only pods that pass readiness are listed as addresses. If a running pod is absent, its readiness probe is failing.

Choosing a probe strategy: checklist, alternatives and cost

Probes are not the only safety net, and often not the best one:

• Exit on fatal errors. If the process exits when it hits an unrecoverable state, the container restart policy restarts it, with backoff. No probe can report a failure more truthfully than the process exiting.
• Application watchdogs. A goroutine or thread that checks whether the event loop or worker is still making progress, and exits the process when it stalls, catches wedges that an HTTP handler would miss. Liveness becomes unnecessary or trivial.
• External synthetic checks. These run through the real ingress, DNS and TLS path and catch failures the kubelet never sees. They should page someone, not restart pods.
• Graceful shutdown. Handle SIGTERM by draining in-flight requests. Endpoint removal and SIGTERM happen concurrently, so a short pause before closing listeners avoids dropped connections during rollouts.

How correct probes save engineering time

Well-chosen probes save time in four places:

• Fewer self-inflicted incidents. A shallow liveness probe, or none at all, removes the restart-storm pattern described above. A dependency slowdown stays a dependency slowdown and doesn't escalate into a fleet-wide outage that pulls several engineers into an incident call.
• Safer rollouts with less babysitting. Because a Deployment waits for new pods to become Ready, a release that cannot serve traffic stalls the rollout instead of taking user traffic. The old pods keep serving while someone investigates, so a bad deploy does not become an emergency rollback.
• Faster diagnosis. When each probe tests one well-defined thing, the symptom points to the cause. An empty endpoint list means readiness is failing, Unhealthy liveness events before a restart mean the probe killed the pod, and OOMKilled means memory. On-call engineers spend less time ruling out possibilities.
• No repeated tuning. Sizing a startup budget once, and keeping liveness cheap, avoids the cycle of raising initialDelaySeconds every time boot time grows and chasing intermittent restarts that only appear under load.

The engineering trade-off is lopsided.

Illustrative estimate (assumed numbers, not a benchmark): net hours saved = (incidents avoided × engineers × hours per incident) − (services × setup hours)

With 20 services, 2 setup hours each, and 10 avoided incidents a year at 3 engineers × 4 hours, the result is (10 × 3 × 4) − (20 × 2) = 120 − 40 = 80 hours.

Most of that cost should be paid once. Ship platform-wide defaults, such as Helm chart or template values with sane timings and no liveness unless a team opts in. This saves each team from rediscovering restart storms on its own.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes liveness readiness probes, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.