What does a liveness probe actually decide, and how is it different from readiness and startup probes?

Each probe answers a different question, and the kubelet takes a different action when it fails. Restart loops often start when an engineer uses the wrong probe for the question they want to ask.

Probe Question it answers Action on failure Who needs it
Liveness Is this process stuck in a state it cannot recover from? Kubelet kills the container. The restartPolicy then restarts it. Processes that can deadlock or wedge while still running
Readiness Can this Pod serve requests right now? Pod is removed from Service endpoints. No restart. Almost every workload that receives traffic
Startup Has initialization finished? Liveness and readiness checks are held off until it succeeds. Repeated failure kills the container. Applications with slow or variable startup

Why each probe exists

Liveness handles one specific failure: the process is alive, but it will never make progress again. Examples are a deadlocked thread pool or a hung event loop. A restart is the only fix for this.

Readiness handles conditions that pass on their own, such as a warming cache, a saturated worker pool, or a slow downstream dependency. Taking the Pod out of rotation gives it time to recover. Readiness keeps running for the container's whole lifetime, not only at startup.

Startup separates "not ready yet" from "broken." Without it, you have two bad options for slow-booting apps. You can set a long initialDelaySeconds on liveness, which delays detection of real hangs for the rest of the Pod's life. Or you can set a short delay, which kills containers that are still booting.

The common mistake

A liveness endpoint that checks the database, a downstream API, or request latency is really a readiness check. When those dependencies slow down under load, every replica fails at once and gets restarted. That adds cold-start load to a system that is already struggling, and the result is often CrashLoopBackOff across the whole Deployment.

Why do liveness probes cause restart loops under load?

Shared dependencies turn one slowdown into a fleet-wide failure

When the liveness handler queries the database or pings a cache, its result depends on something every replica shares. A slow primary, a cache failover, or a saturated connection limit makes every Pod's probe fail within the same few periods. Restarting the container does not fix the dependency, so the restart has no benefit. It also adds reconnection storms and cold caches.

Probes compete with real work

The kubelet's HTTP request goes through the same listener, worker threads, connection pool and event loop as user traffic. If request handling is saturated, the probe waits in the queue like any other request. A process that is slow but making progress misses timeoutSeconds and gets classified as dead. Garbage-collection pauses and CPU throttling under a tight limit cause the same false failure.

Restarts shrink capacity at the worst moment

Each restarted Pod drops out of the Service. Its share of traffic moves to the surviving Pods, which pushes them past their own probe timeouts. A rolling deploy already runs with reduced capacity, and new Pods start cold. The autoscaler adds replicas only after metrics lag, and those replicas also start cold, so they fail probes before they can absorb load.

 load rises ──► probe latency > timeout ──► kubelet restarts Pod
     ▲                                             │
     │                                             ▼
 survivors overload ◄── traffic shifts ◄── capacity drops
 +-----------+     user requests      +---------------------------------+
 |  Service  |----------------------->| Pod A                           |
 +-----------+                        |  listener -> workers -> DB pool |
       ^            GET /healthz      |  probe waits behind user queue  |
       |   +---------+--------------->|                                 |
       |   | kubelet |                +---------------------------------+
       |   +---------+--------------->| Pod B  (same path, same pools)  |
       |                              +---------------------------------+
       |                                    |                |
       |                                    v                v
       |                              +---------------------------------+
       |                              |   shared database / cache       |
       |                              |   (slow primary, failover)      |
       |                              +---------------------------------+
       |                                              |
       |                                              v
       |                              +---------------------------------+
       |                              | all probes exceed timeoutSeconds|
       |                              | -> kubelet restarts containers  |
       |                              +---------------------------------+
       |                                              |
       |                                              v
 +--------------------------+         +---------------------------------+
 | traffic shifts to fewer  |<--------| restarted Pods leave Service;   |
 | survivors; they overload |         | new Pods start cold             |
 +--------------------------+         +---------------------------------+

Liveness probes travel the same request path as user traffic and depend on a shared database, so one slowdown fails every replica's probe, and each restart moves more load onto fewer Pods.

How do timeoutSeconds, periodSeconds and failureThreshold combine into time-to-restart?

The kubelet runs the probe every periodSeconds. Each attempt that does not answer within timeoutSeconds counts as a failure, just like an error response. Once failureThreshold consecutive attempts fail, the kubelet kills the container. In the worst case, the process stops responding right after a successful probe, so the time from hang to restart is roughly:

time-to-restart ≈ periodSeconds × failureThreshold + timeoutSeconds

Graceful termination adds time after that. A single success resets the counter, so you only get a restart when failures happen back to back.

The default 1-second timeout is the field most likely to cause trouble. It measures how fast your health endpoint responds under current conditions, not whether the process is alive. Under load that endpoint waits behind the same thread pool, event loop or connection queue as real requests. CPU throttling at the container limit and garbage-collection pauses also slow it down. Because load is sustained, slow responses come in a run, which is exactly the pattern that satisfies failureThreshold.

Illustrative estimate (assumed numbers, not a benchmark):

  • Assume the health endpoint has a p99 of 0.2 s when idle and 1.5 s at peak load.
  • Defaults (timeout 1 s, period 10 s, threshold 3): the loaded p99 is above the timeout, so three slow probes in a row restart a working Pod in about 10 × 3 + 1 = 31 s.
  • Adjusted (timeout 3 s, period 10 s, threshold 6): the timeout is about twice the loaded p99, and a real hang is caught in about 10 × 6 + 3 = 63 s.

To set these for your own service, measure the probe endpoint's latency at peak load. Then decide how long a deadlocked process can stay up before the cost of waiting is higher than the cost of a restart.

# Container spec fragment: liveness probe tuned using the article's illustrative
# numbers (assumed loaded p99 of 1.5 s). Not a recommended default.
apiVersion: v1
kind: Pod
metadata:
  name: orders-api
spec:
  containers:
    - name: app
      image: registry.example.com/orders-api:1.0.0
      ports:
        - containerPort: 8080
      livenessProbe:
        httpGet:
          path: /healthz
          port: 8080
        # Run a probe every 10 s (same as the default).
        periodSeconds: 10
        # About 2x the assumed loaded p99 (1.5 s), so a slow-but-alive
        # response is not counted as a failure. Default is 1 s.
        timeoutSeconds: 3
        # Require 6 consecutive failures; any single success resets the count.
        # Default is 3.
        failureThreshold: 6
        # Worst-case hang-to-kill time: 10 * 6 + 3 = ~63 s,
        # plus terminationGracePeriodSeconds after the kill is triggered.
      # Graceful shutdown window that follows the restart decision (default 30 s).
  terminationGracePeriodSeconds: 30

A Pod liveness probe that uses the section's illustrative estimate (3 s timeout, 10 s period, threshold of 6), with comments showing how the fields combine into roughly 63 s of worst-case time before a restart.

How should you implement a safe liveness, readiness and startup probe configuration?

Each probe should answer a different question, so give each one its own endpoint.

Liveness should check only whether the process can still make progress. Serve it from a handler that returns a static 200 and touches no database, cache, downstream service or lock shared with request handling. If the liveness handler checks a dependency, a database outage restarts every replica at the same time, and restarting does not fix the database.

Readiness does the traffic gating. It can check connection pools, queue depth or whether a required dependency is reachable. A failed readiness check removes the Pod from Service endpoints without killing it, so the Pod recovers on its own once the pressure drops.

Startup covers slow initialization: JIT warm-up, cache loading, migrations. Liveness and readiness checks do not run until the startup probe succeeds. Set its failureThreshold × periodSeconds to a generous upper bound, such as 30 × 10 s for a five-minute window. This is better than a large initialDelaySeconds because a startup probe lets the Pod proceed as soon as it is ready. After startup, liveness can stay tight, and a fast start does not wait for the full window.

Where the runtime allows it, serve liveness on a dedicated port or on a thread outside the main request pool. Otherwise saturated workers delay the probe, and the kubelet restarts a Pod that is busy but healthy.

Prefer HTTP or TCP checks to exec probes. An exec probe spawns a process in the container on every check. That costs CPU and memory inside the same cgroup that is already under pressure, and it gets slower exactly when the Pod is busiest.

apiVersion: v1
kind: Pod
metadata:
  name: orders-api
spec:
  containers:
    - name: app
      image: registry.example.com/orders-api:1.4.2
      ports:
        - name: http          # main request traffic
          containerPort: 8080
        - name: health        # separate listener so busy workers don't delay probes
          containerPort: 8081
      startupProbe:           # up to 30 x 10s = 5 min for warm-up, cache load, migrations
        httpGet:
          path: /livez
          port: health
        periodSeconds: 10
        failureThreshold: 30
      livenessProbe:          # static 200, no DB/cache/downstream calls, no shared locks
        httpGet:
          path: /livez
          port: health
        periodSeconds: 10
        timeoutSeconds: 2
        failureThreshold: 3
      readinessProbe:         # checks pools/dependencies; failing only removes from endpoints
        httpGet:
          path: /readyz
          port: health
        periodSeconds: 5
        timeoutSeconds: 2
        failureThreshold: 2
        successThreshold: 1

This Pod spec serves all three probes from a dedicated health port. The startup probe allows up to five minutes for initialization, liveness hits a cheap dependency-free endpoint, and readiness does the dependency-aware traffic gating, all over HTTP instead of exec.

How do you confirm a CrashLoopBackOff was caused by a probe?

Start with kubectl describe pod <name> and read two parts of the output: the container's Last State block and the Events list at the bottom.

In Events, a probe-driven restart leaves a clear trail. You will see Unhealthy warnings with messages like Liveness probe failed:, often followed by a timeout or connection error. The kubelet then records that the container failed its liveness probe and will be restarted. If those warnings repeat just before each restart, and Restart Count keeps climbing, the probe is your prime suspect. Events are retained only for a limited time, so capture them early during an incident.

In Last State, compare the termination reason and exit code with what the application would produce on its own:

  • A process that crashes on its own usually exits with its own code, such as 1, and Reason: Error.
  • A container killed after a probe failure typically shows 137 (SIGKILL after the grace period) or 143 (it exited cleanly on SIGTERM).
  • Reason: OOMKilled also shows 137, so don't confuse it with a probe kill.

Then run kubectl logs <pod> --previous to read the terminated container's output. A probe kill under load usually looks like a healthy process that was busy: requests still being served, slow-request warnings, then a shutdown message, with no stack trace or fatal error. A genuine crash ends with the error that caused it.

These two commands take minutes and separate "the app is broken" from "the probe is too strict." Without that distinction, on-call engineers can spend an incident hunting a code bug that doesn't exist, or roll back a release that was never at fault.

#!/usr/bin/env bash
# Triage a CrashLoopBackOff: was it a liveness probe kill or a real crash?
# Usage: ./probe-triage.sh <pod> <container> [namespace]
set -euo pipefail
POD="$1"; CONTAINER="$2"; NS="${3:-default}"

# 1. Save events first. They expire, so keep a copy for the incident record.
kubectl get events -n "$NS" \
  --field-selector involvedObject.name="$POD" \
  --sort-by=.lastTimestamp > "events-$POD.txt"

# 2. Probe failures are recorded as 'Unhealthy' events.
echo "--- Unhealthy events (look for 'Liveness probe failed:') ---"
grep -E 'Unhealthy|Killing' "events-$POD.txt" || echo "none recorded"

# 3. Restart count and how the previous container instance terminated.
JP='{.status.containerStatuses[?(@.name=="'"$CONTAINER"'")]}'
STATUS=$(kubectl get pod "$POD" -n "$NS" -o jsonpath="$JP")
echo "--- Restart count / last termination ---"
kubectl get pod "$POD" -n "$NS" -o jsonpath=\
'{range .status.containerStatuses[*]}{.name}{" restarts="}{.restartCount}{" reason="}{.lastState.terminated.reason}{" exitCode="}{.lastState.terminated.exitCode}{"\n"}{end}'
# Reading the result:
#   reason=Error, exitCode=1 (or another app code)  -> the app crashed on its own
#   exitCode=137 or 143 + Unhealthy events            -> likely a probe kill
#   reason=OOMKilled, exitCode=137                    -> memory limit, not the probe

# 4. Last output from the killed container. A probe kill under load usually
#    shows normal or slow request handling, then a shutdown, with no stack trace.
echo "--- Previous container logs (tail) ---"
kubectl logs "$POD" -n "$NS" -c "$CONTAINER" --previous --tail=50
: "${STATUS:=}"  # raw status JSON kept in case you want to inspect it further

A short triage script that saves the pod's events, pulls out liveness probe failures, prints the restart count and last exit code and reason, and shows the previous container's logs so you can tell a probe kill from a real crash.

When should you avoid a liveness probe, and what are the alternatives?

The simplest fix for a liveness probe that restarts healthy pods is often to delete it. If your process exits on fatal errors, such as an unhandled exception, an out-of-memory kill, or a failed config load at boot, the container restart policy already handles that case. A liveness probe adds nothing there except another way to fail. Liveness is justified mainly for a known hang mode the process cannot detect or escape on its own, for example a deadlocked worker pool or an event loop blocked on a lock. If you can't name that failure mode, you probably don't need the probe.

When you do need hang detection, keep it separate from traffic gating:

Option Handles Main risk Dev and maintenance cost
Restart policy alone Crashes and clean exits Hangs go unnoticed None beyond exiting correctly on fatal errors
Readiness-only gating Temporary overload, warm-up If tied to dependencies, a shared outage marks every pod unready and blackholes all traffic Low, as long as the check stays local to the process
In-process watchdog that exits Deadlocks the app can observe, such as stalled heartbeats from worker threads A badly tuned threshold kills slow but healthy work Moderate: needs a progress signal and tests for the exit path
Monitoring and alerting Rare or ambiguous hangs A person responds slower than an automated restart Moderate: dashboards, alert tuning, runbooks

The watchdog is often the best replacement. It runs inside the process, so it can tell "no progress" apart from "slow progress." A kubelet HTTP probe can't make that distinction.

Trade-offs and a per-workload liveness probe review checklist

Every fix for a restart loop moves risk somewhere else. Make that move on purpose.

Choice What you gain What it can hide or cost
Lenient thresholds and timeouts No restarts during latency spikes A real deadlock goes unnoticed for minutes
Trivial 200 handler Probe never fails because of load A wedged worker pool still reports healthy
Isolated probe thread or port Probe is immune to request queueing Masks saturation; adds a code path that needs its own tests
Long startup window Slow init no longer gets killed Startup-time regressions creep in unnoticed; failed rollouts take longer to detect

The trivial-handler row matters most. If the probe only shows that the HTTP listener can answer, it reports nothing about whether work is getting done. That is why the earlier sections tied liveness to a progress signal, not to reachability. A long startup window has a similar cost. If startup time doubles over several releases, nothing fails, so add an alert on startup duration.

Per-workload review checklist

  • Dependency-free: Liveness never calls databases, caches or downstream services.
  • Cheap and isolated: The handler does no I/O and does not wait behind request queues.
  • Measured values: timeoutSeconds and failureThreshold come from p99 probe latency observed under load, not defaults.
  • Startup probe: Slow initialization is covered by a startup probe, not by a large initialDelaySeconds.
  • Readiness gates traffic: Overload sheds traffic through readiness, not through restarts.
  • Disruption settings: PodDisruptionBudgets and maxSurge/maxUnavailable keep enough capacity if several pods restart at once.
  • Worst-case time-to-restart: The reviewer can state it in seconds and explain why that delay is acceptable.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes liveness probe mistakes, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.