Kubernetes Probes: Liveness vs Readiness vs Startup, Configured Right

What liveness, readiness and startup probes each control

If you run any service on Kubernetes, whether an API, a queue worker or a stateful component, each probe type answers one question for the kubelet. The kubelet takes a different action depending on which probe fails.

Readiness is the only probe that does not restart the container, and that difference is intentional. It handles temporary conditions such as warming a cache, waiting on a dependency or shedding load. In those cases a restart would discard state and make the situation worse. Liveness is for a process that cannot recover by itself, such as a deadlock or a wedged event loop.

The startup probe gates the other two. Until it succeeds, the kubelet does not run liveness or readiness checks. This lets a slow-booting application take the time it needs without loosening the liveness settings that protect it for the rest of its life.

All probes are executed by the kubelet on the pod's node, against the container directly. A passing probe proves the container responds to its own node. It says nothing about Service routing, DNS, network policy, ingress, or whether clients in other namespaces can reach the pod.

Copy Node: the kubelet runs every probe locally against the container
+--------------------------------------------------------------------------+
|  +---------+                                            +-------------+  |
|  | kubelet | startup probe (from container start)       |  container  |  |
|  |         |------------------------------------------->|             |  |
|  |         | fail x failureThreshold -> kill            |             |  |
|  |         |                                            |             |  |
|  |         | gates others until first success           |             |  |
|  |         |                                            |             |  |
|  |         | liveness probe (after startup ok)          |             |  |
|  |         |------------------------------------------->|             |  |
|  |         | fail -> kill; restartPolicy applies        |             |  |
|  |         |                                            |             |  |
|  |         | readiness probe (after startup ok)         |             |  |
|  |         |------------------------------------------->|             |  |
|  |         | fail -> not ready, no restart              |             |  |
|  +---------+                                            +-------------+  |
+--------------------------------------------------------------------------+
         | readiness failure: pod removed from endpoints
         v
  +--------------------------+       +---------------------------------+
  | Service endpoints        |<------| clients, DNS, ingress,          |
  | (ready pods only)        |       | network policy, other namespaces|
  +--------------------------+       +---------------------------------+
  ^ this client path is NOT tested by probes; they test kubelet -> container

The kubelet runs the startup probe first and only then runs liveness (kill and restart on failure) and readiness (remove from Service endpoints on failure), and none of the three probes tests the client path through DNS, ingress, network policy or Service routing.

How probes work: mechanisms and timing fields

Probe mechanisms

exec overhead matters more than it looks. Each probe forks a process in your container's cgroup, and it competes for the same CPU and memory limits as the app. Under tight limits or with many pods per node, a shell script every few seconds is a measurable load that is easy to overlook.

Timing fields

initialDelaySeconds waits before the first probe. periodSeconds sets the interval. timeoutSeconds caps each attempt, and a timeout counts as a failure. failureThreshold is the number of consecutive failures before the kubelet acts. successThreshold is the number of consecutive passes needed to recover, and it must be 1 for liveness and startup probes.

Illustrative estimate: with periodSeconds=10, timeoutSeconds=1 and failureThreshold=3, a process that hangs just after a successful probe is restarted after roughly 30 seconds. Add the timeouts and the termination grace period to that.

Size timeoutSeconds against your p99 latency under peak load, not against idle response time. A 1-second timeout on an endpoint that slows to 1.5 seconds during a traffic spike makes the probe fail exactly when the pod is busiest.

CopyapiVersion: v1
kind: Pod
metadata:
  name: orders-api
spec:
  containers:
  - name: app
    image: registry.example.com/orders-api:1.4.2
    ports:
    - containerPort: 8080   # HTTP API
    - containerPort: 9090   # gRPC, implements the standard health service
    livenessProbe:
      # httpGet proves the handler runs and returns 200-399 in time.
      # Keep /healthz cheap and free of downstream dependencies.
      httpGet:
        path: /healthz
        port: 8080
      periodSeconds: 10
      # Sized above observed p99 under peak load, not idle latency.
      timeoutSeconds: 3
      # 3 consecutive failures x 10s period: roughly 30s+ before restart
      # (illustrative estimate, plus timeouts and termination grace period).
      failureThreshold: 3
      successThreshold: 1   # must be 1 for liveness probes
    readinessProbe:
      # grpc probe asks the health service for SERVING.
      # No exec, so no extra process forked in the container on every probe.
      grpc:
        port: 9090
      periodSeconds: 5
      timeoutSeconds: 3
      failureThreshold: 2   # stop routing traffic after ~10s of failures
      successThreshold: 2   # require 2 passes in a row before routing again
    resources:
      limits:
        cpu: 500m
        memory: 256Mi

A pod spec with an httpGet liveness probe and a grpc readiness probe that shows how periodSeconds, timeoutSeconds, failureThreshold and successThreshold work together, with the timeout set from p99 latency under load.

When to use each probe, and when to leave liveness out

Liveness: only for states the process cannot recover from

A liveness probe should answer one question: is this process stuck in a way that only a restart will fix? Good examples are a deadlocked worker pool, a wedged event loop, or a thread that has stopped processing its queue. The check should exercise that internal machinery and nothing outside the process.

Do not check the database, a cache, or a downstream API from liveness. When a shared dependency goes down, every replica fails the probe at the same moment. The kubelet then restarts all of them, and they come back into the same outage, now also paying startup cost. You have turned a dependency incident into a self-inflicted outage.

Readiness: can this instance serve right now

Readiness controls whether the pod receives traffic, so it can reflect transient conditions such as warming caches, saturated connection pools, or shedding load. Dependency checks belong here, if anywhere. They carry a cost: if every pod reports not-ready because the database is unreachable, the Service has zero endpoints and clients get connection errors instead of fast, meaningful 503s. Include a dependency only if a pod without it truly cannot serve anything.

Startup: protect slow boots without slowing detection

For slow-starting applications, add a startup probe sized so that failureThreshold × periodSeconds covers your worst observed startup time, with margin. This beats a large initialDelaySeconds, which delays checks on every start, including fast ones. Liveness and readiness probes do not run until the startup probe succeeds.

No liveness probe is often correct

If your process exits on fatal errors (panics, unhandled exceptions, failed invariants), the container's restart policy already handles recovery. A liveness probe adds value only for hangs that never crash. Without a known hang mode, leaving liveness out removes a source of false restarts.

Implementing probe endpoints and graceful shutdown

Give each probe its own path, such as /livez and /readyz, so they can answer different questions. Liveness should report only whether the process can make progress. Have it check that the event loop or a worker thread responds, and leave dependencies out of it. Readiness can also report whether this instance should receive traffic right now, for example whether a warm-up has finished or a connection pool is usable. Keep both handlers cheap and unauthenticated, because the kubelet sends no credentials, and an auth failure looks the same as an application failure.

Where the runtime allows, serve probes on a separate port or thread pool. If probes share the request pool, saturation delays probe responses. Liveness then times out and restarts a pod that was only busy, which adds load to the remaining replicas.

Shutdown follows a fixed order.

1. The pod is marked terminating. Two things start in parallel: the control plane begins removing the pod from Service endpoints, and the kubelet runs the preStop hook.
2. A short preStop sleep runs. A few seconds gives kube-proxy and load balancers time to stop routing to the pod before the process is told to exit.
3. The process receives SIGTERM. The handler sets a flag that makes /readyz fail, stops accepting new connections, and waits for in-flight requests to finish.
4. The process exits.

Set terminationGracePeriodSeconds to cover the preStop sleep plus your longest expected drain, with some margin. The grace period countdown includes the hook. If the drain overruns it, the kubelet sends SIGKILL and any remaining requests are dropped.

CopyapiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders-api
  template:
    metadata:
      labels:
        app: orders-api
    spec:
      # Budget: preStop sleep (5s) + longest expected drain (~30s) + margin
      terminationGracePeriodSeconds: 45
      containers:
        - name: app
          image: registry.example.com/orders-api:1.0.0
          ports:
            - name: http
              containerPort: 8080
            - name: probes        # separate listener so saturation on :8080 can't delay probes
              containerPort: 9090
          livenessProbe:          # /livez: process can make progress; no dependency checks
            httpGet: { path: /livez, port: probes }
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3
          readinessProbe:         # /readyz: warm-up done, pool usable, and not shutting down
            httpGet: { path: /readyz, port: probes }
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 1
          lifecycle:
            preStop:
              # Give endpoint removal time to propagate before SIGTERM (image must include sleep)
              exec: { command: ["sleep", "5"] }

A Deployment that serves /livez and /readyz on a dedicated probe port, uses a 5-second preStop sleep so endpoint removal can propagate, and sets terminationGracePeriodSeconds to cover both the sleep and the in-flight request drain.

Trade-offs and failure modes to avoid

Each probe setting trades one failure mode for another. Tuning one side usually shifts the risk to the other, so decide which failure you can tolerate.

Readiness has one more trap. If every pod fails readiness because a shared dependency is down, the Service has no endpoints and clients get connection errors instead of useful responses. For non-critical dependencies, keep the pod Ready and serve degraded responses, such as stale cache, a feature turned off, or an explicit 503 from the application. Fail readiness only when the pod cannot do useful work at all.

Debugging probe failures before loosening thresholds

When a probe starts failing, the first instinct is to raise timeoutSeconds or failureThreshold. That makes the symptom go away while the cause stays. Gather evidence first.

Read the events

kubectl describe pod <name> lists probe failures as Unhealthy events, and the message matters. connection refused means nothing was listening on the port: the process crashed, has not bound yet, or is bound to the wrong interface. context deadline exceeded or a client timeout means the endpoint was reachable but answered too slowly. An HTTP status code means the handler ran and reported unhealthy. Each of these leads to a different fix.

Check restarts and termination reason

The RESTARTS column in kubectl get pod tells you how often this is happening. kubectl get pod <name> -o yaml shows lastState.terminated under containerStatuses, with the reason, exitCode and timestamps. Exit code 137 with reason OOMKilled is a memory problem, not a probe problem. Exit code 137 without OOMKilled, timed shortly after Unhealthy events, usually points to the kubelet killing the container after liveness failures. Then run kubectl logs <name> --previous to see what the killed container was doing just before it died.

Correlate with metrics

Line up the failure timestamps against request latency, GC pause time and CPU throttling. A probe that times out during long GC pauses, or while the container is hitting its CPU limit, tells you the pod is starved. A longer timeout would only hide that. In that case, fix the resource limits, heap sizing or the work happening on the probe's code path. Loosen thresholds only once you have confirmed the endpoint is healthy and just slow for a known reason.

Copy#!/usr/bin/env bash
# Gather probe-failure evidence before touching timeoutSeconds or failureThreshold.
set -euo pipefail

NS="payments"          # placeholder namespace
POD="api-7d9f8c6b5-x2k4q" # placeholder pod name
CONTAINER="api"         # placeholder container name

# 1. Probe failures surface as Unhealthy events. Read the message:
#    'connection refused' = nothing listening; 'context deadline exceeded' = too slow;
#    'HTTP probe failed with statuscode: 503' = handler ran and reported unhealthy.
kubectl get events -n "$NS" \
  --field-selector involvedObject.name="$POD",reason=Unhealthy \
  --sort-by=.lastTimestamp

# Full picture: probe config, events and container state in one view.
kubectl describe pod "$POD" -n "$NS"

# 2. Restart count plus last termination reason, exit code and time.
#    137 + OOMKilled = memory problem. 137 without OOMKilled shortly after
#    Unhealthy events = kubelet killed the container after liveness failures.
kubectl get pod "$POD" -n "$NS" -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}restarts={.restartCount}{"\t"}reason={.lastState.terminated.reason}{"\t"}exit={.lastState.terminated.exitCode}{"\t"}finished={.lastState.terminated.finishedAt}{"\n"}{end}'

# 3. What the killed container was doing just before it died.
kubectl logs "$POD" -n "$NS" -c "$CONTAINER" --previous --tail=200

# 4. Current CPU/memory usage (requires metrics-server). Compare against limits:
kubectl top pod "$POD" -n "$NS" --containers
kubectl get pod "$POD" -n "$NS" -o jsonpath='{.spec.containers[*].resources}{"\n"}'

# Then line up the 'finished' and event timestamps with latency, GC pause and
# CPU throttling graphs (e.g. cAdvisor's container_cpu_cfs_throttled_periods_total)
# before deciding whether the fix is limits, heap sizing, or the probe handler.

A shell sequence that collects Unhealthy events, restart counts, the last termination reason and previous-container logs for a failing pod so you can find the root cause before loosening probe thresholds.

Starter configurations, alternatives, and impact on engineering time

Treat the values below as starting points to tune against your own startup and latency data. The key relationship is that a startup probe allows periodSeconds × failureThreshold before liveness takes over.

Use this checklist for each container:

• Does it receive traffic through a Service? If yes, add readiness.
• Can it deadlock while the process stays alive? If not, skip liveness.
• Is startup slower than your liveness budget? If yes, add a startup probe.

Probes have alternatives and complements:

• Restart policy alone. If the process reliably crashes on fatal errors, the kubelet restarts it without any liveness probe.
• In-process watchdogs. A thread that calls exit when the work loop stalls catches hangs that a shallow HTTP check misses.
• External synthetic monitoring. Probes run from the kubelet on the node. They cannot see network policy, ingress or DNS problems, so they tell you nothing about the health clients actually experience.

On cost, a well-scoped readiness and liveness pair is a small amount of handler code plus a few manifest lines. The return is fewer incidents caused by restart storms during dependency outages, and fewer rollouts stuck because new pods never become ready.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes probes, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.