What is the difference between a liveness, readiness and startup probe?
Each probe asks the kubelet a different question, and the kubelet takes a different action when the answer is "no." Those actions differ in how much damage they can do.
| Probe | Question it asks | Kubelet action on failure | When it runs |
|---|---|---|---|
| Liveness | Is this process stuck and unable to recover without a restart? | Kills the container. It restarts according to the pod's restartPolicy. |
After the startup probe succeeds, for the container's whole life |
| Readiness | Should this pod receive traffic right now? | Marks the pod not ready and removes it from Service endpoints. No restart. | After the startup probe succeeds, for the container's whole life |
| Startup | Has the application finished initializing? | Kills the container once failureThreshold is exceeded. |
From container start until its first success |
Readiness failure is reversible and cheap. The pod stops getting new requests and rejoins when the check passes again. Liveness and startup failures destroy in-memory state, drop in-flight requests, and restart the process. While a startup probe is configured and has not yet passed, liveness and readiness checks do not run at all.
Pointing all three probes at one shared /health endpoint ties a reversible signal to an irreversible one. Suppose the endpoint returns 503 because the API is temporarily overloaded or a dependency is slow. That should only take the pod out of rotation. A liveness probe on the same endpoint will also restart it. Each endpoint needs to answer only its own probe's question.
This affects two groups. Backend engineers own the handlers and decide what each endpoint checks. Platform engineers set cluster defaults and templates, and one shared-endpoint default can repeat this mistake across every service that uses it.
+----------------------------------------------------------------------+
| kubelet |
+----------------------------------------------------------------------+
| | |
| startup probe | liveness probe | readiness probe
| runs first | after startup | after startup
| until success | passes | passes
v v v
+--------------------+ +--------------------+ +--------------------+
| GET /startupz | | GET /livez | | GET /readyz |
| init finished? | | process stuck? | | take traffic now? |
+--------------------+ +--------------------+ +--------------------+
| | |
| fail past | fail past | fail
| failureThreshold | failureThreshold |
v v v
+--------------------+ +--------------------+ +--------------------+
| kill container | | kill container | | remove pod from |
| restart per | | restart per | | Service endpoints |
| restartPolicy | | restartPolicy | | no restart |
+--------------------+ +--------------------+ +--------------------+
IRREVERSIBLE: state lost, in-flight dropped REVERSIBLE: rejoins later
Anti-pattern: all three probes on one shared /health
/health returns 503 (overload) --+--> readiness: out of rotation (intended)
+--> liveness: restart (unintended)
The kubelet runs the startup probe first, then liveness and readiness against separate endpoints. Liveness and startup failures restart the container, while a readiness failure only removes the pod from Service endpoints, so a shared /health endpoint turns a temporary overload into a restart.
Why do liveness probes that check dependencies cause outages?
Consider a liveness endpoint that runs a quick query against the database. The database pauses for a failover or a long lock. Every replica's check times out at about the same moment because they all share that dependency. After the configured number of failures, the kubelet kills the containers. The restart did not cause the outage, but it made it worse. In-flight requests are dropped, connection pools are discarded, and every new container starts at once. They all open fresh connections and warm their caches against a database that is still recovering. That synchronized load can push the database back over, which fails liveness again and produces restart loops across the whole Deployment. Without liveness probes, the API would have degraded briefly and recovered on its own.
The rule follows from this: liveness should check only conditions that a restart can fix. Good examples are a deadlocked event loop, an internal worker pool that is exhausted and never drains, or a process that is wedged but has not exited. Restarting the container clears all of these. Restarting it does nothing for a database that is down.
Build the check to match:
- Cheap. It runs constantly on every replica.
- Served from inside the process. Use the same runtime that would hang, so a hang causes the check to fail.
- No external calls. Do not call databases, caches, queues or downstream services.
Sometimes the right choice is to have no liveness probe at all. This is reasonable for processes that reliably crash on fatal errors, because Kubernetes already restarts containers that exit. The trade-off is that a process that truly hangs without exiting stays up, holding its place in the Deployment, until an alert or a person catches it.
What should a readiness endpoint check for an API?
Readiness answers a narrower question: can this instance serve a request correctly right now? A failing readiness probe never restarts anything. It only removes the pod from the Service's endpoints until the probe passes again. That makes readiness the right place for conditions that are expected to change: warmup finished (caches loaded, connection pools filled, migrations checked), required dependencies reachable, and the process not shutting down.
Dependency checks cut both ways. If every replica checks the same database and that database fails, every replica goes unready at once. The Service then has no endpoints, and clients get connection errors or a proxy's generic error instead of your API's own 503 with a useful body and a Retry-After header. Decide this per dependency. A dependency that is unique to the instance, such as a local sidecar or a per-pod connection, should fail readiness. A dependency that is shared by all replicas usually should not, and the API should return degraded responses instead.
On SIGTERM, start failing readiness while you keep serving in-flight and new requests. Kubernetes already marks a terminating pod's endpoints as not ready. Failing the probe also covers anything that health-checks the pod directly. Removal still propagates asynchronously to kube-proxy, ingress controllers and external load balancers, so a short preStop sleep is often still needed, within terminationGracePeriodSeconds. The cost is slower termination and slower rollouts.
Finally, decide where the probe handler runs:
- Shared request pool: if the pool is saturated, the probe times out and the pod sheds load. That can cascade as traffic shifts onto the remaining replicas.
- Dedicated lightweight path: the probe stays responsive, but it can report ready while real requests are queuing.
const http = require('http');
let warmedUp = false;
let shuttingDown = false;
// Per-pod dependency (local sidecar): failing it should fail readiness.
// The shared database is deliberately NOT checked here; the API returns
// its own 503 + Retry-After when the DB is down.
function sidecarReachable() {
return new Promise((resolve) => {
const req = http.get({ host: '127.0.0.1', port: 15000, path: '/ready', timeout: 500 }, (res) => {
res.resume();
resolve(res.statusCode === 200);
});
req.on('timeout', () => req.destroy()); // destroy emits 'error'
req.on('error', () => resolve(false));
});
}
const api = http.createServer((req, res) => {
res.writeHead(200, { 'content-type': 'application/json' }).end('{"ok":true}');
});
api.listen(8080, async () => {
// load caches, fill connection pools, then report ready
warmedUp = true;
});
// Probes on a separate listener, outside the API's routing and middleware.
http.createServer(async (req, res) => {
if (req.url !== '/readyz') return res.writeHead(404).end();
const ready = warmedUp && !shuttingDown && (await sidecarReachable());
res.writeHead(ready ? 200 : 503).end();
}).listen(8081);
process.on('SIGTERM', () => {
shuttingDown = true; // fail readiness, but keep serving requests
// Give endpoint removal time to reach proxies and load balancers,
// staying well inside terminationGracePeriodSeconds.
setTimeout(() => api.close(() => process.exit(0)), 10000);
});
A Node.js readiness endpoint that requires warmup to finish and a per-pod sidecar to be reachable, skips the shared database check, and starts failing on SIGTERM while the server keeps serving during a drain delay.
How do httpGet, tcpSocket, exec and gRPC probes compare?
The mechanism you pick decides what a passing probe actually proves. Each one exercises a different amount of your application, so a passing result means different things.
| Mechanism | What success proves | Cost per check | Main blind spot |
|---|---|---|---|
tcpSocket |
The kubelet completed a TCP handshake on the port | Very low | A deadlocked app still passes if the listener accepts connections |
httpGet |
A request reached your HTTP handler and returned a status from 200 to 399 | Low; one request through your server stack | Proves only the code path the endpoint runs |
exec |
A command inside the container exited with status 0 | A new process spawned in the container every period | CPU and memory overhead, plus timeouts under load |
grpc |
The server answered the standard gRPC health checking service | Low; one RPC | Requires cluster support and a server that implements the health service |
A TCP check gives false confidence. The kernel completes the handshake from the socket's accept backlog, so a process whose worker threads are all blocked can keep passing. An HTTP check is stronger because a request has to be scheduled, routed and answered by your own code.
An exec probe runs as a child process in your container's cgroup. Under memory pressure it competes with the app it is checking. When the node is busy it can also time out and record a failure even though the app is healthy.
Guidance for typical APIs:
- HTTP APIs: use
httpGetfor all three probes. - gRPC services: use the native
grpcprobe where your cluster supports it, and register the standard health service on the server. - Fallbacks: use
tcpSocketonly for services with no request-level endpoint. Useexeconly when nothing else can express the check.
How do probe timing fields combine, and when should you use a startup probe?
Each probe runs on its own schedule, and five fields decide how quickly Kubernetes reacts:
| Field | What it controls | Default |
|---|---|---|
initialDelaySeconds |
Wait after container start before the first probe | 0 |
periodSeconds |
Interval between attempts | 10 |
timeoutSeconds |
How long one attempt may take before it counts as failed | 1 |
failureThreshold |
Consecutive failures before acting | 3 |
successThreshold |
Consecutive successes to recover (must be 1 for liveness and startup) | 1 |
Detection time is roughly failureThreshold × periodSeconds, plus up to one timeoutSeconds for the last attempt. Illustrative estimate: with periodSeconds=10, failureThreshold=3 and timeoutSeconds=2, a hung process is restarted after about 30 to 32 seconds.
Startup probe vs. a large initial delay
Slow-starting APIs, such as those doing JVM warmup, cache loading or boot-time migrations, need a long grace period. A large initialDelaySeconds on liveness provides it, but the delay is fixed. Pods that boot fast still wait, and a pod that hangs during boot is not detected any sooner. A startup probe blocks liveness and readiness until it succeeds, which allows a boot window of up to failureThreshold × periodSeconds. For example, 30 × 10 gives 300 seconds. The probe passes as soon as the app is up, and after that liveness can stay tight.
Trade-offs
Aggressive timings produce false positives. GC pauses, CPU throttling and saturated request threads all slow probe responses. The resulting restarts remove capacity from an already overloaded service, which makes the overload worse. An overly generous startup window has the opposite problem: pods that will never start take longer to be detected, and doomed rollouts take longer to fail. Size the window from your observed boot-time distribution, such as the slowest boots you actually see plus some headroom, rather than from a guess.
apiVersion: v1
kind: Pod
metadata:
name: orders-api
spec:
containers:
- name: api
image: registry.example.com/orders-api:1.4.2
ports:
- containerPort: 8080
# Boot window: up to 30 x 10s = 300s for JVM warmup, cache load, migrations.
# Size failureThreshold from observed boot times, not a guess.
# Liveness and readiness do not run until this succeeds.
startupProbe:
httpGet:
path: /healthz/live
port: 8080
periodSeconds: 10
failureThreshold: 30
timeoutSeconds: 2
# Tight after startup: a hung process is restarted in ~30-32s
# (3 x 10s, plus up to one 2s timeout). No initialDelaySeconds needed.
livenessProbe:
httpGet:
path: /healthz/live
port: 8080
periodSeconds: 10
failureThreshold: 3
timeoutSeconds: 2
successThreshold: 1 # must be 1 for liveness
# Readiness only removes the pod from Service endpoints; it never restarts it.
# Shorter period lets traffic shift away quickly under load.
readinessProbe:
httpGet:
path: /healthz/ready
port: 8080
periodSeconds: 5
failureThreshold: 3
successThreshold: 2 # require two passes before taking traffic again
timeoutSeconds: 2
A slow-starting API uses a startup probe with a 300-second maximum boot window, which lets the liveness probe stay tight once the app is up, while the readiness probe controls traffic without causing restarts.
How do probes affect rolling updates and restart loops?
A Deployment rollout only advances as new pods become ready. With the default rolling update strategy, the controller creates new pods within the maxSurge budget and removes old ones only while it stays within maxUnavailable. That makes readiness the rollout's throttle. If the check is too strict, for example one that requires a slow downstream dependency to answer, new pods never turn Ready. The rollout stalls, old pods keep serving, and progressDeadlineSeconds eventually marks the Deployment as failed without rolling it back. If the check is too lenient, such as returning 200 before caches warm or connection pools fill, old pods are removed while new ones still serve slow first requests. minReadySeconds adds a guard here: a pod must stay Ready for that long before the controller counts it as available. This catches instances that pass once and then fall over.
Liveness failures work differently. Once failureThreshold consecutive checks fail, the kubelet kills the container and restarts it according to the pod's restart policy. If the container keeps failing, the kubelet waits longer before each restart. The delay roughly doubles each time up to a cap of a few minutes, and the pod shows CrashLoopBackOff while it waits.
To diagnose a restart loop, start with kubectl describe pod. Its events tell you whether a liveness probe triggered the kill or the process exited by itself, and the container status shows the restart count, last state, and exit code. Then run kubectl logs --previous to read the output of the container that was killed, because the current container's logs begin after the restart.
Well-designed probes save engineering time at this step. When liveness checks only the process itself, a restart nearly always points to a real fault in the container. Engineers then spend less time chasing false alarms and get to the root cause sooner.
#!/usr/bin/env bash
# Diagnose a stalled rollout or restart loop for a Deployment named "orders-api".
# Placeholder names: adjust NS, DEPLOY, and the label selector to your setup.
set -euo pipefail
NS=prod
DEPLOY=orders-api
# 1. Is the rollout progressing? This times out if new pods never become Ready,
# which usually points to a readiness probe that is too strict.
kubectl -n "$NS" rollout status deployment/"$DEPLOY" --timeout=120s || true
# 2. List pods with restart counts. A rising RESTARTS column means liveness kills
# or crashing processes. STATUS shows CrashLoopBackOff while kubelet waits.
kubectl -n "$NS" get pods -l app="$DEPLOY"
# Pick one affected pod to inspect (here: the first match).
POD=$(kubectl -n "$NS" get pods -l app="$DEPLOY" -o jsonpath='{.items[0].metadata.name}')
# 3. Events and container state. Look for "Liveness probe failed" or
# "Readiness probe failed" events, plus Last State, Reason, and Exit Code.
# No probe events plus a non-zero exit code means the process died on its own.
kubectl -n "$NS" describe pod "$POD"
# Same facts in compact form: restart count and why the last container ended.
kubectl -n "$NS" get pod "$POD" -o jsonpath='{range .status.containerStatuses[*]}{.name}{" restarts="}{.restartCount}{" lastReason="}{.lastState.terminated.reason}{" exitCode="}{.lastState.terminated.exitCode}{"\n"}{end}'
# 4. Logs from the container that was killed. Without --previous you only see
# output written after the most recent restart.
kubectl -n "$NS" logs "$POD" --previous --tail=100
A shell workflow that checks rollout progress, then uses kubectl describe pod, the restart count, and kubectl logs --previous to tell whether a probe failure or the application itself caused a restart loop.
What is a good baseline probe configuration for an HTTP API?
Start with two handlers. /livez returns 200 whenever the process can accept and answer a request. It makes no database, cache or downstream calls. /readyz returns 200 only when this instance can do useful work: required connections are established, warm-up is finished, and shutdown has not begun. On SIGTERM, set readiness to failing before the server stops accepting connections. Endpoints controllers then remove the pod while in-flight requests drain.
For timing, the following values are an assumed starting point that you should tune, not a recommendation for every workload:
- Startup: probe
/livezevery 5 seconds, with afailureThresholdchosen so that period × threshold exceeds your slowest observed boot plus margin. If the worst boot you have measured is 60 seconds, a threshold of 18 allows 90 seconds. - Readiness: period 5s, timeout 2s, failure threshold 2. It reacts quickly because removing a pod from rotation is cheap and reversible.
- Liveness: period 10s, timeout 5s, failure threshold 3. It reacts slowly because a restart discards in-flight work and warm caches.
The development cost is small. You write two short handlers, add a shutdown hook, and measure boot times from pod events or logs across several deploys, including cold nodes and image pulls. Compare that with a shared /health endpoint. A brief dependency blip can fail liveness on every replica at the same moment and turn a partial degradation into a full outage.
Reviewing existing probes
- Does liveness touch anything outside the process?
- Is liveness at least as forgiving as readiness?
- Is the startup budget based on measured boot times?
- Does readiness fail before shutdown?
- Do the probe timeouts exceed the handler's p99 latency under load?
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to kubernetes liveness vs readiness probe, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.