What question does each Kubernetes probe answer?
Each probe answers exactly one question. Most probe-related outages happen when a probe is asked to answer a question that belongs to a different probe.
Liveness: is this process irrecoverably stuck? A deadlocked worker pool or a wedged event loop will not fix itself, and a restart is the only remedy. When the liveness probe fails failureThreshold times in a row, the kubelet kills the container and restarts it according to the pod's restartPolicy. Because a restart is the only action available, a failing liveness probe should only ever mean "restarting will help."
Readiness: should this pod receive traffic right now? The answer can change many times during a pod's life. A cache might be warming, a connection pool might be saturated, or the pod might be shutting down. On failure, the pod is marked not ready and removed from Service endpoints. Nothing is restarted. When the probe passes again, traffic returns. Readiness keeps running for the whole life of the container, not only at boot.
Startup: has initialization finished? While a startup probe is configured and has not yet succeeded, the kubelet does not run the liveness or readiness probes. This lets a slow-booting API take its time without being killed by an aggressive liveness probe. If the startup probe never succeeds within its budget (failureThreshold × periodSeconds), the container is killed and restarted under restartPolicy.
| Probe | Question it answers | Action on failure | Typical endpoint |
|---|---|---|---|
| Liveness | Is the process irrecoverably stuck? | Container restarted per restartPolicy |
/livez (in-process check only) |
| Readiness | Should it receive traffic now? | Removed from Service endpoints, no restart | /readyz |
| Startup | Has initialization finished? | Blocks the other probes, then restarts if its budget runs out | /livez or a dedicated /startupz |
How do Kubernetes probes work?
Probes are run by the kubelet on the node where the pod is scheduled, not by the API server or a central controller. Probes are configured per container, and the kubelet runs each one on its own schedule. For an httpGet or tcpSocket probe, the kubelet connects to the pod's IP address by default. For an exec probe, it runs the command inside the container. Each attempt produces a result of success or failure, and the kubelet counts consecutive results against failureThreshold and successThreshold.
What happens next depends on the probe type:
- Startup probe. While it has not yet succeeded, the kubelet holds off on the other two probes. Once it succeeds, it stops running and liveness and readiness begin. If it exhausts its budget, the kubelet kills the container.
- Liveness probe. When it crosses
failureThreshold, the kubelet kills the container and restarts it according torestartPolicy. Repeated restarts are subject to an increasing back-off delay, which shows up asCrashLoopBackOff. - Readiness probe. Its result sets the container's ready status. A pod is reported
Readyonly when all of its containers are ready. The EndpointSlice controller removes pods that are not ready from the Service's endpoints, and kube-proxy and any load balancer integration stop routing new traffic to them. When readiness passes again, the pod is added back.
Two consequences follow. First, readiness failures are handled through the control plane, so taking a pod out of rotation is not instant. Endpoint changes take time to reach every node and load balancer, which matters during shutdown (covered below). Second, readiness also drives rollouts: a Deployment counts a new pod as available only after it is ready, so a readiness probe that passes too early lets a rolling update replace working pods with ones that cannot yet serve.
How do probe mechanisms and timing fields work?
Each probe uses one of four mechanisms, and the choice determines what a "pass" actually proves.
| Mechanism | Passes when | Trade-off |
|---|---|---|
httpGet |
Response status is 200–399 | Exercises your HTTP stack and handler logic; the handler must stay cheap |
tcpSocket |
A TCP connection is established | Cheapest option, but only proves the port is open, not that requests succeed |
exec |
Command exits with code 0 | Spawns a process every period; on busy nodes this adds CPU overhead and makes timeouts more likely |
grpc |
Server reports SERVING via the gRPC health checking protocol |
Native fit for gRPC services; requires implementing the health service |
Timing fields
initialDelaySecondsis the wait before the first probe.periodSecondsis the interval between probes.timeoutSecondsis how long a single attempt may take before it counts as a failure.failureThresholdis the number of consecutive failures before Kubernetes acts.successThresholdis the number of consecutive successes needed to recover. It must be 1 for liveness and startup probes, because there is no "partially alive" state to recover from.
Time-to-action is roughly periodSeconds × failureThreshold. Each attempt that times out can add up to timeoutSeconds on top of that. With a period of 10 and a threshold of 3, expect around 30 seconds, plus a few seconds of timeout.
Illustrative estimate (assumed inputs): an API's worst observed boot time is 90 seconds. Give the startup probe periodSeconds: 5 and failureThreshold: 24, which is a 120-second budget. Once startup succeeds, liveness can stay tight. Don't inflate liveness initialDelaySeconds to cover boot time, because that delay applies to every restart and still guesses wrong when a boot runs slow.
The cost of a large startup budget is detection time: a pod that will never become healthy can take the full 120 seconds before Kubernetes restarts it.
containers:
- name: api
image: example.com/orders-api:1.0.0
ports:
- containerPort: 8080
# Startup budget = failureThreshold x periodSeconds = 24 x 5 = 120s,
# sized to exceed the assumed 90s worst observed boot time.
# Liveness and readiness do not run until this probe succeeds.
startupProbe:
httpGet:
path: /healthz
port: 8080
periodSeconds: 5
failureThreshold: 24
timeoutSeconds: 2
# Tight liveness once booted: restart after ~3 x 10s = 30s of
# consecutive failures, plus up to 2s per attempt that times out.
# No initialDelaySeconds needed; the startup probe covers boot.
livenessProbe:
httpGet:
path: /healthz
port: 8080
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
successThreshold: 1 # must be 1 for liveness and startup probes
# Readiness only removes the pod from Service endpoints; it never restarts it.
# successThreshold > 1 is allowed here to avoid flapping back into rotation.
readinessProbe:
httpGet:
path: /ready
port: 8080
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 3
successThreshold: 2
A startup probe with a 120-second budget (24 × 5s) covers an assumed 90-second worst-case boot, so liveness can stay tight without inflating initialDelaySeconds; the trade-off is that a pod that will never become healthy is not restarted until that 120-second budget is used up.
Who should use each probe?
Readiness probes fit almost any API that sits behind a Service and runs more than one replica. Without one, a pod receives traffic as soon as its containers are running, which is often before the listener is open or caches are loaded. That causes failed requests on every deploy and scale-up. Teams that rely on rolling updates benefit most, because rollouts wait for readiness before replacing old pods.
Startup probes are for services whose boot time is long or variable: runtimes with slow warm-up, services that load large datasets or models into memory, or processes that run checks against external systems before serving. If a service boots in a second or two, a startup probe adds little.
Liveness probes are for teams that know of a specific failure mode where the process stays running but stops making progress, such as a deadlock, a stuck event loop, or a worker pool that never drains. They also need to be able to detect that state from inside the process with an in-process check. If you can't name the hang you are guarding against, you probably can't write a liveness check that detects it without false positives.
When should probes be avoided?
Each probe has situations where it does more harm than good:
- Avoid liveness probes when the process already exits on fatal errors and has no known hang mode, when the only check you could write depends on an external system, or when the workload has long, legitimate pauses you can't tell apart from a hang. The next sections cover the restart-loop risk and the alternatives in detail.
- Avoid dependency checks in liveness probes in all cases. A shared dependency failure then becomes a fleet-wide restart.
- Avoid readiness checks on shared dependencies when every replica would fail together. Removing all endpoints turns a degraded service into one that refuses connections.
- Readiness probes do little for workloads that receive no Service traffic, such as batch Jobs or queue consumers that pull their own work. Readiness controls routing, and there is nothing to route.
- Avoid
execprobes on densely packed nodes or with short periods, since each attempt spawns a process. - Skip the startup probe when boot is fast and predictable. A plain readiness probe handles the brief warm-up.
Why do dependency checks in liveness probes cause restart loops?
The rule is simple: a liveness probe should fail only when restarting the container would fix the problem. A deadlocked event loop or a wedged worker pool qualifies. An unreachable database does not, because a fresh process can't reach it either. When liveness checks a shared dependency, a fault outside the pod turns into a restart of every pod at the same moment.
Here is the sequence, using illustrative timings that assume a 10-second probe period and a failure threshold of 3:
t=0s DB failover begins; connections drop
t=0-30s liveness fails 3x on EVERY replica (same DB, same probe)
t=30s kubelet kills all containers -> zero serving pods
t=35s DB is back, but now every pod boots at once:
connection pools, migrations check, cache warm-up
t=40s+ reconnection storm saturates DB -> new liveness failures
-> more restarts, growing CrashLoopBackOff delays
t=?? recovery depends on backoff timing, not on the DB
A 30-second blip becomes a multi-minute outage. Each restart also resets connection state and drops in-flight requests. CrashLoopBackOff then stretches the delay between restarts, so pods can stay down even after the database is healthy again.
With the same dependency check on readiness instead, every pod is removed from the Service endpoints for about 30 seconds. The processes keep running with their pools intact, and they reconnect gradually as the database recovers. Traffic resumes when the checks pass. No pod restarts, no thundering herd, and the outage is no longer than the original fault.
This failure is common because the bug sits dormant. The liveness check looks responsible in review, passes every day, and only fires during the dependency incident it ends up making worse.
+---------------------------------+
| Shared DB fails over, t=0..30s |
+----------------+----------------+
|
+-----------------+----------------+
v v
+------------------------------+ +------------------------------+
| LIVENESS probe checks DB | | READINESS probe checks DB |
+--------------+---------------+ +--------------+---------------+
| |
v v
+------------------------------+ +------------------------------+
+>| t=30s kubelet kills ALL pods | | t=0s all pods leave endpoints|
| +--------------+---------------+ +--------------+---------------+
| | |
| v v
| +------------------------------+ +------------------------------+
| | t=35s DB up, all pods boot | | processes + pools stay up |
| +--------------+---------------+ +--------------+---------------+
| | |
| v v
| +------------------------------+ +------------------------------+
+-| t=40s reconnect storm -> DB | | t=35s DB up, slow reconnect |
+--------------+---------------+ +--------------+---------------+
| |
v v
+------------------------------+ +------------------------------+
| CrashLoopBackOff, outage>>30s| | ready again, outage ~30s |
+------------------------------+ +------------------------------+
The same 30-second database failover (illustrative timings: 10s period, failure threshold 3) leads to a restart loop when the dependency check sits in the liveness probe, but only to a temporary removal from Service endpoints when it sits in the readiness probe.
How should you design liveness and readiness endpoints for an API?
The fix is to give each probe its own handler. Liveness should ask only whether the process can still make progress. Readiness should ask whether this replica should receive traffic right now.
Liveness: cheap, in-process, dependency-free
A liveness handler should never touch the database, cache or downstream services. A shallow check that returns 200 from the HTTP thread costs almost nothing. It also misses a common failure: the listener keeps answering while the worker pool is deadlocked. A deeper check closes that gap. For example, each worker updates a heartbeat timestamp, and the handler fails if any heartbeat is older than a threshold. That threshold needs care. A deep check can also fire during a long GC pause or a temporary backlog, both of which would clear on their own. A restart throws away that recovery and adds a cold start. Check only for conditions that cannot resolve without a restart.
Readiness: local state and critical dependencies
Readiness can check warm-up state, such as loaded config, primed caches and an open connection pool, and it can check dependencies the replica cannot work without. Be careful with that second part. If a shared database goes down, every replica fails readiness together and the Service has no endpoints left. Clients then get connection refusals instead of fast 503s or degraded responses they could have handled. For dependencies shared by all replicas, it is often better to stay ready and fail individual requests quickly.
import queue, threading, time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
STALE_AFTER_S = 60 # well above worst-case GC pause or backlog drain time
heartbeats, hb_lock = {}, threading.Lock() # worker id -> last progress time
warmed_up = threading.Event() # set after config load, cache priming, pool open
jobs = queue.Queue()
def worker(wid):
while True:
with hb_lock:
heartbeats[wid] = time.monotonic() # proves this worker loop still runs
try:
job = jobs.get(timeout=5) # timeout so an idle worker keeps beating
except queue.Empty:
continue
job() # a deadlock here stops the heartbeat
class Probes(BaseHTTPRequestHandler):
def do_GET(self):
if self.path == "/livez":
# In-process only: no DB, cache or downstream calls.
now = time.monotonic()
with hb_lock:
stuck = [w for w, t in heartbeats.items() if now - t > STALE_AFTER_S]
self.reply(500 if stuck else 200)
elif self.path == "/readyz":
# Local warm-up state only. A shared DB outage is deliberately not
# checked, so replicas stay in the Service and fail requests fast.
self.reply(200 if warmed_up.is_set() else 503)
else:
self.reply(404)
def reply(self, code):
self.send_response(code)
self.send_header("Content-Length", "0")
self.end_headers()
for i in range(4):
threading.Thread(target=worker, args=(i,), daemon=True).start()
warmed_up.set() # placeholder: set this only after real warm-up completes
ThreadingHTTPServer(("0.0.0.0", 8080), Probes).serve_forever()
A standard-library Python sketch with a separate /livez handler, which fails only when a worker heartbeat goes stale, and a /readyz handler that checks local warm-up state but not shared dependencies.
Handling graceful shutdown and probe tuning under resource pressure
Draining on shutdown
When a pod is deleted, the kubelet starts termination while endpoint removal propagates separately through the control plane, kube-proxy, and any external load balancer. These happen concurrently, so for a few seconds traffic can still arrive at a process that has already received SIGTERM. If the app exits immediately, those requests fail.
The usual fix is a short delay before shutdown begins. Use a preStop hook that sleeps for a few seconds, or have the app keep serving for a fixed interval after SIGTERM. During that window, fail readiness and stop accepting new long-lived work. Then close listeners and let in-flight requests finish. Set terminationGracePeriodSeconds to cover the preStop delay plus your slowest expected request, because the grace period includes time spent in preStop. If it is too short, the kubelet sends SIGKILL mid-request. The cost is that every rollout and scale-down takes at least that long per pod, so do not pad it arbitrarily.
Probes under resource pressure
A pod at its CPU limit gets throttled, and a probe handler waiting for CPU time can exceed timeoutSeconds, which defaults to one second. Long garbage collection pauses have the same effect. Set the timeout based on observed pause and latency tails, not the default. Serve probes on a separate port or thread pool so a saturated request queue cannot starve them. Keep handlers cheap: no allocation-heavy work, locks, or I/O.
For liveness, periodSeconds × failureThreshold sets how long a pod can be unresponsive before it restarts. A short window recovers hung pods quickly but restarts healthy pods that are only slow under load, which pushes extra traffic onto the remaining replicas. A longer window tolerates load spikes but leaves genuinely stuck pods in place for longer.
apiVersion: apps/v1
kind: Deployment
metadata:
name: orders-api
spec:
replicas: 3
selector:
matchLabels: {app: orders-api}
template:
metadata:
labels: {app: orders-api}
spec:
# Must cover preStop (10s) + slowest request (~20s) + margin, since preStop time counts against it
terminationGracePeriodSeconds: 40
containers:
- name: api
image: registry.example.com/orders-api:1.4.2
ports:
- {name: http, containerPort: 8080} # user traffic
- {name: health, containerPort: 8081} # probes on a separate listener so a full request queue can't starve them
lifecycle:
preStop:
exec:
# Keep serving while endpoint removal reaches kube-proxy and load balancers (image must include sleep)
command: ["sleep", "10"]
readinessProbe:
httpGet: {path: /ready, port: health} # app returns 503 here once shutdown starts
periodSeconds: 5
timeoutSeconds: 2 # sized from observed p99 latency and GC pauses, not the 1s default
failureThreshold: 2
livenessProbe:
httpGet: {path: /live, port: health} # cheap handler: no locks, I/O, or dependency checks
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 3 # ~30s unresponsive before restart: tolerates load spikes and throttling
resources:
requests: {cpu: 500m, memory: 512Mi}
limits: {cpu: "1", memory: 512Mi}
A Deployment that delays shutdown with a preStop sleep, sizes terminationGracePeriodSeconds to cover that delay plus in-flight requests, serves probes on a dedicated port, and sets probe timeouts and liveness thresholds deliberately rather than relying on defaults.
When should you skip a liveness probe, and what are the alternatives?
If tuning that window feels like guesswork, ask whether you need the probe at all. Without a liveness probe, the kubelet treats a running container as alive and restarts it only when its main process exits. For many APIs that is enough. If the runtime terminates on fatal errors such as out-of-memory or a panic, and you have never seen the process hang while still running, a liveness probe adds no detection the kubelet doesn't already provide. It does add a way for a healthy pod to be killed because of a slow garbage-collection pause or a saturated thread pool.
The alternatives work together:
- Crash on fatal errors. Don't catch and log unrecoverable states like a corrupted connection pool or a failed config reload. Exit non-zero and let the restart policy handle it.
- Readiness-only setups. Keep the readiness probe so traffic shifts away from a struggling pod, but never kill it automatically.
- Alert instead of restarting. Page on sustained readiness failures, rising p99 latency, or stalled request counts, so a human can decide what to do.
| Approach | Recovers from hangs | Risk of false restarts | Ongoing effort |
|---|---|---|---|
| Liveness + readiness | Automatically | Present, depends on tuning | Endpoint code and tuning |
| Readiness + alerting | Manually | None from probes | Alert rules and on-call |
| Exit-on-fatal only | Only if the process exits | None | Error-handling discipline |
The cost is real. A process that is truly hung stays in place until someone notices, so this choice is only as good as your alerting. It also lowers development cost: there are fewer endpoints to write and fewer thresholds to tune. In exchange, you need a clear picture of how your process actually fails.
How do you diagnose restart loops and set a baseline probe config?
Diagnose before you tune
Start with kubectl describe pod <name>. Three fields matter most:
- Restart Count: how often the container has been restarted.
- Last State: the termination reason and exit code.
OOMKilledpoints to memory limits, not probes. - Events:
Unhealthywarnings that name the failing probe and the error, such as a timeout or an HTTP status code.
A liveness failure followed by a kill message means the kubelet restarted the container on purpose.
Next, run kubectl logs <pod> --previous. It shows output from the container that was killed, not the fresh one. Check whether the process was still starting up or slow but working when it died. Compare that timeline to your probe settings. If startup takes 40 seconds and your startup budget is 30, the restart loop is in your config, not your code. Do the same with endpoint latency under load: a p99 near timeoutSeconds will fail probes during traffic spikes.
A baseline for stateless APIs
The accompanying manifest uses three probes, with every timing field set explicitly:
- Startup probe: hits a cheap endpoint with
failureThreshold×periodSecondsset well above your worst observed cold start. - Liveness probe: hits an in-process endpoint that checks no dependencies. It uses a conservative
failureThreshold, so one GC pause or CPU throttle doesn't trigger a restart. - Readiness probe: hits its own endpoint, with a shorter period so traffic shifts away quickly.
Explicit values make the intent clear in code review. They also stop silent drift when the cluster defaults differ from what you assumed.
This baseline saves time in two ways. It removes the most common self-inflicted outages: cascading restarts and pods killed while starting. And when something does break, the probe config is already a known quantity, so triage can rule it out quickly instead of having to reconstruct it.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to kubernetes liveness vs readiness probe, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.