What do startup, liveness and readiness probes each do when they fail?
Each probe answers a different question, and Kubernetes takes a different action when the answer is "no." The kubelet runs all three against each container. What matters is the consequence of failure, because that determines whether a slow boot looks like a normal startup or a crash loop.
| Probe | Question it answers | When it runs | Consequence after failureThreshold consecutive failures |
|---|---|---|---|
| Startup | Has the app finished starting? | From container start until it first succeeds | Container is killed, then handled by the Pod's restartPolicy |
| Liveness | Is the process stuck and in need of a restart? | After startup succeeds, for the container's lifetime | Container is killed, then handled by restartPolicy |
| Readiness | Should this Pod receive traffic right now? | After startup succeeds, for the container's lifetime | Pod is marked not Ready and removed from Service endpoints. No restart. |
The startup probe is a gate. While it is configured and has not yet succeeded, the kubelet does not run the liveness or readiness probes. A cache warm-up or a long schema migration therefore cannot trip a liveness restart before the app has had a chance to come up. Once the startup probe passes, it stops running and the other two take over.
Readiness is the only probe that does not restart anything. A failing readiness check takes the Pod out of rotation and leaves it running. This makes readiness the right signal for temporary conditions, such as an overloaded Pod or a dependency that is briefly unavailable.
If your service boots in seconds on one node and minutes on another, these differences matter to you. Misassigning a check to the wrong probe is a common cause of restart loops and of traffic reaching Pods that are not ready.
container starts
v
+-------------------------------+
| STARTUP: "Has it started?" |
| liveness/readiness not run |
+-------------------------------+
| |
| succeeds once | failureThreshold
| (then stops) | consecutive failures
v v
+------------------+ +----------------------+
| both run for the | | container killed, |
| container's life | | then restartPolicy |
+------------------+ +----------------------+
| | ^
v v |
+-------------+ +-------------+ |
| READINESS | | LIVENESS | fails |
| "Traffic?" | | "Restart?" |------------+
+-------------+ +-------------+
|
| fails
v
+-----------------------------+
| Pod not Ready: removed from |
| Service endpoints. |
| Container keeps running. |
+-----------------------------+
The startup probe holds back liveness and readiness until it first succeeds. A startup or liveness failure kills the container and hands it to restartPolicy, while a readiness failure only removes the Pod from Service endpoints.
How does probing work under the hood?
Probes are run by the kubelet on the node where the Pod is scheduled, not by the API server or a central controller. For each configured probe, the kubelet runs a check every periodSeconds and gives each attempt up to timeoutSeconds to complete. Each attempt counts as either a success or a failure:
httpGet: the kubelet sends an HTTP GET to the container's IP, port and path. Any status code from 200 through 399 is a success. Anything else, or no response before the timeout, is a failure.tcpSocket: success means the kubelet could open a TCP connection to the port. Nothing is sent over the connection.grpc: the kubelet calls the standard gRPC health-checking service, and aSERVINGresponse is a success.exec: the kubelet runs a command inside the container, and exit code 0 is a success.
The kubelet counts consecutive results. A probe flips to failed after failureThreshold consecutive failures and back to passing after successThreshold consecutive successes. For liveness and startup probes, successThreshold must be 1.
What happens next depends on the probe type. A failed startup or liveness probe makes the kubelet kill the container, and the Pod's restartPolicy decides whether it restarts. The container restarts in place inside the same Pod, on the same node. The Pod is not rescheduled. Repeated restarts are spaced out with an increasing back-off, which is what shows up as CrashLoopBackOff. A failed readiness probe sets the Pod's Ready condition to false. The endpoints machinery then removes the Pod's address from the Services that select it, and kube-proxy or the equivalent dataplane on each node stops routing new connections to it.
Because the kubelet runs probes from the node, an HTTP or TCP probe that passes proves only that the container answers on its own network address. It says nothing about whether clients elsewhere in the cluster can reach it.
Who should use a startup probe?
A startup probe is for teams whose containers take long, or unpredictably long, to become healthy, and who also want a liveness probe. Typical cases:
- JVM and other runtime-heavy services that spend significant time on class loading, dependency injection wiring and JIT warm-up before they answer requests.
- Services that load large state at boot, such as in-memory caches, search indexes or machine learning models.
- Applications that run schema checks or migrations in-process before they start serving.
- Legacy applications moved onto Kubernetes whose boot time was never optimized because they used to restart rarely.
- Platform teams that maintain shared Deployment templates or Helm charts for many services. A startup probe in the template lets each team set one boot ceiling without weakening the liveness settings that everyone shares.
Teams whose containers boot in a few seconds, every time, gain little from a startup probe. A small initialDelaySeconds on liveness, or no liveness probe at all, is usually enough for them.
When should you add a startup probe?
Add a startup probe when both of these are true:
- The container has a liveness probe, or you want one.
- Boot time can exceed what that liveness probe tolerates, which is roughly
initialDelaySeconds + failureThreshold × periodSeconds.
Common signals that you have reached that point:
- Pods enter
CrashLoopBackOffduring deploys or node failures,kubectl describe podshows liveness failures during boot, and the app logs show normal startup progress up to the kill. - Someone has raised liveness
initialDelaySecondsmore than once to stop restarts. - Boot time differs a lot between a warm node with a cached image and a busy node pulling the image fresh.
- You want a tight liveness probe for fast deadlock detection, but that tight setting is incompatible with the boot time.
You do not need a startup probe when the container has no liveness probe. Readiness already keeps traffic away from a booting Pod, and nothing else restarts it. You also do not need one when boot is fast and consistent enough that a short liveness delay covers it with room to spare.
Why is a long liveness initialDelaySeconds the wrong fix for slow startup?
Before startup probes existed, the standard workaround was to set initialDelaySeconds on the liveness probe high enough that the kubelet would not start checking until the app had finished booting. That approach has a structural flaw: it applies one fixed delay to a quantity that varies.
Boot time depends on cache warm-up, schema migrations, JIT compilation, image pull state, and how busy the node is. A single number has to cover that whole range, and it fails in both directions:
- Too short: a slow boot hits the first liveness check before it can answer. After
failureThresholdmisses, the kubelet kills the container. The container restarts, boots slowly again, and is killed again. This is aCrashLoopBackOffcaused by the probe configuration, not by a bug in the app. - Too long: every boot waits out the worst case, including restarts after a genuine deadlock. During that window, a container that hangs on startup is not detected at all.
There is also a maintenance cost. Each new dependency or larger dataset changes the right value, and nobody notices until pods start crash-looping in production.
A startup probe separates the two concerns. Liveness and readiness checks do not run until the startup probe succeeds. The startup probe's failureThreshold × periodSeconds sets a ceiling on boot time, not a fixed wait. As an illustrative estimate, with a 10-second period and a threshold of 30, a pod gets up to 300 seconds to boot. If it is ready in 20 seconds, liveness takes over at roughly 20 seconds.
What are the alternatives to a startup probe?
A startup probe is not the only way to handle slow boots. The other options solve different parts of the problem and are often combined with it.
- Long liveness
initialDelaySeconds. This is the fixed-delay approach described above. It works when boot time is stable, and it costs detection speed for the rest of the container's life. - No liveness probe. If the process exits reliably on fatal errors and cannot hang while staying alive,
restartPolicyhandles failures and there is nothing to protect during boot. Readiness alone keeps traffic off a booting Pod. - Init containers. Work that must finish before the app starts, such as waiting for a dependency or running a one-off setup step, can run in an init container. App containers do not start, and their probes do not run, until every init container has completed successfully. This takes that work out of the boot window entirely. It does not help with work the main process itself must do, such as filling its own in-memory cache.
- Running migrations as a separate Job. Moving schema migrations out of the app's boot path into a Job, usually run as a deploy step, shortens and stabilizes boot. It also stops several replicas from racing to migrate at the same time.
- Readiness gates. A Pod's
readinessGatesadd custom conditions that an external controller must set before the Pod counts as Ready. They control traffic. They do not stop liveness restarts, so they complement a startup probe rather than replace it. - Making boot faster. Smaller images, images pre-pulled onto nodes, lazy loading of rarely used data, and enough CPU during startup all shrink the boot time distribution. Faster boots help with rollouts and scale-out as well, which a startup probe does not.
A practical rule: use init containers or Jobs for work that can run separately, make the remaining boot as fast as is reasonable, and use a startup probe to cover whatever variable boot time is left.
How does a Kubernetes startup probe set the maximum boot window?
The kubelet runs the startup probe every periodSeconds and kills the container once it records failureThreshold consecutive failures. That makes the maximum boot window roughly initialDelaySeconds + failureThreshold × periodSeconds. Because the final attempt can also take up to timeoutSeconds before it counts as a failure, treat the result as approximate. Liveness and readiness checks are held off until the startup probe succeeds once, so this window is the only clock running during boot.
Sizing it from observed boot times
Illustrative estimate. Assume a service boots in 40 seconds on a warm node with a cached image and in up to 150 seconds when it runs schema checks, warms caches and competes for CPU. With periodSeconds: 10 and failureThreshold: 30, the window is about 300 seconds. That is double the assumed worst case, which leaves headroom for bad days. A boot that genuinely hangs, for example on a deadlocked connection pool or an unreachable dependency with no timeout, is still killed and restarted within five minutes rather than sitting forever as a pod that never becomes ready.
The period also sets detection resolution. A 10-second period means a successful boot is noticed within about 10 seconds. A shorter period with a proportionally higher threshold gives the same window and faster handoff to liveness, at the cost of more probe traffic.
The trade-off
A generous window delays hang detection. A stuck pod occupies a rollout slot for the full window, which slows the rollout. A tight window is riskier. CPU limits cause throttling during the CPU-heavy boot phase, and node contention can stretch a 40-second boot well past its usual time. A window sized to typical boot times then kills pods that were making progress. Each restart repeats the same expensive work, and the pod can enter CrashLoopBackOff even though nothing is broken. Size the window to the slowest boot you have actually observed under constrained conditions, not to the average.
apiVersion: v1
kind: Pod
metadata:
name: slow-boot-api
spec:
containers:
- name: api
image: registry.example.com/slow-boot-api:1.0.0
ports:
- containerPort: 8080
resources:
requests:
cpu: "500m"
limits:
cpu: "1" # throttling here can stretch boot time
startupProbe:
httpGet:
path: /healthz/started
port: 8080
initialDelaySeconds: 0 # adds directly to the window if set
periodSeconds: 10 # also the detection resolution for a finished boot
timeoutSeconds: 3 # last attempt can add up to this much
failureThreshold: 30 # 30 x 10s = ~300s max boot window (illustrative)
livenessProbe: # held off until startupProbe succeeds once
httpGet:
path: /healthz/live
port: 8080
periodSeconds: 10
failureThreshold: 3
readinessProbe: # also held off until startupProbe succeeds
httpGet:
path: /healthz/ready
port: 8080
periodSeconds: 5
failureThreshold: 3
A pod spec whose startup probe allows roughly 300 seconds (failureThreshold 30 × periodSeconds 10) for a boot assumed to take 40s to 150s, after which tighter liveness and readiness probes take over.
What should liveness and readiness probes actually check?
Once the boot window is handled, each remaining probe needs one clear question to answer. If a probe answers the wrong question, no choice of thresholds will fix it.
Liveness asks: can this process still make progress? It should fail only when a restart would actually help. Examples are a deadlocked event loop, a worker pool where every thread is stuck, or a main loop that has stopped ticking. A liveness probe must not check the database, the cache cluster or a downstream API. Those dependencies are shared by every replica, so a single database outage would fail liveness on all of them together. The kubelet would then restart the whole fleet in the middle of the incident, and every pod would pay the slow-boot cost again just as the dependency recovers.
Readiness asks: should this instance receive traffic right now? A failing readiness probe removes the pod from Service endpoints without restarting it, so it is safe for readiness to flap. Use it to report caches that are still warming, a model that is still loading, or deliberate load shedding when local queues are saturated.
| Liveness | Readiness | |
|---|---|---|
| Result of failure | Container restart | Removed from endpoints |
| Checks | Local progress only | Local capacity and warm state |
| Downstream dependencies | Never | Only with a deliberate policy |
| Flapping | Harmful | Acceptable |
Readiness can depend on downstream services, but that requires an explicit decision. Fail closed only when this instance has a problem that other replicas do not, such as a broken connection pool. When every replica would fail together, marking all of them unready leaves the Service with no endpoints. In that case it is usually better to keep serving in a degraded mode and return fast, explicit errors.
You can omit liveness entirely when the process already exits on fatal errors and has no realistic way to hang while staying alive. In that situation the container's restart policy already does the job, and adding a probe only introduces another way to get false restarts.
How do you implement probe endpoints and handlers?
The manifest that accompanies this section runs one container with three HTTP probes on the same port, each on its own path: /startup with a generous failureThreshold, /live with a short period, and /ready with a lower failure threshold so traffic is withdrawn quickly. Separate paths let each handler answer a different question and let you change one without touching the others.
HTTP is the default choice, but other probe types are a better fit in some cases. A TCP probe suits services with no HTTP surface, such as a database proxy. A gRPC probe suits services that already implement the standard gRPC health-checking protocol. Use exec only when the check needs local state that the process doesn't expose. Each exec probe forks a process inside the container every period, so on a dense node with short periods, those forks add up to measurable CPU and latency.
Handlers must be cheap and isolated. If /live waits on the same lock or worker pool as request traffic, a load spike makes the probe time out. The kubelet then restarts a pod that was only busy, which takes away capacity at the moment you need it most. Serve probes from a dedicated listener or thread, and have them read precomputed state rather than doing work.
Set timeoutSeconds deliberately. The default is one second, and a probe that exceeds it counts as a failure, exactly as an error response would. Measure the handler's tail latency under load and set the timeout above it.
Sharing one endpoint for startup and liveness means fewer code paths to test and document. The cost is that you can't later make startup check one-time initialization while liveness checks for deadlock without splitting the endpoint and redeploying. Separate paths add a few lines now and spare you that migration.
apiVersion: apps/v1
kind: Deployment
metadata:
name: slow-api
spec:
replicas: 3
selector:
matchLabels:
app: slow-api
template:
metadata:
labels:
app: slow-api
spec:
containers:
- name: app
image: registry.example.com/slow-api:1.0.0
ports:
- name: http
containerPort: 8080
# Tolerates up to 30 x 10s = 300s of initialization; liveness and readiness wait until it passes
startupProbe:
httpGet: { path: /startup, port: http }
periodSeconds: 10
failureThreshold: 30
timeoutSeconds: 2 # set above measured handler tail latency; a timeout counts as a failure
# Short period to catch deadlock; restarts after 3 consecutive failures
livenessProbe:
httpGet: { path: /live, port: http }
periodSeconds: 5
failureThreshold: 3
timeoutSeconds: 2
# Lower threshold so the pod leaves Service endpoints quickly; failing never restarts it
readinessProbe:
httpGet: { path: /ready, port: http }
periodSeconds: 5
failureThreshold: 1
successThreshold: 1
timeoutSeconds: 2
A Deployment with separate /startup, /live and /ready HTTP probes on one named port, each with its own period, failure threshold and an explicit timeoutSeconds.
How do readiness probes affect rollouts, shutdown and scale-out?
A Deployment's rolling update advances only as new Pods become Ready, so readiness is the gate. If the readiness endpoint returns 200 as soon as the HTTP server binds, before caches load, connection pools fill, or the JIT warms up, the rollout terminates old Pods while the new ones still serve at a fraction of their normal speed. Latency spikes during every deploy, and the probe reports nothing wrong. Readiness should pass only when the Pod can handle its share of real traffic.
Termination has a similar gap. When a Pod is deleted, the kubelet starts shutdown while the endpoint controller and every node's proxy remove the Pod from Service endpoints. These run in parallel, not in sequence. The process can receive SIGTERM while some nodes are still routing new connections to it. Failing readiness on shutdown doesn't close this gap. You need either a short preStop sleep before SIGTERM arrives, or application logic that keeps accepting requests briefly and then drains. Both must fit inside terminationGracePeriodSeconds, which includes the preStop time.
A startup probe makes boot safe. It does not make boot fast, and it adds no capacity. For slow-booting apps, use these alongside it:
maxSurgecreates new Pods before old ones are removed, so serving capacity doesn't drop while replacements boot.minReadySecondsrequires a Pod to stay Ready for a set time before it counts as available, which catches Pods that pass readiness and then fail.- Pre-warming means loading data and exercising hot code paths before reporting ready, plus keeping spare replicas so traffic spikes don't depend on how fast a new Pod boots.
apiVersion: apps/v1
kind: Deployment
metadata:
name: orders-api
spec:
replicas: 4
minReadySeconds: 20 # must stay Ready 20s before counting as available
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1 # boot a replacement before removing an old Pod
maxUnavailable: 0 # never drop below 4 serving Pods
selector:
matchLabels: {app: orders-api}
template:
metadata:
labels: {app: orders-api}
spec:
terminationGracePeriodSeconds: 45 # includes the preStop sleep
containers:
- name: app
image: registry.example.com/orders-api:1.4.2
ports: [{containerPort: 8080}]
startupProbe: # protects slow boot from liveness restarts
httpGet: {path: /healthz, port: 8080}
periodSeconds: 5
failureThreshold: 36 # up to 180s to boot
readinessProbe: # /ready returns 200 only after caches and pools are warm
httpGet: {path: /ready, port: 8080}
periodSeconds: 5
livenessProbe:
httpGet: {path: /healthz, port: 8080}
periodSeconds: 10
lifecycle:
preStop: # keep serving while endpoints are removed cluster-wide
exec:
command: ["sleep", "10"]
A Deployment that gates rollouts on a warm-up-aware readiness probe, keeps capacity with maxSurge and minReadySeconds, and delays SIGTERM with a preStop sleep.
How do you tune and debug probes without guessing?
Measure the boot window before setting it
Probe values should come from data. In staging, under realistic load and with production-sized data, restart the workload repeatedly and record how long each container takes from start until its readiness endpoint first succeeds. Application logs with timestamps work, and so do Pod events. Set the startup probe's failureThreshold × periodSeconds comfortably above the slowest boot you observed, not the average. Illustrative estimate: if the observed maximum is 90 seconds, a 10-second period with a threshold of 15 gives a 150-second window, which leaves room for a slow node without letting a hung boot run forever.
For liveness, make failureThreshold × periodSeconds longer than your worst expected GC pause or CPU-throttling stall. Otherwise a healthy but briefly stalled process gets killed, and the restart adds more load to the remaining replicas. Measure again whenever resource limits, dependencies or data volume change, because each of these shifts the boot distribution.
Debug from evidence
kubectl describe pod: the events list probe failures by type, with the HTTP status or error, and shows when the kubelet decided to restart the container.kubectl get pod: the restart count shows whether a loop is happening. The container's last state shows the termination reason and exit code.kubectl logs --previous: shows the output of the container instance that was killed, which is usually where the cause appears.
To separate the two common causes, check the termination reason and the events together. An OOMKill shows the reason OOMKilled and has no preceding probe failures. A probe-induced restart has liveness failure events just before it.
Checking these signals in order gives you the cause of a restart loop in minutes. Without them, engineers tend to adjust thresholds by trial and error and redeploy to see what happens.
How can startup probes save engineering time?
Most of the time saved comes from removing a class of incident and a class of recurring maintenance, not from the probe running faster.
- Fewer false crash loops to investigate. A liveness probe that kills slow but healthy boots produces
CrashLoopBackOffalerts that look like application bugs. Someone then has to read logs, rule out OOMKills and code changes, and only then find the probe timing. With a startup probe sized from measured boot times, those alerts largely go away. The restarts that remain are more likely to point at real failures. - No more repeated retuning of
initialDelaySeconds. With a fixed delay, every new dependency, data growth or change to resource limits can mean another round of raising the value, redeploying and waiting to see if it holds. A startup probe's window is a ceiling, so modest growth in boot time fits inside it without a change. The value needs revisiting only when boot time approaches the ceiling. - Liveness settings stay simple and shared. Because boot is handled separately, liveness can use the same short period and threshold across services. Platform teams can keep one template instead of a set of per-service exceptions.
- Faster recovery during incidents. Liveness that checks only local progress, and a boot window that is not too tight, prevent a dependency outage from becoming a fleet-wide restart storm. That storm is often what turns a short incident into a long one.
Illustrative estimate, with assumed inputs only: suppose a team runs 20 slow-booting services. Assume each one has a probe-induced crash-loop investigation once a quarter, and each investigation takes an engineer 2 hours. That is 40 engineer-hours per quarter. Assume also that adding and sizing a startup probe takes about 1 hour per service, including measuring boot times in staging. That is a one-time cost of 20 hours. Under these assumptions the change pays for itself within the first quarter. Replace the assumed frequency and duration with your own incident history before relying on the result.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to kubernetes startup probe, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.