Kubernetes Startup Probe, Liveness and Readiness for Slow-Booting APIs

What startup, liveness and readiness probes each do

Kubernetes gives the kubelet three separate questions to ask a container. Each failure has a different consequence, so each probe needs a different check.

While a startup probe is configured and has not yet passed, liveness and readiness are not run. That gating is what makes it useful for slow-booting APIs.

Copy container start
       |
       v
 [startup probe] --fails N times--> kill --> restart (per restartPolicy)
       | first success
       v
 +-----+------------------+
 |                        |
 v                        v
[liveness]           [readiness]
 | fail N times           | fail N times
 v                        v
 kill + restart      pod NotReady -> removed from
                     Service endpoints (still running)
                          | succeeds again
                          v
                     added back to endpoints

This guide is for backend and platform engineers who run APIs with long or unpredictable boot times, such as large cache warm-ups, migrations, JIT compilation or slow dependency handshakes.

The thesis is that the startup probe should absorb boot-time uncertainty. Liveness should stay strict and cheap. Readiness should report only whether the pod should receive traffic now. Most probe incidents come from one probe doing another's job. A common example is a liveness check that calls the database. When the database has an outage, that check restarts every replica at once.

Copy+-----------------+
| container start |<--------------------------------------------+
+--------+--------+                                             |
         |                                                      |
         v                                                      |
+-----------------+   fails failureThreshold times   +----------+--------+
|  startup probe  |--------------------------------->| kubelet kills     |
| liveness and    |                                  | container, then   |
| readiness wait  |                                  | restartPolicy     |
+--------+--------+                                  +-------------------+
         | first success                                        ^
         |                                                      |
         +--------------------------+                           |
         |                          |                           |
         v                          v                           |
+-----------------+        +-----------------+                  |
| readiness probe |        | liveness probe  |--fails N times---+
+--------+--------+        +-----------------+
         | fails N times
         v
+-----------------------------+
| pod NotReady: removed from  |
| Service endpoints; still    |
| running, not restarted      |
+--------------+--------------+
               | passes again
               v
+-----------------------------+
| added back to endpoints     |
+-----------------------------+

Liveness and readiness wait until the startup probe first succeeds. Startup or liveness failures lead the kubelet to kill the container, which is then handled by restartPolicy. Readiness failures only remove the running pod from Service endpoints until it passes again.

How it works: what the kubelet actually does

Probes are not run by the control plane. The kubelet on the node hosting the pod executes each probe on its own timer. For httpGet and tcpSocket probes, it connects to the pod's IP from the node. For exec, it runs the command inside the container. For grpc, it calls the standard gRPC health service. Each attempt produces a pass or a fail, and the kubelet keeps a count of consecutive results per probe.

The three probes act on different parts of the system:

• Startup probe. The kubelet runs it from container start. On the first success, it marks the container as started, stops running the startup probe for that container instance, and begins running liveness and readiness. If it reaches failureThreshold consecutive failures first, the kubelet kills the container. The startup probe state belongs to a container instance, so after any restart the new instance goes through the startup phase again, with the full budget.
• Liveness probe. When it reaches failureThreshold consecutive failures, the kubelet kills the container and the pod's restartPolicy decides whether it comes back. The pod object stays the same. Only the container is replaced, and its restart count goes up.
• Readiness probe. Its result sets the container's ready status, which feeds the pod's Ready condition. The control plane watches that condition and updates the Service's EndpointSlices. kube-proxy, ingress controllers and load balancers then add or remove the pod from rotation. Nothing is restarted.

This split explains the advice in the rest of this article. Startup and liveness failures cost a restart and a fresh boot. Readiness failures cost only traffic, and that traffic returns as soon as the check passes again.

Who should use a startup probe

Use a startup probe if your containers take long enough to boot, or vary enough in boot time, that a liveness probe with tight settings would kill them before they finish. Typical cases:

• JVM and other JIT-compiled services where class loading and warm-up take tens of seconds, and longer under a low CPU limit.
• APIs that load large data at boot, such as in-memory caches, lookup tables, search indexes or ML models.
• Services with slow dependency handshakes at startup, such as fetching secrets, establishing connection pools or registering with a discovery system.
• Services whose boot time depends on the node, where image pull, cold disk caches or CPU contention make some boots several times slower than others.

Platform teams that maintain shared Helm charts or base manifests should also care. A default template that pairs a startup probe with strict liveness settings gives every team a safe starting point, instead of each team discovering the initialDelaySeconds problem on its own.

If your service boots in a second or two and the time hardly varies, a startup probe adds little. Liveness with a short default delay already covers it.

Why a startup probe beats a longer initialDelaySeconds

The older fix for a slow boot is to raise initialDelaySeconds on the liveness probe until restarts stop. That setting applies only once, at container start, so it doesn't weaken steady-state checks directly. The problem is that it's a single fixed guess. Set it below your slowest boot and pods get killed mid-startup. Set it above your typical boot and every pod that hangs during startup sits there for the full delay before liveness even begins counting failures. Teams tend to keep raising it after each incident, so it only grows.

A startup probe makes boot a separate phase with its own budget. While it runs, Kubernetes holds off liveness and readiness checks. Once it succeeds, it stops, and liveness runs with its normal, tight settings. The ceiling is failureThreshold × periodSeconds, and the probe passes on its first success, so a fast boot isn't held to the worst case.

Illustrative estimate. Assume you measured a p99 boot time of 90 seconds and want roughly 60 seconds of headroom. With periodSeconds: 10 and failureThreshold: 15, the ceiling is 150 seconds. A pod that boots in 25 seconds registers as started on the third check, about 30 seconds in. The short period keeps detection fast, and the high threshold provides the headroom.

The trade-off is that a pod that deadlocks during boot now takes the full 150 seconds to be killed. Size the ceiling from observed data instead of padding it heavily. Boot time also shifts with node size, image pull time, cold caches and CPU limits. A lower CPU limit can stretch JVM or interpreter warm-up significantly, so revisit the budget whenever you change requests or limits.

CopyapiVersion: v1
kind: Pod
metadata:
  name: slow-api
spec:
  containers:
    - name: api
      image: registry.example.com/slow-api:1.0.0
      ports:
        - containerPort: 8080
      resources:
        requests:
          cpu: "500m"
          memory: 512Mi
        limits:
          cpu: "1"     # lowering this can stretch boot; re-measure and resize the budget
          memory: 1Gi
      # Boot phase: liveness and readiness are held off until this succeeds.
      # Ceiling = failureThreshold x periodSeconds = 15 x 10 = 150s
      # (illustrative: assumed p99 boot of 90s plus ~60s headroom).
      startupProbe:
        httpGet:
          path: /healthz
          port: 8080
        periodSeconds: 10      # short period: a fast boot registers as started quickly
        failureThreshold: 15   # high threshold: the headroom lives here
      # Steady state: tight settings, no initialDelaySeconds guess needed.
      livenessProbe:
        httpGet:
          path: /healthz
          port: 8080
        periodSeconds: 10
        failureThreshold: 3    # roughly 30s to detect a hung process after startup
      readinessProbe:
        httpGet:
          path: /ready
          port: 8080
        periodSeconds: 5
        failureThreshold: 2

A pod spec where a startup probe gives boot a 150-second ceiling (an illustrative 15 × 10s budget) while liveness and readiness keep tight steady-state settings with no initialDelaySeconds.

What each health endpoint should check

Each probe answers a different question, so each endpoint should check only what its answer depends on.

A liveness handler should do almost nothing. It returns 200 if the request reached a worker in time. For a worker pool, you can also have each worker update a heartbeat timestamp and return 503 when the timestamp goes stale. The startup handler reads a flag that the boot sequence sets as its last step.

Keep the database out of liveness. If liveness depends on a shared database, a database outage fails the check on every replica, and the kubelet restarts all of them. When the database comes back, every pod reconnects and warms up at the same moment. That thundering herd can push the database straight back over.

Both choices have costs. A shallow liveness check misses a process that still answers /healthz but cannot do useful work. Catch that case with alerts on error rate and latency, not with restarts. A deep readiness check can mark every replica unready at the same time, which leaves the Service with zero endpoints. In that situation, serving degraded responses is often better than serving nothing.

CopyapiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  replicas: 3
  selector:
    matchLabels: {app: orders-api}
  template:
    metadata:
      labels: {app: orders-api}
    spec:
      containers:
        - name: api
          image: registry.example.com/orders-api:1.0.0
          ports:
            - containerPort: 8080
          # /startupz returns 200 once the boot sequence sets its final flag.
          # Allows up to 30 x 10s = 300s to boot. Liveness and readiness wait until it passes.
          startupProbe:
            httpGet: {path: /startupz, port: 8080}
            periodSeconds: 10
            failureThreshold: 30
          # /healthz is in-process only: 200 if a worker answered and the heartbeat is fresh.
          # It never touches the database, so a DB outage cannot restart every replica.
          livenessProbe:
            httpGet: {path: /healthz, port: 8080}
            periodSeconds: 10
            timeoutSeconds: 2
            failureThreshold: 3
          # /readyz checks local state: config loaded, cache warm, local pool initialised.
          # Leave shared dependencies out. Otherwise all pods go unready together
          # and the Service is left with zero endpoints.
          readinessProbe:
            httpGet: {path: /readyz, port: 8080}
            periodSeconds: 5
            timeoutSeconds: 2
            failureThreshold: 3

A Deployment that wires a boot-flag startup probe, an in-process liveness probe and a local-state readiness probe to three separate endpoints, with comments stating what each handler must and must not check.

Probe tuning: timing parameters and probe mechanisms

The kubelet runs a probe every periodSeconds, waits up to timeoutSeconds for a response, and acts only after failureThreshold consecutive failures. That makes worst-case detection time roughly periodSeconds × failureThreshold, plus up to one timeout. successThreshold sets how many consecutive passes are needed to flip back to healthy. It must be 1 for liveness and startup probes, so it is only tunable for readiness, where requiring 2 or 3 passes stops a flapping pod from rejoining the Service too early.

Keep timeoutSeconds above the default of 1 second for most runtimes. A stop-the-world GC pause, or CFS throttling when the container hits its CPU limit, can stall a perfectly healthy handler past one second. The kubelet counts that as a failure. If several of these stalls land in a row, liveness restarts a pod that was only busy, and the restart adds more load to the remaining replicas.

Aggressive settings, such as a short period and a low threshold, find real failures sooner but multiply probe traffic and false restarts. Lenient settings absorb transient stalls at the cost of slower recovery from real hangs. Choose based on what a missed detection costs you compared with what a spurious restart costs you.

Copycontainers:
  - name: api
    image: registry.example.com/orders-api:1.4.2
    ports:
      - containerPort: 8080
    resources:
      limits:
        cpu: "500m"      # throttling under this limit can stall probe handlers
        memory: 512Mi
    startupProbe:
      httpGet:
        path: /healthz
        port: 8080
      periodSeconds: 5
      timeoutSeconds: 3      # above the 1s default to absorb GC pauses
      failureThreshold: 36   # up to ~180s (5 x 36) to finish booting
      # successThreshold must be 1 for startup probes (the default)
    livenessProbe:
      httpGet:
        path: /healthz      # cheap check: is the process responsive at all?
        port: 8080
      periodSeconds: 10
      timeoutSeconds: 3
      failureThreshold: 3    # restart after ~30s of consecutive failures
      # successThreshold must be 1 for liveness probes (the default)
    readinessProbe:
      httpGet:
        path: /ready         # deeper check: can it serve traffic right now?
        port: 8080
      periodSeconds: 5
      timeoutSeconds: 2
      failureThreshold: 2    # leave the Service endpoints quickly
      successThreshold: 2    # rejoin only after 2 consecutive passes

A container spec with a long startup budget, a lenient liveness probe that tolerates GC and throttling stalls, and a readiness probe whose successThreshold of 2 keeps a flapping pod from rejoining the Service too early.

Readiness during rollouts and graceful shutdown

During a rolling update, readiness decides how fast the rollout moves. The Deployment controller counts a new pod as available only after it has been Ready for minReadySeconds, and maxUnavailable limits how many old pods can be removed before replacements count. If readiness returns 200 as soon as the HTTP server binds, a version with a bad database credential is marked available and the rollout replaces every healthy pod. If readiness requires every optional downstream to respond, one flaky dependency stops the deploy with old and new ReplicaSets running side by side. Setting minReadySeconds to a value long enough to catch a crash in early traffic, for example 10 to 30 seconds, catches failures that a single successful probe would miss.

Readiness also signals drain. When a pod is deleted, Kubernetes marks it terminating and starts removing it from endpoints. That removal reaches kube-proxy, ingress controllers and external load balancers asynchronously, so traffic can keep arriving for several seconds. The safe sequence is:

1. A preStop hook sleeps briefly, around 5 to 15 seconds, while endpoint removal propagates.
2. On SIGTERM, the app flips readiness to failing and keeps serving.
3. It finishes in-flight requests, closes idle keep-alive connections, then exits.

terminationGracePeriodSeconds must cover the preStop delay plus the longest expected request. The grace-period countdown includes time spent in preStop. If the grace period runs out, the kubelet sends SIGKILL, and any requests still in flight are cut off.

When to skip or limit probes, and the alternatives

Probes are optional, and a liveness probe has to justify its place. If your process exits on fatal errors, such as a failed config load, an unrecoverable panic or a connection pool it cannot rebuild, the kubelet already restarts it according to the pod's restartPolicy. A liveness probe adds value only for failures where the process stays up but stops making progress, such as a deadlock. A badly tuned liveness probe can turn a long garbage-collection pause or a saturated downstream dependency into simultaneous restarts across every replica. That outage is worse than the one the probe was meant to prevent.

Restarts also hide root causes. A memory leak that forces a restart every few hours can look like a healthy service on a dashboard. When the cause persists, as with a database that stays unreachable, the container enters CrashLoopBackOff. The kubelet then waits progressively longer between attempts, so recovery slows down at the moment you need it most.

Probes also have a narrow view. The kubelet runs them from the node against the pod's IP. They tell you nothing about Services, ingress, DNS, TLS or the latency clients actually experience.

In practice, rely on self-exit and metrics first. Add a liveness probe only for a hang you have actually observed, and use synthetic checks to cover what the kubelet cannot see.

When a startup probe should be avoided

A startup probe is the wrong tool in several situations:

• Fast, predictable boots. If the container is ready in a second or two every time, the extra probe adds configuration and another endpoint to maintain without protecting against anything.
• Boot work that belongs elsewhere. Schema migrations and one-off data loads usually belong in an init container or a separate Job, not in the API's startup phase. If they run inside the API, every replica and every restart repeats them, and the startup budget must grow to cover the slowest migration. Move that work out and keep the API's boot short.
• Pods that are not long-running servers. Jobs and CronJobs that run to completion are not behind a Service and are not restarted for being slow. They gain little from startup or readiness probes. Use a deadline on the Job instead.
• No measured boot times. A startup budget picked without data is the same guess as a large initialDelaySeconds, just under a different name. Measure first, then set the ceiling.
• A budget used to hide a hang. If the only way to stop restarts is a very large failureThreshold, the probe now hides deadlocks during boot for that whole window. Investigate why boot is slow or variable before widening the budget again.

Debugging probe failures and the cost of getting probes right

A debugging workflow

Start with kubectl describe pod <pod>. The Events list shows Unhealthy entries that name the failing probe (startup, liveness or readiness) and the reason, such as a timeout or an HTTP status code. Check Restart Count and Last State in the container status as well. A climbing restart count alongside liveness failures usually means the kubelet is killing a container that is slow, not one that is broken.

Next, check what readiness is doing to traffic. Run kubectl get endpoints <service>, or kubectl get endpointslices -l kubernetes.io/service-name=<service> to see per-endpoint readiness conditions. If pods are Running but missing from the ready addresses, the readiness check is failing. A common cause is that it depends on something the pod does not need in order to serve requests.

Finally, run kubectl logs <pod> --previous to read output from the container instance that was killed. The current container's logs only start after the restart, so the slow query or blocked thread that caused the timeout is usually only visible in the previous instance's logs.

What it costs

Separating probes properly takes a small amount of upfront work. You build distinct endpoints, measure real boot times, and review thresholds whenever CPU or memory limits change. Lower limits slow startup and can push it past the startup probe's failure budget.

Illustrative estimate (assumed numbers, not a benchmark): Incident cost = incidents per year × engineers involved × hours per incident. Assume 4 × 2 × 3 = 24 engineer-hours per year, compared with roughly 4 hours of setup.

How it saves engineering time

The time savings come from removing specific, recurring kinds of work:

• No more initialDelaySeconds tuning cycles. With a startup probe, the boot budget is a separate setting derived from measured boot times. You do not raise the liveness delay after each slow-boot incident, and steady-state liveness settings stay fixed.
• Fewer false-restart incidents. Keeping liveness in-process and out of shared dependencies removes the class of incident where a database or downstream outage restarts every replica at once, followed by a thundering-herd recovery.
• Faster diagnosis. With three separate endpoints, a kubectl describe pod event that names the failing probe tells you which phase broke: boot, a hung process, or a pod that is unable to serve. You don't need to reverse-engineer a single overloaded health check.
• Safer rollouts with less manual watching. Readiness that reflects real serving ability, combined with minReadySeconds, lets a rolling update stall on a broken version instead of replacing every healthy pod. Fewer rollouts need to be caught and reverted by hand.
• Clean shutdowns. A correct preStop and readiness drain sequence removes intermittent errors during deploys. Otherwise those errors tend to be investigated repeatedly as one-off client issues.

Illustrative estimate (assumed inputs): assume each incident in the formula above involves the patterns listed here, such as a false restart or a cascading liveness failure, and that probe separation prevents most of them. Under those assumed inputs, roughly 4 hours of setup and periodic threshold review replace on the order of 24 engineer-hours per year of incident work for one service. The real ratio depends on your incident rate and team size, so substitute your own numbers.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes startup probe, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.