Kubernetes Startup Probe: Stop Restart Loops from Slow Startups

How does a slow startup turn into a liveness probe restart loop?

The sequence is mechanical, which is why it repeats so reliably.

1. The container starts. The kubelet waits initialDelaySeconds, then runs the liveness probe every periodSeconds. It does not know or care whether the application has finished starting.
2. The app is still initializing. It might be loading configuration, running migrations, filling caches, waiting on a dependency, or not yet listening on the probed port. The HTTP request is refused or times out, and the probe fails.
3. Failures accumulate. After failureThreshold consecutive failures, the kubelet kills the container and restarts it according to the pod's restart policy.
4. The restart begins from zero. The new process has no memory of the previous attempt's progress. It hits the same probe at the same point in its startup and is killed again.
5. The pod enters CrashLoopBackOff. Repeated restarts trigger an increasing back-off delay between attempts, capped at several minutes by default. The pod spends most of its time waiting, and it never becomes Ready.

Here is an illustrative estimate with assumed inputs. Suppose initialDelaySeconds: 10, periodSeconds: 10, and failureThreshold: 3. The container is killed roughly 30–40 seconds after it starts. If the app needs 90 seconds to warm up, it cannot succeed, no matter how many times it retries.

The root mistake is a category error. Liveness answers "is this process stuck and beyond recovery?" Here it is being asked "has this process finished starting?" When one probe has to answer both questions, it gets at least one of them wrong.

Copy+--------------------+     +-----------------------------+
| Container starts   |---->| kubelet waits               |
| (fresh process)    |     | initialDelaySeconds (10s)   |
+--------------------+     +-----------------------------+
          ^                               |
          |                               v
          |                +-----------------------------+
          |                | Liveness probe every        |
          |                | periodSeconds (10s)         |
          |                +-----------------------------+
          |                               |
          |                               v
          |                +-----------------------------+
          |                | App still initializing:     |
          |                | port not bound, warming up  |
          |                | -> probe refused / timeout  |
          |                +-----------------------------+
          |                               |
          |                               v
          |                +-----------------------------+
          |                | failureThreshold (3) hit    |
          |                | kubelet kills container     |
          |                | (~30-40s, app needs ~90s)   |
          |                +-----------------------------+
          |                               |
          |                               v
+--------------------+     +-----------------------------+
| CrashLoopBackOff:  |<----| Repeated exits:             |
| growing back-off,  |     | restarted per restartPolicy |
| pod never Ready    |     | fails at same point again   |
+--------------------+     +-----------------------------+

Illustrative estimate with assumed settings: liveness checks start before a 90-second warm-up finishes, so the kubelet kills each fresh container at the same point, and the pod ends up in CrashLoopBackOff.

What are startup, liveness, and readiness probes for?

Kubernetes splits these questions across three probes, and each one has a single job.

Startup probe: "Has the app finished starting?" It runs only until it succeeds once. While it is running, the kubelet does not run the liveness or readiness probes, so a slow boot cannot be mistaken for a hang. After the first success it never runs again for that container. If it fails failureThreshold times in a row, the kubelet kills the container. That makes it a hard ceiling on startup time, not an open-ended wait.

Liveness probe: "Is this process stuck in a way only a restart will fix?" Deadlocks, a wedged event loop, and an exhausted thread pool that never drains all qualify. A slow dependency or a busy period does not, because restarting will not make either of those go away.

Readiness probe: "Should this pod get traffic right now?" It runs for the container's whole lifetime and can flip back and forth. That makes it the right place to express temporary conditions, such as a downstream outage or a full work queue.

The difference in consequences matters most. A readiness failure is cheap and reversible. A liveness failure throws away all in-memory state and starts the boot over.

Teams whose services have expensive startup paths gain the most from this separation. Examples include JIT warmup, loading large caches, and running schema migrations on boot. A restart puts these services right back into the slow phase that triggered the restart.

How do you size a Kubernetes startup probe from measured startup time?

The startup probe's effective budget is about failureThreshold × periodSeconds. Kubernetes runs the probe every periodSeconds and restarts the container only after failureThreshold consecutive failures. Until the startup probe succeeds, liveness and readiness checks don't run, so this product is the time your app has to come up.

Size that budget from the slowest startup you can observe, not the typical one. Measure from container start to the first successful health check, using the same CPU and memory requests and limits as production. A CPU limit can throttle JIT compilation and cache loading enough that a fast local start becomes a slow one in the cluster. Capture the bad cases deliberately:

• Cold caches and empty local volumes.
• Busy or undersized nodes.
• Boots that run schema migrations.
• Full warmup paths.

Take the worst of these and add margin, because the next deploy may land on a slower node than any you tested.

Illustrative estimate (assumed numbers, not a benchmark):

• Assumed worst-case startup: 90s
• Headroom: 2x, giving a 180s target
• periodSeconds: 10
• failureThreshold = 180 / 10 = 18

Substitute your own measured worst case and headroom into failureThreshold = (worst case × headroom) / periodSeconds.

The example below uses these values. The startup probe hits the same endpoint as the liveness probe, so passing it shows the process can answer that check. The liveness probe uses a short threshold suited to a running app. The readiness probe controls traffic separately.

A fixed ceiling has a limit. If startup blocks on an external dependency with no upper bound, such as a database that may be down for an hour, no failureThreshold is large enough. Change the app so startup doesn't wait on that dependency: start the process, report not-ready through readiness, and retry the connection in the background.

CopyapiVersion: v1
kind: Pod
metadata:
  name: orders-api
spec:
  containers:
    - name: app
      image: registry.example.com/orders-api:1.4.2
      ports:
        - containerPort: 8080
      # Measure worst-case startup with these same requests/limits.
      resources:
        requests: { cpu: "500m", memory: "512Mi" }
        limits: { cpu: "1", memory: "1Gi" }
      # Illustrative estimate: 90s worst case x 2 headroom = 180s budget.
      # Budget ~= failureThreshold x periodSeconds = 18 x 10 = 180s.
      startupProbe:
        httpGet: { path: /healthz, port: 8080 }
        periodSeconds: 10
        failureThreshold: 18
      # Runs only after startupProbe succeeds; short budget for a running app.
      livenessProbe:
        httpGet: { path: /healthz, port: 8080 }
        periodSeconds: 10
        failureThreshold: 3
      # Gates traffic separately; failing removes the pod from endpoints, no restart.
      readinessProbe:
        httpGet: { path: /ready, port: 8080 }
        periodSeconds: 5
        failureThreshold: 3

A pod spec that sizes the startup probe at about 180s (failureThreshold 18 × periodSeconds 10) from an illustrative 90s worst case with 2x headroom, alongside a short-threshold liveness probe and a separate readiness probe.

Why is a longer initialDelaySeconds the wrong fix, and what are the alternatives?

Raising initialDelaySeconds on the liveness probe looks like the smallest change, but it is a fixed wait applied to every container start. Suppose you set it to 120 seconds to cover your slowest observed boot. A pod that normally starts in 10 seconds and then deadlocks at second 15 runs unchecked for almost two minutes. The delay is also still a guess. A start that hits a cold cache or a slow volume mount and takes 130 seconds gets killed anyway, and the restarted container faces the same odds.

A startup probe avoids both problems. It runs only until its first success and then hands control to liveness. Fast starts come under liveness checks within seconds, and slow starts get the full failureThreshold × periodSeconds ceiling. The trade-off is that a container that hangs during startup survives until that ceiling expires. With a 10-minute ceiling, a stuck pod can sit for up to 10 minutes before it is restarted.

Omitting liveness entirely is defensible when the process reliably exits on fatal errors. In that case the pod's restart policy already restarts crashed containers, and a badly written probe would only add false kills. What you give up is recovery from hangs. A deadlocked thread pool or a stuck event loop leaves the container Running indefinitely, and nothing restarts it.

How should you tune the liveness probe once the app is running?

The liveness endpoint should answer one question: is this process still able to do work? Serve it in-process, with no database queries, cache lookups, or calls to other services. It should cost almost nothing, so its latency reflects the health of the process and not of the network.

Set timeoutSeconds to cover the pauses a healthy process actually goes through. A stop-the-world GC pause or a container hitting its CPU limit can delay a trivial handler by more than the default timeout. Set failureThreshold above 1 so that one slow response does not trigger a restart. The trade-off is easy to calculate. With periodSeconds: 10 and failureThreshold: 3, a true deadlock takes roughly 30 seconds to detect (illustrative estimate). Tighter settings catch hangs sooner, but they also restart pods that are only slow under load. That pushes their traffic onto fewer replicas, which then slow down too.

A shallow check has a blind spot. An HTTP server on its own thread can keep returning 200 while the worker pool behind it is deadlocked. To close that gap, have the work loop update a heartbeat timestamp on every iteration. The liveness handler then fails if that timestamp is older than a threshold.

Never put dependency checks in liveness. If the database goes down, every replica fails liveness at the same moment, and Kubernetes restarts all of them. The outage gets worse, and the restarted pods can still enter CrashLoopBackOff while the database is unavailable. Dependency status belongs in readiness, which removes pods from Service endpoints without killing them, or in no probe at all.

Copyimport queue
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

MAX_HEARTBEAT_AGE = 30  # seconds; must exceed the slowest legitimate job
jobs = queue.Queue()
last_beat = time.monotonic()  # monotonic: immune to wall-clock jumps

def work_loop():
    global last_beat
    while True:
        last_beat = time.monotonic()  # proves the loop is still iterating
        try:
            # The timeout keeps an idle worker beating; a blocking get() would look stale.
            job = jobs.get(timeout=5)
        except queue.Empty:
            continue
        job()

class Health(BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path != "/livez":
            self.send_error(404)
            return
        # In-process only: no DB, cache or downstream calls here.
        age = time.monotonic() - last_beat
        self.send_response(200 if age < MAX_HEARTBEAT_AGE else 503)
        self.end_headers()

    def log_message(self, *args):
        pass  # keep probe traffic out of the logs

if __name__ == "__main__":
    threading.Thread(target=work_loop, daemon=True).start()
    ThreadingHTTPServer(("0.0.0.0", 8080), Health).serve_forever()

A liveness handler that returns 503 when the worker loop's heartbeat goes stale, so a deadlocked worker fails the probe even though the HTTP server thread stays responsive.

How do you tell which probe is killing your pod?

Start with kubectl describe pod <name> and read the container's Restart Count and Last State. Last State records how the previous container instance ended: a Terminated reason, an exit code, and start and finish timestamps. The finish time matters because you will line it up against the Events list at the bottom of the same output.

A probe kill leaves a trail in Events. You will see repeated Liveness probe failed or Startup probe failed warnings, followed by a message saying the container failed its probe and will be restarted. The kubelet then sends SIGTERM, and SIGKILL if the container outlives its grace period. Because of that, the exit code reflects how your process handled the signal, not the cause of the kill. Rely on the event sequence rather than the code.

Next, run kubectl logs <pod> --previous to read the killed container's output. If the last lines show the app still initializing, such as migrations running or caches warming, the probe fired too early. If they show a stack trace or a "missing config" error, the app crashed without any help from a probe.

Events are only retained for a limited time. On a pod that has been looping for hours, the probe messages may already be gone. When that happens, catch a fresh restart with kubectl get events --watch.

CrashLoopBackOff is a symptom, not a cause. Bad config, a missing Secret or ConfigMap key, a wrong command, and failed dependencies all produce it. Loosening probes before you confirm a probe is responsible does not fix those failures. It only hides them, and real crashes then take longer to surface.

Copy#!/usr/bin/env bash
# Triage a restarting pod: probe kill, app crash, or OOMKilled?
set -euo pipefail
POD="${1:?usage: $0 <pod> [namespace] [container]}"
NS="${2:-default}"
CONTAINER="${3:-}"
C_FLAG=()
[[ -n "$CONTAINER" ]] && C_FLAG=(-c "$CONTAINER")

# 1. How did the previous instance end? Note reason, exit code, finishedAt.
kubectl get pod "$POD" -n "$NS" -o jsonpath='{range .status.containerStatuses[*]}{.name}{"  restarts="}{.restartCount}{"  reason="}{.lastState.terminated.reason}{"  exit="}{.lastState.terminated.exitCode}{"  finished="}{.lastState.terminated.finishedAt}{"\n"}{end}'

# 2. Events for this pod, oldest first. Line timestamps up with finishedAt.
#    Probe kill: repeated 'Liveness probe failed' / 'Startup probe failed'
#    followed by 'failed ... probe, will be restarted' just before finishedAt.
#    No probe events + reason=Error  -> the app exited on its own.
#    reason=OOMKilled (exit 137)     -> memory limit, not a probe.
echo '--- probe-related events ---'
kubectl get events -n "$NS" \
  --field-selector "involvedObject.name=$POD" \
  --sort-by=.lastTimestamp \
  | grep -Ei 'probe|killing|oom|back-off' || echo '(none retained; see step 4)'

# 3. Last output of the killed container.
#    Still initializing (migrations, cache warm-up) -> probe fired too early.
#    Stack trace or missing config/Secret error     -> real crash; fix that,
#    do NOT loosen probes to hide it.
echo '--- previous container logs (tail) ---'
kubectl logs "$POD" -n "$NS" "${C_FLAG[@]}" --previous --tail=50

# 4. Events expire. If step 2 was empty on a long-looping pod, watch the
#    next restart live (Ctrl-C to stop):
# kubectl get events -n "$NS" --field-selector "involvedObject.name=$POD" --watch

A triage script that reads the previous container's termination reason, exit code and finish time, lists the pod's probe-related events to line up against that time, and tails the killed container's logs to separate probe kills from app crashes and OOMKilled restarts.

When should you revisit probe settings, and what does it save?

Treat probe settings as a statement about the workload's startup path, not as one-time boilerplate. Use this checklist when you first set them and whenever that path changes:

• Measure startup under realistic resources. Time it with the same CPU and memory limits as production, on a node that is busy rather than idle. A laptop or an empty test cluster hides CPU throttling, which often dominates startup time for JIT-compiled runtimes and cache warm-up.
• Give the startup probe a ceiling above the worst case. Its failure threshold multiplied by its period should exceed the slowest startup you observed, with headroom. The ceiling only affects pods that are actually stuck, so making it generous costs little.
• Keep liveness lightweight and tolerant. It should check only that the process can respond. It should not touch downstream dependencies, and it should allow several consecutive failures before a restart.
• Use readiness to gate traffic. Dependency checks belong in the readiness probe, which removes the pod from Service endpoints without killing it.
• Re-measure when startup work changes. New schema migrations, larger caches loaded at boot, and new dependencies resolved during init all lengthen startup.

Infrastructure changes can bring loops back without any code change. Moving to smaller nodes, autoscaling onto nodes that are already busy, or tightening CPU limits all slow startup, and none of them shows up in an application diff.

The upfront cost is small: measure startup and write down why each threshold has its value. In return, you spend less time chasing restarts blamed on a "flaky app," and you have fewer outages caused by your own probes.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to Kubernetes startup probe, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.