How do you confirm the liveness probe is causing CrashLoopBackOff?
CrashLoopBackOff is not an error. It is a waiting state: the kubelet restarted a container, the container stopped again, and each further restart is delayed by an exponentially growing interval up to a cap of a few minutes. The real cause shows up in the pod's events and the container's last state.
Start with kubectl get pod and note the RESTARTS column. A steadily rising count across replicas suggests something systemic, such as probe settings that are too strict for the workload. Then run kubectl describe pod <name>. In the Events section, look for Liveness probe failed, often with a timeout message such as context deadline exceeded, followed by a note that the container will be restarted. In the container status, read Last State: the Reason and Exit Code fields. Finally, run kubectl logs <name> --previous to see what the killed instance printed before it stopped.
| Cause | Events | Last State reason / exit code | Previous logs |
|---|---|---|---|
| Liveness probe kill | Liveness probe failed, restart notice |
Error, typically 143 (SIGTERM) or 137 (SIGKILL after grace period) |
Normal output that stops abruptly, no stack trace |
| Application crash | No probe failures | Error, often exit code 1 |
Uncaught exception or explicit exit |
| OOMKilled | No probe failures | OOMKilled, 137 |
Truncated output, often during load |
Expect slow feedback. The back-off delay grows with each failure, so waiting for the next crash takes longer the more restarts have already happened. Probe settings are part of the pod spec. Editing them changes nothing for running pods until the Deployment rolls out new pods that use the updated template.
#!/usr/bin/env bash
set -euo pipefail
NS="shop" # placeholder namespace
APP="orders-api" # placeholder app label
POD="$(kubectl get pod -n "$NS" -l app="$APP" -o jsonpath='{.items[0].metadata.name}')"
kubectl get pod -n "$NS" -l app="$APP"
kubectl get events -n "$NS" \
--field-selector involvedObject.name="$POD",reason=Unhealthy
kubectl get pod -n "$NS" "$POD" -o jsonpath=\
'{.status.containerStatuses[0].lastState.terminated.reason}{" exit="}{.status.containerStatuses[0].lastState.terminated.exitCode}{"\n"}'
kubectl logs -n "$NS" "$POD" --previous --tail=50
kubectl describe pod -n "$NS" "$POD"
kubectl rollout status deployment/"$APP" -n "$NS"
A triage script that checks restart counts, probe-failure events, the last termination reason and exit code, and previous logs to tell a liveness probe kill apart from an application crash or an OOMKill.
Who should use this guidance?
This article is for engineers who run Node.js HTTP services on Kubernetes and see CrashLoopBackOff or steadily rising restart counts that don't match any crash in the application logs. In practice that means three groups:
- Service teams that own a Node.js API or web server and write its Deployment manifest or Helm values, including the probe settings.
- Platform and cluster owners who ship default probe settings, chart templates or CPU and memory limit policies that many services inherit. A strict default copied across dozens of services can produce false restarts everywhere at once.
- On-call engineers who need a quick way to tell a probe kill from an application crash or an OOMKill before digging into code.
The probe-design advice applies most directly to long-running servers with a single event loop per process. It matters less for workloads that don't define a liveness probe, such as batch Jobs that run to completion, and for services that already exit cleanly on every fatal error and rely on the restart policy alone. Much of the mechanics (probe timing, startup probes, the liveness/readiness split) applies to any language, but the event-loop sensitivity described below is specific to Node.js and other single-threaded runtimes.
Why are Node.js pods especially sensitive to liveness probe timeouts?
The kubelet's HTTP probe is just another request to your process. In Node.js, your JavaScript runs on one thread, the event loop. The probe gets no special treatment. The kernel can accept the TCP connection, but your handler can't run until whatever synchronous work currently holds the loop finishes.
event loop (one thread)
running: [ JSON.parse(40 MB body) ...................... ]
queued: [ GET /orders ] [ GET /healthz <- kubelet ] [ POST /cart ]
|
timeoutSeconds expires here, before the handler runs
Common blockers include synchronous CPU work like templating, sorting large arrays or regex on untrusted input. Others are parsing or serializing big JSON payloads, the synchronous crypto and zlib APIs, and long garbage-collection pauses. During any of these the process is healthy and will answer once the task completes. The kubelet only sees a probe that missed its deadline. After failureThreshold misses, it kills the container.
The restart makes things worse. The killed pod's traffic moves to the remaining replicas, and each of them is now more likely to stall. Meanwhile the new pod pays its warm-up cost: module loading, JIT compilation and cold caches. A single slow request can spread into restarts across the whole Deployment.
Two resource problems look the same from outside:
- CPU limits. When a container uses up its CFS quota for a scheduling period, the kernel pauses it until the next period. A 50 ms handler can take several hundred milliseconds that way.
- Memory pressure. A heap close to its limit triggers more frequent and longer GC pauses. If the container goes over its memory limit, it is OOMKilled (exit code 137), which is not a probe failure even though it also ends in CrashLoopBackOff.
Check requests, limits and the container's last termination reason before you change any probe timeouts.
+-----------+ HTTP GET /healthz every periodSeconds
| kubelet |------------------------------------+
+-----------+ |
^ v
| +----------------------------------------------------+
| | Node.js container (one JS thread: the event loop) |
| | |
| | running +------------------------------------+ |
| | ------> | sync work: JSON.parse, crypto, GC | |
| | +------------------------------------+ |
| | queued [GET /orders][GET /healthz][POST /cart] |
| | ^ still waiting (no reply)|
| +----------------------------------------------------+
| ^ ^
| +------------------+ +------------------+
| | CPU limit hit: | | memory near limit|
| | CFS quota spent, | | -> longer GC |
| | paused till next | | pauses; over the |
| | period (throttle)| | limit: OOMKill |
| | | | (exit 137) |
| +------------------+ +------------------+
|
+-- timeoutSeconds missed x failureThreshold --> kill + restart
|
v
+----------------------------------------------------------------+
| traffic shifts to other replicas -> they stall -> more misses |
| restarted pod warms up: module load, JIT, cold caches |
+----------------------------------------------------------------+
The kubelet's /healthz request waits on the same event loop behind synchronous work, CPU throttling or GC pauses; once timeoutSeconds is missed failureThreshold times the container is restarted and its load moves to the other replicas.
How does a liveness probe work?
Probes are run by the kubelet on the node where the pod is scheduled, not by the control plane. For each container that defines a livenessProbe, the kubelet runs a check on a timer and records the result. Kubernetes supports several probe mechanisms:
httpGet: the kubelet sends an HTTP GET to the given path and port on the pod's IP. Any status code from 200 up to 399 counts as success. Anything else, a refused connection, or no response withintimeoutSecondscounts as failure.tcpSocket: success means a TCP connection to the port could be opened. This only proves something is listening. For a Node.js server it says little about the event loop, because the kernel can complete the TCP handshake while JavaScript is blocked.exec: the kubelet runs a command inside the container. Exit code 0 is success.grpc: the kubelet calls the standard gRPC health-checking service on the given port, for servers that implement it.
When the number of consecutive failures reaches failureThreshold, the kubelet treats the container as unhealthy and kills it. It sends SIGTERM, waits for the termination grace period, then sends SIGKILL if the process is still running. That is why probe kills show exit code 143 or 137. What happens next depends on the pod's restartPolicy. For Deployments it is Always, so the kubelet restarts the container in place, on the same node and in the same pod, and increments the restart count. The pod is not rescheduled. If the restarted container keeps failing, the kubelet applies the growing back-off delay, and the pod reports CrashLoopBackOff while it waits.
Readiness and startup probes use the same mechanisms and the same timing fields but have different consequences. A failed readiness probe removes the pod from Service endpoints without touching the container. A startup probe suppresses liveness and readiness checks until it succeeds once. The rest of this article relies on keeping those three roles separate.
How does liveness probe timing work, and how fast will a stall trigger a restart?
Liveness probes don't run until initialDelaySeconds has passed. After that, the kubelet probes every periodSeconds and waits up to timeoutSeconds for each response. Once failureThreshold consecutive probes have failed, it kills the container. If the event loop hangs, the time until a restart is therefore roughly periodSeconds × failureThreshold, plus up to timeoutSeconds for the last probe to time out. The container's termination grace period is added on top of that before the process actually exits.
| Parameter | Controls | Effect of raising it |
|---|---|---|
initialDelaySeconds |
Wait before the first probe | Slower boots survive, but real hangs during that window go undetected |
periodSeconds |
Interval between probes | Fewer probes and slower detection |
timeoutSeconds |
How long one probe may take | Tolerates slow responses, so brief stalls stop counting as failures |
failureThreshold |
Consecutive failures before a restart | Tolerates longer stalls but delays recovery from real hangs |
successThreshold |
Successes needed to reset to healthy | Must be 1 for liveness |
Illustrative estimate, using assumed values:
- Aggressive: period 5s, timeout 1s, threshold 2. The pod restarts about 10–11s into a hang.
- Tolerant: period 10s, timeout 5s, threshold 6. The pod restarts about 60–65s into a hang.
The aggressive configuration fails on normal behaviour. Any synchronous work that blocks the event loop for more than 1s fails the probe that lands during it. Examples include a large JSON.parse, a long garbage-collection pause or a CPU-heavy request. If the block lasts slightly longer than one period, two consecutive probes fail and the container restarts, even though it would have recovered on its own. The tolerant configuration absorbs the same stall without a restart.
Readiness probes use the same timing parameters. With readiness, though, crossing the threshold removes the pod from Service endpoints instead of restarting it.
containers:
- name: api
image: registry.example.com/node-api:1.4.2
ports:
- containerPort: 3000
livenessProbe:
httpGet:
path: /healthz # cheap handler: no DB or downstream calls
port: 3000
initialDelaySeconds: 15 # let the app finish booting first
periodSeconds: 10 # probe every 10s
timeoutSeconds: 5 # a 1-4s event-loop stall still passes
failureThreshold: 6 # ~60-65s of continuous hang before restart
successThreshold: 1 # must be 1 for liveness probes
readinessProbe:
httpGet:
path: /ready # may check dependencies; failure only removes the pod from endpoints
port: 3000
periodSeconds: 5 # same knobs, but tuned to react faster
timeoutSeconds: 2
failureThreshold: 3 # ~15s to stop receiving traffic, no restart
# An aggressive liveness probe would look like this:
# periodSeconds: 5, timeoutSeconds: 1, failureThreshold: 2
# A 6s synchronous block (large JSON.parse, GC pause) fails two
# consecutive probes and restarts a container that would have recovered.
A Node.js container spec with a tolerant liveness probe (about 60–65s to restart on a sustained hang) and a faster readiness probe using the same timing fields. The aggressive alternative is shown in comments for comparison.
What should a liveness endpoint check, and what belongs in readiness instead?
Each probe has one job, and mixing them is where restart loops come from.
Liveness answers a single question: is this process irrecoverably stuck? If the answer is yes, a restart is the only fix. The handler should exercise the event loop and nothing else. A response proves the loop is turning and the HTTP server is accepting connections. That is enough.
Readiness answers a different question: should this pod receive traffic right now? A failing readiness probe takes the pod out of Service endpoints and leaves the container running, so it can recover by itself and rejoin.
Putting a database or Redis ping inside the liveness handler mixes these up. When the database goes down, every replica fails liveness at about the same time, and the kubelet restarts all of them. The restarted containers find the database still unavailable, fail again, and back off into CrashLoopBackOff. The outage was outside your pods, but you now have a fleet-wide restart cascade, plus a burst of reconnect attempts when the database comes back.
Readiness is the right place for dependency checks, but it has its own failure mode. If every replica depends on the same database, they all go unready together, and the Service ends up with zero endpoints. Clients then get connection errors instead of your application's error responses. For that reason, some teams keep readiness limited to local state, such as "warmed up" or "not draining," and return degraded responses (cached data or an explicit 503 with a clear body) when a dependency is down.
The fragment below defines both probes, each pointing to its own path with its own thresholds.
containers:
- name: api
image: registry.example.com/orders-api:1.4.2
ports:
- name: http
containerPort: 3000
# Liveness: is the event loop still turning? No DB/Redis calls here,
# so a dependency outage never triggers a restart.
livenessProbe:
httpGet:
path: /livez
port: http
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3 # ~30s of consecutive failures before restart
# Readiness: should this pod get traffic right now? Failing removes it
# from Service endpoints but leaves the container running.
readinessProbe:
httpGet:
path: /readyz # local state: warmed up, not draining
port: http
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 2 # pull from endpoints quickly
successThreshold: 1 # rejoin on first success
# Optional: hold off liveness during slow boot instead of a long
# initialDelaySeconds; liveness starts only after this succeeds.
startupProbe:
httpGet:
path: /livez
port: http
periodSeconds: 5
failureThreshold: 12 # allow up to ~60s to start
A container spec that points liveness at a cheap event-loop check (/livez) and readiness at a separate local-state check (/readyz), each with its own thresholds, plus a startup probe that holds off liveness during boot.
When should you use a startup probe instead of a long initialDelaySeconds?
A startup probe changes when the other two probes begin. While a startupProbe is defined and has not yet succeeded, the kubelet does not run liveness or readiness checks at all. Once it succeeds, it stops running for the life of that container, and the normal liveness and readiness settings take over. This lets a pod spend several minutes warming caches, compiling templates, or waiting on a migration without loosening the liveness settings that must catch a wedged event loop at 3 a.m.
A long initialDelaySeconds on the liveness probe is a weaker substitute because it is a fixed guess. Set it to 120 and a pod that boots in 8 seconds still has no liveness coverage for nearly two minutes. A pod that needs 150 seconds after a cold cache gets killed mid-boot and enters the same restart loop you were trying to avoid. A startup probe polls, so it hands off as soon as the app is actually up.
A typical configuration points the startup probe at the same cheap liveness path, with periodSeconds: 10 and failureThreshold: 30. That allows roughly 300 seconds to boot (illustrative estimate: 30 × 10s). It does not change liveness timing afterwards.
The cost is that this window is also how long a genuine startup hang goes undetected. If boot deadlocks on a connection that never resolves, Kubernetes waits the full five minutes before restarting. Size the window from observed worst-case boot times plus some margin. Do not pick an arbitrarily large number.
How do you implement a cheap, honest health handler in Node.js?
The liveness handler should answer one question: is this process's event loop still turning? If the request reaches the handler and the response goes out, the answer is yes. So the handler must return synchronously. Do not query the database, read files, or call downstream services. Any await in the handler makes the probe depend on something other than the process itself, and a slow dependency then looks like a dead pod.
Register the route before authentication, body parsing, request logging and tracing middleware. Each layer adds latency and possible failure on every probe. A failing auth service should not cause kubelet to kill a healthy process. Logging every probe also fills your logs with noise.
An exec probe avoids HTTP entirely, but kubelet spawns a new process inside the container on every check. At a short periodSeconds across many pods, that process creation adds steady CPU and memory overhead. A spawned process also can't see whether Node's event loop is responsive, only whether the container can run a command. For a Node server, an HTTP probe is usually the more honest signal.
To detect a wedged loop deliberately, measure event-loop delay inside the app. Use monitorEventLoopDelay from perf_hooks, or compare a setInterval timer's expected and actual firing times. Have the handler return 503 when lag stays above a threshold. The trade-off is that the threshold needs tuning. If you set it below the lag of your normal GC pauses or CPU-heavy requests, you recreate the false restarts you were trying to remove. Base it on observed lag under peak load, and judge sustained lag rather than single spikes.
What are the alternatives to tuning liveness probes, and what do they cost?
Each option moves the restart-loop problem somewhere else. Decide where you want it to live.
| Option | What it fixes | Risk it adds | Effort | Who decides |
|---|---|---|---|---|
| Tighter probes | Faster recovery from real hangs | More false restarts during GC or CPU-heavy requests | Low | Service team |
| Looser probes | Stops restarts caused by brief stalls | Slower recovery from real deadlocks | Low | Service team |
| No liveness probe | Removes false restarts entirely | A silently wedged event loop may never be restarted | Low, but needs an audit of fatal-error paths | Service team plus on-call |
| Event-loop-lag liveness | Separates "busy" from "stuck" | Wrong threshold brings the false restarts back | Medium | Service team |
| Raise or remove CPU limits | Less throttling-induced lag | Lower scheduling density, more noisy-neighbor contention | Low to change, high to evaluate | Platform/cluster owners |
Dropping liveness is reasonable for a service that reliably exits on uncaught exceptions and fatal errors. In that case the restart policy does the work. It is a poor fit if any code path can block or spin without throwing.
Illustrative estimate (assumed numbers, not a benchmark). Worst-case detection time is roughly periodSeconds × failureThreshold. With an assumed period of 10 s and a threshold of 3, a true deadlock is detected in about 30 s. Loosen that to 20 s and 6, and the same deadlock serves errors for about 120 s before the pod restarts.
The case for fixing the probe design is mostly about people's time. A probe that kills busy-but-healthy pods produces restart counts that look like application crashes. On-call engineers then pull logs, heap snapshots and recent deploys, looking for a bug that does not exist. An honest probe makes a restart mean the process was actually stuck, so the signal deserves the investigation it triggers.
How can getting probes right save engineering time?
The savings come from removing false work, not from making any single task faster. Three places stand out.
Fewer misleading investigations. When liveness only fails on a truly stuck event loop, a restart becomes a trustworthy signal. On-call engineers can skip the step of proving that the application did not crash, and the triage script above usually settles the cause in one pass: probe kill, application crash or OOMKill.
No fleet-wide cascades from external outages. Keeping dependency checks out of liveness means a database or cache outage stays a dependency incident. Engineers deal with one problem instead of two: the outage itself plus a Deployment stuck in CrashLoopBackOff with back-off delays slowing recovery after the dependency returns.
Less repeated tuning. A startup probe sized from observed boot times, plus a liveness probe with a cheap handler, rarely needs revisiting when the app's startup gets slower or a heavier endpoint ships. When platform teams put sensible defaults into shared chart templates, each service team stops rediscovering the same failure.
Illustrative estimate (assumed inputs, not measured). Assume a team sees 4 false-restart alerts a month, each taking an engineer 2 hours to investigate before someone notices the probe. That is about 8 engineer-hours a month. Assume that fixing the probe design takes one engineer one day: splitting the endpoints, adding a startup probe, and rolling it out. Under those assumptions the fix pays for itself within the first month or so. Your own alert count and investigation time will differ. Substitute them before using this to justify the work.
A step-by-step debugging sequence for Node pod restart loops
Work through these in order. Each step narrows the cause before you change anything.
-
Get the termination reason and exit code. Run
kubectl describe pod <pod>and readLast Stateunder the container.OOMKilledmeans the memory limit, not the probe, ended the process. Exit code 137 or 143 with reasonErroris consistent with the kubelet stopping the container, which needs confirming in step 2. -
Read the probe events.
kubectl get events --field-selector involvedObject.name=<pod>showsLiveness probe failedmessages with the failure detail, such as a timeout or a non-2xx status. Checkkubectl logs <pod> --previousfor what the process was doing just before the kill. -
Measure the health endpoint under load. Exec into the pod, or run
kubectl port-forward pod/<pod> 8080:8080and time requests withcurl -s -o /dev/null -w '%{time_total}\n'while real traffic runs. Port-forward adds its own hop, so treat an in-pod measurement as more faithful. Compare the numbers againsttimeoutSeconds. -
Check CPU throttling and memory headroom.
kubectl top pod <pod> --containersgives usage if metrics-server is installed. Inside the container,cat /sys/fs/cgroup/cpu.staton cgroup v2 showsnr_throttledandthrottled_usec. If those counters climb during probe failures, the event loop is being starved by the CPU limit. -
Change one probe parameter, then roll out. Apply the change, wait for
kubectl rollout status deployment/<name>, and watch restart counts withkubectl get pods -wover a period that includes peak load. If you change several parameters at once, you can't tell which one fixed the problem.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to kubernetes crashloopbackoff liveness probe, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.