What happens when Kubernetes terminates a pod?
Graceful shutdown means a pod stops accepting new work, finishes the requests it already has, releases its resources, and exits on its own before Kubernetes forcibly kills it. To get there, you need to know what Kubernetes does when a pod is deleted, whether by a rolling update, a scale-down, a node drain or kubectl delete.
Two paths that run at the same time
When the API server marks a pod for deletion, the pod enters the Terminating state and two independent processes begin.
On the node, the kubelet sees the deletion and runs the container's preStop hook, if one is defined. When the hook finishes, the kubelet sends SIGTERM to PID 1 in each container. The grace period (terminationGracePeriodSeconds, 30 seconds by default) starts at deletion, not at SIGTERM, so time spent in preStop comes out of the same budget. If the process is still running when the grace period expires, the kubelet sends SIGKILL. Nothing is drained after that.
In the control plane, the endpoints controller removes the pod from the Service's EndpointSlices. Then every consumer of that data reacts on its own schedule:
- kube-proxy on each node rewrites its forwarding rules.
- Ingress controllers reload their upstream lists.
- External load balancers run their own deregistration and health checks.
Why traffic arrives after SIGTERM
Nothing orders these two paths. SIGTERM can reach your process while some node's kube-proxy, or a cloud load balancer, is still sending it new connections. If the app closes its listener immediately, those requests fail with connection refused or resets, even if the shutdown code is correct.
That race causes most dropped requests during deploys. The fix is mostly about timing, not about rewriting application logic.
+--------------------------------+
| API server: pod marked deleted |
| Terminating; grace clock starts|
+---------------+----------------+
|
+---------------------+----------------------+
| (node) (control plane) |
v v
+---------------------+ +--------------------------+
| kubelet: preStop | | endpoints controller |
| hook (if defined) | | removes pod from Service |
+---------------------+ | EndpointSlices |
| +-------------+------------+
v |
+---------------------+ +------------+------------+
| SIGTERM -> PID 1 | v v v
| stop accepting, | +-----------+ +---------+ +-----------+
| drain, exit | | kube-proxy| | ingress | | external |
+---------------------+ | rewrites | | reloads | | LB |
| | rules | | upstream| | deregister|
v (grace expires) +-----------+ +---------+ +-----------+
+---------------------+
| SIGKILL: no further | Nothing orders the two paths:
| draining | SIGTERM can arrive while any of
+---------------------+ these still routes new requests to the pod.
After deletion, the kubelet's preStop/SIGTERM/SIGKILL sequence and the EndpointSlice update that kube-proxy, ingress controllers and load balancers each consume run independently, so a pod can receive new traffic after it gets SIGTERM.
Why does my app never see SIGTERM?
The previous section covered the routing side. This one covers the process side. When the kubelet stops a container, the container runtime sends SIGTERM to the container's main process, which is PID 1 inside the container. If that process is not your application, your shutdown handler never runs. The pod waits out terminationGracePeriodSeconds, then gets SIGKILL, and every in-flight request dies with it.
The usual cause is a shell-form ENTRYPOINT or CMD, such as ENTRYPOINT node server.js. Docker wraps shell form in /bin/sh -c, so the shell becomes PID 1 and your app runs as its child. Most shells do not forward SIGTERM to children. Some shells exec a single simple command directly, but you should not depend on that. Exec form, such as ENTRYPOINT ["node", "server.js"], starts the binary directly as PID 1.
Wrapper scripts have the same problem. If entrypoint.sh renders config and then starts the app, make the last line exec your-binary "$@". That replaces the shell with your process, so your process inherits PID 1 and receives the signal.
PID 1 has a second quirk. The kernel does not apply default signal actions to it. An app without an explicit SIGTERM handler simply ignores the signal. A minimal init such as tini or dumb-init fixes this. It runs as PID 1, forwards signals to your process, and reaps zombie processes. Kubernetes has no equivalent of docker run --init, so you must build the init into the image.
To verify, run kubectl exec <pod> -- ps to see what PID 1 is. Then run kubectl exec <pod> -- kill -TERM 1 and watch kubectl logs -f for your shutdown messages. Minimal images may lack ps or kill. In that case, use an ephemeral debug container that targets the app container.
FROM node:20-alpine
# tini will run as PID 1. It forwards signals to its child and reaps zombies.
# On Alpine the binary is installed at /sbin/tini.
RUN apk add --no-cache tini
WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY . .
# entrypoint.sh renders config and then hands off to the app.
# Its last line must be: exec "$@"
# exec replaces the shell with node. Without exec, sh stays in the
# process tree as node's parent, so SIGTERM from tini is delivered to sh
# and never reaches node.
COPY entrypoint.sh /app/entrypoint.sh
RUN chmod +x /app/entrypoint.sh
# BAD (shell form): Docker runs this as /bin/sh -c "node server.js".
# sh becomes PID 1 and may not forward SIGTERM to node.
# ENTRYPOINT node server.js
# GOOD (exec form): no implicit shell. tini is PID 1. It runs the wrapper,
# which execs node, and tini forwards SIGTERM to node.
# "--" ends tini's own option parsing.
ENTRYPOINT ["/sbin/tini", "--", "/app/entrypoint.sh"]
CMD ["node", "server.js"]
# Verify in a running pod:
# kubectl exec <pod> -- ps # PID 1 should be tini, with node as its child
# kubectl logs -f <pod> # in a second terminal, watch for shutdown logs
# kubectl exec <pod> -- kill -TERM 1 # tini forwards SIGTERM to node
# If the image has no ps or kill, run the same checks from an ephemeral
# debug container that targets the app container.
This Dockerfile uses an exec-form ENTRYPOINT with tini as PID 1 and a wrapper script that ends in exec "$@", so SIGTERM reaches the Node process, and the trailing comments show how to check this with kubectl exec and kill.
How do you bridge the endpoint removal race with a preStop hook and grace period budget?
Kubernetes removes the pod from Service endpoints and starts the preStop hook at the same time. Endpoint removal then has to propagate to kube-proxy, ingress controllers and any external load balancer, and until it does, new requests can still arrive. A preStop sleep keeps the process serving normally during that window, because SIGTERM is only sent after the hook returns. If you sleep after SIGTERM instead, most frameworks have already stopped accepting connections, so late-routed requests get refused. An exec hook such as sleep 10 also requires a sleep binary in the image.
The grace period countdown starts when termination begins, and it includes the preStop time:
terminationGracePeriodSeconds ≥ preStop + max request duration + buffer
Illustrative estimate (assumed numbers, not a benchmark): 10s preStop + 20s maximum request + 5s buffer = 35s. That exceeds the 30s default. With the default left in place, the kubelet sends SIGKILL partway through the drain, and any requests still running are cut off. In the pod spec, set terminationGracePeriodSeconds: 35 alongside a lifecycle.preStop.exec command that runs the sleep.
| Approach | Requires | Weakness |
|---|---|---|
| preStop exec sleep | sleep binary in image |
Fixed guess |
| In-app delay after SIGTERM | Application code | Same timing assumptions |
| No delay | Nothing | Drops late-routed requests |
There is no universal sleep value. Measure how long your proxies and load balancers take to converge, and choose the sleep from that measurement. Even then, a fixed sleep is a heuristic and cannot guarantee that no request arrives late. For distroless images that have no shell or sleep binary, an in-app delay works, but it depends on the same timing assumptions.
apiVersion: v1
kind: Pod
metadata:
name: web
spec:
# Budget covers the whole shutdown, including preStop:
# 10s preStop + 20s max request + 5s buffer = 35s (illustrative estimate).
# Left at the 30s default, SIGKILL would land mid-drain.
terminationGracePeriodSeconds: 35
containers:
- name: web
image: registry.example.com/web:1.4.2
ports:
- containerPort: 8080
lifecycle:
preStop:
exec:
# Keep serving normally while endpoint removal propagates to
# kube-proxy, ingress and external load balancers.
# SIGTERM is sent only after this hook returns.
# Requires a sleep binary in the image (not present in distroless).
# 10s is an assumed value: measure your own convergence time.
command: ["sleep", "10"]
readinessProbe:
httpGet:
path: /healthz
port: 8080
periodSeconds: 5
A pod spec that runs a 10s preStop sleep before SIGTERM and raises terminationGracePeriodSeconds to 35s so the sleep and the in-flight request drain both fit inside the budget.
How do you drain in-flight HTTP, gRPC and streaming connections on SIGTERM?
Once the delay has elapsed, the application performs the actual drain, and that drain has a deadline that must finish inside the grace period.
In Go, signal.NotifyContext gives you a context that is cancelled on SIGTERM. When it fires, call http.Server.Shutdown with a context deadline set a few seconds short of the remaining grace period. Shutdown closes the listeners, closes idle keep-alive connections, and waits for active requests to complete. If the deadline expires first, call Close to force the remaining connections shut. Exiting before SIGKILL lets you log what was abandoned.
Ending the listener alone does not drain traffic. Keep-alive, HTTP/2 and gRPC clients reuse connections, so removing the pod from the endpoints does not move connections that are already open. The server has to tell clients to reconnect elsewhere. For HTTP/1.1, send Connection: close on responses during the drain. For HTTP/2, send GOAWAY, which Go's built-in HTTP/2 server does during Shutdown. For gRPC, call GracefulStop, which sends GOAWAY and waits for in-flight RPCs, and fall back to Stop when your deadline passes. Node, Java and Python servers follow the same sequence: stop accepting connections, signal close, wait with a timer, then force-close.
Streams and WebSockets can stay open indefinitely, and Shutdown does not track hijacked connections. Give them an explicit maximum lifetime and close them using RegisterOnShutdown or your own tracking, so clients learn to reconnect.
Log timestamps for three events: SIGTERM received, listener closed, and last request completed. The gaps between them show where the shutdown budget actually goes.
// Go: drain on SIGTERM with a deadline, then force-close.
package main
import (
"context"
"errors"
"log"
"net/http"
"os/signal"
"syscall"
"time"
)
func main() {
log.SetFlags(log.LstdFlags | log.Lmicroseconds) // timestamps for the shutdown timeline
srv := &http.Server{Addr: ":8080", Handler: http.DefaultServeMux}
// Hooks run after Shutdown closes the listeners. Close streams/WebSockets here,
// because Shutdown does not track hijacked connections.
srv.RegisterOnShutdown(func() { log.Print("listener closed; signalling long-lived streams") })
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM)
defer stop()
go func() {
if err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
log.Fatal(err)
}
}()
<-ctx.Done()
log.Print("SIGTERM received")
// Assumes a 30s grace period with a 5s preStop delay; leave margin before SIGKILL.
drainCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
// Shutdown closes listeners and idle keep-alives, sends GOAWAY on HTTP/2, waits for active requests.
if err := srv.Shutdown(drainCtx); err != nil {
log.Printf("drain deadline exceeded, force-closing: %v", err)
srv.Close()
return
}
log.Print("last request completed")
}
A Go HTTP server that catches SIGTERM with signal.NotifyContext, drains with http.Server.Shutdown under a 20-second deadline, falls back to Close, and logs when SIGTERM arrives, when the listener closes and when the last request completes.
Should readiness fail during shutdown, and what about liveness?
Failing readiness when SIGTERM arrives is common advice, but it buys less than people expect. When a pod is deleted, the control plane marks its endpoints as not ready because the pod is terminating, whether or not any readiness probe has failed. The kubelet's next probe result does not trigger that removal and cannot speed it up. That is why the preStop delay matters more than the probe.
Flipping readiness still helps consumers that check readiness themselves instead of following pod deletion. Examples include an external load balancer health-checking the pod directly, a custom discovery client, or a proxy that keeps routing to terminating endpoints that still report as serving. For those consumers, a failing readiness check is the only signal that the pod is leaving.
Liveness is different. It must keep passing until the process exits. A failing liveness probe tells the kubelet to kill the container, which cuts off the drain partway through. A frequent cause is a single /healthz handler used for both probes that returns 503 once shutdown begins. Keep the two endpoints separate.
| Probe | During drain | Why |
|---|---|---|
| Readiness | May fail | Signals consumers that watch the probe; endpoint removal happens anyway |
| Liveness | Must pass | Failure kills the container and aborts in-flight work |
There is also a capacity cost. If readiness fails as soon as SIGTERM arrives, the pod stops counting as available before its replacement is ready. As an illustrative estimate, with two replicas and maxUnavailable: 1, the remaining pod carries all traffic until the new pod passes its readiness probe. Check that a single replica can handle that load before you fail readiness early.
Draining workers, sidecars and rolling update settings
Queue consumers and background workers
A worker that receives SIGTERM should first stop pulling new messages, because anything it fetches after that point is work it may not finish. For messages already in hand, either finish them within the grace period or explicitly nack or release them so the broker can redeliver them promptly instead of waiting for a visibility or session timeout. Then commit offsets or acknowledgements for completed work. If you skip the commit, another consumer will reprocess those messages. Handlers must be idempotent anyway, but a clean commit keeps duplicates rare.
Shutdown order inside the process
Shut down resources in reverse dependency order. Stop accepting requests and messages, wait for in-flight work to finish, then close database and cache pools. Flush and close telemetry exporters last. If you close the pool early, the final requests fail. If you flush telemetry early, you lose the logs and traces that explain a failed shutdown.
Sidecars and mesh proxies
A sidecar proxy has its own SIGTERM handling and its own timing. If the proxy exits while your app is still draining, the in-flight requests and outbound calls routed through it fail, even though your code handled the signal correctly. How the app and the proxy are ordered depends on your Kubernetes setup and mesh configuration, so verify it in your environment. Don't assume it.
Rollout and disruption settings
maxUnavailable sets how many pods can be draining at once without replacements. maxSurge lets new pods start before old ones go away. For node drains, a PodDisruptionBudget limits how many pods are evicted at the same time. Run enough replicas that the pods left during a drain can absorb the traffic shifted from the pods that are shutting down.
How do you test graceful shutdown, and when is it not enough?
Testing the drain
Point a load generator at the Service from inside the cluster and keep a steady request rate going. While it runs, trigger kubectl rollout restart on the Deployment. When the rollout finishes, count non-2xx responses and connection-level errors such as resets and refused connections separately. Status codes show what your handlers did. Connection errors show what the network did to requests that never reached a handler. A correct setup should show zero of both. Then remove the preStop hook and run the test again. Errors that appear only in this second run come from the endpoint removal race. That confirms the sleep is doing its job and is not just ritual.
Who needs it, and when to hold back
This matters most for services behind load balancers that deploy often or autoscale, because every pod termination is a chance to drop requests. Skip long delays for batch jobs, which should checkpoint instead. Also skip them where you need fast scale-down. A long sleep slows every rollout, and terminating pods keep holding node capacity while they wait.
| Approach | Strength | Risk |
|---|---|---|
| Drain + preStop | Fixes errors at the server | Slower rollouts |
| Client retries + idempotency keys | Also covers crashes | Hides bugs, duplicates non-idempotent writes |
Use retries alongside draining, not instead of it. Neither one helps with SIGKILL, OOM kills, node failure or preemption, because all of these skip the drain entirely. The cost is a small upfront engineering effort and slower deploys. In return, you spend less time chasing error spikes at deploy time.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to graceful shutdown kubernetes sigterm, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.