How does Kubernetes graceful shutdown work when a pod terminates?

Graceful shutdown means a pod stops taking new work, finishes what it already accepted, and exits on its own before Kubernetes forcibly kills it. It rests on two processes that run at the same time and do not coordinate with each other.

The kubelet's termination sequence

When a pod is deleted, whether by a rollout, a scale-down, an eviction or a node drain, the API server sets a deletion timestamp and grace period on the pod object, and the pod shows as Terminating. The kubelet on that node sees the change and acts in a fixed order:

  1. It runs each container's preStop hook, if one is defined, and waits for it to finish.
  2. It sends SIGTERM to PID 1 in each container. This is the container's configured stop signal, and SIGTERM is the default.
  3. If containers are still running when terminationGracePeriodSeconds expires, it sends SIGKILL. SIGKILL cannot be caught or delayed.

Endpoint removal runs in parallel

The same deletion event also starts a separate chain of updates:

  • The control plane marks the pod's endpoint as terminating in its EndpointSlices.
  • kube-proxy or the cluster's dataplane rewrites forwarding rules on every node.
  • Ingress controllers update their upstream lists.
  • External load balancers deregister the target.

Each of these components watches for changes and reacts on its own schedule. None of them reports back to the kubelet. As a result, your process can receive SIGTERM while some parts of the network are still sending it traffic.

One shared budget

The grace period starts counting down when termination begins, not when SIGTERM arrives. Time spent in the preStop hook comes out of the same budget. If the hook takes 20 seconds of a 30-second grace period, the application has about 10 seconds after SIGTERM to drain before it is killed.

     pod deleted: rollout / scale-down / eviction / node drain
                                   v
        +-----------------------------------------------------+
        | API server: sets deletionTimestamp + grace period   |
        | pod shows Terminating                (t = 0)        |
        +-----------------------------------------------------+
                 |                                  |
     kubelet track (in order)          endpoint track (async, parallel)
                 v                                  v
        +-----------------+             +-----------------------------+
        | 1. run preStop  |             | EndpointSlice: endpoint     |
        |    hook, wait   |             | marked terminating          |
        +-----------------+             +-----------------------------+
                 |                                  |
                 v                                  v
        +-----------------+             +-----------------------------+
        | 2. SIGTERM to   |             | kube-proxy / dataplane      |
        |    PID 1        |             | rewrites rules on each node |
        +-----------------+             +-----------------------------+
                 |                                  |
                 v                                  v
        +-----------------+             +-----------------------------+
        | 3. SIGKILL if   |             | ingress upstreams updated,  |
        |  still running  |             | external LB deregisters     |
        +-----------------+             +-----------------------------+
       (no feedback between tracks: traffic can still arrive after SIGTERM)
 t=0                                                 grace period expires
  |<--- preStop --->|<--- app drains after SIGTERM --->| SIGKILL
  |<----------- terminationGracePeriodSeconds -------->|

Pod deletion starts two uncoordinated tracks at t = 0: the kubelet's preStop, SIGTERM, SIGKILL sequence and asynchronous endpoint removal across the network, with preStop and application drain sharing a single terminationGracePeriodSeconds budget.

Why do requests get dropped during rollouts and node drains?

Deleting a pod, whether from a rolling update, a scale-down, or kubectl drain evicting it, starts two independent processes at the same moment. The kubelet on the pod's node runs the preStop hook and then sends SIGTERM. Separately, the control plane marks the pod as terminating, and the EndpointSlice controller removes it from the Service's endpoints. That change then has to reach every consumer: kube-proxy on each node rewriting its iptables or IPVS rules, ingress controllers reloading their upstream lists, and external load balancers deregistering targets. Each consumer watches and applies changes on its own schedule. Nothing makes the kubelet wait for them.

If the application closes its listener as soon as SIGTERM arrives, there is a window in which some nodes still route new connections to it. Those connections are refused or reset. The client sees a connection error or a 502, even though every component did what it was designed to do.

Failing the readiness probe on SIGTERM does not close this window. A terminating pod is already pulled from ready endpoints, and a readiness failure travels the same propagation path. It also adds probe-period and failure-threshold delay on top.

Endpoint updates change routing only for new connections. Existing TCP connections keep flowing: kube-proxy's conntrack entries persist, and HTTP keep-alive pools, gRPC channels, and ingress upstream pools keep reusing sockets to the old pod until something closes them.

This affects anyone running services behind Services, ingress, or cloud load balancers. Node drains make it worse, because they terminate many pods at once.

How do you use a preStop hook to wait out endpoint propagation?

The kubelet runs the preStop hook to completion before it sends SIGTERM. That ordering lets you buy time. While the hook waits, the endpoints controller removes the pod from EndpointSlices, kube-proxy and ingress controllers reprogram their routing, and new connections stop arriving. Your process keeps serving the stragglers normally. A delay of a few seconds is usually enough. The right value depends on how quickly your proxies and load balancers converge, so measure it rather than guessing.

There are two ways to express the delay. The traditional form is an exec hook that runs sleep. This only works if the image contains a sleep binary, and a shell if you wrap it in sh -c. Distroless and scratch images have neither, so the hook fails. Newer clusters offer a native sleep action under lifecycle.preStop, which takes a seconds field. The kubelet performs that wait itself, so the image needs no binaries. Support depends on the cluster version and feature gates. Check your version's documentation before relying on either form.

A broken hook does not fail loudly. If the command is missing or hangs, the hook still consumes the grace period. The only trace is a FailedPreStopHook event in kubectl describe pod, and nothing appears in your application logs.

Approach Image requirements Cluster support Failure visibility
exec sleep sleep binary, and a shell if wrapped All versions Pod events only
Native sleep action None Newer versions only; check docs Pod events only
App-side delay after SIGTERM None All versions Application logs
apiVersion: v1
kind: Pod
metadata:
  name: web-shutdown-demo
spec:
  # Total budget for preStop + SIGTERM handling. The hook delay counts against it.
  terminationGracePeriodSeconds: 30
  containers:
    # Form 1: exec hook. The image must contain a sleep binary,
    # and a shell too if you wrap the command in sh -c.
    # On distroless or scratch images this fails, and the failure
    # appears only as a FailedPreStopHook pod event.
    - name: app-debian
      image: registry.example.com/web-app:1.0-debian
      ports:
        - containerPort: 8080
      lifecycle:
        preStop:
          exec:
            # Hold off SIGTERM while EndpointSlices and proxies drop this pod.
            command: ["sleep", "5"]

    # Form 2: native sleep action. The kubelet performs the wait itself,
    # so the image needs no binaries. Requires cluster support:
    # check your version's documentation and feature gates first.
    - name: app-distroless
      image: registry.example.com/web-app:1.0-distroless
      ports:
        - containerPort: 8081
      lifecycle:
        preStop:
          sleep:
            seconds: 5

A pod spec showing the same 5-second preStop delay written as an exec sleep hook, which needs a sleep binary in the image, and as the native sleep action, which needs no binaries but does require cluster support.

How should your application handle SIGTERM and drain in-flight work?

Once SIGTERM arrives, shut down in a fixed order. Each step removes a source of new work before you tear down what that work depends on:

  1. Stop accepting new connections.
  2. Let in-flight requests finish.
  3. Close idle and keep-alive connections.
  4. Stop queue consumers and background workers.
  5. Flush buffers and commit offsets or transactions.
  6. Close database and client pools.
  7. Exit.

If you close the database pool before handlers finish, the requests you were trying to save will fail.

In Go, http.Server.Shutdown covers the first three steps. It closes the listeners, closes idle connections, and waits for active ones to go idle. Pass it a context whose deadline is shorter than the grace period left after your preStop delay. That leaves time for steps 4–7 before the kubelet sends SIGKILL. In Node, server.close() stops new connections, but open keep-alive sockets can keep the process alive, so close idle connections explicitly.

Long-lived connections need an explicit signal to move elsewhere:

  • HTTP/1.1: set Connection: close on responses.
  • HTTP/2 and gRPC: send GOAWAY so clients finish current streams and open new ones on another pod. Go's gRPC GracefulStop does this.
  • WebSockets: Shutdown does not track hijacked connections. Send a close frame and have clients reconnect with jitter.

Queue consumers should stop fetching first. Then they either finish the current message or nack it for redelivery. Because a SIGKILL can still interrupt processing, handlers must be idempotent.

Runtimes and frameworks differ in default signal handling. Some exit immediately on SIGTERM, and some install handlers that conflict with yours. Validate the shutdown behavior separately for each stack you run.

const http = require('node:http');
// App-specific dependencies: a queue consumer and a DB pool (placeholders).
const { consumer, pool } = require('./deps');

let shuttingDown = false;

const server = http.createServer(async (req, res) => {
  // During shutdown, tell HTTP/1.1 clients not to reuse this socket.
  if (shuttingDown) res.setHeader('Connection', 'close');
  res.end('ok\n');
});
server.listen(8080);

process.once('SIGTERM', async () => {
  shuttingDown = true;

  // Assumed: terminationGracePeriodSeconds 30, preStop sleep 5s, so ~25s remain.
  // Give up at 20s to leave headroom before SIGKILL.
  const deadline = setTimeout(() => {
    server.closeAllConnections();
    process.exit(1);
  }, 20_000);

  // Steps 1-3: stop accepting, let in-flight requests finish, drop idle keep-alives.
  const httpClosed = new Promise((resolve) => server.close(resolve));
  server.closeIdleConnections();
  await httpClosed;

  // Step 4: stop fetching; finish or nack the current message (handlers are idempotent).
  await consumer.stop();
  // Step 5: commit offsets / flush buffered work.
  await consumer.commit();
  // Step 6: close the pool only after nothing can still use it.
  await pool.end();

  // Step 7: exit cleanly.
  clearTimeout(deadline);
  process.exit(0);
});

A Node.js SIGTERM handler that drains in this order: HTTP connections, then the queue consumer, then pending commits and the database pool, with a hard deadline set shorter than the grace period left after the preStop delay.

Why does SIGTERM never reach your process? Fixing the PID 1 problem

The kubelet asks the container runtime to send SIGTERM to the container's PID 1. If PID 1 is not your server, the signal never reaches the server, and the pod sits idle until the grace period ends and SIGKILL arrives.

Shell-form entrypoints

A shell-form instruction such as CMD node server.js runs as /bin/sh -c "node server.js". That makes the shell PID 1 and your server its child. Many shells do not forward SIGTERM to child processes. Some shells exec a single simple command directly, but you should not rely on that. Use exec form, CMD ["node", "server.js"], so your binary becomes PID 1.

If you need a wrapper script for setup work such as rendering config or reading secrets, end it with exec "$@" or exec /app/server. exec replaces the shell process with your server, so the server takes over PID 1 and receives signals directly.

PID 1 ignores default signal actions

The kernel treats PID 1 differently from other processes. A signal with no registered handler is ignored instead of triggering its default action, which for SIGTERM is termination. Runtimes such as Node.js and Python do not install a SIGTERM handler by default. As PID 1, they ignore SIGTERM unless your code registers a handler.

Use a minimal init

A small init such as tini, set with ENTRYPOINT ["tini", "--", "/app/server"], runs as PID 1 and forwards signals to its child. It also reaps zombie processes. This is the right choice for containers that spawn subprocesses.

By default, tini forwards signals only to its direct child. Its -g flag sends them to the child's whole process group instead. If a container runs several long-lived processes, it needs a real supervisor that coordinates shutdown across all of them. Splitting them into separate containers is often the cleaner fix.

FROM node:20-alpine

# tini: a minimal init that runs as PID 1, forwards signals to its child
# and reaps zombie processes. Alpine installs it at /sbin/tini.
RUN apk add --no-cache tini

WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY . .

# Wrapper script for setup work. Its contents:
#
#   #!/bin/sh
#   set -e
#   # Setup: load a mounted secret into the environment
#   if [ -f /run/secrets/db_password ]; then
#     export DB_PASSWORD="$(cat /run/secrets/db_password)"
#   fi
#   # Replace this shell with the server so the server, not sh,
#   # becomes tini's direct child and receives SIGTERM.
#   exec "$@"
COPY docker-entrypoint.sh /usr/local/bin/docker-entrypoint.sh
RUN chmod +x /usr/local/bin/docker-entrypoint.sh

# Exec form (JSON array): no implicit /bin/sh -c, so tini is PID 1.
# Add "-g" after /sbin/tini to signal the child's whole process group.
ENTRYPOINT ["/sbin/tini", "--", "/usr/local/bin/docker-entrypoint.sh"]

# Also exec form. Node.js installs no SIGTERM handler by default, so
# server.js must call process.on('SIGTERM', ...) to drain and exit cleanly.
CMD ["node", "server.js"]

A Dockerfile that uses exec-form ENTRYPOINT and CMD, runs tini as PID 1, and calls a wrapper script that ends with exec "$@", so SIGTERM reaches the Node.js server.

How long should terminationGracePeriodSeconds be?

Treat the grace period as a sum, not a guess. The countdown starts when the pod is marked for deletion, and the preStop hook runs inside it. Size it as preStop delay + longest drain you will tolerate + cleanup + safety margin. When the timer expires, the kubelet sends SIGKILL regardless of what the process is doing.

Illustrative estimate. Assume these inputs: endpoint removal takes about 10s to reach every load balancer and proxy, the slowest legitimate request takes 20s, and closing connections plus flushing buffers takes 5s. That adds up to 35s. A 10s margin brings it to 45s. The default of 30s would kill this pod during cleanup.

Workload type Main sizing driver Typical direction vs. 30s default
Stateless HTTP API, short requests Endpoint propagation delay Shorter or equal
Long-polling / streaming / WebSockets Acceptable connection cutoff Longer, or reconnect on client side
Queue consumer Max time to finish or nack one message Depends on job length
Batch or report generation Checkpoint interval Longer, or checkpoint instead

Longer is not free. A preStop sleep adds its full duration to every pod replacement during rollouts, scale-downs and node drains. A pod that ignores SIGTERM holds its slot for the whole grace period, so node drains and autoscaler scale-downs run slower.

Measure your propagation delay and request latency, then set the numbers from those results. That keeps deploys from taking longer than they need to. It also means a SIGKILL during shutdown points to a real regression in your drain logic, not to a timeout someone guessed at.

When should you avoid relying on graceful shutdown, and what are the alternatives?

Some failures cannot be fixed by drain logic, however careful it is. A request that runs longer than the grace period gets cut off. A SIGKILL gives the process no chance to run code, whether it comes from the kubelet after the deadline, the OOM killer or a crashed node. Cloud preemption, spot reclaims and some autoscaler scale-downs can also allow less time than terminationGracePeriodSeconds. In those cases the node is removed on the infrastructure's schedule, not the pod's.

Platform settings change how much load lands on the pods that stay. maxUnavailable and maxSurge control how many pods terminate at once. A high maxUnavailable can overload the remaining replicas while old pods are still draining. PodDisruptionBudgets limit only voluntary disruptions, such as drains that go through the eviction API. They do nothing against node failures or preemption.

Sidecar and mesh proxies run their own shutdown sequence. If the proxy exits on SIGTERM before the app finishes, in-flight responses and outbound calls from the draining app fail. If the proxy waits on idle keep-alive connections, it can hold the pod open until SIGKILL. The proxy's drain window should start before the app stops accepting traffic and end after the app has finished.

Failure that shutdown can't prevent Complementary approach Why it helps
Connection reset mid-request Client retries with idempotency keys The retry lands on a healthy pod without double-applying writes
Consumer killed before ack Idempotent handling of redelivered messages The broker redelivers and the second run is harmless
Job longer than the grace period Checkpointing The replacement resumes instead of restarting
Long-lived connections Connection-level draining at proxies Clients reconnect elsewhere before the pod exits

Each of these approaches assumes that failures are rare. Graceful shutdown keeps them rare, so you need both, not one in place of the other.

How do you verify that graceful shutdown actually works?

The only reliable evidence is a test that reproduces the conditions under which requests get dropped. Reading the manifest and concluding that the hook "should" work isn't enough.

Run load while you terminate pods

Point a load generator at the service through the same path real clients use: the ingress or load balancer, not a port-forward. Keep the request rate constant so that any error spike is caused by termination and not by changes in load.

While the load runs, trigger two scenarios:

  • Rollout. kubectl rollout restart deployment/<name> replaces pods through the normal update path.
  • Node drain. kubectl drain <node> --ignore-daemonsets evicts pods and exercises a different sequence. Several pods can terminate at once, and node-level networking may be affected.

Count every non-2xx response and every connection reset or refused connection. Resets matter most because they usually mean the process exited while connections were still open, and they never appear in application logs.

Check hook failures and the timeline

kubectl describe pod and kubectl get events will show a failed or timed-out preStop hook (FailedPreStopHook) and pods that were killed at the end of the grace period.

Have the application log a timestamp at three points:

  1. preStop start
  2. SIGTERM receipt
  3. Process exit

Compare these timestamps against your budget. If the gap between SIGTERM and exit regularly comes close to terminationGracePeriodSeconds, the margin is too small, even when the test passes.

Make it a regression check

Shutdown behavior can break without any visible change. A new base image may swap the init process, or a sidecar may start exiting first. Run this test in CI or staging on every image or chart change, and fail the build on any dropped request.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes graceful shutdown sigterm, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.