How does Kubernetes pod termination actually work?

When you delete a pod, the API server does not remove it right away. It sets deletionTimestamp and a grace period on the pod object. Two independent control loops watch for that change and act on it at the same time.

The kubelet path

The kubelet on the pod's node sees the deletion and starts the grace period countdown. If a container defines a preStop hook, the kubelet runs it first. When the hook finishes, the kubelet sends SIGTERM to the container's main process. If the process is still running when terminationGracePeriodSeconds expires, the kubelet sends SIGKILL. The preStop hook's runtime counts against the same grace period. It does not get a separate budget.

The endpoint path

At the same moment, the EndpointSlice controller sees that the pod is terminating and marks its endpoint as not ready. That change then has to reach every component that routes traffic:

  • kube-proxy on every node rewrites its iptables or IPVS rules.
  • Ingress controllers reload their upstream lists.
  • Service mesh control planes push new configuration to their sidecars.
  • Cloud load balancers deregister the target, often with their own draining delay.

Each of these steps adds latency, and the latency varies with cluster size and load.

Why requests get dropped

Nothing connects the two paths. The kubelet does not wait for kube-proxy, an ingress controller, or a load balancer to confirm that traffic has stopped. If SIGTERM arrives and the application exits quickly, some routers are still sending it new connections. Those requests get refused or reset.

Readiness probes do not fix this. Endpoint removal is triggered by the deletion itself, not by a failed probe, so the endpoint is already marked not ready. Failing a probe faster changes nothing. The gap that drops requests is the time it takes for that change to propagate downstream.

+------------------------------------------------------------------+
| API server: sets deletionTimestamp + grace period on the pod     |
+------------------------------------------------------------------+
           |                                       |
           | watch                                 | watch
           v                                       v
+----------------------+            +------------------------------+
| KUBELET PATH         |            | ENDPOINT PATH                |
| grace timer starts   |            | EndpointSlice controller     |
+----------------------+            | marks endpoint not ready     |
           |                        +------------------------------+
           v                                       |
+----------------------+                           v
| 1. run preStop hook  |            +------------------------------+
|    (shares budget)   |            | propagate to every router:   |
+----------------------+            |  - kube-proxy: iptables/IPVS |
           |                        |  - ingress: upstream reload  |
           v                        |  - mesh: push to sidecars    |
+----------------------+            |  - cloud LB: deregister+drain|
| 2. SIGTERM to app    |            +------------------------------+
+----------------------+                           |
           |                                       v
           v                        traffic stops reaching pod
+----------------------+            only after variable latency
| 3. SIGKILL at grace  |
|    period expiry     |  <-- X -- no confirmation sent back
+----------------------+

After the API server sets deletionTimestamp, the kubelet runs preStop, SIGTERM and SIGKILL on its own timer while endpoint removal propagates separately to kube-proxy, ingress controllers, meshes and cloud load balancers, and nothing makes the kubelet wait for that propagation to finish.

What is a preStop hook and why use it for graceful shutdown?

A preStop hook is a lifecycle handler that the kubelet runs inside a container before sending it SIGTERM. It comes in two forms. An exec handler runs a command in the container's filesystem and namespaces. An httpGet handler sends a request to a path and port on the container. The kubelet waits for the hook to finish, and only then signals the main process. Hook runtime counts against terminationGracePeriodSeconds.

That ordering is why a short sleep works. While the hook sleeps, the application is untouched: it keeps accepting connections and serving requests normally. During that window, kube-proxy, ingress controllers, and external load balancers finish removing the pod from their routing tables. When SIGTERM finally arrives, little or no new traffic is still being sent to the pod. The sleep doesn't make the endpoint removal faster. It makes the process outlive the period when the removal is still in progress.

This matters for any workload behind a Service, an ingress, or an external load balancer that takes live traffic. The race appears whenever pods are deleted while healthy, so it shows up most during:

  • Rolling deployments
  • Node drains
  • Autoscaler scale-down

The example spec sets lifecycle.preStop.exec.command to run sleep for a fixed duration. A starting value of 5 to 15 seconds is an illustrative estimate. Tune it against how quickly your own load balancers converge.

Minimal and distroless images often have no sleep binary or shell, so an exec hook fails. Newer Kubernetes releases offer a native sleep action on lifecycle hooks. Confirm that your cluster version supports and enables it before relying on it. If it doesn't, copy a small static sleep binary into the image and call it directly.

apiVersion: v1
kind: Pod
metadata:
  name: web
  labels:
    app: web
spec:
  # Total budget for preStop plus the app's own SIGTERM shutdown.
  # Keep it comfortably larger than the preStop sleep.
  terminationGracePeriodSeconds: 45
  containers:
    - name: web
      image: registry.example.com/web:1.0.0
      ports:
        - containerPort: 8080
      lifecycle:
        preStop:
          exec:
            # Keep serving while kube-proxy, ingress and external LBs
            # remove this pod from routing. SIGTERM is sent only after
            # this command exits. 10s is an illustrative starting value
            # (try 5-15s) to tune against your own LB convergence time.
            command: ["sleep", "10"]
            # For distroless or minimal images with no sleep binary,
            # copy a small static sleep binary into the image and call
            # it by absolute path instead, for example:
            # command: ["/bin/sleep", "10"]
          # Alternative: the native sleep action, with no binary needed.
          # Use it only if your cluster version supports and enables it,
          # and replace the exec block above, since a handler takes one action:
          # sleep:
          #   seconds: 10

A pod spec whose exec preStop hook sleeps for an illustrative 10 seconds so the container keeps serving while routing state converges, with notes on distroless images and the native sleep action.

How does a preStop hook work during termination?

The hook is a blocking step at the start of the kubelet path. The sequence for a single container looks like this:

  1. Deletion is recorded. The API server sets deletionTimestamp on the pod. The grace period clock starts, and the EndpointSlice controller begins removing the endpoint in parallel.
  2. The kubelet invokes the hook. For an exec handler, the command runs inside the container, as if you had run it with kubectl exec. It uses the container's own binaries, environment and filesystem. For an httpGet handler, the kubelet itself makes the HTTP request to the container's IP and the port you configure.
  3. The kubelet waits. SIGTERM is not sent until the hook returns. For an exec sleep, that means the full sleep duration. For an httpGet handler, it means until the application sends its response. The main process keeps running and serving traffic the whole time.
  4. SIGTERM is delivered. After the hook completes, the container runtime sends SIGTERM to PID 1 in the container. This happens whether the hook succeeded or failed. A failed hook produces a FailedPreStopHook event on the pod, but it does not stop termination.
  5. SIGKILL at the deadline. The hook and the application's shutdown share the same grace period. If the hook is still running when the period ends, or the app hasn't exited by then, the container is killed.

The hook does not observe the endpoint path, and it receives no signal when routing has converged. It works only because it uses time to cover propagation that Kubernetes doesn't track. That is why its duration has to be chosen and verified, as the following sections describe.

With multiple containers in a pod, each container's preStop hook runs for that container only. A sidecar proxy that exits before the application container can still cut traffic. If you run sidecars, check that their shutdown order and hooks fit the same budget.

Who should use a preStop hook for graceful shutdown?

Use a preStop delay if your workload meets all three of these conditions:

  • It receives routed traffic. It sits behind a Service, an ingress controller, a service mesh, or a cloud load balancer, and clients reach it through those layers rather than through a fixed address.
  • Its pods are replaced while healthy. Rolling deployments, node drains for upgrades or maintenance, and autoscaler scale-down all delete pods that are serving normally. Those are the deletions where the endpoint race drops requests.
  • Dropped requests matter. User-facing HTTP and gRPC APIs fit, as do internal services whose callers don't retry, and any endpoint that handles non-idempotent writes.

The groups that benefit most:

  • Application teams that deploy often. Each rollout replaces every pod, so the race is triggered on every replica on every release.
  • Platform teams that maintain base charts, templates or admission defaults. A preStop delay and a matching grace period, set once in a shared template, protect every service that adopts it.
  • Teams on autoscaled or frequently upgraded node pools. Scale-down and node rotation terminate pods without any application change, so the race occurs even during weeks with no deploys.

Workloads that don't need it are covered later in "When should you avoid or limit preStop delays?". Those include batch jobs, queue consumers and pods with no Service in front of them.

How should you budget terminationGracePeriodSeconds?

The grace period is a single clock shared by everything that happens during termination. It starts when the API server marks the pod for deletion, not when your application receives SIGTERM. The kubelet runs the preStop hook first and sends SIGTERM only after the hook returns. Every second spent in the hook is therefore subtracted from the time your process has to finish its work. When the clock runs out, the container runtime sends SIGKILL. Any request still in flight at that point is cut off.

Size the period from the work it has to cover:

terminationGracePeriodSeconds ≥ preStop delay + longest in-flight request + drain/cleanup margin

The cleanup margin covers closing connection pools, flushing buffered logs or metrics, and committing consumer offsets.

Illustrative estimate (assumed numbers, not a benchmark):

Component Assumed value Why it's there
preStop sleep 10s Time for endpoint removal to reach load balancers and proxies
Longest in-flight request 20s Requests accepted just before SIGTERM must complete
Drain/cleanup margin 5s Pool shutdown, flushes, final acknowledgements
Total 35s Exceeds the default of 30s

With these inputs, the default 30-second period would kill the process 5 seconds into cleanup, or earlier if a slow request runs long. Substitute your own measured p99 or timeout-bounded request duration for the 20s figure.

Don't simply raise the value to be safe. A longer grace period also means a hung pod that ignores SIGTERM occupies its slot longer, which slows rollouts and recovery. On spot or preemptible nodes, the provider's reclamation notice may be shorter than your grace period. In that case the node disappears before the budget is ever used.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders-api
  template:
    metadata:
      labels:
        app: orders-api
    spec:
      # Illustrative budget (assumed values, not measured):
      #   10s preStop sleep + 20s longest request + 5s cleanup = 35s
      # Rounded up to 40s for headroom; the default would be 30s.
      # This clock starts at deletion and covers preStop AND app shutdown.
      terminationGracePeriodSeconds: 40
      containers:
        - name: app
          image: registry.example.com/orders-api:1.4.2
          ports:
            - containerPort: 8080
          lifecycle:
            preStop:
              exec:
                # Hold off SIGTERM so endpoint removal reaches proxies
                # and load balancers. These 10s count against the 40s above.
                # Requires a sleep binary in the image.
                command: ["sleep", "10"]
          env:
            # App-side limits should fit inside what remains after preStop:
            # 40s - 10s = 30s for in-flight requests (20s) plus cleanup (5s).
            - name: REQUEST_TIMEOUT_SECONDS
              value: "20"
            - name: SHUTDOWN_CLEANUP_SECONDS
              value: "5"

A Deployment that sets terminationGracePeriodSeconds from the illustrative budget (10s preStop + 20s request + 5s cleanup, rounded up to 40s) and caps app-side timeouts so they fit in the time left after the preStop hook.

How do you implement SIGTERM handling in the application?

Once the preStop hook returns and SIGTERM arrives, the application owns the rest of the shutdown. It has four jobs. It must stop accepting new connections by closing the listener. It must let in-flight requests finish. It must close idle keep-alive connections so clients reconnect elsewhere. And it must stop pulling from queues or consumer groups, then acknowledge or release the messages it already holds. Most HTTP server libraries handle the first three in one graceful-shutdown call. Pass that call a deadline shorter than the time left in your grace budget, which is the grace period minus the preStop duration. If the deadline expires, force-close the remaining connections and exit yourself. That way the cleanup runs under your control rather than ending in SIGKILL.

Long-lived connections need explicit handling, because endpoint removal only affects new connections. A client holding an open keep-alive, HTTP/2, or gRPC connection keeps sending requests to the terminating pod.

  • HTTP/1.1: the server adds Connection: close to its responses.
  • HTTP/2 and gRPC: the server sends a GOAWAY frame, which tells the client to finish its current streams and open new ones on another connection.
  • WebSockets: there is no protocol-level drain. Your code must send a close frame, and clients must reconnect with backoff.

The PID 1 problem

SIGTERM only helps if your process receives it. With a shell-form entrypoint, /bin/sh runs as PID 1 and usually does not forward signals to its child. A wrapper script that launches the app as a child process has the same problem. A PID 1 process also gets no default signal handling from the kernel, so a SIGTERM it does not explicitly handle is simply ignored. The app then runs until SIGKILL. There are three fixes:

  • Use exec-form entrypoints.
  • End wrapper scripts with exec so the app replaces the shell.
  • Run a minimal init process that forwards signals and reaps zombie processes.
const http = require('node:http');

// Assumed budget: terminationGracePeriodSeconds 30 minus a 10s preStop sleep
// leaves about 20s. Stop well before that so we exit on our own, not via SIGKILL.
const SHUTDOWN_DEADLINE_MS = 15_000;

let shuttingDown = false;

const server = http.createServer((req, res) => {
  // Ask keep-alive clients to reconnect, which lands them on a pod still in Endpoints.
  if (shuttingDown) res.setHeader('Connection', 'close');
  res.end('ok\n');
});

server.listen(8080);

process.on('SIGTERM', () => {
  shuttingDown = true;

  // Fallback: if in-flight requests outlive the deadline, cut them and exit ourselves.
  const timer = setTimeout(() => {
    server.closeAllConnections();
    process.exit(1);
  }, SHUTDOWN_DEADLINE_MS);

  // Stop accepting new connections. The callback runs once every connection has closed.
  server.close((err) => {
    clearTimeout(timer);
    process.exit(err ? 1 : 0);
  });

  // Drop idle keep-alive sockets now. Busy ones close after their current response.
  server.closeIdleConnections();

  // Stop queue consumers here and ack or release any messages already held.
});

A Node.js HTTP server that handles SIGTERM. It closes the listener and idle keep-alive sockets, adds Connection: close to responses, and force-closes remaining connections at a deadline set below the remaining grace budget.

What are the alternatives to a preStop sleep?

The exec sleep used so far is the most common pattern, but it isn't the only way to hold a pod in service while endpoint removal propagates. The options differ in where the delay lives and what breaks when the same image runs outside a cluster.

Approach Image dependency App coupling Outside Kubernetes What it guarantees
exec preStop sleep Needs a sleep binary, which distroless images lack None No effect A fixed delay before SIGTERM
Native sleep action None None No effect Same as exec, but only on clusters that support it
HTTP preStop handler None App must serve the endpoint Endpoint is unused Delay plus app-defined drain logic, if the handler is correct
In-process delay after SIGTERM None Full Ctrl+C hangs for the delay unless it is configurable Delay counted inside the grace period
Readiness failure only None Low N/A Nothing about timing, because propagation is still asynchronous
Client retries only None Moves the work to callers N/A Nothing on the server side

A preStop delay keeps Kubernetes-specific behavior out of the binary. An in-process delay makes the binary portable to other orchestrators but needs a flag so local runs exit promptly.

Client retries belong in every system, but they are a poor substitute for draining correctly. They hide shutdown bugs because dropped requests never show up as errors. During rollouts they multiply load at the moment capacity is lowest. They also can't safely replay non-idempotent requests, such as a payment POST without an idempotency key.

When should you avoid or limit preStop delays?

A preStop sleep is paid on every termination, not just the ones that would have dropped traffic. Its cost scales with how many pods you replace and how many at a time. As an illustrative estimate: a 15-second sleep on a 100-replica Deployment rolled one pod at a time adds 25 minutes to the rollout before any shutdown work begins. Node drains and autoscaler scale-downs pay the same tax. If the sleep plus application shutdown exceeds a drain timeout or the autoscaler's own termination limit, the tooling can give up or force the pod down. You lose the graceful exit the sleep was meant to protect.

There is no universally correct value. A delay that is too short still drops requests. One that is too long consumes the grace budget, so in-flight work gets SIGKILLed when the period expires. Either way, the sleep is a guess about propagation time. A slow external load balancer or a busy API server under heavy churn can exceed any fixed number.

Hook failures are quiet. If the preStop command is missing, crashes or times out, the kubelet records an event and sends SIGTERM anyway. Unless you alert on those events, you won't notice the protection is gone.

Many termination paths never run preStop. These include node crashes, kernel OOM kills, hard node-pressure evictions and forced deletes with zero grace. Clients still need timeouts, bounded retries and idempotent handlers.

Finally, skip the delay where it buys nothing. Batch workers, queue consumers and pods that sit behind no Service receive no routed traffic, so there is no endpoint race to close.

How can a preStop hook save engineering time?

The endpoint race is expensive to debug, more than to fix. Its symptoms are intermittent 502s, 503s and connection resets that appear only during deploys, drains or scale-down. They often show up only in client or load balancer logs and disappear before anyone can reproduce them. Teams can spend a long time on the wrong suspects, such as the ingress controller, network policies or application bugs, before they connect the errors to pod termination. A correctly budgeted preStop hook and SIGTERM handler remove this class of error at its source, so that investigation doesn't happen at all.

The time savings come from a few concrete places:

  • Fewer deploy-time incidents and pages. If rollouts don't drop requests, on-call engineers stop triaging error spikes that line up with release windows.
  • Less defensive code in callers. Without clean draining, client teams add retries, extra timeouts and deduplication to absorb errors caused by the server's shutdown. Fixing termination once on the server is smaller than patching every caller, although callers still need basic retries for the failure paths preStop can't cover.
  • Deploys stop being events. When a rollout is known to be safe, teams can ship during working hours and in smaller batches. They no longer need to schedule releases for low-traffic windows or watch dashboards through each rollout.
  • Configure once, reuse everywhere. A platform team can put the preStop delay, the grace period and the init process into a shared base chart or template. Each new service then gets correct shutdown behavior without its owners having to rediscover the race.
  • Faster incident reviews. With a written, measured shutdown budget, a deploy-related error can be checked against known numbers instead of investigated from scratch.

Illustrative estimate (assumed inputs, not measured): assume a team ships 20 deploys a month, and one in four produces a short burst of errors that someone investigates for an assumed 2 hours. That is 5 investigations and about 10 engineer-hours a month spent on a known cause. The one-time cost to fix it is adding the hook, sizing the grace period, fixing PID 1 signal handling and running the verification tests described below. Under these assumptions, that cost is recovered within the first few months, and the fix applies to every later deploy. Substitute your own deploy frequency and incident history to see whether the same holds for you.

The tradeoff, covered above, is slower rollouts and drains. You are spending pipeline wall-clock time, which is cheap and predictable, to save engineer time on unpredictable incidents, which is expensive.

Treat every number in your shutdown budget as a hypothesis until a test confirms it. Run sustained load at realistic concurrency against the Service, then trigger a rolling restart with kubectl rollout restart and, separately, a node drain with kubectl drain. Count two things on the client side: non-2xx responses and connection resets. Resets matter because they indicate traffic reaching a process that had already closed its listener, and they often show up only in client logs, not in server metrics.

To find the real propagation window, run kubectl get endpointslices --watch alongside kubectl get events --watch and record timestamps. The gap between a pod's termination event and its endpoint being marked not ready or removed is your in-cluster delay. Repeat the test a few times, because this gap varies with control-plane load.

Then tune the controls that shape the window:

  • maxUnavailable and maxSurge decide how many pods drain at once. Lower values spread the load on remaining replicas but lengthen rollouts.
  • PodDisruptionBudgets stop a drain from evicting too many replicas at once. Without one, a drain can terminate several pods together.
  • External load balancer deregistration delay and health check intervals often take longer than in-cluster propagation. If the balancer needs several failed checks before it stops routing, your preStop sleep must cover that time as well.

Write the measured budget down, including each component and the reason for it. This turns future incident reviews into a comparison against a known baseline instead of another round of guessing. The cost is slower rollouts. Make that cost explicit in deployment SLOs and pipeline timeouts so the tradeoff is visible to the team.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes prestop hook graceful shutdown, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.