How does Kubernetes pod termination actually work?

When you delete a pod, or a rolling update replaces it, the API server does not stop the container. It sets a deletionTimestamp and a grace period on the pod object. Several independent components watch for that change and react on their own schedules.

Two tracks, no coordination

Track 1, on the node. The kubelet sees the deletion, runs the container's preStop hook if one exists, and sends SIGTERM to the container's main process once the hook returns. The grace period countdown starts when deletion begins and includes the time spent in preStop. When it expires, any remaining processes get SIGKILL.

Track 2, in the control plane and network. The EndpointSlice controller marks the pod's endpoint as terminating. kube-proxy on every node, ingress controllers, service mesh sidecars and cloud load balancers then pick up that change through their own watch loops, sync intervals or API calls.

No component waits for Track 2 to finish before Track 1 sends SIGTERM. If your process exits quickly, some proxies still route requests to an address with nothing listening. Those requests become connection refusals or resets, which the proxy reports as a 502 during deploy.

Endpoint conditions

EndpointSlices describe each endpoint with three conditions:

  • ready: the endpoint should receive normal traffic. It is set to false once the pod is terminating.
  • serving: it reflects readiness-probe status regardless of termination, so consumers can tell when a terminating pod can still handle requests.
  • terminating: the pod has a deletion timestamp.
              +------------------------------------+
              | API server: pod gets               |
              | deletionTimestamp + grace period   |
              +------------------------------------+
                     |                       |
        Track 1      v                       v      Track 2
  +-------------------------+   +---------------------------------+
  | kubelet (node)          |   | EndpointSlice controller        |
  | 1. run preStop hook     |   | endpoint conditions:            |
  | 2. SIGTERM after hook   |   |   ready       = false           |
  | 3. SIGKILL at grace end |   |   serving     = readiness probe |
  |  (grace includes hook)  |   |   terminating = true            |
  +-------------------------+   +---------------------------------+
               |                              |  watch / sync / API
               v                              v  (async, own timing)
  +-------------------------+   +---------------------------------+
  | process exits, stops    |   | kube-proxy | ingress | mesh     |
  | listening on pod IP     |   | cloud load balancers            |
  +-------------------------+   +---------------------------------+
               ^                              |
               |  requests still routed here  |
               +------------------------------+
         no listener -> refused / reset -> proxy returns 502

  Nothing makes Track 1 wait for Track 2 to finish before SIGTERM.

Pod deletion triggers two uncoordinated tracks, kubelet shutdown on the node and asynchronous endpoint propagation to proxies, so traffic can reach a pod that has already stopped listening.

Why doesn't a readiness probe prevent 502s during deploy?

The intuitive fix is to make the readiness probe fail during shutdown. That doesn't address the actual cause. Once a pod has a deletion timestamp, the control plane already marks its endpoint as not ready in the EndpointSlice, without waiting for any probe. Kubernetes detects that the pod is leaving almost immediately.

The failures come from propagation. The EndpointSlice update is only a change in the API server. Every consumer has to observe it and act on it:

  • kube-proxy on each node must rewrite its iptables or IPVS rules.
  • An ingress controller must refresh its upstream list.
  • A cloud load balancer must deregister the target, often on its own health-check schedule.

Each of these runs on its own watch, queue and reconcile loop. Meanwhile, the kubelet starts the shutdown sequence in parallel. Nothing makes the kubelet wait until every consumer has stopped routing to the pod. If the process exits or closes its listener first, requests already in flight to that IP fail with connection resets, which the proxy reports as 502s or 503s.

A failing readiness probe makes this worse. It adds periodSeconds × failureThreshold of delay before readiness changes, and it still feeds the same propagation path. You get extra latency and no ordering guarantee.

You need a deliberate fix if you run HTTP or gRPC services behind kube-proxy, an ingress controller or a cloud load balancer, and you see errors during rollouts, node drains or scale-down. Long-lived connections and high request rates make the window easier to hit, but every such service is exposed to it.

How does a preStop hook work?

A preStop hook is a handler declared under lifecycle.preStop on a container. The kubelet calls it when the container is being terminated through the API or by a management event, such as a pod deletion, a rolling update, an eviction during a node drain, or a liveness-probe failure. It does not run when a container exits on its own, for example when a process crashes or a Job's container completes.

The call is synchronous and blocking. The kubelet runs the handler and waits for it to finish, and only then sends SIGTERM to the container's main process. That ordering is the whole mechanism: any time spent inside the hook is time the application keeps running and serving normally while Track 2 catches up. The hook does not remove the pod from load balancing itself. It just holds back SIGTERM while the EndpointSlice change reaches the proxies.

Three properties matter in practice:

  • It runs inside the grace period. The hook does not get its own timer. If it runs until terminationGracePeriodSeconds is used up, the container is killed without a normal SIGTERM shutdown.
  • It is best-effort. If the handler fails, for example because the command does not exist in the image, the kubelet records a FailedPreStopHook event and continues with termination. A broken hook never blocks a rollout, which also means it can fail without anyone noticing.
  • It runs per container. Each container in the pod has its own hook. A delay on the application container does nothing for a sidecar that has no hook and exits on SIGTERM straight away.

Handlers come in a few forms. exec runs a command inside the container's filesystem. httpGet sends a request to an endpoint on the container, which lets the application itself decide when to return. The built-in sleep action, available on newer clusters, pauses without needing anything in the image. The next section covers how to choose between them.

Who should use a preStop hook?

A preStop delay suits workloads that receive traffic through something that learns about endpoint changes asynchronously:

  • Services behind a Kubernetes Service, an ingress controller or a cloud load balancer that serve HTTP, gRPC or other request/response traffic.
  • Workloads that are replaced often. Frequent deploys, horizontal autoscaling that scales down regularly, spot or preemptible nodes, and routine node drains for upgrades all trigger the same termination path and the same race.
  • Teams that cannot change every application. A hook is pod-spec configuration. Operators can add it to existing services, including third-party images, without touching application code, as long as the image or cluster supports the chosen handler.
  • Platform teams that own shared templates. Setting the hook once in a base Helm chart, Kustomize base or policy gives every service the same protection, and the delay can be adjusted to the cluster's networking without involving each service team.

A preStop hook does not replace SIGTERM handling in the application. It only buys time before shutdown starts. The application still has to drain correctly once SIGTERM arrives, as described later.

How do you implement a kubernetes preStop hook delay?

The most common implementation is an exec-form preStop hook that runs sleep for a few seconds. The kubelet runs the hook before it sends SIGTERM, so the main process keeps accepting and serving requests while endpoint slice updates reach kube-proxy, ingress controllers and external load balancers. Once the hook returns, SIGTERM arrives and normal shutdown begins. The sleep time counts against terminationGracePeriodSeconds, which section 4 covers.

The exec form needs a sleep binary inside the container image, because the hook runs in the container's filesystem. If the binary is missing, the hook fails. The kubelet records a failed-hook event and sends SIGTERM anyway, so the delay disappears without stopping the rollout. You may not notice until the 502s come back.

Distroless and scratch images usually have no shell or sleep. You have two options:

  • Delay in the application. When the process receives SIGTERM, it keeps serving for a configured interval and only then starts draining.
  • Built-in sleep action. Newer Kubernetes releases offer a native sleep handler for lifecycle hooks, which needs nothing in the image. Use it only if your cluster version and provider documentation confirm it is available and enabled.
Approach Image size Attack surface App/platform coupling
Exec sleep Needs a shell or coreutils layer Adds binaries an attacker could use None; the platform owns the delay
In-app delay No change No change High; each service must implement and configure it
Built-in sleep action No change No change None, but depends on cluster support

The in-app delay costs the most development effort. Every language and framework needs its own implementation, and that code has to be kept in step with how your infrastructure routes traffic.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      # Covers the preStop delay plus the app's own drain time after SIGTERM.
      terminationGracePeriodSeconds: 30
      containers:
        - name: web
          image: registry.example.com/web:1.4.2
          ports:
            - containerPort: 8080
          lifecycle:
            preStop:
              # Option A: exec form. Runs inside the container, so the image
              # must contain a sleep binary. No shell is needed because the
              # command is called directly. While this runs, the app keeps
              # serving and routing removes the pod from load balancing.
              # SIGTERM is sent only after the hook returns.
              exec:
                command: ["sleep", "5"]
              # Option B: built-in sleep action. It needs nothing in the image,
              # so it suits distroless or scratch images. Use it only if your
              # cluster version and provider docs confirm support. Replace the
              # exec block above with:
              # sleep:
              #   seconds: 5

A Deployment with an exec-form preStop sleep that delays SIGTERM while endpoints converge, plus the built-in sleep action as a commented alternative for images without a sleep binary.

How should you size terminationGracePeriodSeconds?

terminationGracePeriodSeconds is a single budget for the whole shutdown. The countdown starts when the pod is marked for deletion, so the preStop hook runs inside it. SIGTERM is sent only after preStop finishes, and the application's shutdown uses whatever time remains. When the period runs out, the kubelet sends SIGKILL, whether the process is still draining requests or not. The default is 30 seconds. A long preStop sleep therefore takes time away from the application's drain instead of adding to it.

Worked example (illustrative estimate)

Assumptions:

  • preStop delay: 10 s, enough time for endpoint removal to reach your load balancers and proxies
  • Longest expected in-flight request: 20 s, for example a slow report export
  • Safety margin: 5 s, to cover connection-pool teardown, log flushing and scheduling jitter

Total: 10 + 20 + 5 = 35 seconds. With the 30-second default, that request would be killed 5 seconds before it finished. If you raise only the preStop sleep to 15 seconds and leave the grace period at 35, the application gets 20 seconds, which leaves no margin at all.

What longer periods cost

Every pod in a rollout can take up to the full grace period to exit, so long periods slow deploys and node drains. A higher pod-spec value also doesn't guarantee your pods that time. Graceful node shutdown has its own time budget. Cluster autoscalers can apply their own cap when evicting pods during scale-down. Spot or preemptible reclamation gives a fixed notice set by the provider. Any of these can end termination early. Set the grace period close to what you actually measure, and fix requests that run for minutes instead of extending the grace period to fit them.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: report-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: report-api
  template:
    metadata:
      labels:
        app: report-api
    spec:
      # One budget for the whole shutdown, counted from deletion:
      #   10s preStop + 20s longest request + 5s margin = 35s
      # When it runs out, the kubelet sends SIGKILL.
      terminationGracePeriodSeconds: 35
      containers:
        - name: app
          image: registry.example.com/report-api:1.4.2
          ports:
            - containerPort: 8080
          lifecycle:
            preStop:
              exec:
                # Keep serving while endpoint removal reaches load balancers
                # and proxies. These 10s come out of the 35s, not on top of it.
                # The image must contain a sleep binary.
                command: ["sleep", "10"]
          # SIGTERM arrives only after preStop exits. The app then has
          # 35 - 10 = 25s to finish in-flight requests and exit.
          # Raising the sleep without raising the grace period cuts
          # into that window.

Deployment for the illustrative estimate: a 10-second preStop sleep inside a 35-second terminationGracePeriodSeconds, which leaves 25 seconds after SIGTERM for a 20-second request plus a 5-second margin.

What must the application do when it receives SIGTERM?

SIGTERM marks the start of the drain, and the process should run the shutdown steps in order:

  1. Stop accepting new connections. Close the listener so that any traffic still arriving from stale endpoints is refused quickly rather than accepted and then dropped.
  2. Bound or reject new long-running work. Refuse new uploads, streams and jobs that cannot finish within the remaining grace period.
  3. Finish in-flight requests under a deadline shorter than terminationGracePeriodSeconds.
  4. Close idle connections actively. HTTP/1.1 keep-alive connections stay open after the listener closes. Send Connection: close on the next response, or close idle connections, so clients reconnect to a healthy pod.
  5. Send GOAWAY on HTTP/2 and gRPC. GOAWAY tells clients to open new streams elsewhere while existing streams complete.
  6. Flush and exit. Commit offsets, flush buffered logs and metrics, then exit with status 0.

None of this runs if the application never receives the signal. Kubernetes sends SIGTERM to PID 1 in the container. With a shell-form entrypoint or a wrapper script, PID 1 is the shell. A shell usually does not forward SIGTERM to its child, so the app runs until SIGKILL. The kernel also gives PID 1 no default signal handling, so an unhandled SIGTERM does nothing. Have wrapper scripts exec the real binary, or run the process under a minimal init that forwards signals.

Client-side retries of idempotent requests on connection reset add a second layer of protection, but they do not replace server-side draining. Retries can hide shutdown bugs and multiply load on the remaining pods during a rollout. They can also duplicate non-idempotent operations, such as charging a payment twice.

// server.js: run with exec-form CMD ["node", "server.js"] so node is PID 1 and receives SIGTERM.
const http = require('http');

const DRAIN_DEADLINE_MS = 20_000; // keep below terminationGracePeriodSeconds (e.g. 30s)
let draining = false;

async function flushTelemetry() { /* placeholder: commit offsets, flush logs and metrics */ }

const server = http.createServer((req, res) => {
  if (draining) {
    // Tell keep-alive clients to drop this connection and reconnect to a healthy pod.
    res.setHeader('Connection', 'close');
    // Refuse new long-running work that cannot finish within the grace period.
    if (req.url.startsWith('/upload')) {
      res.statusCode = 503;
      return res.end('shutting down\n');
    }
  }
  res.end('ok\n');
});
server.listen(8080);

process.on('SIGTERM', () => {
  draining = true;
  // Close the listener. The callback runs once every open connection has closed.
  server.close(async () => {
    await flushTelemetry();
    process.exit(0);
  });
  // Close keep-alive connections that have no request in flight.
  server.closeIdleConnections();
  // Hard stop if in-flight requests overrun the drain deadline.
  setTimeout(() => {
    server.closeAllConnections();
    process.exit(1);
  }, DRAIN_DEADLINE_MS).unref();
});

A Node.js HTTP/1.1 server handles SIGTERM by closing its listener, sending Connection: close, rejecting new uploads, closing idle keep-alive connections and finishing in-flight requests under a deadline before it flushes and exits.

When is a preStop sleep the wrong tool, and what are the alternatives?

A fixed sleep is a guess. Nothing bounds how long it takes kube-proxy, ingress controllers and other EndpointSlice watchers to react. A loaded control plane or a slow controller can exceed any value you pick. If the delay is too short, you get residual 502s. If it is too long, every rollout, scale-down and node drain waits that long per pod, multiplied across replicas.

External load balancers and ingress controllers that target pod IPs directly add their own timing. They deregister a target based on their own health checks and deregistration or draining delay, not on the moment Kubernetes marks the endpoint terminating. Your preStop delay must cover the slowest consumer, and terminationGracePeriodSeconds must exceed that consumer's draining window. If the grace period is shorter, the pod is killed while the load balancer is still sending it traffic.

Service meshes add an ordering problem. If the sidecar proxy exits before the application, in-flight requests and the app's outbound calls fail. The sidecar must outlive the app, either through mesh drain settings or by declaring it as a native sidecar, which Kubernetes stops after the main containers.

Approach What it covers Cost
preStop sleep In-cluster propagation lag Fixed delay on every termination; still a guess
Application-level delay Same, plus app-controlled drain logic Code in every service
LB deregistration or draining External LB traffic to pod IPs Must be tuned to match the grace period
Client retries Residual errors from any cause Extra load; unsafe for non-idempotent calls
Mesh-managed draining Proxy-level connection drain Mesh complexity and correct sidecar ordering

When to avoid a preStop delay entirely

Skip the hook, or remove it, in these cases:

  • The pod receives no traffic through Services or load balancers. Queue consumers, cron-style workers, Jobs and controllers have no endpoints to propagate. A sleep only delays shutdown. They need correct SIGTERM handling, such as stopping message fetches and committing offsets, not a delay.
  • The application already implements the delay. Stacking an in-app delay on top of a preStop sleep doubles the wait and eats into the grace period with no benefit.
  • Clients bypass endpoint propagation. Clients that do their own load balancing, for example gRPC clients resolving a headless Service through DNS and holding long-lived connections, do not react to kube-proxy changes. A sleep does not move them. GOAWAY and client-side reconnection logic do.
  • The termination budget is fixed and short. On spot or preemptible nodes with a short provider notice, or under an autoscaler cap, a sleep can consume most of the window and leave too little for the drain itself.
  • Fast scale-down or fast rollouts matter more than a handful of errors. In development or ephemeral preview environments, the added time per pod may cost more than the errors it prevents.
  • The image has no sleep and the cluster has no built-in sleep action. An exec hook that fails gives a false sense of safety. Use an in-app delay instead.

How can a preStop hook save engineering time?

The savings come from moving a recurring, hard-to-diagnose problem into one piece of configuration:

  • One fix instead of one per service. A hook in a shared deployment template protects every service that uses it. The alternative, an in-app delay, has to be written, reviewed and tested once per language and framework, then kept consistent as infrastructure changes.
  • Fewer intermittent-error investigations. Deploy-time 502s are brief, scattered across pods and often missing from server logs, because the request never reached a running process. Each occurrence tends to trigger a round of log searching and dashboard comparisons. Removing the cause removes those investigations.
  • Deploys that teams trust. When rollouts reliably produce no errors, teams stop scheduling deploys for quiet hours, stop watching dashboards during each rollout and stop manually reverting changes that only looked broken because of the shutdown race.
  • Less pressure to add retries everywhere. Without a drain delay, the usual workaround is adding retries to every client. That spreads work across many teams and introduces duplicate-operation risk. A server-side fix keeps retries as a backstop, not the main defense.
  • Tuning in one place. When the cluster's networking changes, for example a new ingress controller or a different load balancer, the platform team adjusts the delay in the template rather than asking each service team to change code.

The time spent setting it up is mostly the verification below. Size the delay against your slowest traffic consumer, test it once per cluster and reuse the result.

Graceful termination verification checklist

Run these checks before you trust a configuration. A deliberate test takes minutes; tracing intermittent 502s from production logs after the fact usually takes much longer.

  • Generate continuous load through the real ingress path. Keep a steady request stream running through the full rollout, then repeat it during a kubectl drain of a node hosting the workload. A rollout and a drain evict pods through different paths, so passing one does not prove the other.
  • Count failures at the client, not the server. Record non-2xx responses, connection resets and timeouts on the load generator. A pod that was already gone when the request arrived logs nothing, so server-side metrics undercount exactly the errors you are hunting.
  • Watch endpoint propagation live. Run kubectl get endpointslices -w alongside the test. Confirm that a terminating pod's endpoint flips to not-ready before the process stops accepting connections.
  • Check how each container exited. Exit code 137 (SIGKILL) during a rollout means the grace period expired before shutdown finished. Capture this with kubectl get events -w or your log pipeline while the test runs, because a deleted pod's status disappears quickly.
  • Look for FailedPreStopHook events. Hooks are best-effort. A failing hook does not block termination, so a broken sleep can fail without any visible effect except returning 502s.
  • Confirm capacity holds during the drain. Verify that PodDisruptionBudgets, maxUnavailable and maxSurge leave enough ready replicas to absorb traffic while old pods sit in their preStop delay. Delayed pods still count against surge, but they serve nothing new.
  • Repeat per cluster. kube-proxy modes, ingress controllers and cloud load balancers propagate endpoint changes on different timelines. A delay that works in one cluster can be too short in another.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes prestop hook, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.