Kubernetes preStop Hook Checklist: Stop 502s on Pod Shutdown

What happens when a pod terminates, and why it causes 502s

Deleting a pod sets a deletion timestamp on the object in the API server. Two independent control loops react to that change, and nothing coordinates them.

• The kubelet on the node runs the container's preStop hook, if one is defined, then sends SIGTERM to the container. If the container is still running when the grace period ends, the kubelet sends SIGKILL.
• The EndpointSlice controller marks the pod's endpoint as no longer ready. Each consumer then has to notice that change on its own schedule: kube-proxy reprograms iptables or IPVS rules on every node, ingress controllers reload their upstream lists, service-mesh control planes push new configuration to their proxies, and cloud load balancers deregister the target and wait out their own health checks.

The kubelet does not wait for any of those consumers. If your application exits as soon as it receives SIGTERM, which is the correct default behavior, it can be gone while some nodes and proxies still route traffic to its IP. New connections are refused. In-flight requests are cut off. Clients see connection resets, and proxies report them as 502s or 503s.

A failing readiness probe does not fix this. The endpoint is already being removed, so marking it unready changes nothing. The delay is in propagation to the consumers, and a probe cannot speed that up.

This affects any team running HTTP, gRPC or WebSocket services behind a Service, Ingress or cloud load balancer. It happens on every rolling update, node drain, autoscaler scale-down and spot instance reclamation.

Copy                 kubectl delete pod / drain / scale-down
                                 |
                                 v
                 +-------------------------------+
                 | API server: deletionTimestamp |
                 +-------------------------------+
                  |   no coordination           |
                  v                             v
         +------------------+    +-----------------------------+
         | kubelet (node)   |    | EndpointSlice controller    |
         | 1. run preStop   |    | marks endpoint not ready    |
         | 2. send SIGTERM  |    +-----------------------------+
         | 3. SIGKILL after |                   | each consumer on
         |    grace period  |                   v its own schedule
         +------------------+    +-----------------------------+
                  |              | kube-proxy (every node)     |
                  v              | ingress controller reload   |
         +------------------+    | mesh proxy config push      |
         | app exits on     |    | cloud LB deregistration     |
         | SIGTERM promptly |    +-----------------------------+
         +------------------+                   |
                  |                             | still routing
                  v                             v to pod IP
         +-----------------------------------------------+
         | Gap: pod gone, traffic still arriving         |
         | -> resets, refused conns, 502 / 503           |
         +-----------------------------------------------+

Pod deletion triggers the kubelet's preStop/SIGTERM sequence and EndpointSlice removal in parallel, and because the kubelet never waits for kube-proxy, ingress, mesh and load balancers to catch up, a promptly exiting pod can still receive traffic that fails as resets or 502s.

How a preStop hook buys time: exec sleep vs the built-in sleep action

When a pod is marked for deletion, the kubelet runs the container's preStop hook before it sends SIGTERM. The endpoint controller removes the pod from Service endpoints at the same moment, and kube-proxy, ingress controllers and load balancers each pick up that change on their own schedule. A preStop delay keeps the process serving while that removal spreads. Nothing tells the kubelet when every proxy has actually stopped routing to the pod, so the delay is a heuristic. You pick a duration you expect to exceed propagation time, not a point where the two sides synchronize.

Option 1: exec hook running sleep

Set lifecycle.preStop.exec.command to something like ["sleep", "10"]. The kubelet runs this inside the container, so the image must contain a sleep binary. On distroless or scratch images it does not, and the exec fails at once. The kubelet records the failure as a pod event and moves straight on to SIGTERM. Rollouts still succeed, so the only symptom is that the 502s come back.

Option 2: built-in sleep action

Newer clusters support lifecycle.preStop.sleep.seconds. The kubelet performs the wait itself, so the image needs no binaries. Older API servers reject or ignore this field, so check your cluster version and feature-gate status before you rely on it.

With the exec variant, retest the hook every time the base image changes. A move to a slimmer image can remove sleep without any other visible change.

Copy# Option 1: exec hook. Requires a `sleep` binary inside the image.
apiVersion: v1
kind: Pod
metadata:
  name: web-exec-sleep
spec:
  # Must cover the preStop delay plus the app's own shutdown time.
  terminationGracePeriodSeconds: 30
  containers:
    - name: web
      image: registry.example.com/web:1.4.2
      lifecycle:
        preStop:
          exec:
            # Fails on distroless/scratch images. The kubelet logs an event
            # and sends SIGTERM immediately.
            command: ["sleep", "10"]
---
# Option 2: built-in sleep action. The kubelet waits, so the image needs no binaries.
# Confirm your cluster version and feature support first.
apiVersion: v1
kind: Pod
metadata:
  name: web-builtin-sleep
spec:
  terminationGracePeriodSeconds: 30
  containers:
    - name: web
      image: registry.example.com/web-distroless:1.4.2
      lifecycle:
        preStop:
          sleep:
            # Delay before SIGTERM so endpoint removal can propagate.
            seconds: 10

Two pod specs that delay SIGTERM by 10 seconds: one uses an exec hook that depends on a sleep binary in the image, and the other uses the kubelet's built-in sleep action, with the grace period set longer than the delay in both.

Budgeting terminationGracePeriodSeconds

terminationGracePeriodSeconds is a single countdown. It starts when the pod is marked for deletion, not when your app receives SIGTERM. The preStop hook runs inside that window, so every second spent sleeping is a second the application no longer has to drain. When the countdown expires, the kubelet sends SIGKILL whether or not in-flight requests have finished.

The rule: grace period ≥ preStop delay + app drain time + headroom.

These fields work together in the pod spec:

• spec.terminationGracePeriodSeconds: the total budget, set per pod (default 30).
• spec.containers[].lifecycle.preStop: the delay that consumes the first part of that budget.
• The app's own shutdown timeout, usually in config or code, which must fit in what remains.

Illustrative estimate (assumed numbers, not a benchmark):

• Assumed preStop sleep: 10 s
• Assumed longest in-flight request plus connection close: 20 s
• Assumed headroom: 5 s
• Required grace period: 10 + 20 + 5 = 35 s

At the default of 30 s, these inputs would kill the slowest requests mid-response.

Don't inflate the number "to be safe." Every terminating pod holds its slot for up to the full grace period, which slows rolling updates, node drains and autoscaler scale-down. Some paths also enforce their own caps. Autoscalers can limit how long they wait, and spot or preemptible reclamation gives a fixed notice set by the provider. Under those caps, a generous grace period still ends in SIGKILL.

Size the preStop delay the same way. Too short, and busy or large clusters still route traffic to the pod after it stops accepting connections. Too long, and every deploy waits on it.

CopyapiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      # Total budget, counted from deletion: 10 (preStop) + 20 (drain) + 5 (headroom)
      # Illustrative estimate from assumed inputs, not a measured value.
      terminationGracePeriodSeconds: 35
      containers:
        - name: web
          image: registry.example.com/web:1.4.2
          ports:
            - containerPort: 8080
          env:
            # The app reads this and stops waiting on in-flight requests after 20s.
            # It must fit in what is left after preStop: 35 - 10 = 25s.
            - name: SHUTDOWN_TIMEOUT_SECONDS
              value: "20"
          lifecycle:
            preStop:
              exec:
                # Keep serving while endpoints and load balancers stop routing here.
                # This sleep counts against terminationGracePeriodSeconds.
                # Requires a sleep binary in the image.
                command: ["sleep", "10"]
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            periodSeconds: 5

A Deployment whose 35-second grace period covers a 10-second preStop sleep, a 20-second app drain timeout and 5 seconds of headroom, using the illustrative estimate's assumed inputs.

What the application must do on SIGTERM

The preStop delay only keeps new traffic away. The process itself still has to shut down correctly once SIGTERM arrives. A correct handler does five things in order:

1. Stop accepting new connections.
2. Let in-flight requests finish.
3. Close keep-alive connections cleanly, with Connection: close on HTTP/1.1 responses or a GOAWAY frame on HTTP/2.
4. Flush pending work such as queued messages, buffered logs and open transactions.
5. Exit before the grace period ends.

Most frameworks provide this as one call. In Go it is http.Server.Shutdown, and in gRPC it is GracefulStop. Your job is to wire that call to the signal and set an internal timeout shorter than the remaining budget.

The PID 1 problem

The handler is useless if the signal never reaches your process. A shell-form CMD node server.js runs under /bin/sh -c, so the shell becomes PID 1 and receives SIGTERM. Many shells do not forward it to the child. The kernel also treats PID 1 specially: signals it has no handler for are ignored, so an unhandled SIGTERM does nothing until SIGKILL arrives. There are three ways to avoid this:

• Use exec form: CMD ["node", "server.js"].
• End any wrapper script with exec "$@" so the app replaces the shell.
• Run a small init process, such as tini or dumb-init, as PID 1 to forward signals and reap zombie processes.

Long-lived connections

Endpoint removal affects only new connections. Existing connections stay open. This includes WebSockets, gRPC streams and pooled HTTP keep-alive clients, which keep sending to the terminating pod until something closes the connection. Some graceful-shutdown helpers do not close these. Go's Shutdown, for example, ignores hijacked connections such as WebSockets. The server must therefore close them itself, using a WebSocket close frame or GOAWAY, and stagger the closes to avoid a reconnect storm. Clients must reconnect and retry idempotent requests, because some requests will still be in flight when the connection closes.

Copyconst http = require('node:http');

let shuttingDown = false;

const server = http.createServer((req, res) => {
  if (shuttingDown) {
    // Tell keep-alive clients to reconnect, which sends them to a live pod.
    res.setHeader('Connection', 'close');
  }
  res.end('ok\n');
});

server.listen(8080);

async function flushPendingWork() {
  // Placeholder: drain queues, flush buffered logs, commit open transactions.
}

process.on('SIGTERM', () => {
  shuttingDown = true;

  // Stop accepting new connections. The callback runs once every
  // existing connection has closed.
  server.close(async () => {
    await flushPendingWork();
    process.exit(0);
  });

  // Close keep-alive sockets that have no request in flight.
  server.closeIdleConnections();

  // Hard deadline. Assumes terminationGracePeriodSeconds: 30 and a 5s
  // preStop sleep, which leaves about 25s. Exit before SIGKILL arrives.
  setTimeout(() => {
    console.error('drain timeout, forcing exit');
    process.exit(1);
  }, 20_000).unref();
});

A Node.js SIGTERM handler that stops accepting connections, sends Connection: close on remaining keep-alive responses, flushes pending work and exits before an internal deadline set shorter than the grace period.

Layers outside kube-proxy: ingress, cloud load balancers and sidecars

Kube-proxy is only one consumer of EndpointSlice updates. Any component that routes straight to pod IPs keeps its own copy of the backend list and updates it on its own schedule. The preStop delay has to cover the slowest of these layers, because a request routed by a stale layer still reaches the terminating pod.

Ingress controllers that bypass the Service's virtual IP watch endpoints and then reload or reconfigure their proxy. That adds a lag on top of the API server's propagation time. Cloud load balancers that target pods directly add a second stage. A controller in the cluster has to deregister the target. Then the load balancer applies its own deregistration delay, the period during which it lets existing connections drain. If it falls back on health checks instead, removal takes roughly the check interval multiplied by the unhealthy threshold.

To align these values, size the preStop sleep to the longest time any layer takes to stop sending new requests. Keep the load balancer's deregistration delay no longer than the preStop sleep plus the application's drain time. Otherwise the load balancer holds connections open to a process that has already exited. The grace period must still exceed that total.

Sidecar proxies add their own shutdown ordering. If the mesh proxy receives SIGTERM and exits before the application finishes draining, in-flight and late-arriving requests fail, which brings back the original problem. The fix depends on the specific mesh and is out of scope here.

Alternatives and when to avoid a preStop sleep

A preStop sleep is one of several ways to cover the gap between a pod being marked for deletion and every proxy noticing. Each option fails differently.

Retries hide the symptom but do not fix it. A failed readiness probe only starts the same endpoint-removal path that deletion already started, so it cannot make that path faster.

When to skip or shorten the delay

• Batch workers and Jobs not behind a Service. No endpoint lists them, so there is nothing to wait for.
• Workloads with no inbound traffic, such as queue consumers. These need to finish or requeue in-flight work. They don't need to wait out propagation.
• Short preemption windows, as with spot or preemptible nodes. When the node will vanish on a fixed deadline, spend that time finishing in-flight work rather than idling in a sleep.
• An in-app delay that is already standardized in a shared framework. Adding a preStop sleep on top only stacks the two delays and uses up the grace period.

Verification, related settings and the final checklist

Prove it under load. Run a steady request stream against the service, trigger kubectl rollout restart deployment/<name>, then repeat with kubectl drain <node> --ignore-daemonsets. Count non-2xx responses and connection resets separately. Resets point to sockets closed under the client, while 502/503s point to routing to a dead backend. In a second terminal, run kubectl get endpointslices -l kubernetes.io/service-name=<svc> -w alongside kubectl get events -w and confirm that each pod leaves the EndpointSlice before its process exits. Repeat the test several times. Propagation delay varies with cluster load, so a single clean deploy proves little. Tune against the worst run, not the best one.

Check the related settings next:

• maxUnavailable and maxSurge control how much capacity disappears during a rollout.
• PodDisruptionBudgets stop a drain from evicting every replica at once.
• Readiness probes must fail until the app can actually serve, or new pods take traffic too early.

Tight capacity can also produce overload errors even when every pod drains perfectly.

A platform-level default costs a small, one-time effort: a shared pod template or policy, plus a load-test step in CI. It saves repeated investigations of intermittent deploy-time 502s. Per-service in-app draining needs ongoing review in every codebase.

Final checklist

• preStop hook present and working in this image
• terminationGracePeriodSeconds ≥ preStop + drain + headroom
• App handles SIGTERM as PID 1 or through an init process
• Keep-alive and long-lived connections closed
• External load balancer deregistration aligned
• Verified under load during rollout and drain

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes prestop hook, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.