Kubernetes Graceful Shutdown for APIs: preStop Hooks and SIGTERM

What is Kubernetes graceful shutdown and how does pod termination work?

Graceful shutdown means an API pod stops receiving new requests, finishes the ones it already accepted, and exits before Kubernetes forcibly kills it. The goal is that clients see no connection resets or 5xx responses caused by the pod going away.

The termination sequence

When a pod is deleted, whether by a rollout, a scale-down, or a node drain, the API server sets a deletion timestamp and the pod is marked terminating. From that point, two independent processes react to the same event:

1. Endpoint removal. The endpoint for the pod is marked as no longer ready in its EndpointSlice. kube-proxy on every node, ingress controllers, and service mesh sidecars each watch that change and update their own routing tables. Each one does this on its own schedule.
2. Container shutdown. The kubelet on the pod's node runs any preStop hook. It then sends SIGTERM to the main process of each container. The terminationGracePeriodSeconds countdown starts at deletion and includes the time the hook takes. If containers are still running when it expires, they receive SIGKILL.

Nothing makes path 2 wait for path 1. Your process can receive SIGTERM and close its listener while some proxies are still sending it new traffic. Those requests are refused or reset. Draining in-flight requests does not help, because the failing requests arrive after the drain has started.

This is the usual cause of errors during rollouts, more often than a process that ignores SIGTERM. It matters for any request-serving workload behind a Service, ingress, or mesh. Teams that deploy often, autoscale, or drain nodes routinely feel it the most.

Copy                 +-----------------------------------------+
                 | Pod deleted: rollout, scale-down, drain |
                 | API server sets deletionTimestamp       |
                 | Pod marked Terminating                  |
                 +--------------------+--------------------+
                                      |
            +-------------------------+-------------------+
            v  in parallel, nothing orders these paths    v
+-----------------------+                     +-----------------------+
| PATH 1: endpoints     |                     | PATH 2: kubelet       |
| EndpointSlice marks   |                     | runs preStop hook     |
| endpoint not ready    |                     | grace clock running   |
+-----------+-----------+                     +-----------+-----------+
            | each watcher updates                        |
      +-----+--------+--------------+                     v
      v              v              v         +-----------------------+
+------------+ +------------+ +------------+  | SIGTERM to containers |
| kube-proxy | | ingress    | | mesh       |  | app closes listener   |
| per node   | | controller | | sidecars   |  | and drains in-flight  |
+-----+------+ +-----+------+ +-----+------+  +-----------+-----------+
      |              |              |                     |
      +--------------+--------------+                     v
                     v                        +-----------------------+
        +-------------------------+           | grace period expires: |
        | stale routes still send |           | SIGKILL if running    |
        | NEW requests to the pod |           +-----------------------+
        | after listener closes:  |
        | refused or reset        |
        +-------------------------+

After a pod is marked terminating, endpoint removal and container shutdown run concurrently, so proxies that have not yet updated their routes can send new requests to a pod that has already received SIGTERM and closed its listener.

Why use a preStop hook, and how long should the drain delay be?

A pod in Terminating state is removed from EndpointSlices at roughly the same moment the kubelet starts shutting it down. These two things happen in parallel, not in sequence. kube-proxy, ingress controllers, and mesh sidecars each learn about the endpoint change on their own schedule. If SIGTERM reaches your API immediately, it can stop accepting connections while some proxies are still sending it new requests. Clients then see resets and 502s.

A preStop hook closes that gap. The kubelet runs the hook to completion before sending SIGTERM, so a short sleep keeps the app serving normally while routing tables converge. The pod spec in the example below shows both forms.

Failed or hung hooks

The grace period countdown starts when termination begins, not when SIGTERM is sent. A hook that hangs or fails still uses up that time. A hook that runs past the grace period means the process is killed without ever receiving SIGTERM, so it gets no chance to drain.

Sizing the delay

No fixed sleep is provably safe. A preStop sleep is a heuristic. Don't copy 5, 10, or 15 seconds from another team's manifest. Measure how long your own ingress or mesh takes to stop routing to a deleted pod under realistic load. Add some margin to that number. Leave room inside the grace period for draining after SIGTERM.

Copy# Option A: exec sleep. Requires a sleep binary in the image.
apiVersion: v1
kind: Pod
metadata:
  name: orders-api-exec
spec:
  # Must cover the preStop delay plus in-app draining after SIGTERM.
  terminationGracePeriodSeconds: 45
  containers:
    - name: api
      image: registry.example.com/orders-api:1.4.2
      lifecycle:
        preStop:
          exec:
            # Fails if the image has no sleep binary (distroless, scratch).
            command: ["sleep", "8"]  # placeholder: measured convergence + margin
---
# Option B: built-in sleep action. No binary needed.
# The API server rejects this if your cluster version does not support it.
apiVersion: v1
kind: Pod
metadata:
  name: orders-api-sleep
spec:
  terminationGracePeriodSeconds: 45
  containers:
    - name: api
      image: registry.example.com/orders-api-distroless:1.4.2
      lifecycle:
        preStop:
          sleep:
            seconds: 8  # placeholder: measured convergence + margin

Two pod specs showing the same preStop drain delay, once as an exec sleep that needs a sleep binary and once as the built-in sleep action that works with distroless images, each inside a grace period that leaves room for draining after SIGTERM.

How does terminationGracePeriodSeconds work and how do you size it?

terminationGracePeriodSeconds is one countdown, not two. The clock starts when the pod is marked for deletion. The preStop hook runs first, and SIGTERM reaches your process only after the hook returns. When the budget runs out, the kubelet sends SIGKILL, whatever state the application is in. Every second your preStop sleep uses is a second your application no longer has for draining requests. The default is 30 seconds, and that is often too short once a drain delay is added.

Illustrative estimate, with these assumed inputs: a 10s preStop delay, a 15s p99 request duration, and a 5s margin for closing connection pools and flushing telemetry. That gives 10 + 15 + 5 = at least 30s. Use your own measured p99 rather than the average. The requests that get cut off are the slow ones.

Long-lived connections break this calculation. Streaming responses, long polling and WebSockets may never finish on their own. A larger grace period only delays the cut. For these workloads, close the connections from the server side during shutdown and make sure clients reconnect with backoff.

Long grace periods have costs. Each terminating pod keeps its resources reserved while it drains. That slows rolling deploys, node drains and autoscaler scale-down, and it means extra capacity stays allocated through every rollout.

CopyapiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders-api
  template:
    metadata:
      labels:
        app: orders-api
    spec:
      # One shared budget for preStop + app shutdown (illustrative estimate):
      #   10s preStop delay + 15s p99 request + 5s cleanup margin = 30s
      # Raise this if your measured p99 or the preStop delay grows.
      terminationGracePeriodSeconds: 30
      containers:
        - name: api
          image: registry.example.com/orders-api:1.4.2
          ports:
            - containerPort: 8080
          lifecycle:
            preStop:
              exec:
                # Hold off SIGTERM so endpoint removal can propagate.
                # These 10s count against the 30s above. The image must
                # contain a sleep binary.
                command: ["sleep", "10"]
          # After preStop returns, the app gets SIGTERM and has roughly
          # 30 - 10 = 20s to finish in-flight requests, close pools and
          # flush telemetry before the kubelet sends SIGKILL.
          readinessProbe:
            httpGet:
              path: /healthz/ready
              port: 8080
            periodSeconds: 5

A Deployment where a 10s preStop sleep and the application's own shutdown both fit inside a single 30s terminationGracePeriodSeconds budget, matching the illustrative estimate.

How should an API handle SIGTERM correctly?

The application's half of the contract is a fixed sequence. In pseudocode terms:

1. Catch SIGTERM and start a shutdown deadline. Make it shorter than whatever grace period remains after the preStop delay, so cleanup has time to run before SIGKILL.
2. Optionally fail readiness. This is a backstop for endpoint removal, not a substitute for the preStop delay.
3. Stop accepting new connections by closing the listener.
4. Drain in-flight requests until they finish or the deadline expires.
5. Shed connection reuse. On HTTP/1.1, send Connection: close so clients do not reuse a socket to a dying pod. HTTP/2 and gRPC servers should send GOAWAY, which tells clients to open new streams on a healthy pod.
6. Stop queue consumers. Stop pulling new messages first. Then finish or nack the in-flight ones. Redelivery will happen, so make handlers idempotent.
7. Clean up and exit. Flush logs and metrics, close database pools, and exit 0.

In Go, http.Server.Shutdown with a context deadline covers steps 3 to 5. It closes listeners and idle connections, then waits for active ones to go idle or for the context to expire. Hijacked connections such as WebSockets are not tracked, so register their cleanup with RegisterOnShutdown.

The deadline is a trade-off. If you set it close to the grace period, SIGKILL can arrive during the pool close or the log flush, and you lose telemetry exactly when you need it. If you set it too short, the server cuts off legitimate slow requests. Size it from your real p99-plus latency, and leave a few seconds of headroom.

Copy// Go (net/http): drain on SIGTERM with a deadline shorter than the remaining grace period
package main

import (
	"context"
	"errors"
	"log"
	"net/http"
	"os/signal"
	"syscall"
	"time"
)

func main() {
	srv := &http.Server{Addr: ":8080", Handler: http.DefaultServeMux}
	// Shutdown does not track hijacked conns (e.g. WebSockets); close them from this hook.
	srv.RegisterOnShutdown(func() { /* tell WebSocket hubs to send close frames */ })

	go func() {
		if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
			log.Fatalf("listen: %v", err)
		}
	}()

	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM)
	defer stop()
	<-ctx.Done() // SIGTERM arrives after the preStop hook finishes

	// Illustrative: 30s grace - 5s preStop = 25s left; drain 20s, keep ~5s for cleanup.
	drainCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
	defer cancel()
	// Closes listeners and idle conns, then waits for active requests to finish.
	if err := srv.Shutdown(drainCtx); err != nil {
		log.Printf("drain deadline hit, forcing close: %v", err)
		srv.Close()
	}
	// Next: stop queue consumers, flush logs/metrics, close DB pools.
	// Returning from main exits with status 0.
}

A minimal Go standard-library server that catches SIGTERM, drains in-flight requests with http.Server.Shutdown under a 20-second deadline, and force-closes any stragglers so cleanup can finish before SIGKILL.

Why does my container ignore SIGTERM? PID 1 and entrypoint pitfalls

If the API never logs that it received SIGTERM and the pod always runs out the full grace period, check what is actually running as PID 1 in the container.

Shell form vs. exec form

In a Dockerfile, ENTRYPOINT node server.js is shell form. The runtime wraps it as /bin/sh -c "node server.js", so the shell becomes PID 1 and your server runs as a child. Kubernetes sends SIGTERM only to PID 1, and many shells do not forward it to the child. Exec form, ENTRYPOINT ["node", "server.js"], starts the server directly as PID 1, so it receives the signal itself.

Wrapper scripts cause the same problem. If an entrypoint script exports variables and then runs the binary as a child, the script holds PID 1. End the script with exec "$@" or exec /app/server so the server replaces the shell process.

Why PID 1 is special

The kernel does not apply default signal actions to PID 1 in a PID namespace. A normal process with no SIGTERM handler is terminated. PID 1 with no handler ignores the signal. Your server must register a handler explicitly. If it doesn't, or if you need zombie reaping, run a minimal init such as tini, either baked into the image or added with docker run --init.

Verify locally

Start the container with docker run -d --name api <image>. Then run time docker stop api, which sends SIGTERM and escalates to SIGKILL after 10 seconds by default. Check the result with docker inspect --format '{{.State.ExitCode}}' api.

• Good result: the stop returns quickly and the exit code is 0.
• Signal ignored: the stop takes about 10 seconds and the exit code is 137 (128 + 9, SIGKILL). This means SIGTERM never reached a handler.

When should you avoid relying on these mechanisms, and what are the alternatives?

These mechanisms only cover a planned, orderly stop. An OOM kill, kernel panic, node loss or forced eviction skips the preStop hook and SIGTERM entirely, so in-flight requests fail regardless of how well the drain logic works. Every API still needs idempotent writes and clients that retry. Retries have their own costs. They hide shutdown bugs because a retried request shows up as a success on dashboards. They add load during exactly the moments the system is already shrinking. They are only safe for idempotent operations.

Failing readiness on SIGTERM is not the main protection. A terminating pod is removed from Service endpoints whether or not its probe fails. The readiness change mainly helps external load balancers or health checkers that probe the pod directly and do not watch the Kubernetes API.

In a service mesh, the order in which containers stop matters. If the proxy sidecar exits before the application finishes draining, requests fail even though the app handles SIGTERM correctly. Make the proxy wait for the app, either through mesh configuration or a preStop delay on the proxy container.

How do you verify graceful shutdown and save engineering time?

The mechanisms above only help if you can prove they work, and the proof is a repeatable test, not a quiet afternoon in production.

A rollout test you can repeat

Run a steady synthetic load against the Service from inside the cluster. A constant request rate makes failures easy to attribute. While it runs, trigger kubectl rollout restart deployment/<name> -n <namespace> and wait for kubectl rollout status to finish. Then compare the load tool's report with what the pods recorded:

• Responses: Count non-2xx responses, connection resets, and timeouts. The target is zero. A handful of 502s "during deploys" is a bug in the shutdown path, not background noise. Accepting it is how real regressions hide.

• Events: Check kubectl get events for the restarted pods. Look for Killing events and any FailedPreStopHook events.

• Logs: Confirm this order in each old pod's logs:

  1. preStop start
  2. SIGTERM received
  3. Listener closed
  4. In-flight requests drained
  5. Exit code 0

  Kubernetes does not emit an event when a preStop hook succeeds, so your application has to log each of these steps itself.

• Exit path: Make sure no pod reached the grace-period deadline and was killed with SIGKILL.

Why the setup cost pays back

The work is mostly one-time:

• A shared shutdown library for each language you run, so every service handles SIGTERM, readiness, and draining the same way.
• A base-image convention that fixes the entrypoint and PID 1 behavior.
• This rollout test as a CI or staging gate.

After that, a new service inherits correct behavior instead of rediscovering it. The recurring costs go away: incident triage for deploy-time error spikes, flaky alerts that teams learn to ignore, and long debates about whether clients should simply retry harder.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes graceful shutdown, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.