Spring Boot Graceful Shutdown on Kubernetes: preStop and Draining

What happens when Kubernetes terminates a pod

When a pod is deleted, whether by a rolling update, a scale-down, or a node drain, the API server sets a deletion timestamp on the pod object and kubectl starts reporting it as Terminating. From that moment, two independent chains of work begin.

The kubelet's chain

The kubelet on the pod's node runs the container's preStop hook, if one is defined. When the hook returns, it sends SIGTERM to the main process (PID 1) in each container. It then waits for the containers to exit, up to terminationGracePeriodSeconds (30 seconds by default). That countdown starts when termination begins, so time spent in preStop is part of the same budget. Any container still running when the budget expires receives SIGKILL.

The networking chain

At the same time, the endpoints controller removes the pod from the Service's endpoints. Each component that routes traffic learns about that change on its own schedule:

• kube-proxy on every node rewrites its forwarding rules.
• Ingress controllers reload their upstream lists.
• External load balancers deregister the target after their own health-check or sync interval.

Each of these steps watches the API and reacts asynchronously. Nothing in Kubernetes makes the kubelet wait for them before sending SIGTERM. As a result, a process that exits immediately on SIGTERM can still be receiving new connections that are already in flight from stale routing tables.

What graceful shutdown means

Graceful shutdown means two things: stop accepting new traffic, then finish in-flight work (HTTP requests, consumed messages, open transactions) before the process exits. Any team running request-serving or message-consuming Spring Boot services that are regularly rolled, scaled down, or drained needs it. Without it, every deployment produces a small burst of errors.

Copy      pod deleted: rolling update / scale-down / node drain
                              |
                              v
        +----------------------------------------------+
        | API server sets deletionTimestamp;           |
        | pod reported as Terminating                  |
        +----------------------------------------------+
             |                                 |
             | kubelet chain                   | networking (concurrent)
             v                                 v
+--------------------------+   +-------------------------------+
| 1. run preStop hook      |   | endpoints controller removes  |
+--------------------------+   | pod from Service endpoints    |
             |                 +-------------------------------+
             v                        |            |          |
+--------------------------+          v            v          v
| 2. SIGTERM -> PID 1      |    +----------+ +---------+ +---------+
+--------------------------+    |kube-proxy| | ingress | | ext. LB |
             |                  | rewrites | | reloads | | deregs  |
             v                  | rules    | | upstream| | target  |
+--------------------------+    +----------+ +---------+ +---------+
| 3. wait for exit, up to  |     each on its own schedule; stale
| terminationGracePeriod-  |     routes can still send new requests
| Seconds (default 30s)    |
+--------------------------+     budget starts at Terminating;
             |                   preStop time counts against it
             v
+--------------------------+     nothing links the two chains:
| 4. SIGKILL if still up   |     SIGTERM does not wait for routing
+--------------------------+

Pod termination runs two independent chains: the kubelet's preStop, SIGTERM, grace period and SIGKILL sequence, and asynchronous endpoint removal propagating to kube-proxy, ingress controllers and external load balancers on their own schedules.

Why server.shutdown=graceful alone still drops requests

Kubernetes does not wait for the network to forget about a pod before telling the pod to stop. When a pod is marked for deletion, two things start in parallel: the kubelet sends SIGTERM to the container, and the control plane removes the pod's address from the Service's EndpointSlices. That removal then has to propagate to every component that routes traffic, including kube-proxy rules on each node, ingress controllers, and service mesh sidecars. Each one watches the API server and updates on its own schedule.

With server.shutdown=graceful, Spring reacts to SIGTERM immediately. The embedded server stops accepting new work and lets in-flight requests finish. Depending on the server, a new connection is either refused at the network layer or accepted and answered with a 503. Meanwhile, a proxy that has not yet processed the endpoint update still considers the pod healthy and keeps sending requests to it. For that window, clients see connection refused, connection reset, or 503 errors from a pod that is technically shutting down correctly.

This matters because pods are terminated far more often than most teams assume. A rolling update terminates every replica. A scale-down by the Horizontal Pod Autoscaler terminates pods during normal traffic, not during a maintenance window. A node drain for an upgrade evicts everything on the node at once. Each of these events produces a small burst of errors.

Load tests often miss the problem. An ingress or mesh that retries failed upstream requests can hide these errors from the client, so dashboards look clean. However, a retry after a reset is only safe when the backend did not already process the request. If the backend processed a POST before the connection dropped, the retry can create a duplicate order or payment. Whether a retry happens depends on the proxy's configuration, so check it rather than assuming either way.

Configuring Spring Boot for graceful shutdown

Two properties do the work. server.shutdown: graceful makes the embedded server stop accepting new connections on SIGTERM and wait for in-flight requests to finish. spring.lifecycle.timeout-per-shutdown-phase caps how long each lifecycle phase, including that wait, may take before Spring moves on. The default is short, so look it up for your version and set it deliberately. It should exceed your slowest legitimate request and still fit inside the pod's grace period.

Readiness during shutdown

With Actuator health probes enabled, /actuator/health/liveness and /actuator/health/readiness are exposed as separate health groups. When shutdown begins, Spring sets the readiness state to REFUSING_TRAFFIC, and the readiness probe starts failing. This adds little. Deleting the pod already starts endpoint removal, so the probe failure only confirms what the control plane is already doing. Liveness is different and must stay healthy throughout the drain. If liveness fails, the kubelet restarts the container and cuts off the requests you are trying to finish.

What the drain does not cover

Graceful server shutdown only covers HTTP. Other components need their own settings:

• Message listener containers should stop consuming and finish or nack the current message.
• Scheduled tasks need spring.task.scheduling.shutdown.await-termination and an await period.
• Async executors need the matching spring.task.execution.shutdown settings.
• Outbound clients and connection pools must close after the work that uses them finishes, not before.

Shutdown order matters. If the database pool closes while a listener is still processing a message, the drain fails even though HTTP shut down cleanly.

Copy# application.yml
server:
  # On SIGTERM, stop accepting new connections and wait for in-flight requests
  shutdown: graceful

spring:
  lifecycle:
    # Max time per shutdown phase, including the HTTP drain. Set it explicitly:
    # longer than the slowest legitimate request, but preStop sleep plus all
    # phases must still fit inside terminationGracePeriodSeconds (e.g. 60s).
    timeout-per-shutdown-phase: 30s
  task:
    scheduling:
      shutdown:
        # Let a running @Scheduled task finish instead of being interrupted
        await-termination: true
        await-termination-period: 20s
    execution:
      shutdown:
        # Let queued and running @Async work finish before the context closes
        await-termination: true
        await-termination-period: 20s

management:
  endpoints:
    web:
      exposure:
        include: health
  endpoint:
    health:
      probes:
        # Exposes /actuator/health/liveness and /actuator/health/readiness.
        # Readiness reports REFUSING_TRAFFIC once shutdown starts. Liveness
        # must stay UP during the drain, so do not add checks of shared
        # dependencies (database, broker) to the liveness group.
        enabled: true

An application.yml that enables graceful HTTP shutdown, sets an explicit per-phase timeout, gives scheduled and async executors time to finish, and exposes separate liveness and readiness probes.

Adding a preStop hook and making SIGTERM reach the JVM

The pod spec needs two additions. First, a preStop hook with an exec action such as command: ["sleep", "10"]. Kubernetes runs this hook before it sends SIGTERM, so the JVM keeps serving while endpoint removal propagates to kube-proxy, ingress controllers, and any service mesh sidecars. Those components learn about the terminating pod asynchronously, and the sleep covers that window. Second, set terminationGracePeriodSeconds explicitly. The preStop duration counts against this deadline, so the grace period must cover both the sleep and Spring's drain.

The exec hook only works if the image contains a sleep binary. Distroless and scratch images usually lack one. On those images the hook fails, and Kubernetes proceeds straight to SIGTERM, which removes the protection the hook was meant to add. Newer Kubernetes releases provide a built-in sleep action for lifecycle hooks that needs no binary in the image. Confirm that your cluster version supports it, and that it is enabled, before you rely on it.

The hook buys time, but the JVM still has to receive SIGTERM. Use the exec form of ENTRYPOINT, for example ENTRYPOINT ["java", "-jar", "/app/app.jar"], so Java runs as PID 1. The shell form wraps the command in /bin/sh -c, and that shell becomes PID 1. Many shells do not forward SIGTERM to their children. If you need a wrapper script to assemble flags, end it with exec java ... so the shell replaces itself with the JVM. You can also run a minimal init process that forwards signals. Without one of these, Spring never sees SIGTERM. The kubelet sends SIGKILL at the deadline, and in-flight requests are dropped.

CopyapiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders-api
  template:
    metadata:
      labels:
        app: orders-api
    spec:
      # Budget: 10s preStop sleep + 20s Spring drain + 15s margin.
      # The preStop duration counts against this deadline.
      terminationGracePeriodSeconds: 45
      containers:
        - name: app
          # Image must use exec-form ENTRYPOINT, e.g.
          # ENTRYPOINT ["java", "-jar", "/app/app.jar"],
          # or a wrapper script ending in `exec java ...`,
          # so SIGTERM reaches the JVM.
          image: registry.example.com/orders-api:1.0.0
          ports:
            - containerPort: 8080
          env:
            - name: SERVER_SHUTDOWN
              value: graceful
            - name: SPRING_LIFECYCLE_TIMEOUTPERSHUTDOWNPHASE
              value: 20s
          lifecycle:
            preStop:
              # Keep serving while endpoint removal propagates.
              # Requires a `sleep` binary: fails on distroless/scratch.
              exec:
                command: ["sleep", "10"]
              # Built-in alternative on newer clusters, no binary needed.
              # Confirm your version supports and enables it first:
              # sleep:
              #   seconds: 10

A Deployment with an exec-based preStop sleep, an explicit terminationGracePeriodSeconds that covers both the sleep and Spring's drain timeout, and notes on the built-in sleep action and the exec-form ENTRYPOINT the image needs.

Budgeting timeouts inside terminationGracePeriodSeconds

The grace period is a single clock, and it starts when the kubelet begins terminating the pod, not when the JVM receives SIGTERM. The preStop hook runs on that same clock. Whatever time the hook uses is time Spring no longer has for draining.

The budget therefore has three parts:

terminationGracePeriodSeconds ≥ preStop sleep + spring.lifecycle.timeout-per-shutdown-phase + cleanup margin

The cleanup margin covers work that happens after the web server stops: closing connection pools, flushing message producers, and running shutdown hooks.

Illustrative estimate (assumed numbers, not a benchmark)

When a timeout expires, each layer behaves differently. When Spring's phase timeout elapses, Spring stops waiting and continues shutting down, which abandons any requests still running. At the grace-period deadline, Kubernetes sends SIGKILL, and nothing after that point runs. Non-idempotent work, such as a payment write followed by an unsent event, can be left half-done, so size the budget for your slowest legitimate request rather than the average one.

Do not copy the 10-second figure. Measure how long your ingress, service mesh, or cloud load balancer takes to stop routing to a terminating pod, then add headroom.

Verifying zero-downtime shutdown

Run the test against the Service from outside the cluster, or through the same ingress your users hit. A load generator running in a separate pod inside the cluster can bypass the ingress, so it will not see the routing-lag problem you just budgeted for. Keep the request rate steady and low enough that the pods are not saturated. You want any errors to come from shutdown, not from overload.

Before you start, disable retries in the ingress, service mesh, or client for the duration of the test. A proxy that silently retries a reset connection on another pod will report success even when your shutdown is broken. The failure then shows up later, on non-idempotent requests that cannot safely be retried.

With load running, trigger kubectl rollout restart deployment/<name> and watch three things:

• Error rate. Watch the load generator's count of 5xx responses, connection resets, and timeouts. The target is zero, not "low."
• Spring's shutdown log lines. Each terminating pod should log that graceful shutdown has started and, later, that it has completed. If the completion line is missing, the process was killed before draining finished.
• Termination timing and exit codes. Old pods are deleted once they terminate, so run kubectl get events --watch during the rollout, or kubectl describe pod while a pod is still terminating. An exit code of 137 means the pod received SIGKILL, which tells you the budget in terminationGracePeriodSeconds was exceeded. A JVM that shuts down cleanly after SIGTERM typically exits with 143.

Add this check to CI or to a pre-release script. Running it on demand takes minutes. The alternative is correlating intermittent 502s with deploy timestamps after each release, which can cost days.

Copy#!/usr/bin/env bash
# Zero-downtime shutdown check. Disable ingress/mesh/client retries first, or errors get masked.
set -euo pipefail
NS=shop; DEPLOY=orders-api; SELECTOR=app=orders-api
URL=   # hit the public ingress, not the pod network
LOG=$(mktemp)
trap 'kill $(jobs -p) 2>/dev/null || true' EXIT

# Steady, low-rate load (~10 req/s). curl prints 000 for resets and timeouts.
( while true; do
    curl -s -o /dev/null -w '%{http_code}\n' --max-time 5 "$URL" >> "$LOG" || true
    sleep 0.1
  done ) &

# Follow Spring's shutdown log lines from the pods that are about to be replaced.
for pod in $(kubectl -n "$NS" get pods -l "$SELECTOR" -o name); do
  kubectl -n "$NS" logs -f --prefix "$pod" | grep --line-buffered -i 'graceful shutdown' &
done

# Exit codes as pods terminate: 143 = clean exit after SIGTERM, 137 = SIGKILL (grace period exceeded).
kubectl -n "$NS" get pods -l "$SELECTOR" --watch \
  -o custom-columns='POD:.metadata.name,PHASE:.status.phase,EXIT:.status.containerStatuses[0].state.terminated.exitCode' &
# Kill and probe-failure events, with timestamps, while old pods still exist.
kubectl -n "$NS" get events --watch --field-selector involvedObject.kind=Pod &

sleep 10   # baseline before the restart
kubectl -n "$NS" rollout restart "deployment/$DEPLOY"
kubectl -n "$NS" rollout status "deployment/$DEPLOY" --timeout=10m
sleep 60   # let the last old pod finish draining

total=$(wc -l < "$LOG")
failed=$(grep -cv '^2' "$LOG" || true)
echo "requests=$total failed=$failed"
[ "$failed" -eq 0 ] || { echo 'FAIL: errors during rollout'; exit 1; }
echo 'PASS: zero errors during rollout'

A repeatable rollout test that sends steady traffic through the ingress, follows Spring's graceful-shutdown log lines and pod exit codes during kubectl rollout restart, and fails if any request returns a non-2xx status, a reset, or a timeout.

Trade-offs, when to avoid it, and alternatives

Every preStop sleep is paid on every pod termination, not only during deploys. Assume 10 replicas rolled one at a time and a 10-second sleep. Each rollout then gains roughly 100 seconds (illustrative estimate). Surge settings change the multiplier, and node drains pay the same cost per pod.

A larger terminationGracePeriodSeconds also slows node maintenance and autoscaler scale-down. A shutdown that hangs holds the pod in Terminating for the full period, and the node can't be reclaimed during that time.

The mechanism has hard limits:

• Planned termination only. Graceful shutdown covers deploys, drains and scale-down. Crashes, OOM kills and node loss bypass it entirely.
• Long-lived connections. WebSockets, server-sent streams, long polls and pooled keep-alive connections can outlast any drain window you would accept.
• Little benefit without live traffic. Skip a long preStop delay for batch jobs, queue consumers and services that receive no live requests. Endpoint propagation lag has no in-flight requests to break, so the sleep only adds delay.

These approaches complement each other. Retries handle the failures that shutdown can't prevent. Graceful shutdown keeps retries from firing on every deploy. The setup is a few properties and a few lines of manifest, which costs far less than one afternoon tracing intermittent 502s.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to spring boot graceful shutdown kubernetes, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.