What is Spring Boot graceful shutdown, and why does server.shutdown=graceful alone leave gaps?

Graceful shutdown is not one property. It is a timeline that spans the orchestrator, the network layer, and every component inside the JVM that holds work in flight. The goal is that a pod stops receiving new work before it stops being able to finish old work, and that it finishes or hands back that work before the process exits.

server.shutdown=graceful covers one part of that timeline. When the application context begins closing, the embedded web server stops taking new work. Depending on the server, it either stops accepting connections or rejects new requests. It then waits for in-flight requests to complete, up to a bounded timeout. That is useful, but it operates entirely inside the pod. It has no influence on kube-proxy rules, ingress controllers, or load balancers that still list the pod as a valid endpoint. If those components keep routing traffic after the server has stopped accepting it, clients see connection resets or 502/503 responses. The property also does nothing for Kafka listener containers. They have their own lifecycle, and if they stop mid-batch without committing offsets, the next consumer to own those partitions reprocesses those records.

This matters most to backend and platform engineers running HTTP and Kafka workloads in the same service on Kubernetes. Pods terminate routinely during rolling deploys, autoscaler scale-down, and node drains. Each of these events is a chance to drop requests or duplicate message processing. The failures are intermittent and easy to misattribute to the network or the broker. They appear as bursts of 502/503 errors and duplicate side effects during every deploy.

Who should use this shutdown setup?

The full setup in this article, covering preStop delay, probe wiring, lifecycle timeouts, Kafka container tuning and a sized grace period, is aimed at teams with these characteristics:

  • Spring Boot services on Kubernetes that serve HTTP traffic through a Service, an ingress controller or an external load balancer. These are the services exposed to the endpoint-removal race.
  • Services that consume from Kafka with Spring Kafka listener containers, especially when processing has side effects such as database writes, outbound calls or produced messages, where reprocessing is visible.
  • Teams that deploy often or scale dynamically. Frequent rolling updates, Horizontal Pod Autoscaler scale-down, cluster autoscaler consolidation and routine node maintenance all terminate pods. The more often pods terminate, the more often the gaps show up.
  • Platform teams that own shared base images, Helm charts or deployment templates. Getting the entrypoint, probes, preStop hook and grace period right once in a template applies the fix to every service built on it.

It is less important for workloads that rarely terminate during active traffic, for run-to-completion Jobs that are not interrupted mid-run, and for internal services whose callers already retry and whose handlers are fully idempotent. Those services still benefit from correct signal handling and a sized grace period, but the preStop delay and Kafka tuning matter less.

How does the Kubernetes termination sequence work, and why do endpoint removal and SIGTERM race?

When you delete a pod or a rolling update replaces it, the API server sets a deletion timestamp and the pod shows as Terminating. Two independent processes then start at the same time.

On the node, the kubelet runs the container's preStop hook, if one is defined. When the hook returns, it sends SIGTERM to PID 1 in each container. In a Spring Boot app, SIGTERM triggers the JVM shutdown hook, which closes the application context. If the containers are still running when terminationGracePeriodSeconds expires, the kubelet sends SIGKILL. The countdown starts at deletion and includes the time spent in preStop.

In the control plane, the endpoints controller removes the pod from the Service's EndpointSlices. That change then has to reach every consumer of it: kube-proxy on each node rewriting its rules, ingress controllers reloading upstreams, and external load balancers running their own deregistration and health checks. Nothing makes the kubelet wait for this propagation. SIGTERM can arrive while traffic is still being routed to the pod. If the app stops accepting connections immediately, those requests fail.

Two other problems turn a graceful shutdown into an abrupt one. If an entrypoint shell script starts the JVM as a child process instead of using exec, the shell is PID 1. A shell does not forward SIGTERM to its child by default, so the JVM gets no signal and is killed outright when the grace period ends. Service mesh sidecars cause a similar failure. The proxy receives SIGTERM alongside the app and may exit first, cutting off in-flight requests and broker connections while Spring is still draining.

+----------------------------------------------------+
| Pod delete / rolling update: deletionTimestamp set |
| Pod = Terminating; grace period clock starts (t=0) |
+----------------------------------------------------+
        v  NODE (kubelet)              v  CONTROL PLANE
+-------------------------+  +-------------------------------+
| 1. preStop hook runs    |  | Endpoints controller removes  |
+-------------------------+  | pod from EndpointSlices       |
        |                    +-------------------------------+
        v                                    |  async, kubelet
+-------------------------+                  v  does not wait
| 2. SIGTERM -> PID 1     |  +-------------------------------+
|  shell w/o exec: JVM    |  | kube-proxy rules on each node |
|  never gets the signal  |  | ingress controller upstreams  |
+-------------------------+  | external LB deregistration    |
        |                    +-------------------------------+
        v                                    |
+-------------------------+   stale routes   |
| 3. JVM shutdown hook -> |<-----------------+
|    Spring context close |   requests can still arrive
+-------------------------+
| Mesh sidecar also gets  |
| SIGTERM, may exit first |
+-------------------------+
        v
+-------------------------+
| 4. Grace period ends:   |
|    SIGKILL if running   |
+-------------------------+

After a pod is marked Terminating, the kubelet's preStop, SIGTERM and SIGKILL sequence runs in parallel with asynchronous endpoint and load balancer deregistration, so traffic can still reach the pod while Spring is shutting down.

How do SmartLifecycle phases and spring.lifecycle.timeout-per-shutdown-phase control shutdown order?

Spring stops beans that implement SmartLifecycle in order of their phase. Beans with a higher phase stop first. Spring Boot's graceful web server shutdown and Spring Kafka's listener containers each register in their own phase, so they stop at different points in the shutdown sequence. spring.lifecycle.timeout-per-shutdown-phase caps how long Spring waits for each phase. It does not cap the whole shutdown. Because every phase gets its own timeout, the worst case is the sum of all the phases, and that sum has to fit inside terminationGracePeriodSeconds.

When a listener container stops, the consumer finishes processing the records from its current poll. It then commits pending offsets according to the ack mode, closes, and leaves the group, which triggers a rebalance. If the process is killed before that commit, the uncommitted records are delivered again to whichever consumer takes over the partition:

Ack mode When offsets commit Re-delivered if killed mid-processing
RECORD After each record The in-flight record
BATCH After the whole poll batch The entire current batch
MANUAL / MANUAL_IMMEDIATE When your code acknowledges Everything not yet acknowledged

The time a consumer needs to stop is bounded by:

consumer stop ≈ (max.poll.records × per-record time) + container shutdownTimeout

Illustrative estimate (assumed numbers, not a benchmark):

  • Inputs: max.poll.records = 500, 50 ms per record, shutdownTimeout = 10 s.
  • Consumer stop: 500 × 0.05 s = 25 s, plus 10 s, gives 35 s.
  • Web server phase: 20 s.
  • Worst-case total: 55 s.

With a 30 s grace period, the kubelet sends SIGKILL partway through the batch. Nothing in that batch is committed, so under BATCH mode all of it is processed again after the restart.

How does graceful shutdown work end to end?

The previous sections describe the pieces separately. When everything is configured, one pod termination runs in this order:

  1. Pod marked Terminating. The grace-period clock starts. In parallel, the control plane begins removing the pod from EndpointSlices, and that change starts propagating to kube-proxy, ingress controllers and load balancers.
  2. preStop sleep. The kubelet runs the preStop hook. The application is still fully running and still serving any requests that reach it over stale routes. The sleep exists only to give endpoint removal time to propagate before the app reacts.
  3. SIGTERM reaches the JVM. Because the entrypoint uses the exec form, the JVM is PID 1 and receives the signal directly. The JVM runs its shutdown hooks, and Spring Boot's hook starts closing the application context.
  4. Readiness flips to refusing traffic. As the context closes, the readiness endpoint stops reporting ready. By this point the pod should already be out of the endpoints. This acts as a backstop for anything still routing to it. Liveness keeps passing, so the kubelet does not restart the container.
  5. SmartLifecycle phases stop in order. The graceful web server phase stops taking new requests and waits for in-flight ones. Kafka listener containers finish their current batch, commit offsets according to the ack mode, and leave the consumer group, which lets the partitions be reassigned. Each phase is capped by spring.lifecycle.timeout-per-shutdown-phase.
  6. Executors and beans shut down. Task executors and schedulers configured to await termination finish their queued work. Then singleton beans are destroyed: connection pools close, producers flush, and clients disconnect.
  7. The process exits before SIGKILL. If the grace period was sized to cover steps 2 through 6, the JVM exits on its own and the kubelet never needs to kill it.

The central idea is ordering. New traffic stops arriving before the app stops accepting it. In-flight work finishes or is committed before the process exits. Each layer has a time budget that fits inside the one around it.

How do you implement graceful shutdown in Spring Boot and Kubernetes?

Step-by-step

  1. Application properties. Set server.shutdown=graceful so the web server stops accepting new connections and lets in-flight requests finish. Set spring.lifecycle.timeout-per-shutdown-phase to the longest drain you budgeted. Lower spring.kafka.consumer.max-poll-records so a single batch can finish inside that window. Raise the listener container's shutdownTimeout through ContainerProperties on the container factory, because the consumer thread only stops between polls.

  2. Entrypoint. Use the exec form (a JSON array) in the Dockerfile. With the shell form, /bin/sh runs as PID 1, may not forward SIGTERM, and the JVM never starts its shutdown hooks.

  3. Probes. Point readiness at /actuator/health/readiness and liveness at /actuator/health/liveness. When the context closes, readiness changes to refusing traffic, which removes the pod from endpoints. Liveness must keep passing during the drain. If it fails, the kubelet restarts the container in the middle of a request.

  4. preStop and grace period. A preStop sleep holds back SIGTERM until endpoint removal has propagated. terminationGracePeriodSeconds must cover the sleep plus every shutdown phase. Distroless images have no shell, so call the image's own sleep binary directly if one exists. If there is none, use the native sleep lifecycle action on clusters that support it, or add a small static sleep binary to the image.

  5. Your own threads. Configure drain behaviour for @Async executors, custom pools and scheduled tasks. For Boot-managed executors, use spring.task.execution.shutdown.await-termination and spring.task.scheduling.shutdown.await-termination with a termination period. For custom pools, set the equivalent explicitly. Without these settings, tasks are either interrupted or block shutdown until SIGKILL arrives.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-service
spec:
  replicas: 3
  selector:
    matchLabels: { app: orders-service }
  template:
    metadata:
      labels: { app: orders-service }
    spec:
      # Budget: 10s preStop + 30s timeout-per-shutdown-phase + 5s margin
      terminationGracePeriodSeconds: 45
      containers:
        - name: app
          image: registry.example.com/orders-service:1.0.0
          ports:
            - containerPort: 8080
          lifecycle:
            preStop:
              # Delay SIGTERM until endpoint removal has propagated.
              # Distroless with no sleep binary: use `sleep: { seconds: 10 }`
              # where the cluster supports it, or ship a static sleep binary.
              exec:
                command: ["sleep", "10"]
          readinessProbe:
            # Reports refusing traffic once the context starts closing
            httpGet: { path: /actuator/health/readiness, port: 8080 }
            periodSeconds: 5
            failureThreshold: 1
          livenessProbe:
            # Must keep passing during the drain, so the kubelet does not restart the container
            httpGet: { path: /actuator/health/liveness, port: 8080 }
            periodSeconds: 10
            failureThreshold: 3

A Deployment that delays SIGTERM with a preStop sleep and sizes terminationGracePeriodSeconds to cover the sleep and the Spring shutdown phase. It points readiness and liveness at the Spring Boot availability endpoints, so the pod drops out of endpoints during the drain without being restarted.

How long should terminationGracePeriodSeconds be? A worked shutdown budget

Each component of shutdown takes time, and in the worst case they run one after another. Kubernetes starts the grace-period clock when the pod is marked for deletion, so the preStop hook is counted inside terminationGracePeriodSeconds, not added to it. The budget is:

grace period ≥ preStop sleep + web drain timeout + Kafka stop time + bean destroy time + safety margin

Kafka stop time is bounded by the batch already in flight: max.poll.records × worst-case per-record processing time.

Illustrative estimate (assumed numbers, not a benchmark)

Component Assumed input Seconds
preStop sleep Endpoint propagation guess 10
Web drain Longest expected request 20
Kafka stop 100 records × 150 ms 15
Bean destroy Pool and client close 3
Safety margin Headroom for GC pauses and slow I/O 7
Total 55 → set 60

spring.lifecycle.timeout-per-shutdown-phase must cover the slowest single phase, which is 20 s here. It does not need to cover the whole total.

The preStop value is the weakest number in the budget. It is a guess about how long endpoint removal takes to reach kube-proxy, ingress controllers and load balancers, and that delay differs between clusters and ingress setups. Measure it in your environment rather than copying a value.

These seconds have a cost. Node drains wait for every evicted pod to finish terminating. A rollout that waits on terminations repeats the wait for every replica. With 30 replicas replaced one at a time, a 60 s grace period can add up to roughly 30 minutes to a rollout.

max.poll.records trades throughput against stop time:

  • Small values shorten the worst-case stop and reduce reprocessing after a forced kill. They cost more poll round trips.
  • Large values improve throughput. They lengthen the shutdown bound and raise the risk of exceeding max.poll.interval.ms.

How do you verify graceful shutdown during a rolling restart?

Treat verification as a load test with a deploy in the middle. Start a steady request stream against the Service, then run kubectl rollout restart deployment/<name> and let every pod cycle. Count non-2xx responses by status code and record client-side errors separately. A 502 or 503 usually means traffic reached a pod after its listener closed, so the endpoint-removal delay is too short. A connection reset points to the process exiting with in-flight sockets still open. The target is zero of both, held across several runs, because a single clean run can hide a race.

For Kafka, produce messages with unique keys throughout the restart. Run kafka-consumer-groups.sh --describe --group <group> before, during and after. Lag should spike briefly during rebalances and then return to baseline. Have the consumer log each processed key, then check for duplicates. Duplicates mean offsets were not committed before the partitions were revoked.

Then read the shutdown logs from a terminated pod. The web server should stop accepting requests first, listener containers should stop next, and the context should close last. Each phase should finish inside the configured lifecycle timeout.

Run this check in Kubernetes, not only locally. A local SIGTERM does not reproduce node drains, which can apply a different grace period. It also does not reproduce sidecars that exit before your application and cut off its network. Running it in staging or CI turns an intermittent post-deploy error into a deterministic failing check. Debugging that error from production logs takes far longer than fixing a failing check.

#!/usr/bin/env bash
set -euo pipefail
NS=staging
DEPLOY=orders-api
URL=
BOOTSTRAP=kafka.staging.svc.cluster.local:9092
GROUP=orders-consumer
RESULTS=$(mktemp)

lag() { kafka-consumer-groups.sh --bootstrap-server "$BOOTSTRAP" --describe --group "$GROUP"; }
lag > lag-before.txt

(
  while true; do
    code=$(curl -s -o /dev/null -w '%{http_code}' --max-time 5 "$URL") || code="curl_exit_$?"
    echo "$code" >> "$RESULTS"
  done
) &
LOAD_PID=$!

kubectl -n "$NS" rollout restart deployment/"$DEPLOY"
until kubectl -n "$NS" rollout status deployment/"$DEPLOY" --timeout=15s; do
  lag >> lag-during.txt
done

sleep 30   # let the last terminating pods drain and lag settle
kill "$LOAD_PID"
lag > lag-after.txt

sort "$RESULTS" | uniq -c
if grep -qv '^200$' "$RESULTS"; then
  echo "FAIL: non-2xx responses or connection errors during rollout"; exit 1
fi
echo "PASS: no errors; compare lag-before.txt and lag-after.txt"

A repeatable CI or staging script that sends steady traffic through a rolling restart, separates HTTP status codes from connection resets, and takes consumer group lag snapshots before, during and after the rollout.

How can graceful shutdown save engineering time?

Most of the cost of poor shutdown behaviour falls on engineers, not on infrastructure. A correctly configured and verified shutdown reduces that cost in several ways:

  • Less incident triage after deploys. Bursts of 502/503 errors and connection resets during rollouts look like network, ingress or broker problems. Engineers spend time correlating dashboards, ingress logs and deploy timestamps before they find that pod termination is the cause. Once the shutdown sequence is correct and checked, a deploy-time error spike points to a real regression rather than background noise.
  • Less cleanup of duplicate side effects. When a Kafka consumer is killed mid-batch, uncommitted records are processed again. Without idempotent handlers, that can mean duplicate rows, repeated notifications or repeated downstream calls, and someone has to find and reconcile them by hand. Committing offsets before exit reduces how often this happens and how many records are involved each time.
  • Deploys stop needing special handling. If rollouts reliably drop requests, teams compensate with low-traffic deploy windows, manual monitoring during each release, or batched releases. A shutdown that drains cleanly lets deploys run during working hours as routine automation, without someone watching each one.
  • Intermittent failures become deterministic checks. The rolling-restart script in the previous section turns a race that appears only occasionally in production into a pass/fail result in staging or CI. A failing check with a clear cause is faster to fix than a sporadic production error that has to be reconstructed from logs.
  • Configuration becomes reviewable. Writing the shutdown budget as a sum, with an explicit grace period, per-phase timeout and Kafka batch bound, gives reviewers concrete values to check. A change to max.poll.records or a new long-running endpoint can be judged against the budget in code review instead of being found after an incident.
  • One fix applies across many services. When the entrypoint, probes, preStop hook and grace period live in a shared base image or deployment template, each service inherits correct behaviour without its own investigation.

The time saved depends on deploy frequency, replica counts and how much work each terminated pod holds. Estimate it from your own incident history and deploy cadence rather than from a general figure.

When should you avoid relying on graceful shutdown, and what are the alternatives?

Graceful shutdown is a best-effort optimization. It shrinks the window in which work is lost or duplicated, but it cannot close that window. Long-running requests, streaming responses and WebSockets can hold the web server's drain until spring.lifecycle.timeout-per-shutdown-phase expires. At that point the connection is closed anyway. Any client on such a connection needs reconnect or retry logic regardless of how carefully the pod shuts down.

Kafka has the same limit. Committing offsets before the consumer leaves the group narrows the duplicate window. Records that were processed but not yet committed are still re-delivered in three cases:

  • the kubelet sends SIGKILL at the end of the grace period
  • a long GC pause or slow handler misses the heartbeat or max.poll.interval.ms deadline
  • a rebalance revokes partitions mid-batch

Kafka consumers give at-least-once delivery, so idempotent processing is required. Graceful shutdown does not remove that requirement. Treat the following as complements to graceful shutdown:

Technique Effect during failures Effect during scale-down
Idempotent consumer / dedup key Re-delivered records become no-ops, so a crash causes no corruption Duplicates from revoked partitions are harmless
Client-side retries with backoff Cut connections and 503s are absorbed by the caller Requests that hit a draining pod succeed on retry
Cooperative rebalancing (CooperativeStickyAssignor) Only the affected partitions move, so other consumers keep processing A departing pod no longer pauses the whole group
Static membership (group.instance.id) A pod that restarts within session.timeout.ms rejoins without a rebalance No benefit for permanent removal, and the departure is detected later

Static membership has a tradeoff on scale-down. A pod that leaves permanently keeps its partitions until the session times out, which delays their reassignment. Tune session.timeout.ms with that in mind.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to spring boot graceful shutdown, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.