Why do Kubernetes rolling updates return 502s even with readiness probes?
A zero-downtime rollout means that no client request fails because of the rollout: no 502s or 503s from a proxy, no connection resets, and no truncated responses. Readiness probes cover half of this. They keep new pods out of rotation until they can serve. They do nothing for old pods on their way out, and that is where most rollout errors come from.
When a pod is marked for deletion, the API server sets its deletion timestamp, and two independent processes react to it:
- The kubelet path. The kubelet on the node runs the preStop hook if one is defined. It then sends SIGTERM to the containers. If they are still running when
terminationGracePeriodSecondsexpires, it sends SIGKILL. - The endpoint path. The EndpointSlice controller marks the pod's endpoint as not ready. That change then has to propagate to every consumer that routes traffic. kube-proxy rewrites rules on every node, ingress controllers reload their upstream lists, and cloud load balancers deregister targets. Each consumer has its own watch latency and update cycle.
Nothing coordinates these two paths. If the application exits on SIGTERM before every proxy has stopped routing to it, requests still in flight hit a closed socket. The client sees a 502 or a reset.
Readiness probes don't close this gap. A terminating pod is pulled from endpoints because it is terminating, not because a probe failed. Failing readiness on SIGTERM changes nothing, because removal is already underway and propagation speed stays the same.
This affects any team running HTTP or gRPC services behind a Service or an ingress.
+--------------------------------------+
| kubectl rollout / Deployment update |
+----------------+---------------------+
|
v
+--------------------------------------+
| API server: pod deletionTimestamp set|
+---------+------------------+---------+
| no coordination |
v v
+--------------------------+ +----------------------------------+
| KUBELET PATH (node) | | ENDPOINT PATH (cluster-wide) |
| 1. run preStop hook | | 1. EndpointSlice controller |
| 2. send SIGTERM | | marks endpoint not ready |
| 3. SIGKILL when grace | | 2. async propagation via watches:|
| period expires | | - kube-proxy rules per node |
+------------+-------------+ | - ingress upstream reload |
| | - cloud LB target deregister |
v +----------------+-----------------+
+--------------------------+ |
| app exits, socket closed | v
+------------+-------------+ +----------------------------------+
| | proxies still route to old pod |
| | until their own update lands |
| +----------------+-----------------+
+---------------+-----------------+
v
+--------------------------------------+
| in-flight request hits closed socket |
| client sees 502 / connection reset |
+--------------------------------------+
Once a pod's deletion timestamp is set, the kubelet shutdown path and the asynchronous endpoint-removal path run independently, so the app can close its socket while proxies are still sending it traffic.
How does a zero-downtime termination sequence work?
The fix does not coordinate the two paths, because Kubernetes offers no mechanism for that. It orders the pod's own shutdown so that the endpoint path has time to finish before the process stops serving. With the configuration described in this article, a terminating pod goes through these stages:
- Deletion. A rollout, scale-down or eviction sets the pod's deletion timestamp. The grace-period clock starts, and endpoint removal begins on its own.
- preStop delay. The kubelet runs the preStop hook, which waits a few seconds. The process is still listening and still answers any request that a slow proxy sends it. Meanwhile kube-proxy, ingress controllers and load balancers drop the pod from rotation.
- SIGTERM. After the hook returns, the kubelet sends SIGTERM. By now little or no new traffic should be arriving.
- Graceful drain. The application closes its listening socket, finishes in-flight requests, tells keep-alive and HTTP/2 clients to reconnect elsewhere, and releases resources.
- Clean exit. The process exits with status zero before
terminationGracePeriodSecondsruns out, so the kubelet never sends SIGKILL.
Around this per-pod sequence, the Deployment's rollout strategy makes sure replacement pods are ready before old ones start terminating, so overall capacity doesn't dip while old pods drain. Each of the following sections covers one of these pieces: the preStop hook, the grace-period budget, the application's signal handling, long-lived connections and external load balancers, and rollout settings.
Who should use preStop hooks and graceful shutdown?
This pattern is for teams whose pods receive request traffic through a Kubernetes Service, an ingress controller or a cloud load balancer that targets pods, and who replace or move those pods regularly. In practice that includes:
- Teams running HTTP, gRPC or WebSocket services that deploy often. Each rollout terminates every old replica, so a small per-pod race occurs on every deploy.
- Platform and cluster operators who run node drains for upgrades, autoscaler scale-downs or maintenance. Evictions follow the same termination path, so workloads without graceful shutdown drop requests during routine cluster work, even when no application team is deploying anything.
- Teams with user-facing or non-idempotent endpoints, such as payments, form submissions or uploads. Here a failed request can't be safely retried, so errors reach users directly.
- Teams with strict error budgets or SLOs, where deploy-time 5xx spikes eat into the budget or trigger alerts.
It matters less for workloads with no Service in front of them, such as batch jobs and queue consumers. Those need a SIGTERM handler but no preStop delay. It also matters less for internal services whose callers already retry idempotent requests. The section on when to avoid preStop delays covers these cases in more detail.
How does a preStop hook prevent traffic from reaching a terminating pod?
A preStop hook is an ordering tool. When a pod is deleted, the kubelet runs the container's preStop hook first and sends SIGTERM only after the hook returns. Meanwhile, the endpoint removal triggered by the same deletion continues on its own path through the control plane. A short delay in the hook keeps the process running and accepting connections while kube-proxy, ingress controllers and load balancers drop the pod from their routing tables. Requests that arrive during that window still get answered instead of being refused.
The usual implementation is an exec handler under lifecycle.preStop that runs something like sleep 10, as the example below shows. Some clusters also offer a built-in sleep action that needs no binary in the image. Support depends on your cluster's version and configuration, so check what your cluster accepts before relying on it. The exec form depends on a sleep binary, and distroless or scratch images often lack one. Without it, the hook fails and the delay you planned never happens. Your options are to add a minimal static binary, use the built-in action where it exists, or delay inside the application itself.
The hook's duration counts against terminationGracePeriodSeconds. A 10-second sleep inside a 30-second grace period leaves about 20 seconds for the application to drain before SIGKILL.
The sleep is a heuristic. No signal tells the pod that every proxy has stopped routing to it. Propagation time depends on cluster size, control-plane load and how many proxy layers sit in front of the pod. The value you pick is an estimate, so test it under realistic load.
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
replicas: 3
selector:
matchLabels:
app: web
template:
metadata:
labels:
app: web
spec:
# Total budget for preStop + graceful shutdown before SIGKILL.
# 10s sleep leaves roughly 20s for the app to drain after SIGTERM.
terminationGracePeriodSeconds: 30
containers:
- name: web
image: registry.example.com/web:1.4.2
ports:
- containerPort: 8080
lifecycle:
preStop:
# Keep serving while endpoint removal propagates to
# kube-proxy, ingress controllers and load balancers.
# SIGTERM is sent only after this command returns.
# Requires a sleep binary in the image (absent in scratch
# and many distroless images). 10s is a heuristic: tune it
# under realistic load for your cluster.
exec:
command: ["sleep", "10"]
# Alternative on clusters that support the built-in action
# (no binary needed); check your cluster before using it:
# sleep:
# seconds: 10
A Deployment whose preStop hook runs a 10-second exec sleep before SIGTERM, with the delay counted against a 30-second terminationGracePeriodSeconds and the built-in sleep action shown as a commented alternative.
How should you size terminationGracePeriodSeconds?
The grace period is a single budget that starts when the pod is marked for deletion. It is not a timer that starts after SIGTERM. The preStop hook runs inside that budget, so every second spent waiting for endpoint propagation is a second the application no longer has for shutdown. Size it as the sum of three components, plus headroom.
| Budget component | Assumed value | What drives it |
|---|---|---|
| preStop delay | 10s | Endpoint propagation across kube-proxy, ingress and load balancer layers |
| Longest in-flight request | 20s | Slowest legitimate endpoint, such as an export or a large upload |
| Application cleanup | 5s | Flushing buffers, closing DB pools, committing queue offsets |
| Total | 35s | Set the grace period above this, for example 45s |
Illustrative estimate. These inputs are assumptions, not measurements.
When the budget runs out, the kubelet sends SIGKILL. Open connections are reset, and clients see errors or truncated responses. This is the failure you were trying to avoid, and the pod's own logs won't record it.
Long values are not free. Every terminating pod holds its slot for the full shutdown, so rollouts, scale-downs and node drains all slow down. A drain waits on its slowest pod. If you set a ten-minute grace period across a fleet, routine cluster upgrades can stall or hit the timeouts of whatever tool is orchestrating them.
A hard cap on request duration is usually the better design. A server-side timeout of 20s gives you a known upper bound for the budget. Work that can't fit under the cap belongs in a background job, which can checkpoint and resume, not in a request that pins a pod during shutdown.
How do you implement graceful shutdown in the application?
Sending a request elsewhere is the load balancer's job. The process's job is to stop cleanly once SIGTERM arrives. The sequence matters:
- Stop accepting new connections. Close the listening socket so that any late arrivals get a fast connection refusal, which most proxies retry, rather than an accepted connection that is then dropped.
- Finish in-flight requests, bounded by the server-side timeout from the previous section.
- Close idle keep-alive connections. On HTTP/1.1, send
Connection: closeon the final response for each connection. On HTTP/2, send GOAWAY so the client opens new streams elsewhere. Without this step, a client can reuse a pooled connection just as you close it. - Exit with status zero.
Most standard libraries bundle steps 1–3 into a single "shutdown" call. The handler only needs to catch the signal, call shutdown with a deadline slightly shorter than the remaining grace period, and then release resources such as database pools.
None of this runs if the process never receives the signal. With shell-form ENTRYPOINT java -jar app.jar, the image runs /bin/sh -c, and the shell becomes PID 1. A shell does not forward SIGTERM to its child, so the app runs until the kubelet sends SIGKILL. Use the exec form, ENTRYPOINT ["java", "-jar", "app.jar"]. If you need a wrapper script, end it with exec "$@". Alternatively, run a minimal init process as PID 1 to forward signals and reap zombies.
Failing readiness on SIGTERM adds nothing. Kubernetes removes terminating pods from endpoints whatever the probe reports. If liveness shares the same path, the flipped probe can also trigger a pointless restart during shutdown.
// Run with exec form so node is PID 1 and receives SIGTERM directly:
// ENTRYPOINT ["node", "server.js"]
const http = require('node:http');
let shuttingDown = false;
const server = http.createServer((req, res) => {
// HTTP/1.1: tell the client not to reuse this connection after this response.
if (shuttingDown) res.setHeader('Connection', 'close');
res.end('ok\n');
});
server.listen(8080);
process.once('SIGTERM', () => {
shuttingDown = true;
// 1 + 2: close the listening socket and wait for in-flight requests to finish.
// The callback runs once every connection has closed.
server.close((err) => {
// 4: release other resources here (DB pools, queues), then exit cleanly.
process.exit(err ? 1 : 0);
});
// 3: drop keep-alive sockets that are idle right now instead of waiting
// for keepAliveTimeout to expire.
server.closeIdleConnections();
// Backstop: keep this below terminationGracePeriodSeconds minus preStop time,
// so the process exits on its own terms before the kubelet sends SIGKILL.
setTimeout(() => {
server.closeAllConnections();
process.exit(1);
}, 20000).unref();
});
A Node.js SIGTERM handler drains an HTTP server. It stops accepting connections, lets in-flight requests finish, marks final responses with Connection: close, closes idle keep-alive sockets, and enforces a deadline shorter than the grace period.
What about long-lived connections, ingress controllers and cloud load balancers?
Endpoint removal only changes where new connections go. kube-proxy rewrites its rules for fresh flows, but an established TCP connection keeps its path. HTTP keep-alive pools, gRPC channels and WebSockets can therefore keep sending requests to a pod that left endpoints seconds ago, right up until the process exits and the client sees a reset.
The server has to end these connections deliberately. During shutdown, send Connection: close on HTTP/1.1 responses, send GOAWAY on HTTP/2 and gRPC, and close WebSockets with a close frame so clients reconnect elsewhere. On the client side, cap connection lifetime with a pool max-age or the gRPC max-connection-age setting. That way connections rebalance regularly instead of pinning to one pod for hours. Client retries cover what remains, but only for idempotent requests. They can also hide shutdown bugs: a rollout that looks clean may depend on retries absorbing resets that should never have happened.
Ingress controllers and cloud load balancers watch endpoints or run their own health checks on their own schedule, and they are often slower than kube-proxy. Base the preStop delay on the slowest hop, not the fastest. Set the load balancer's deregistration or drain timeout to fit inside terminationGracePeriodSeconds.
| Layer | Reacts to | What to tune |
|---|---|---|
| kube-proxy | Endpoint changes, new connections only | preStop delay |
| Ingress controller | Its own endpoint watch and reload | preStop delay, upstream keep-alive timeout |
| Cloud load balancer | Target deregistration and health checks | Drain timeout, health-check interval and threshold |
| Clients and pools | Connection close, GOAWAY, resets | Max connection age, idempotent retries |
| Mesh sidecar | Its own SIGTERM | Shutdown ordering relative to the app |
A service mesh sidecar adds a separate problem. If the proxy exits before the application finishes draining, in-flight requests fail regardless of everything above. Sidecar shutdown ordering needs its own coordination.
Which rollout settings interact with pod termination?
The rollout strategy decides how many terminating pods exist at once, and therefore how much capacity is draining at any moment.
maxSurge and maxUnavailable
maxUnavailable: 0 with maxSurge: 1 (or a percentage) means a new pod must become available before an old one starts terminating. Serving capacity never drops below the desired replica count while old pods sit in their preStop delay. The cost is headroom. Each surge pod needs schedulable CPU, memory and quota. If the cluster has no room, the surge pod stays Pending, no old pod is removed, and the rollout stalls until progressDeadlineSeconds marks it as failed. Kubernetes does not roll back automatically when that happens. A larger surge finishes faster, but it requires proportionally more spare capacity.
minReadySeconds and readiness probes
A pod counts as available only after it has been Ready for minReadySeconds. This gives a brief buffer against pods that pass one probe and then crash. It only helps if readiness is accurate. A probe that reports Ready before caches, connection pools or JIT warm-up are in place lets the controller terminate old pods while the new ones are still slow.
PodDisruptionBudgets
Node drains use the eviction API, and evicted pods follow the same termination path: preStop, SIGTERM, grace period. A PDB limits how many pods can be evicted at once. Rolling updates are governed by the Deployment strategy, not by the PDB. Do not set minAvailable equal to the replica count, because that blocks drains entirely. The example below pairs a surge-only strategy with a PDB that allows one disruption at a time.
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
spec:
replicas: 4
minReadySeconds: 10 # new pod must stay Ready 10s before it counts as available
progressDeadlineSeconds: 600 # stalled rollout is marked failed; no automatic rollback
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1 # one extra pod at a time; needs spare CPU, memory and quota
maxUnavailable: 0 # an old pod terminates only after a new one is available
selector:
matchLabels: {app: web}
template:
metadata:
labels: {app: web}
spec:
terminationGracePeriodSeconds: 45 # must cover the preStop delay plus in-flight work
containers:
- name: web
image: registry.example.com/web:1.4.2
resources:
requests: {cpu: 250m, memory: 256Mi} # the surge pod must fit these requests
readinessProbe:
httpGet: {path: /ready, port: 8080} # report Ready only after caches and pools are warm
lifecycle:
preStop:
exec: {command: ["sleep", "10"]} # wait for endpoint removal before SIGTERM
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: web-pdb
spec:
maxUnavailable: 1 # drains evict one pod at a time and never block completely
selector:
matchLabels: {app: web}
This example pairs a surge-only rolling update, which relies on minReadySeconds, an accurate readiness probe and a preStop delay, with a PodDisruptionBudget that allows node drains to evict one pod at a time.
When should you avoid preStop delays, and what are the alternatives?
A preStop delay solves one problem: traffic that arrives after the pod starts terminating but before every proxy and load balancer has dropped its endpoint. If that traffic doesn't exist, the delay only slows rollouts.
Skip it for batch jobs and queue consumers with no Service in front of them. They need a SIGTERM handler that stops pulling work and finishes or requeues in-flight items. Shorten it for very large Deployments. A delay of a few seconds per pod adds up across hundreds of replicas and limited surge. Accept the errors for internal services whose callers already retry idempotent requests and tolerate a rare failed call.
| Approach | Effectiveness | Cost | Caveats |
|---|---|---|---|
| preStop sleep | High for the endpoint race | A few lines of config | Lengthens every termination |
| Client-side retries | High for idempotent calls | Changes in every client | Unsafe for non-idempotent writes; can amplify load |
| Fail readiness on SIGTERM | Low on its own | Small app change | Endpoint removal already starts on deletion; mainly helps external health checkers |
| Mesh-managed draining | Medium to high | Mesh operations overhead | Other proxies still need time to receive config updates |
| LB deregistration delay | High when the LB targets pods directly | Cloud-specific setup | Does nothing for in-cluster Service traffic |
| Capped connection age | High for long-lived HTTP/2 or gRPC | Server setting | Clients must handle reconnects cleanly |
On cost: a preStop hook and a signal handler take an afternoon. Intermittent 502s that appear only during deploys can take days to trace. Put the settings in a shared Deployment template so each team doesn't have to rediscover the race on its own.
How can this save engineering time?
Most of the time saved comes from avoiding a class of bug that is expensive to diagnose, not from making deploys faster. Termination-race errors are intermittent and short, and they show up only during rollouts or drains. Proxy and load balancer logs record them, but the pod that caused them often logs nothing, because it either exited cleanly or was killed. Engineers who chase these errors tend to look first at the application, the network or the ingress controller, and only later at shutdown ordering.
Getting the setup right once removes several recurring costs:
- Fewer deploy-time investigations. When rollouts produce no 5xx spikes, nobody has to correlate error bursts with deploy timestamps, file incidents or explain alerts that clear on their own.
- Deploys during working hours. Teams that see errors on every rollout often batch releases into low-traffic windows or add manual watch steps. Clean rollouts remove the need for both.
- Routine cluster maintenance without coordination. Platform teams can drain nodes for upgrades without asking each service owner to stand by, because evictions follow the same graceful path.
- Less retry and workaround code. Services don't need ad hoc client retries added just to hide deploy-time resets, and those retries no longer mask real shutdown bugs.
- One fix applied everywhere. A shared Deployment template with a preStop hook, a sized grace period and a surge-only strategy, plus a standard signal-handling pattern in each language's service skeleton, means individual teams inherit the fix instead of each rediscovering the race.
A worked example makes the trade-off concrete. Illustrative estimate, with assumed inputs only: assume a team spends two engineer-days tracing one episode of deploy-time 502s, and that such an episode recurs twice a quarter across its services. Assume the one-time fix takes one engineer-day for the shared template and signal handlers, plus half a day to add the verification test described below. Under those assumptions, the fix costs 1.5 engineer-days once and avoids about four engineer-days of investigation per quarter. Replace the assumed inputs with your own incident history before relying on the result.
The verification test is what keeps the savings. Without it, a cluster upgrade or a new sidecar can quietly bring back the race, and the expensive investigation starts again.
How do you verify a Kubernetes zero downtime deployment?
Treat the configuration as unproven until a test shows zero failed requests.
Generate constant-rate load
Point any HTTP load generator at the Service through the same path real clients use: ingress, cloud load balancer, or both. Use a constant arrival rate, not a fixed number of concurrent workers. Closed-loop workers slow down when requests stall, so they send fewer requests during the exact window you are testing. Record every non-2xx response and every connection-level error (resets, refused connections, timeouts) separately. A 502 and a TCP reset usually point to different gaps.
Trigger the events
Run the load while you start a rollout with kubectl rollout restart deployment/<name>. Then run it again while you evict pods with kubectl drain <node> --ignore-daemonsets, and afterwards run kubectl uncordon <node>. A drain evicts pods through a different path than a rollout and often removes several replicas at once. In a second terminal, run kubectl get endpointslices -l kubernetes.io/service-name=<svc> -w. This lets you match error timestamps against the moment each pod's endpoint turns not-ready or disappears.
Tune, then stress
If errors cluster just after termination begins, lengthen the preStop delay. If they appear near the end of the grace period, raise terminationGracePeriodSeconds or shorten shutdown. Repeat until errors reach zero at your target rate. Then retest while the control plane is busy, for example during several simultaneous rollouts. Endpoint propagation slows under load, and that is when a marginal delay fails.
Add this test to CI or to your pre-upgrade checklist. Cluster upgrades, ingress changes and new sidecars can each break the setup without any visible sign.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to kubernetes zero downtime deployment, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.