How does a spot interruption reach a Kubernetes pod?
Spot capacity is spare compute that a cloud provider sells at a steep discount. In exchange, the provider can take it back whenever it needs the capacity. That discount is the main reason teams put Kubernetes nodes on spot. The catch is that reclamation is not a rare failure; it is part of the contract.
The interruption path
When a node is reclaimed, the signal passes through several components before it reaches your process:
- Provider notice. The provider announces the reclamation. On AWS this is a two-minute warning, exposed through instance metadata and an event stream. Other providers give a shorter window, often around 30 seconds.
- Cordon and drain. A termination handler, or Karpenter watching an interruption queue, marks the node unschedulable. It then evicts pods through the Eviction API, which respects PodDisruptionBudgets.
- Pod termination. Each evicted pod gets a deletion timestamp. The kubelet runs any
preStophook and then sends SIGTERM to the containers. - Endpoint removal. At the same time, the pod is removed from EndpointSlices. kube-proxy, ingress controllers and service meshes learn about this asynchronously, so traffic can keep arriving for a few seconds after SIGTERM.
- SIGKILL. Once
terminationGracePeriodSecondsruns out, the kubelet kills whatever is still running.
Why the window matters
Detection delay, drain ordering, the preStop hook and your shutdown logic all have to fit inside the provider's notice. If they don't, the instance disappears mid-shutdown. A generous grace period cannot extend that deadline. This is why moving a node pool to spot is not a pricing toggle. Whether an interruption is safe depends on how each pod behaves during those seconds.
<-- provider notice window: hard ceiling for every step below -->
+--------------------------+ +--------------------------+
| Cloud provider notice |----------->| Termination handler or |
| (reclaim warning) | | Karpenter (intr. queue) |
+--------------------------+ +------------+-------------+
| cordon node
v
+--------------------------+
| drain via Eviction API |
| (respects PDBs) |
+------------+-------------+
| deletion timestamp
+---------------------------------+
v v
+--------------------------+ +--------------------------+
| kubelet | | EndpointSlice update |
| preStop hook -> SIGTERM | | -> kube-proxy, ingress, |
+------------+-------------+ | mesh (async lag) |
| +------------+-------------+
v | traffic can
+--------------------------+ | still arrive
| App shutdown logic |<------------------+
| (finish in-flight work) |
+------------+-------------+
| terminationGracePeriodSeconds expires
v
+--------------------------+
| SIGKILL / instance gone |
+--------------------------+
A spot interruption moves from the provider notice through cordon and drain, eviction, and then SIGTERM in parallel with asynchronous endpoint removal, ending in SIGKILL. All of these steps must finish inside the provider's notice window.
Which workloads should run on spot instances, and when should you avoid them?
The deciding question is what happens when a pod disappears with about two minutes of warning, possibly along with several siblings on the same node. Spot suits workloads that lose nothing and recover fast. It does not suit workloads that lose state or coordination, or that need minutes to replace lost capacity.
| Workload | Spot fit | Why |
|---|---|---|
| Stateless HTTP/gRPC services | Good | Replicas are interchangeable, so the remaining pods absorb traffic while the terminating one drains |
| Idempotent queue consumers with visibility timeouts | Good | Unacknowledged messages reappear after the timeout, and reprocessing them is harmless |
| Checkpointed batch jobs | Good | A restart resumes from the last checkpoint instead of starting over |
| Singleton controllers | Poor | Reconciliation stalls until a replacement schedules and acquires the leader lease |
| Stateful databases | Poor | Losing a primary forces failover, and rebuilding a replica costs hours of I/O |
| Long-lived connection holders | Poor | Every client on the node reconnects at once and hits the survivors as a burst |
| Slow starters (large images, JVM warmup, cache loading) | Poor | Replacements are not ready before the old pod is gone, so effective capacity dips on every interruption |
Avoid spot entirely in these cases:
- No spare replicas. The service cannot lose a whole node's worth of pods and still meet its latency target.
- One required instance type. The workload needs a single type, such as a specific GPU. Interruptions then tend to hit together, and replacement capacity may not exist when you need it.
- Unsafe retries. Requests are not idempotent, so a request retried after an interruption can corrupt data.
How do you make backend pods shut down gracefully on spot interruption?
Kubernetes delivers the interruption as an ordinary pod deletion. Whatever handles the notice (Karpenter's interruption handling or a node termination handler) cordons and drains the node, and each container receives SIGTERM. The application must treat SIGTERM as "stop accepting, finish, exit", not as "exit now".
The shutdown sequence should run in this order:
- preStop sleep of a few seconds. Endpoint removal and SIGTERM happen in parallel. kube-proxy, ingress controllers and cloud load balancers update asynchronously, so they can keep routing to the pod briefly after deletion. The sleep keeps the listener open while that change propagates.
- Fail readiness and stop accepting new work. Return failures from the readiness endpoint so external health checks drop the pod. Stop pulling from queues.
- Drain in-flight requests with a hard deadline, close database pools, then exit with a zero status.
Budget the grace period explicitly. terminationGracePeriodSeconds includes the preStop time, and it must fit inside the notice window after detection and drain overhead. As an illustrative estimate with a 120-second notice:
- Allow about 20 seconds for detection and eviction.
- Use a 10-second preStop sleep.
- Allow 45 seconds for draining.
That gives a grace period of about 60 seconds and leaves margin. If you set it above the remaining notice time, the node disappears mid-drain.
Long-lived connections cannot be drained in time. For WebSockets and streaming gRPC, send a close frame or GOAWAY so the server signals the shutdown to clients. Clients still need reconnect logic with jittered backoff, and they need idempotent retries.
Misconfiguration fails silently. Missing SIGTERM handling or an oversized grace period causes no errors during normal rollouts. These mistakes only show up as dropped requests during a real interruption.
apiVersion: apps/v1
kind: Deployment
metadata:
name: orders-api
spec:
replicas: 3
selector:
matchLabels:
app: orders-api
template:
metadata:
labels:
app: orders-api
spec:
# Illustrative budget for a 120s spot notice:
# ~20s detection + eviction leaves ~100s.
# 10s preStop + 45s drain + 5s margin = 60s, well inside that.
# The grace period includes preStop time.
terminationGracePeriodSeconds: 60
containers:
- name: api
image: registry.example.com/orders-api:1.4.2
ports:
- containerPort: 8080
env:
# Hard deadline the app enforces on in-flight work after SIGTERM.
- name: SHUTDOWN_DRAIN_SECONDS
value: "45"
readinessProbe:
httpGet:
path: /readyz # app returns 503 once SIGTERM is received
port: 8080
periodSeconds: 5
failureThreshold: 1
lifecycle:
preStop:
exec:
# Keep serving while endpoint removal reaches kube-proxy, ingress and LBs.
# SIGTERM is sent only after this returns. Requires a sleep binary in the image.
command: ["sleep", "10"]
A Deployment that budgets a 60-second grace period inside a 120-second spot notice (illustrative estimate), uses a 10-second preStop sleep so endpoint removal can propagate, and passes the application a 45-second drain deadline for its SIGTERM handler.
What can PodDisruptionBudgets and topology spread actually protect against?
A PodDisruptionBudget is enforced by the Kubernetes eviction API, so it only governs voluntary disruptions such as kubectl drain, node upgrades, and Karpenter consolidation. When a cloud provider reclaims a spot node, the machine disappears whether or not the eviction succeeded. A handler that cordons and drains the node on the interruption notice will respect your PDB only until the deadline. After that, minAvailable has no effect. Treat PDBs as protection against your own automation, not against the provider.
topologySpreadConstraints with topologyKey: topology.kubernetes.io/zone and a small maxSkew keep replicas from piling into one zone. The scheduler evaluates the constraint only when it places a pod, though. It does not move running pods to restore balance, so skew builds up after a few interruptions until something reschedules them.
Spreading only helps if the surviving replicas can carry the load:
Illustrative estimate (assumed numbers, not a benchmark)
- Required surviving capacity =
N − ceil(N / Z) - Assumed inputs: N = 6 replicas, Z = 3 zones, peak load needing 4 replicas
- Result: losing one zone leaves
6 − 2 = 4replicas, with zero headroom.
Spot reclaims are often correlated. A single instance type in one zone shares a capacity pool, and that pool can be drained all at once. When several nodes go together, every affected pod reschedules simultaneously. Each one pulls images onto fresh nodes, and caches and connection pools start cold. Diversifying instance types and sizing for the loss of a full pool protects you here. A PDB does not.
# PDB: limits voluntary evictions (drain, upgrades, consolidation).
# It cannot stop the provider from reclaiming a spot node.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: api-pdb
spec:
minAvailable: 4 # matches the assumed peak load in the estimate
selector:
matchLabels:
app: api
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
spec:
replicas: 6 # 2 per zone across 3 zones; losing one zone leaves 4
selector:
matchLabels:
app: api
template:
metadata:
labels:
app: api
spec:
# Evaluated only at scheduling time; running pods are not rebalanced.
topologySpreadConstraints:
- maxSkew: 1
topologyKey: topology.kubernetes.io/zone
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: api
containers:
- name: api
image: registry.example.com/api:1.0.0
resources:
requests:
cpu: 500m
memory: 512Mi
A PodDisruptionBudget that limits only voluntary evictions, together with a zone spread constraint that balances the six replicas when they are scheduled but does not rebalance them later.
How do you configure Karpenter for spot with on-demand fallback?
…against correlated reclaims. Karpenter supports this directly: let a single NodePool choose from many instance types and both capacity types.
A NodePool that prefers spot and falls back
Set the karpenter.sh/capacity-type requirement to allow both spot and on-demand, and keep instance requirements broad by specifying categories, generations, and architectures rather than exact types. When both capacity types are allowed, Karpenter favors spot. If the spot request fails because capacity is unavailable, Karpenter launches on-demand instead of leaving pods pending. A narrow list such as one instance family in one zone defeats this fallback, because Karpenter keeps hitting the same exhausted pools.
For interruption handling, Karpenter needs an interruption queue that receives the cloud provider's spot interruption warnings and instance health events. When a warning arrives, it cordons and drains the node and provisions a replacement before the instance is reclaimed. Consolidation also removes or replaces nodes to cut cost. These are voluntary disruptions. They respect PodDisruptionBudgets and NodePool disruption budgets, but they still restart your pods.
Mixed placement
To keep a baseline of replicas on on-demand capacity, run a separate on-demand NodePool. Pin one Deployment to it with required node affinity on karpenter.sh/capacity-type, and run a second Deployment with surplus replicas on spot. Taint the spot NodePool so that only workloads with a matching toleration can land there.
Trade-offs
Broad instance flexibility means replicas can run on different CPU generations and memory sizes, which makes per-pod latency less predictable. You also take on more NodePools, interruption infrastructure, and more frequent pod churn.
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
name: spot-flex
spec:
template:
spec:
nodeClassRef:
group: karpenter.k8s.aws
kind: EC2NodeClass
name: default
requirements:
# Both capacity types: Karpenter prefers spot, falls back to on-demand
- key: karpenter.sh/capacity-type
operator: In
values: ["spot", "on-demand"]
# Broad requirements give many spot pools to choose from
- key: karpenter.k8s.aws/instance-category
operator: In
values: ["c", "m", "r"]
- key: karpenter.k8s.aws/instance-generation
operator: Gt
values: ["4"]
- key: kubernetes.io/arch
operator: In
values: ["amd64", "arm64"]
# Only pods that tolerate this taint can land here
taints:
- key: example.com/spot
value: "true"
effect: NoSchedule
disruption:
# Consolidation adds voluntary disruptions; budget caps how many at once
consolidationPolicy: WhenEmptyOrUnderutilized
consolidateAfter: 5m
budgets:
- nodes: "10%"
# Opted-in Deployment pod spec adds:
# tolerations: [{key: example.com/spot, operator: Equal, value: "true", effect: NoSchedule}]
A Karpenter v1 NodePool on AWS that allows spot and on-demand across broad instance categories, generations, and architectures, taints its nodes so only opted-in workloads schedule there, and caps consolidation-driven voluntary disruptions with a node budget.
How much do spot instances really save, and what does it cost to run them?
Illustrative estimate (assumed numbers, not a benchmark):
- On-demand price per node: $0.20/hour
- Spot discount: 60%
- Baseline: 10 on-demand nodes
- Share of capacity moved to spot: 70%
- Extra spot headroom to absorb interruptions: 2 nodes
Net savings = 1 − [(1 − s) + (s + h) × (1 − d)]
Here s is the spot share (0.7), h is headroom as a fraction of baseline (0.2), and d is the discount (0.6). That gives 1 − [0.3 + 0.9 × 0.4] = 34%, well below the 60% headline discount.
| Scenario | Nodes | Hourly cost | vs. baseline |
|---|---|---|---|
| All on-demand | 10 OD | $2.00 | — |
| Mixed, spot available | 3 OD + 9 spot | $1.32 | −34% |
| Spot unavailable, full fallback | 12 OD | $2.40 | +20% |
The last row is easy to miss. When spot capacity dries up, Karpenter replaces spot nodes with on-demand nodes, and you pay on-demand prices for the headroom too. Discounts and interruption rates vary by instance type, region, and time, so treat any estimate as a hypothesis. Compare it against real billing data after a few weeks and again whenever you change instance families.
Engineering time is the larger hidden cost. Someone has to build and test termination handling, tune alerts, and answer pages when a capacity shortage hits several pools at once.
Much of that work pays off beyond spot. Correct SIGTERM handling, readiness gating, and connection draining are the same mechanisms that make every voluntary drain, node upgrade, and rolling deploy clean. If you get shutdown right once, spot becomes the most frequent test of code you needed anyway.
How do you verify interruption safety, and what are the alternatives to spot?
Rehearse the interruption under load
Don't wait for the cloud provider to test your shutdown path. Run a load test at realistic traffic, then pick a spot node and run kubectl cordon followed by kubectl drain. Cordoning stops new pods from being scheduled on the node. Draining evicts the existing pods through the Eviction API, so it honors your PodDisruptionBudgets and each pod's termination grace period, as a node termination handler or Karpenter would. While the drain runs, watch three things:
- Error rate at the load balancer or ingress. A spike means traffic still reaches pods after they stop accepting connections.
- p99 latency. It shows whether the remaining replicas can absorb the load while replacements start.
- Termination time per pod. Compare it with the grace period. Pods killed with SIGKILL instead of exiting cleanly have a shutdown path longer than your budget allows. That budget must also fit inside your provider's interruption notice.
Repeat the test with several nodes drained at once. A single drain rarely breaks anything. Correlated reclaims do.
When something other than spot fits better
| Approach | Why it saves money | Fits better than spot when |
|---|---|---|
| On-demand plus rightsizing | Removes overprovisioned requests | Workloads are stateful or can't tolerate interruption, and requests are padded |
| Committed or reserved capacity | Discount in exchange for a term commitment | Baseline load is steady and predictable |
| Horizontal and vertical autoscaling | Matches capacity to actual demand | Traffic swings daily and idle capacity is the main waste |
| Scale-to-zero | Pays nothing while idle | Services are rarely called and tolerate cold starts |
These approaches combine well. A common pattern is committed capacity for the floor, autoscaling for the shape of demand, and spot for the burst.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to spot instances kubernetes, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Consumers learn about a breaking change at review time rather than from production errors.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.