Spot Instances in Kubernetes: Handling Interruptions Safely

What spot and preemptible nodes are and how reclamation works

Spot capacity is spare compute that a cloud provider sells at a steep discount on the condition that it can take the machine back whenever it needs the capacity. The names vary: AWS calls them Spot Instances, Google Cloud has Spot VMs (and the older preemptible VMs), and Azure has Spot Virtual Machines. To Kubernetes, these are ordinary nodes that can disappear on short notice.

How reclamation works

When demand for on-demand or reserved capacity rises, the provider reclaims spot machines to serve it. Some providers can also evict when the market price rises above a maximum price you set. The provider first publishes a notice through the instance metadata service and its event APIs, then stops or terminates the machine. The notice is short. AWS documents two minutes. Google Cloud and Azure document roughly 30 seconds. The provider treats the notice as best-effort and does not guarantee delivery. The window is short because the provider is freeing capacity for customers who pay full price and won't wait for your batch job.

Why use it anyway

You pay substantially less for compute. That trade only pays off if you treat interruption as a routine event rather than a failure. A spot node going away belongs in the same category as a rolling deploy or a node upgrade: expected, frequent and handled automatically. If your workloads can be rescheduled within seconds without losing correctness, spot is cheap capacity. If they can't, it is an outage waiting for a capacity crunch.

Which workloads belong on spot and which should stay on on-demand

The test is whether a workload can lose any single pod at an arbitrary moment and keep working without a human. That depends less on the application than on where its state lives and how long one unit of work runs.

Stateful workloads are risky for reasons beyond the pod restart. A block volume usually has to detach from the reclaimed node before it can attach elsewhere, and that can take far longer than the notice window. Most block storage is zone-bound, so the pod can only reschedule into the zone that just lost capacity. That zone is also the one most likely to be short of spot instances. Replicated databases on spot can enter failover churn: a primary is reclaimed, a replica is promoted, and then the new primary is reclaimed too.

Long batch jobs fail more quietly. A six-hour job without checkpoints that runs on capacity reclaimed every few hours may never finish, and you still pay for every attempt.

Who should use spot nodes

Spot nodes suit teams whose compute bill is large enough that a discount matters, and whose workloads already pass the test above. In practice that usually means:

• Platform teams running shared clusters. They control node pools, taints and autoscaler configuration, and can offer spot capacity as an opt-in for teams whose workloads tolerate it.
• Teams running batch, data processing, rendering or CI. These workloads are retryable by design, and they often account for a large share of compute hours.
• Service owners with replicated, stateless deployments. If a service already runs several replicas behind a Service, has readiness probes and shuts down cleanly on SIGTERM, most of the work is already done.

A team also needs some operational maturity: someone who owns the termination handler and autoscaler, metrics that show error rates and pending pods, and time to run interruption drills. A team that has none of that in place should build it on on-demand nodes first. Spot exposes missing graceful shutdown and missing redundancy, and it does so at unpredictable times.

When to use spot and when to avoid it

Use spot when

• The workload can lose any pod at any moment without data loss or manual recovery. Examples include replicated stateless services, queue consumers with redelivery, checkpointed batch jobs and CI.
• A replacement pod can start and become Ready quickly. That depends on image size, startup work and warm-up time. If the replacement takes much longer than the notice window, users will see degraded service.
• You can keep an on-demand baseline that carries critical traffic by itself, so spot capacity is an addition rather than a dependency.
• You run a termination handler and an autoscaler with on-demand fallback, and you have rehearsed interruptions in staging.
• Compute spend is high enough that the savings cover the engineering work. The payback estimate later in this article shows how to check that.

Avoid spot when

• The workload holds primary state on node-attached or zone-bound storage. Examples are database primaries, message broker leaders and anything that waits on volume detach and attach to recover.
• The workload is a singleton without a standby. Every reclamation causes a gap that lasts until rescheduling completes.
• Jobs run longer than the typical interval between interruptions and cannot checkpoint. They may never complete.
• Latency or availability targets cannot absorb the rescheduling gap. Displaced pods stay Pending while new nodes provision, and the surviving replicas carry extra load during that time.
• Startup is slow. Large images, long cache warm-up or lengthy migrations at boot can make every interruption a visible incident.
• Shutdown cannot fit inside the notice. If cleanup reliably needs more time than the provider gives, and the work cannot be made idempotent, spot will cut it off.
• The cluster is small, or spend is low. The fixed engineering cost of handlers, node pools, PDBs and drills may never pay back.
• The team has no capacity to run drills or watch interruption metrics. Untested spot configurations tend to fail during provider capacity shortages, which are also the times when on-demand fallback is busiest.

How a node termination handler turns reclamation into a drain

Without a handler, Kubernetes learns about a reclaimed node only after it disappears. The kubelet stops reporting, the node controller marks the node NotReady, and pods are deleted only after a timeout. During that gap, Services can still route traffic to endpoints that no longer exist, and replacement pods have not been scheduled yet.

A termination handler closes that gap by acting on the provider's advance notice. It typically runs either as a DaemonSet that polls each node's instance metadata endpoint, or as a central controller that consumes interruption events from the provider's event stream. When a notice arrives, the handler takes four steps:

1. Cordon the node so the scheduler stops placing new pods on it.
2. Drain the node by calling the Eviction API for each pod. This makes evictions respect PodDisruptionBudgets instead of bypassing them.
3. Let each pod shut down gracefully. The kubelet sends SIGTERM, and the pod has its terminationGracePeriodSeconds to finish in-flight work while its endpoints are removed.
4. Reschedule. Owning controllers, such as Deployments and StatefulSets, create replacements, and the scheduler places them on healthy nodes.

The handler adds components you have to run and monitor. It needs RBAC permissions to evict pods, and it is itself a workload that can crash, be evicted, or poll too slowly. A drain that a strict PodDisruptionBudget blocks for longer than the notice window still ends with a hard kill. Track handler health and drain duration, and alert when a drain is still running when the node disappears.

Copy+-----------------------------+       +------------------------------+
| Cloud provider              |       | Termination handler          |
| interruption notice         |------>| DaemonSet polling metadata,  |
| (instance metadata or       |       | or controller reading the    |
|  event stream)              |       | provider event stream        |
+-----------------------------+       +------------------------------+
                                      1. cordon node |  (needs RBAC)
                                      2. evict pods  |
                                                     v
+-----------------------------+       +------------------------------+
| PodDisruptionBudgets        |<----->| Kubernetes API server        |
| allow or delay each eviction|       | Eviction API                 |
+-----------------------------+       +------------------------------+
                                                     v
+-----------------------------+       +------------------------------+
| Service endpoints           |<------| Spot node (cordoned)         |
| pod removed, no new traffic |       | 3. kubelet sends SIGTERM;    |
+-----------------------------+       |    grace period, then exit   |
                                      +------------------------------+
                                                     v
                                      +------------------------------+
                                      | 4. Deployment/StatefulSet    |
                                      | creates replacement pod;     |
                                      | scheduler places it on a     |
                                      | healthy node                 |
                                      +------------------------------+
 Risk: handler crashes or polls slowly, or a strict PDB blocks the drain
 past the notice window -> node disappears -> remaining pods hard-killed

A termination handler turns the provider's interruption notice into a cordon and an Eviction API drain, which respects PodDisruptionBudgets, so pods shut down gracefully and are rescheduled before the node disappears. If the handler fails or the drain stalls, pods are still hard-killed.

Scheduling controls: node pools, taints, affinity and topology spread

Placement starts with separate node pools. Run at least one on-demand pool for system components and anything that cannot tolerate reclamation, and one or more spot pools for interruptible capacity. Each pool carries a label that identifies its capacity type. The key and values vary by provider, so this guide uses the placeholder <CAPACITY_LABEL_KEY> with values <SPOT_VALUE> and <ON_DEMAND_VALUE>. Check your provider's documentation for the real label.

Taints keep unprepared pods off spot

Apply a taint such as <SPOT_TAINT_KEY>=true:NoSchedule to spot nodes. Without a taint, any pod can land on spot, including ones that have no graceful shutdown and no budget. Workloads that are ready for spot add a matching entry under tolerations. A toleration only permits scheduling on spot. It does not attract the pod there.

Affinity expresses preference, not requirement

To pull tolerant pods toward spot, use preferredDuringSchedulingIgnoredDuringExecution under nodeAffinity, matching <CAPACITY_LABEL_KEY> in <SPOT_VALUE>, with a weight such as 80. Use a preference rather than required... so that pods still schedule on on-demand nodes when spot capacity is unavailable.

Topology spread limits correlated loss

Add two topologySpreadConstraints:

• One with topologyKey: topology.kubernetes.io/zone spreads replicas across zones, since reclamation often hits one zone at a time.
• One with topologyKey: <CAPACITY_LABEL_KEY> and maxSkew: 1 balances replicas between spot and on-demand. A single reclamation event then cannot take out every replica.

Set whenUnsatisfiable: ScheduleAnyway when availability matters more than exact balance. Use DoNotSchedule when an even spread is mandatory, and accept that some pods will stay pending. Both constraints need a labelSelector that matches the workload's own pods.

CopyapiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 6
  selector:
    matchLabels: {app: web}
  template:
    metadata:
      labels: {app: web}
    spec:
      tolerations:              # permits spot nodes; does not attract the pod
        - key: "<SPOT_TAINT_KEY>"
          operator: Equal
          value: "true"
          effect: NoSchedule
      affinity:
        nodeAffinity:           # soft preference, so on-demand is still a fallback
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 80
              preference:
                matchExpressions:
                  - {key: "<CAPACITY_LABEL_KEY>", operator: In, values: ["<SPOT_VALUE>"]}
      topologySpreadConstraints:
        - maxSkew: 1            # spread replicas across zones
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector: {matchLabels: {app: web}}
        - maxSkew: 1            # balance replicas between spot and on-demand
          topologyKey: "<CAPACITY_LABEL_KEY>"
          whenUnsatisfiable: ScheduleAnyway
          labelSelector: {matchLabels: {app: web}}
      containers:
        - name: web
          image: registry.example.com/web:1.0.0
      terminationGracePeriodSeconds: 30

A Deployment that tolerates the spot taint, softly prefers spot nodes, and uses two topology spread constraints to balance replicas across zones and across spot and on-demand capacity.

Pod disruption budgets and graceful shutdown inside the notice window

A PodDisruptionBudget limits how many replicas a voluntary eviction can remove at once. Setting minAvailable or maxUnavailable makes the node termination handler's drain wait until replacements are Ready elsewhere before evicting more pods. A strict budget can block the drain entirely: maxUnavailable: 0, or minAvailable equal to the replica count, never permits an eviction. PDBs also have no authority over the provider. When the reclamation deadline arrives, the VM stops whether or not the budget was satisfied. A PDB shapes how the drain proceeds; it cannot extend the drain.

Shutting down inside the window

Shutdown on an evicted pod happens in this order:

1. The preStop hook runs.
2. SIGTERM is sent.
3. After terminationGracePeriodSeconds, SIGKILL follows.

The grace period includes the preStop time. On SIGTERM, the application should:

• fail its readiness probe so endpoints stop routing to it
• stop accepting new work
• finish in-flight requests
• flush state
• exit

A short preStop sleep covers the delay before load balancers notice the endpoint removal.

Size the grace period against the notice, not against what the app would like to have:

grace ≤ notice − detection latency − drain/eviction overhead

Illustrative estimate (assumed numbers, not a benchmark):

The formula gives grace ≤ 120 − 10 − 20 = 90 s. With a 10 s preStop, that leaves about 80 s for application cleanup.

If shutdown logic needs more time than the notice allows, it will be cut off mid-operation. Any work that cannot finish in that time must be made idempotent or checkpointed.

CopyapiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  maxUnavailable: 1   # drain evicts one pod at a time; 0 would block the drain entirely
  selector:
    matchLabels:
      app: api
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 4
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      # Illustrative budget: 120s notice - 10s detection - 20s eviction overhead = 90s
      terminationGracePeriodSeconds: 90   # covers preStop + app shutdown, then SIGKILL
      containers:
        - name: api
          image: registry.example.com/api:1.4.2
          readinessProbe:
            httpGet:
              path: /ready     # app returns 503 here once SIGTERM is received
              port: 8080
            periodSeconds: 5
            failureThreshold: 1
          lifecycle:
            preStop:
              exec:
                # give load balancers time to see the endpoint removal before SIGTERM
                command: ["sleep", "10"]   # requires a sleep binary in the image

A PodDisruptionBudget that lets a drain proceed one replica at a time, plus a pod spec with a 10-second preStop sleep. Its 90-second grace period is sized to fit inside an assumed 120-second interruption notice.

Capacity diversification, autoscaling and fallback to on-demand

Spot capacity is priced and reclaimed per pool, where a pool is one instance type in one availability zone. If every spot node comes from the same pool, a single capacity squeeze can reclaim all of them together. Spreading node groups across several similarly sized instance types and at least two or three zones means one reclamation event takes out only a fraction of the fleet. Keep an on-demand baseline sized to carry critical traffic by itself, so a large reclamation reduces capacity without causing an outage.

When nodes disappear, the autoscaler sees pods it cannot schedule and requests replacements. That recovery is slow. Provisioning the VM, joining the cluster, reaching Ready and pulling container images can add up to minutes. Displaced pods stay Pending for that whole time while the surviving replicas absorb their load. That is when latency spikes. Small images and pre-warmed spare capacity shorten this gap.

If spot capacity runs out, configure the autoscaler to fall back to an on-demand node group. Fallback protects availability, but its cost is unpredictable. Shortages tend to hit many tenants simultaneously, and you may run at full on-demand prices for hours.

Testing interruptions and estimating cost versus risk

Before trusting the setup in production, rehearse reclamation in staging. Run kubectl cordon followed by kubectl drain --ignore-daemonsets --delete-emptydir-data on a spot node to replay the graceful path your termination handler triggers. Then run kubectl delete node, or terminate the instance through your provider, without draining first. This simulates a missed or late notice. During each run, watch three signals:

• Error rates at the ingress or load balancer. A spike means readiness or shutdown handling is wrong.
• Time until replacement pods are Ready. This shows whether the autoscaler and image pulls fit inside the notice window.
• Job retry counts. Unexpected failures point to non-idempotent work.

Illustrative estimate (assumed numbers, not a benchmark). Assume 40 nodes, on-demand at $0.40/hour, spot at $0.12/hour, 730 hours per month, and 120 engineering hours at $100/hour ($12,000) for handlers, node pools and testing.

Monthly savings = nodes × spot share × hours × (on-demand price − spot price) Payback (months) = engineering cost ÷ monthly savings

Higher shares pay back faster, but they increase exposure if the provider reclaims many nodes at once. Subtract an allowance for fallback hours billed at on-demand rates, using your own observed interruption rate.

The engineering cost is mostly paid once. Without that work, the cost shows up later and keeps recurring: unexplained 5xx bursts, half-finished jobs and on-call time spent tracing failures back to a node that disappeared. Designing for interruption up front turns those incidents into routine rescheduling.

Copy#!/usr/bin/env bash
set -euo pipefail

CONTEXT="staging-cluster"        # placeholder kube context
SPOT_LABEL="node-pool=spot"        # placeholder label on your spot node pool
NAMESPACE="web"                     # namespace of the service under test
APP_SELECTOR="app=checkout"         # pods whose rescheduling time we measure

kubectl config use-context "$CONTEXT"

NODE=$(kubectl get nodes -l "$SPOT_LABEL" -o jsonpath='{.items[0].metadata.name}')
echo "Target node: $NODE"

START=$(date +%s)
kubectl cordon "$NODE"
kubectl drain "$NODE" --ignore-daemonsets --delete-emptydir-data --timeout=120s

kubectl wait pod -n "$NAMESPACE" -l "$APP_SELECTOR" --for=condition=Ready --timeout=300s
echo "Graceful reschedule took $(( $(date +%s) - START ))s"

kubectl get jobs -n "$NAMESPACE" \
  -o custom-columns=NAME:.metadata.name,SUCCEEDED:.status.succeeded,FAILED:.status.failed

NODE2=$(kubectl get nodes -l "$SPOT_LABEL" -o jsonpath='{.items[1].metadata.name}')
START=$(date +%s)
kubectl delete node "$NODE2"
kubectl wait pod -n "$NAMESPACE" -l "$APP_SELECTOR" --for=condition=Ready --timeout=300s
echo "Abrupt reschedule took $(( $(date +%s) - START ))s"

A staging drill that drains one spot node gracefully, deletes another without draining, and records rescheduling time and job failure counts for each path.

How designing for interruption saves engineering time

Spot pricing reduces cost. Most of the time savings come from the work spot forces you to do, because that work also covers every other kind of node disruption.

• Routine maintenance becomes uneventful. Node upgrades, kernel patching, autoscaler scale-down and instance retirement all remove nodes. A workload that already handles a spot drain correctly, with graceful SIGTERM handling, a sensible PDB, readiness that fails on shutdown and replicas spread across zones, handles these the same way. Fewer node rotations need a maintenance window or someone watching them.
• Debugging time drops. Without interruption handling, a reclaimed node shows up as scattered 5xx errors and jobs that failed partway through. Engineers then spend time correlating those failures with node events. With a termination handler, a cordon and drain is logged against a named node, so the cause is visible immediately.
• Retries replace manual reruns. Idempotent jobs, queue redelivery and checkpointing let interrupted work resume automatically. Without them, someone reruns failed batch jobs by hand or works out which partial output to discard. With checkpoints, a long job resumes rather than starting over, which saves the engineers waiting on its results as well as the compute.
• Drills replace incident response. Running the staging drill above on a schedule finds broken shutdown logic, oversized images and blocking PDBs when the problem is cheap to fix. Otherwise it surfaces during a provider capacity shortage, in the middle of an on-call shift.
• Scheduling policy is written once. Taints, tolerations, affinity and topology spread rules live in shared templates or platform defaults. Individual service teams opt in by adding a toleration rather than each designing their own placement and failure handling.

These savings depend on the setup being maintained. If the termination handler goes unmonitored, drills stop running, or new services go onto spot without graceful shutdown, the recurring costs described in the previous section come back.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to spot instances kubernetes, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.