Spot Instances in Kubernetes: Interruption Handling and Real Savings

What are spot instances for Kubernetes workers and how does reclamation work?

Spot capacity is the provider's unused compute, sold at a discount on the condition that it can be taken back when on-demand customers need it. The instance types are the same as on-demand. The difference is the contract: you get a lower price and accept that your node can disappear on the provider's schedule. The price varies by instance type and Availability Zone, and so does the likelihood of reclamation.

On AWS, reclamation comes with a two-minute interruption notice, which is exposed through instance metadata and emitted as an event. The notice is best-effort. Usually you get close to the full two minutes. Occasionally you get less, and nothing guarantees you will get it at all. Any shutdown path that needs the whole window to finish will fail some of the time.

AWS also emits a rebalance recommendation, which is a separate and earlier signal. It means an instance is at elevated risk of interruption. It is not a notice that reclamation is coming. It can arrive well before the two-minute notice, or it can arrive and be followed by no interruption at all. Acting on it buys you time to drain the node, but you will sometimes replace nodes that would have survived.

Teams put Kubernetes worker nodes on spot because stateless replicas, batch jobs and CI runners already tolerate losing a node, so they want that capacity at a lower price. The catch is that the discount on the invoice is not the saving. The real number is that discount minus the cost of interruptions: rescheduling, cold starts, retried work, and extra headroom you keep provisioned to absorb node loss. The rest of this article measures that difference.

How much do spot instances actually save once interruptions are counted?

Each of those costs can be estimated, so treat the discount as one input to a formula:

Net cost ratio = [(1 − f)(1 − d) + f] × (1 + r + h) × (1 + w)

Here d is the spot discount and f is the on-demand fallback share. r is the extra replicas you run for resilience, and h is the scheduling headroom you keep for absorbing node loss. w is wasted node-hours as a fraction of the total: interruptions per node-month × (rescheduling time + lost partial work) ÷ hours per month.

Illustrative estimate (assumed numbers, not a benchmark)

In this illustrative estimate, with the assumed inputs above, an assumed 65% headline discount becomes a net saving of about 30%. That would still be worth having, but it is less than half the headline figure. Under these assumptions, the largest reductions come from fallback and over-provisioning, not from interruptions themselves. Your own figures will differ, so plug in your measured discount, fallback share, headroom and interruption rate rather than reusing these values.

The formula also leaves out churn costs that never show up as compute spend:

• Cold caches. A rescheduled pod starts with an empty cache, which pushes extra load onto databases.
• Image pulls. New nodes must download images again, which costs registry egress and adds startup latency.
• JIT warm-up. JIT-compiled runtimes run slowly until they warm up, so you need extra replicas to cover the slow period.
• Cross-zone traffic. Pods rescheduled into a different zone generate inter-zone data charges.
• On-call noise. Alerts triggered by expected churn cost engineer time.

Silent fallback is the biggest risk. When spot capacity runs short, the provisioner launches on-demand nodes, and those nodes stay in place. Over time f can drift from 25% toward 100%, and nobody notices. Your bill then rises above an all-on-demand baseline, because you are still paying for the resilience overhead. Track f as a metric and alert on it.

Which workloads should run on spot, and when should you avoid it?

Spot fits when losing a node costs a reschedule and nothing more. That means stateless services running enough replicas across zones that one node's departure stays within the PodDisruptionBudget. It also covers batch and CI jobs that checkpoint, so a restart repeats minutes of work instead of hours. Queue consumers fit as well: an unacknowledged message returns to the queue and another worker picks it up.

It fits poorly wherever recovery is slower or riskier than the interruption itself. Stateful workloads look cheap on spot, but their costs come from places outside the compute line:

• Volume reattachment latency. A replacement pod waits for the old node's volume to detach and reattach before it can start.
• Zone-bound persistent volumes. Block volumes are usually tied to a single zone. If spot capacity in that zone dries up, the pod stays Pending even though other zones have nodes available.
• Replication rebuild load. A database or broker replica that rejoins the cluster must resync. That consumes I/O and network on the surviving members, often at exactly the moment they are carrying extra traffic.

A few of these events per month can erase the discount and add on-call pages on top.

How does a spot interruption flow through a Kubernetes cluster?

The clock starts when the provider sends its signal. On AWS that is either a rebalance recommendation, an early and optional hint that capacity is at elevated risk, or the interruption notice, which gives a fixed two-minute warning before the instance is reclaimed. Every step below spends part of that window:

1. Event delivery. A handler picks up the signal from instance metadata or a provider event queue. Polling intervals and queue latency take a few seconds off the window before Kubernetes knows anything.
2. Cordon. The node is marked unschedulable so no new pods land on it. This step is fast.
3. Drain via the Eviction API. Evictions respect PodDisruptionBudgets. If a PDB allows zero disruptions, the eviction is refused and retried. The provider does not wait for these retries, so the instance terminates anyway and the pods die without a graceful exit.
4. Pod shutdown. The preStop hook runs first, then the pod receives SIGTERM. Both must fit inside terminationGracePeriodSeconds, and both are effectively capped by whatever time remains in the notice window.
5. Traffic removal. A terminating pod is pulled from Service endpoints, but kube-proxy and external load balancers learn about this asynchronously. A short preStop sleep keeps the pod serving until routing catches up. Without it, in-flight and newly routed requests get connection resets.
6. Replacement. A new node is requested in parallel with the drain. Boot time and image pulls often take longer than the remaining window, so evicted pods sit Pending and the surviving replicas carry the load until the new node is ready.

Requests and work are lost in three places: drains blocked by PDBs, grace periods that run past the window, and batch work that has not been checkpointed.

Copy+--------------------------------------------------------------+
| Provider signal: rebalance recommendation (early, optional)  |
| or interruption notice: fixed 2-minute warning               |
+--------------------------------------------------------------+
          |
          v
+-------------------+   +----------------+   +------------------------+
| 1 Event delivery  |-->| 2 Cordon node  |-->| 3 Drain: Eviction API  |
| IMDS poll / queue |   | unschedulable  |   | respects PDBs          |
| (seconds lost)    |   | (fast)         |   | refused -> retry  [X1] |
+-------------------+   +----------------+   +------------------------+
          |                                              |
          | in parallel                                  v
          v                                  +------------------------+
+-------------------+                        | 4 Pod shutdown         |
| 6 Replacement     |                        | preStop -> SIGTERM     |
| new node request  |                        | in grace period        |
| boot, image pull  |                        | grace > window    [X2] |
| pods Pending      |                        | lost batch work   [X3] |
+-------------------+                        +------------------------+
          |                                              |
          v                                              v
+-------------------+                        +------------------------+
| Surviving pods    |                        | 5 Traffic removal      |
| carry the load    |<-- load shifts --------| endpoints updated;     |
| until node Ready  |                        | kube-proxy / LB lag;   |
+-------------------+                        | preStop sleep bridges  |
                                             +------------------------+
|<------------- 2-minute notice window ------------->| instance reclaimed
[X1] PDB blocks drain  [X2] grace > window  [X3] no checkpoint = lost

A spot interruption runs from the provider signal through event delivery, cordon, PDB-aware drain, pod shutdown and traffic removal, with replacement provisioning in parallel, all inside the two-minute window; X1-X3 mark where requests and work are lost.

Node termination handler vs Karpenter interruption handling: what are the alternatives?

Three options exist, and they differ mainly in where the interruption signal comes from and what has to exist on the cloud side to deliver it.

IMDS mode is the quickest to adopt. It only sees what the node itself can see, though, and a node that is already unhealthy cannot report its own termination. Queue mode and Karpenter both get events from the control plane, so they keep working when the node is degraded. The cost is maintaining the queue, the rules, and the permissions. Karpenter also has an advantage the handler lacks: the same component that drains the node launches its replacement, so the new capacity request goes out as soon as the notice arrives. On-demand nodes avoid this machinery entirely, and the price premium you pay for them is the baseline the other options are measured against.

Pick exactly one interruption path per node pool. If you run the handler and Karpenter against the same nodes, both will cordon and evict at the same time. Their eviction retries compete with each other, and Karpenter may delete a node the handler is still draining. If both read one SQS queue, they behave as competing consumers. Each message is deleted by whichever component receives it first, so each component misses some events at random.

Configuring Karpenter spot NodePools with diversification and on-demand baseline

With Karpenter as the single consumer of interruption events, most of the remaining work is in the NodePool definition. Field names and API groups have changed across Karpenter releases, so check them against the version you have installed.

A diversified spot NodePool

Start with a NodePool whose karpenter.sh/capacity-type requirement allows both spot and on-demand. When both are permitted, Karpenter favors spot and uses on-demand only when it cannot get spot capacity. Keep the instance shape requirements broad:

• Use several instance categories or families rather than a single family.
• Allow a range of sizes.
• Include every zone your subnets cover.

Each instance type and zone combination is a separate spot pool. A narrow requirement, such as one family in one size, fails without obvious errors. Launches fall back to on-demand, or stall when that pool is reclaimed, so you lose both availability and the savings you expected.

Broad requirements have a cost. Pods will run on different CPU generations and memory-to-core ratios, so you need to load-test across that range and set requests that hold up on the slowest node you allow.

Consolidation interacts with spot too. It deletes underused nodes and can replace on-demand nodes with spot. Replacing one spot node with a cheaper spot node is gated in some releases and requires wide instance flexibility. Check whether your release supports it before you rely on it.

On-demand baseline

Create a second NodePool that requires on-demand and applies a taint, for example workload-tier=critical:NoSchedule.

• Give critical pods a matching toleration.
• Add a required node affinity on karpenter.sh/capacity-type: on-demand.

The toleration lets critical pods schedule onto the baseline pool. The taint keeps other pods off those nodes. The affinity keeps critical pods off spot nodes.

CopyapiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-spot
spec:
  template:
    spec:
      nodeClassRef: {group: karpenter.k8s.aws, kind: EC2NodeClass, name: default}
      requirements:
        # Allow both; Karpenter prefers spot and uses on-demand only if spot is unavailable.
        - {key: karpenter.sh/capacity-type, operator: In, values: [spot, on-demand]}
        # Broad categories, sizes and zones give many spot pools to draw from.
        - {key: karpenter.k8s.aws/instance-category, operator: In, values: [c, m, r]}
        - {key: karpenter.k8s.aws/instance-size, operator: NotIn, values: [nano, micro, small]}
        - {key: topology.kubernetes.io/zone, operator: In, values: [us-east-1a, us-east-1b, us-east-1c]}
  disruption:
    # Removes underused nodes and can replace on-demand nodes with spot.
    consolidationPolicy: WhenEmptyOrUnderutilized
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: critical-on-demand
spec:
  template:
    spec:
      nodeClassRef: {group: karpenter.k8s.aws, kind: EC2NodeClass, name: default}
      requirements:
        # Baseline capacity for critical pods is never spot.
        - {key: karpenter.sh/capacity-type, operator: In, values: [on-demand]}
        - {key: karpenter.k8s.aws/instance-category, operator: In, values: [m, r]}
      # Keeps general workloads off the baseline; critical pods need a matching toleration
      # plus required node affinity on karpenter.sh/capacity-type: on-demand.
      taints:
        - {key: workload-tier, value: critical, effect: NoSchedule}

Two Karpenter NodePools: a diversified pool that allows both spot and on-demand with broad instance categories, sizes and zones, and a tainted on-demand-only baseline pool for critical pods.

How do you harden workloads and verify spot savings are real?

Hardening the workload

Run at least three replicas so one reclaimed node never takes out a majority. Add topologySpreadConstraints on both topology.kubernetes.io/zone and kubernetes.io/hostname. Spot pools in a single zone are often reclaimed together, so spreading across zones and nodes limits how many replicas one event can remove.

Set the PodDisruptionBudget to maxUnavailable: 1 rather than minAvailable equal to the replica count. A PDB that allows zero disruptions blocks the drain, the notice window runs out, and the node is terminated with pods still on it. That is worse than an orderly eviction. A PDB with no real limit lets a consolidation pass evict every replica at once.

Keep terminationGracePeriodSeconds well inside the notice window, which is two minutes on AWS. Leave headroom for a preStop sleep, so endpoints are removed before the process stops accepting connections, and for connection draining. Readiness probes stop the replacement pod from receiving traffic before it is warm.

What to measure

• Interruption events per pool and per instance type
• Pod restarts and evictions
• Time pods spend Pending, and the full time to reschedule each one
• 5xx and latency during interruption windows compared with baseline
• Blended spot/on-demand share of node-hours
• Effective cost per workload, using this formula:

Effective cost = (spot node-hours × spot price + on-demand node-hours × on-demand price + overprovisioned headroom) ÷ requests served

Compare this effective cost against the on-demand baseline, not against the list discount.

When metrics tie each interruption to its error budget impact, an interruption becomes a dashboard entry instead of a page. Engineers review trends weekly instead of draining nodes by hand at 3 a.m.

CopyapiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
spec:
  replicas: 3  # one reclaimed node never removes a majority
  selector:
    matchLabels: {app: checkout-api}
  template:
    metadata:
      labels: {app: checkout-api}
    spec:
      terminationGracePeriodSeconds: 60  # includes preStop; well inside the 2-minute AWS notice
      topologySpreadConstraints:
        - maxSkew: 1  # spread across zones; soft so a zone shortage does not leave pods Pending
          topologyKey: topology.kubernetes.io/zone
          whenUnsatisfiable: ScheduleAnyway
          labelSelector: {matchLabels: {app: checkout-api}}
        - maxSkew: 1  # never stack replicas on one node
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: DoNotSchedule
          labelSelector: {matchLabels: {app: checkout-api}}
      containers:
        - name: app
          image: registry.example.com/checkout-api:1.4.2
          ports: [{containerPort: 8080}]
          lifecycle:
            preStop:  # let endpoint removal propagate before SIGTERM; image must include sleep
              exec: {command: ["sleep", "10"]}
          readinessProbe:  # replacement pod gets traffic only once warm
            httpGet: {path: /healthz/ready, port: 8080}
            periodSeconds: 5
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-api
spec:
  maxUnavailable: 1  # drains proceed one pod at a time instead of being blocked
  selector:
    matchLabels: {app: checkout-api}

A three-replica Deployment spread across zones and nodes, with a preStop sleep, a readiness probe and a grace period that fits the spot notice window, paired with a PodDisruptionBudget that allows one eviction at a time.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to spot instances kubernetes, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Consumers learn about a breaking change at review time rather than from production errors.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.