Kubernetes Spot Instances for Backends: Interruptions and Real Savings

What spot capacity is and how it differs from on-demand

To Kubernetes, a spot node is an ordinary node. The kubelet registers, pods schedule, and nothing in the API marks the machine as temporary unless you add a label or taint yourself. The difference is in the contract with the cloud provider. Spot capacity is spare capacity sold at a discount, and the provider can take it back whenever it needs that capacity for on-demand customers.

Two properties of that reclamation shape everything later in this article. First, the warning is short, from tens of seconds to a couple of minutes depending on the provider. It is also best-effort, so your design must survive a node that disappears with no notice at all. Second, interruptions are correlated. Price and availability are set per capacity pool, roughly an instance type in a zone. When a pool runs dry, many of your nodes in it can be reclaimed together. That is a different risk from random, independent hardware failure.

Teams adopt spot for the large nominal discount on compute. This article's thesis is that the discount is only the starting figure. Real savings equal that discount minus three costs: the engineering work to make interruptions routine, the extra headroom you provision to absorb them, and the incidents caused by failure modes you didn't anticipate.

Which backend workloads fit spot and which to keep off it

Those costs stay small only when the workload already treats a lost node as routine. The test is simple: can a replica disappear without warning and be replaced before anyone notices?

Good fits share three traits. There are spare replicas, so losing one doesn't hurt. Startup is fast enough that a replacement is serving before the remaining replicas saturate. Work can be retried without harm.

• Stateless HTTP services behind a load balancer fit well when they run several replicas.
• Queue consumers fit well because an unacknowledged message returns to the queue, as long as processing is idempotent.
• Batch jobs that checkpoint lose only the work done since their last save point.

Poor fits break one of those properties:

• Single-replica services go down outright when their node goes.
• Long jobs without checkpoints restart from zero.
• Services with slow warm-up, such as large caches, JIT compilation or model loading, leave a capacity gap after each interruption.
• Services with long in-flight requests can't drain inside a short notice window.

Stateful workloads usually cost more engineering time than they save. A persistent volume has to detach from the dead node and reattach to the new one, which can take minutes. Block storage is usually tied to one zone, so the replacement pod can only schedule where spare capacity exists in that zone. Replicated databases and consensus systems can lose quorum if one interruption wave takes out several members at once.

How an interruption flows through the cluster

The sequence starts with the cloud provider, not Kubernetes. On AWS, a spot interruption notice arrives roughly two minutes before reclamation. Other providers give less warning, often around 30 seconds. That window is the whole budget for the steps below.

1. Detection. A node termination handler or Karpenter's interruption controller picks up the notice. It reads it either from the instance metadata endpoint or from an event queue. Polling intervals and queue delivery add seconds you cannot get back.
2. Cordon and drain. The node is marked unschedulable and its pods are evicted through the Eviction API. That API respects PodDisruptionBudgets. A tight PDB can stall eviction while the clock keeps running. The provider will not wait for your budget.
3. Graceful shutdown. Each pod receives SIGTERM and gets up to terminationGracePeriodSeconds to finish. A grace period longer than the notice is a promise the platform cannot keep.
4. Rescheduling. Evicted pods go Pending until they fit somewhere. If spare capacity exists, this is fast. If not, they wait for a replacement node to launch, join, pull images and pass readiness. That is usually the largest gap.

If the notice never arrives, or detection misses it, the node simply disappears. The control plane marks it NotReady after missed heartbeats. Pods are only evicted after the default not-ready/unreachable toleration expires, which is 300 seconds unless you shorten it. Meanwhile, traffic may still route to dead endpoints until readiness updates propagate.

Pick exactly one interruption mechanism. If you use Karpenter with its native interruption handling enabled, do not also run a standalone termination handler. Both will react to the same event and issue overlapping cordons and drains. They can also make conflicting decisions about the same node.

Copy+--------------------------------+
| Cloud provider: spot notice    |  window: ~2 min on AWS = whole budget
+---------------+----------------+
                v
+--------------------------------+
| 1. Detection (pick ONE)        |  <- lost: polling / queue delivery
|  Karpenter interruption ctrl   |     never run both: duplicate
|  OR node termination handler   |     cordons and conflicting drains
+---------------+----------------+
                v
+--------------------------------+
| 2. Cordon + drain              |  <- lost: tight PDB stalls eviction
|  Eviction API honors PDBs      |     while the clock keeps running
+---------------+----------------+
                v
+--------------------------------+
| 3. SIGTERM, then up to         |  <- grace > notice: pod is killed
|  terminationGracePeriodSeconds |     before shutdown finishes
+---------------+----------------+
                v
+--------------------------------+
| 4. Pending -> rescheduled      |  <- usually largest gap: launch,
|  spare capacity or new node    |     join, image pull, readiness
+--------------------------------+

No notice / missed: node vanishes -> NotReady (missed heartbeats)
  -> wait out not-ready/unreachable toleration (300s default) -> evict
  -> step 4; traffic may hit dead endpoints until readiness updates

A spot interruption runs from provider notice through detection by one chosen handler, cordon and PDB-limited drain, graceful shutdown and rescheduling, with time lost at each step and a slower fallback when no notice arrives.

Making pods interruption-safe

Every pod on a spot node can be evicted with only a short warning. The pod spec and the application each have to handle that.

Shutdown sequence

When the kubelet stops a pod, it runs the preStop hook first and sends SIGTERM to the container after the hook finishes. A short sleep in preStop gives endpoint controllers and load balancers time to remove the pod from rotation. Without it, new requests can still reach a process that is already shutting down. Once SIGTERM arrives, the application should stop accepting new connections, finish the requests it already has, flush buffers and exit.

terminationGracePeriodSeconds covers the preStop hook and the SIGTERM drain together. Size it so that both, plus the time the drain takes to reach the pod, fit inside the provider's notice window. On AWS that window is two minutes, and some other providers give less. A grace period longer than the notice gives you nothing, because the node disappears when the window closes. Readiness probes should also fail as soon as shutdown begins, so a restarted or rescheduled pod gets traffic only when it can serve it.

The Deployment example below sets these fields together with a PodDisruptionBudget. The minAvailable setting stops a drain from evicting every replica at once. The budget applies only to evictions made through the API, such as a drain. It cannot hold back a node that the cloud provider reclaims at the end of the window.

Work that outlives the notice

A request or job that runs longer than the notice window will be cut off regardless of how well the pod handles shutdown. When the window closes, the provider powers off the machine and the process stops wherever it is. For this kind of work, make operations idempotent so retries are safe. Checkpoint long jobs so a replacement pod can resume from the last saved state. If neither is possible, schedule the work on on-demand nodes.

CopyapiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  replicas: 4
  selector:
    matchLabels: {app: orders-api}
  template:
    metadata:
      labels: {app: orders-api}
    spec:
      # preStop sleep + SIGTERM drain must finish well inside a 120s notice
      terminationGracePeriodSeconds: 90
      containers:
        - name: api
          image: registry.example.com/orders-api:1.4.2
          ports: [{containerPort: 8080}]
          lifecycle:
            preStop:
              # Keep serving while endpoints and LBs drop this pod (needs sh in image)
              exec: {command: ["sh", "-c", "sleep 15"]}
          readinessProbe:
            # App returns 503 here as soon as it receives SIGTERM
            httpGet: {path: /ready, port: 8080}
            periodSeconds: 5
            failureThreshold: 1
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: orders-api
spec:
  # Limits API-initiated evictions such as drains; cannot stop a provider reclaim
  minAvailable: 3
  selector:
    matchLabels: {app: orders-api}

A Deployment whose preStop sleep and grace period fit inside a two-minute spot notice, a readiness probe that removes the pod from rotation during shutdown, and a PodDisruptionBudget that keeps a drain from evicting all replicas at once.

Scheduling controls that limit blast radius

Spot reclamation is correlated. When a provider needs capacity back, it takes it from a specific instance type in a specific zone, so every node in that pool can receive notices within the same few minutes. If your node group allows only one or two instance types, all replicas of a service may sit in the same pool, and one reclamation event removes all of them at once. Draining logic cannot help when there is nowhere left to reschedule. Allow a broad set of instance types with similar CPU-to-memory ratios across every available zone. Each pool then holds a small share of your capacity, and losing one pool costs only that share.

Spread at the pod level too. A topology spread constraint on topology.kubernetes.io/zone with a maxSkew of 1 keeps replicas balanced across zones. Preferred pod anti-affinity on kubernetes.io/hostname stops the scheduler from packing several replicas onto one large node.

For critical services, keep a floor of replicas on on-demand capacity. Use the capacity-type node label to place them: karpenter.sh/capacity-type (spot or on-demand) with Karpenter, or eks.amazonaws.com/capacityType with EKS managed node groups. The example below runs two Deployments for the same service:

• Baseline: a required node affinity for on-demand nodes.
• Burst: a toleration for a spot taint and a preferred affinity for spot nodes.

The taint keeps workloads that have not opted in off spot nodes entirely.

CopyapiVersion: apps/v1
kind: Deployment
metadata: {name: api-baseline}
spec:
  replicas: 2
  selector: {matchLabels: {app: api, tier: baseline}}
  template:
    metadata: {labels: {app: api, tier: baseline}}
    spec:
      affinity:  # Baseline: critical floor must run on on-demand nodes, never spot
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions: [{key: karpenter.sh/capacity-type, operator: In, values: [on-demand]}]
      containers: [{name: api, image: registry.example.com/api:1.4.2}]
---
apiVersion: apps/v1
kind: Deployment
metadata: {name: api-burst}
spec:
  replicas: 6
  selector: {matchLabels: {app: api, tier: burst}}
  template:
    metadata: {labels: {app: api, tier: burst}}
    spec:
      tolerations: [{key: spot, operator: Equal, value: "true", effect: NoSchedule}]  # opt in to tainted spot nodes
      affinity:
        nodeAffinity:  # prefer spot, but fall back to on-demand if spot capacity is gone
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            preference:
              matchExpressions: [{key: karpenter.sh/capacity-type, operator: In, values: [spot]}]
        podAntiAffinity:  # avoid packing several replicas onto one large node
          preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 50
            podAffinityTerm: {topologyKey: kubernetes.io/hostname, labelSelector: {matchLabels: {app: api}}}
      topologySpreadConstraints:  # keep replicas balanced so one zone's reclamation removes only a share
      - {maxSkew: 1, topologyKey: topology.kubernetes.io/zone, whenUnsatisfiable: ScheduleAnyway, labelSelector: {matchLabels: {app: api, tier: burst}}}
      containers: [{name: api, image: registry.example.com/api:1.4.2}]

Two Deployments for the same service: a baseline pinned to on-demand nodes via the karpenter.sh/capacity-type label, and a burst tier that tolerates a spot=true:NoSchedule taint (assumed to be applied to spot nodes by the node pool), prefers spot, and spreads across zones and hosts.

Karpenter spot provisioning versus cluster autoscaler node groups

Where cluster autoscaler scales node groups you define in advance, Karpenter provisions nodes directly from pending pod requirements. A NodePool states constraints, not a fixed instance type. Conceptually, the requirements are a list of label selectors: karpenter.sh/capacity-type In [spot, on-demand], a range of instance families and sizes, and both architectures if your images support them. A wide type list matters because spot pools are reclaimed independently. Fifteen acceptable shapes give the provisioner more pools to choose from than three do. When both capacity types are allowed, Karpenter prefers spot and launches on-demand only when spot cannot be obtained.

Consolidation replaces underused nodes with cheaper or fewer ones. Disruption budgets cap how many nodes Karpenter may voluntarily disrupt at once.

The trade-offs work against each other. Fallback keeps pods running, but it can move a large share of the fleet onto on-demand for days, so alert on the capacity-type mix and not only on total spend. Turning fallback off keeps the bill predictable but shifts the failure to Pending pods. Aggressive consolidation combined with spot reclamation can reschedule the same pods several times in an hour. Set a disruption budget and a consolidation delay so voluntary node moves do not stack on top of involuntary reclamations.

CopyapiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: backend-general
spec:
  template:
    spec:
      nodeClassRef:            # cloud-specific node class, defined separately
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
        # Allow both: spot is preferred, on-demand is the fallback.
        # Remove on-demand to cap cost and accept Pending pods instead.
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        # Wide family list = more independent spot pools to draw from.
        - key: karpenter.k8s.aws/instance-family
          operator: In
          values: ["m5", "m6i", "m7i", "c6i", "c7i", "r6i", "m6g", "m7g", "c7g"]
        - key: karpenter.k8s.aws/instance-size
          operator: In
          values: ["large", "xlarge", "2xlarge"]
        # Both architectures only if every image is multi-arch.
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64", "arm64"]
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    # Wait before consolidating so freshly rescheduled pods are not moved again.
    consolidateAfter: 10m
    budgets:
      # Voluntarily disrupt at most 10% of this pool's nodes at once.
      # Spot reclamation is involuntary and is not limited by this budget.
      - nodes: "10%"

A conceptual Karpenter NodePool that allows spot with on-demand fallback across a wide set of instance shapes, and limits consolidation with a delay and a disruption budget.

Impact on cost and engineering time: a worked model and what to measure

Illustrative estimate (assumed numbers, not a benchmark): on-demand price P = $0.20 per node-hour, spot discount D = 65%, on-demand baseline B = 30% of required capacity, spot headroom H = 10% extra nodes to absorb reclaims, a fleet that requires 100 nodes, and a 5% daily interruption rate on spot nodes. None of these inputs are measured values. Substitute your own prices, discount and interruption history.

Blended cost = P × [B + (1 − B)(1 + H)(1 − D)] = 0.20 × [0.30 + 0.70 × 1.10 × 0.35] ≈ $0.114 per required node-hour.

With these assumed inputs, the illustrative result is roughly 43% savings, not the assumed 65% headline discount. In this example, the baseline and headroom together take about a third of the discount. The assumed 100-node fleet puts 70 required nodes on spot, which becomes 77 spot nodes once you add the 10% headroom. At the assumed 5% daily rate, that is about four reclaims a day. Each reclaim brings image pulls, cold caches, reset connections and alerts that someone has to triage. Bugs also get harder to reproduce when pods move constantly. These costs are paid in engineer hours, so add them to the model before you call the savings real.

Engineering time stays bounded when every service uses the same interruption-safe patterns: graceful shutdown, readiness gating and disruption budgets. It also helps to have one interruption mechanism, either the provisioner or a node termination handler but not both. Each new service then inherits tested behaviour, and on-call engineers deal with one failure mode.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes spot instances, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Consumers learn about a breaking change at review time rather than from production errors.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.