What is Karpenter consolidation and how does it work?

Consolidation is Karpenter's node autoscaling behavior for removing capacity. Provisioning adds nodes when pods are pending. Consolidation works in the other direction: it looks for nodes the cluster no longer needs, or nodes that cost more than their workload requires, and disrupts them. It takes one of two actions:

  • Delete: if every pod on a node can be rescheduled onto spare capacity on other existing nodes, Karpenter drains the node and terminates it.
  • Replace: if the pods can't all fit elsewhere but would fit on a single cheaper instance, Karpenter launches that replacement first, then drains the original. This catches the common case of a large node left mostly idle after a scale-down.

Karpenter doesn't act on utilization thresholds. It runs a scheduling simulation: it removes the candidate node from its model of the cluster and checks whether the displaced pods would schedule. The check honors their requests, node selectors, affinities, taints and topology spread constraints. A replacement counts as valid only when it is strictly cheaper than what it removes. Without that rule, a replacement could later become a candidate for replacement itself, and Karpenter would loop.

Karpenter ranks candidates by disruption cost, preferring nodes with fewer pods, lower-priority pods, or pods that are cheap to evict. It considers empty nodes, single nodes and groups of nodes. A group can sometimes be collapsed into one replacement when no single node in it could be removed on its own.

 candidate node
      │
      ▼
 simulate rescheduling its pods
      │
      ├── fits on existing nodes ──────────► DELETE
      ├── fits on one cheaper new node ────► REPLACE
      └── neither ─────────────────────────► SKIP
┌──────────────────────────────────────────────────────────┐
│ Candidate nodes: empty, single, or multi-node groups     │
│ Ranked by disruption cost: fewer pods, lower priority    │
└───────────────────────────┬──────────────────────────────┘
                            ▼
┌──────────────────────────────────────────────────────────┐
│ Simulate: remove candidate(s) from the cluster model     │
│ and reschedule displaced pods, honoring requests,        │
│ node selectors, affinities, taints, topology spread      │
└───────────────────────────┬──────────────────────────────┘
                            ▼
                ┌────────────────────────┐
                │ All pods fit on        │── yes ──►┌──────────────────┐
                │ existing nodes?        │          │ DELETE: drain,   │
                └───────────┬────────────┘          │ terminate node   │
                            │ no                    └──────────────────┘
                            ▼
                ┌────────────────────────┐
                │ Fit on one new node    │── yes ──►┌──────────────────┐
                │ strictly cheaper?      │          │ REPLACE: launch  │
                └───────────┬────────────┘          │ new, then drain  │
                            │ no                    │ and terminate    │
                            ▼                       └──────────────────┘
                ┌────────────────────────┐
                │ SKIP: keep node,       │
                │ try next candidate     │
                └────────────────────────┘

Karpenter ranks candidate nodes by disruption cost and simulates rescheduling their pods. It deletes the node if the pods fit on existing capacity, replaces it if they fit on one strictly cheaper node, and otherwise skips it.

Why does bin packing depend on resource requests, not actual usage?

Every branch in that simulation is evaluated against pod requests, not live utilization. The kube-scheduler places pods by subtracting requested CPU and memory from a node's allocatable capacity, and Karpenter uses the same arithmetic when it decides whether pods "fit" elsewhere. A pod that requests 1 vCPU but idles at 100m still reserves the full vCPU. Inflated requests make half-idle nodes look full, so there is nothing for consolidation to remove.

This makes right-sizing requests from observed usage the biggest lever on EKS costs. Size CPU from a high percentile with headroom, not the average, because averages under-provision for rare peaks. CPU overruns are throttled, but memory overruns get the container OOM-killed. Size memory from high percentiles or observed maximums.

Illustrative estimate. These inputs are assumed and the results are not measured. Assume 40 services with 3 replicas each (120 pods), nodes with 3.9 vCPU allocatable, memory is not the binding constraint, and DaemonSet overhead is ignored.

Before right-sizing After right-sizing
CPU request per pod 1000m 400m (p95 of 250m plus headroom)
Total requested CPU 120 vCPU 48 vCPU
Nodes required (÷ 3.9, rounded up) 31 13

Teams with many over-requested services benefit most, because the waste compounds across every replica. This hygiene lowers infrastructure cost in production and in every development and staging cluster before you tune any autoscaler settings.

How do you configure disruption policy on a NodePool?

Disruption settings live in the disruption block of each NodePool. The first decision is consolidationPolicy. WhenEmpty removes a node only after its last non-daemon pod leaves, which is safe but rarely reclaims much on a busy cluster. WhenEmptyOrUnderutilized also evicts pods from partly filled nodes so Karpenter can repack them onto fewer or cheaper machines. That is where most bin-packing savings come from.

Setting Effect Trade-off
WhenEmpty Removes idle nodes only Low churn, modest savings
WhenEmptyOrUnderutilized Repacks partly used nodes Higher savings, more pod restarts
consolidateAfter Waits before acting on a node Longer delay means less churn but slower savings
budgets Caps concurrent disruptions Protects capacity, slows consolidation

consolidateAfter stops Karpenter from reacting to every short dip in load. Without a delay, a deployment that scales down and back up within minutes can cause nodes to be created and removed repeatedly. Budgets set how many nodes, or what percentage, may be disrupted at once. A budget with nodes: "0", a cron schedule and a duration blocks voluntary disruption during a business-critical window such as month-end processing. You can also limit a budget to specific reasons so that removing empty nodes continues while repacking pauses.

The requirements block determines which instances Karpenter can choose from. Allow several instance categories and sizes through karpenter.k8s.aws/instance-category and instance-size, and set karpenter.sh/capacity-type to both on-demand and spot. If you pin a single instance type, Karpenter cannot replace two half-empty large nodes with one medium node, and bin packing stops working.

Spot capacity costs less, but AWS can reclaim it at short notice. Each reclaim moves pods in addition to the moves consolidation already causes. Run only workloads that tolerate frequent rescheduling on spot capacity.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-purpose
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
      requirements:
        # Broad choice lets Karpenter swap two half-empty nodes for one smaller one
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-size
          operator: NotIn
          values: ["nano", "micro", "small"]  # too small to be worth packing
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]  # spot is cheaper but can be interrupted
  disruption:
    # Repack partly used nodes, not just idle ones
    consolidationPolicy: WhenEmptyOrUnderutilized
    # Ignore short load dips so scale-down/scale-up cycles don't churn nodes
    consolidateAfter: 10m
    budgets:
      # Never disrupt more than 10% of this pool's nodes at once
      - nodes: "10%"
      # Weekday business hours (UTC): pause repacking, still remove empty nodes
      - nodes: "0"
        reasons: ["Underutilized"]
        schedule: "0 9 * * mon-fri"
        duration: 8h
      # First day of the month: block all voluntary disruption for processing
      - nodes: "0"
        schedule: "0 0 1 * *"
        duration: 24h

A Karpenter v1 NodePool that consolidates underutilized nodes after a 10-minute delay, caps concurrent disruptions with budgets (including scheduled freeze windows), and allows broad instance families, sizes and both spot and on-demand capacity so bin packing has room to work.

What blocks Karpenter from consolidating a node?

workloads, such as stateless workers and batch jobs.

Consolidation only works if pods can leave a node. These workload-side settings decide whether they can:

Control Effect on consolidation Typical reason it exists
PodDisruptionBudget with zero allowed disruptions Eviction refused, node stays Over-strict minAvailable
karpenter.sh/do-not-disrupt: "true" Node skipped entirely Long jobs, fragile pods
Pod with no owning controller Not evicted, since nothing would recreate it Ad-hoc debug pods
Local storage (hostPath, local PVs) Data is tied to the node Caches, scratch data
Zonal volume (e.g. EBS) Replacement must be in the same zone Databases, queues
Required anti-affinity or strict topology spread Fewer valid target nodes High availability

There is a direct trade-off between high availability and dense packing. Anti-affinity and zone spread deliberately keep replicas apart, and bin packing tries to pull them together. Decide how much spread each service actually needs. If you can, prefer preferredDuringScheduling affinity rules and ScheduleAnyway spread constraints over their hard versions.

Blocks are easy to miss. If PDBs are too strict or many pods carry do-not-disrupt, consolidation stops without raising any errors. The cluster looks correctly configured, but the bill stays the same.

To find out what is pinning a node:

  • Run kubectl get pdb -A and look for ALLOWED DISRUPTIONS set to 0.
  • Run kubectl get pods -A --field-selector spec.nodeName=<node> -o wide to list the node's pods.
  • Run kubectl describe pod on a suspect pod to check its annotations, owner references and volumes.
  • Run kubectl describe node <node> and kubectl describe nodeclaim <name>, then read the Events for disruption-blocked messages from Karpenter.

Put databases and other stateful systems on a separate NodePool. Give it conservative disruption settings, so that general-purpose pools can consolidate aggressively.

#!/usr/bin/env bash
# Show which workload-side controls are keeping a node from being consolidated.
# Usage: ./why-pinned.sh ip-10-0-42-17.ec2.internal
set -euo pipefail
NODE="${1:?usage: $0 <node-name>}"

echo "== PDBs that currently allow zero disruptions (cluster-wide) =="
kubectl get pdb -A --no-headers \
  -o custom-columns=NS:.metadata.namespace,NAME:.metadata.name,ALLOWED:.status.disruptionsAllowed \
  | awk '$3 == 0'

echo
echo "== Pods on ${NODE}: owner, do-not-disrupt annotation, volumes =="
# OWNER=<none> means a bare pod that nothing would recreate.
# DO_NOT_DISRUPT=true makes Karpenter skip the node.
# HOSTPATH and PVC point to node-local data or zonal volumes.
kubectl get pods -A --field-selector spec.nodeName="${NODE}" \
  -o custom-columns='NS:.metadata.namespace,POD:.metadata.name,OWNER:.metadata.ownerReferences[0].kind,DO_NOT_DISRUPT:.metadata.annotations.karpenter\.sh/do-not-disrupt,HOSTPATH:.spec.volumes[*].hostPath.path,PVC:.spec.volumes[*].persistentVolumeClaim.claimName'

echo
echo "== Recent events for the node (Karpenter reports blocked disruption here) =="
kubectl get events -A --field-selector involvedObject.name="${NODE}" \
  --sort-by=.lastTimestamp | tail -n 20

# Next steps: run 'kubectl describe pod -n <ns> <pod>' on any suspect pod above
# to check affinity and topology spread rules, then read the Events section
# of 'kubectl describe nodeclaim <name>' for the NodeClaim that backs this node.

A shell script that lists the PDBs, bare pods, do-not-disrupt annotations, local or zonal volumes and node events that can stop Karpenter from consolidating a given node.

How can you observe consolidation decisions and measure savings?

Karpenter records its reasoning in three places. Check them in this order.

Events. Karpenter attaches events to Node and NodeClaim objects. kubectl describe nodeclaim <name> or kubectl get events --field-selector involvedObject.kind=NodeClaim shows when a node was disrupted. More usefully, it shows why a node was not disrupted: a pod with the do-not-disrupt annotation, a PodDisruptionBudget allowing zero evictions, or a pod that no other node can schedule. These messages tell you which workload to fix.

Controller logs. The logs record each decision: which NodeClaims were selected, the disruption reason (empty, underutilized, drifted), and whether Karpenter deleted the node or replaced it with a cheaper one. Use them to correlate a consolidation with a latency spike or a burst of pod restarts.

Metrics. Karpenter exports Prometheus counters for disruption decisions and disrupted NodeClaims, labeled by reason. Metric names have changed between releases, so check the list for your installed version. Graph them alongside node count to see trends that individual events don't show.

To measure the effect of a change, track node count and the request ratio before and after:

request ratio = sum of pod CPU requests ÷ sum of node allocatable CPU

Illustrative estimate (assumed numbers, not a benchmark): 180 vCPU requested on 20 nodes with 15 allocatable vCPU each gives 180 ÷ 300 = 60%. After consolidation, 14 nodes gives 180 ÷ 210 ≈ 86%. Six fewer nodes at an assumed $0.68/hour saves 6 × 0.68 × 730 ≈ $2,980/month.

Without this data, finding out why a node stays up means guessing and checking pods one at a time. A single blocked event names the cause directly.

#!/usr/bin/env bash
# Snapshot consolidation signals. Run before and after a change, then diff.
set -euo pipefail
KARPENTER_NS="${KARPENTER_NS:-kube-system}"   # namespace depends on how Karpenter was installed

# 1. Events: why NodeClaims were (or were not) disrupted.
#    -A because events for cluster-scoped objects are not in your current namespace.
kubectl get events -A --field-selector involvedObject.kind=NodeClaim \
  --sort-by=.lastTimestamp | tail -n 20

# 2. Controller logs: recent disruption decisions (reason, replace vs delete).
kubectl logs -n "$KARPENTER_NS" deploy/karpenter --since=1h \
  | grep -i 'disrupt' | tail -n 20 || true

# 3. Node count for Karpenter-managed nodes (karpenter.sh/nodepool label).
NODES=$(kubectl get nodes -l karpenter.sh/nodepool --no-headers | wc -l)

# 4. Request ratio = sum(pod CPU requests) / sum(node allocatable CPU).
#    Counts scheduled, non-terminated pods across all nodes; ignores init containers.
CPU='def cpu: if endswith("m") then (.[:-1]|tonumber/1000) else tonumber end;'

REQUESTED=$(kubectl get pods -A -o json | jq "$CPU"'
  [.items[]
   | select(.spec.nodeName != null)
   | select(.status.phase != "Succeeded" and .status.phase != "Failed")
   | .spec.containers[] | (.resources.requests.cpu // "0") | cpu] | add // 0')

ALLOCATABLE=$(kubectl get nodes -o json | jq "$CPU"'
  [.items[] | .status.allocatable.cpu | cpu] | add // 0')

awk -v n="$NODES" -v r="$REQUESTED" -v a="$ALLOCATABLE" 'BEGIN {
  printf "karpenter_nodes=%d requested_vcpu=%.1f allocatable_vcpu=%.1f request_ratio=%.0f%%\n",
         n, r, a, (a > 0 ? 100 * r / a : 0)
}'

A bash snapshot script that prints recent NodeClaim events and Karpenter disruption log lines, then computes node count and the CPU request-to-allocatable ratio so you can compare runs before and after a consolidation change.

When should you avoid aggressive consolidation?

Consolidation assumes a pod can be moved cheaply. When that assumption fails, the savings come at the cost of user-visible latency.

Workloads that pay for every move

  • Slow starters. A JVM service that needs 90 seconds to boot, or a cache that takes ten minutes to warm, serves degraded traffic after every eviction.
  • Pods without readiness probes. Kubernetes routes traffic to them as soon as the container starts, before the process can actually serve requests.
  • Bursty services. Requests size the steady state, so packing nodes tightly removes the spare capacity a spike would have used. Pods with CPU limits get throttled. Pods without memory limits push the node into memory pressure and kubelet evictions.
  • Pods that land in Pending. If the remaining nodes can't fit a displaced pod, it waits for a new node to boot and pull its images.

Churn

Churn is a node removed and then re-provisioned minutes later, often because a daily traffic pattern crosses the utilization threshold back and forth. Each cycle adds image pulls, cold starts, scheduler work and calls to the Kubernetes and cloud APIs. The instance-hours saved can be small next to the disruption caused.

Trading savings for stability

A longer consolidateAfter delay makes Karpenter wait out short dips in demand, so it acts only on sustained low usage. Disruption budgets limit how many nodes can be drained at once, or block draining during peak hours. Both settings leave some idle capacity running. That idle capacity is the price of predictable behavior.

For NodePools that run stateful, latency-critical or slow-starting workloads, use the empty-only policy (WhenEmpty), so only nodes with no workload pods are removed. Where any disruption is unacceptable, disable consolidation for that NodePool and keep it enabled for stateless pools that tolerate rescheduling.

What is a safe rollout sequence, and what are the alternatives?

Stage the rollout

  1. Empty-node consolidation first. Set consolidationPolicy: WhenEmpty. Removing nodes that run no workload pods carries almost no risk, and it shows you how quickly Karpenter reclaims capacity left behind after scale-down.
  2. Harden critical services. Before Karpenter moves running pods, give each service a PodDisruptionBudget and an accurate readiness probe. The PDB limits how many replicas can be evicted at once. The probe keeps traffic away from a replacement pod until that pod can serve it.
  3. Right-size requests. Bin packing works on the numbers you declare. If requests are inflated, nodes look full and Karpenter has nothing to consolidate. If requests are too low, Karpenter packs pods so densely that they get throttled or OOM-killed.
  4. Enable underutilized consolidation with tight budgets. Switch to WhenEmptyOrUnderutilized and allow only one node, or a small percentage of nodes, to be disrupted at a time. Widen the budget only after a quiet period.

Each stage removes one class of failure before the next stage can trigger it. This keeps unplanned incident work small. A bad eviction caught at stage 4 points to a single cause, rather than to an unclear mix of missing PDBs and wrong requests.

Alternatives

Approach Bin packing ability Operational effort Fit
Cluster Autoscaler with node groups Limited to the fixed instance types in each group Medium: you maintain the node groups Clusters already standardized on a few instance types
Manual right-sizing alone None on its own, because nodes are not replaced Ongoing and manual Small, stable clusters
Overprovisioning with low-priority placeholder pods Works against packing by reserving idle headroom on purpose Low Workloads where scale-up latency costs more than idle nodes
Scheduled scaling None. Capacity follows a clock, not the scheduler Low once set up Predictable daily or weekly traffic patterns

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to karpenter consolidation, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.