What is Karpenter consolidation and why does idle capacity build up?

Karpenter consolidation is a continuous bin-packing loop. Karpenter repeatedly checks whether the pods on its nodes could run on fewer or cheaper nodes. If they can, it drains the surplus nodes and terminates them. Consolidation is part of how Karpenter manages capacity all the time, not a cleanup job you schedule.

Idle capacity builds up because scale-out and scale-in are not symmetric. The Cluster Autoscaler adds nodes to a node group when pods are pending, and it does this quickly. It removes nodes cautiously. A node becomes a candidate only after its requested utilization stays below a threshold for a sustained period, and only if every pod on it can move elsewhere. It also cannot replace a node with a smaller one, because each group has a fixed instance shape. After a traffic spike subsides or a rolling deploy ends, pods end up spread thinly across many nodes. Most of those nodes are too busy to remove but too empty to justify their cost.

Karpenter takes a different approach. For each provisioning decision, it picks an instance type from the set your NodePool allows, based on the pending pods' requests and constraints. Later, it can move those pods onto fewer or smaller nodes.

Aspect Cluster Autoscaler Karpenter
Unit of scaling Predefined node group Individual node
Instance choice Fixed per group Chosen per scheduling decision
Scale-in Removes underused nodes conservatively Deletes or replaces nodes to repack
Right-sizing No Can swap to a cheaper instance

Platform teams benefit most when their clusters have variable load, many heterogeneous workloads, or frequent deploys. These are the conditions where fragmentation builds up fastest.

How does Karpenter consolidation decide which nodes to remove?

Karpenter treats every candidate node as a scheduling question: if this node disappeared, could its pods land somewhere else? It answers that by simulating the pods against the remaining nodes and, if needed, one new node it would launch. It acts only when two conditions hold: every displaced pod still fits, and the resulting set of nodes costs less than the current one. If either check fails, the node stays.

Candidates are evaluated in three ways:

  • Empty-node deletion. A node running only daemonset pods is removed outright. This is the cheapest decision because nothing needs rescheduling.
  • Multi-node deletion. Karpenter looks for a group of nodes whose pods can be absorbed by the rest of the cluster, optionally plus a single cheaper replacement. This is where most bin packing gains come from, because several half-empty nodes often collapse into one.
  • Single-node replacement. If one node's pods can't fit elsewhere but would fit on a smaller or cheaper instance type, Karpenter launches that instance, then drains and deletes the original.

When choosing among candidates, Karpenter favors the nodes that are cheapest to disrupt, such as those running fewer pods or lower-priority pods.

The simulation only uses the pod specs it can see. A mutating admission webhook that injects a sidecar, or raises resource requests, changes the pods after the decision is made. Underestimated daemonset overhead causes the same problem. Either way, the replacement node turns out too small and pods go pending. Karpenter then provisions more capacity, and later consolidates it again. The result is a replacement loop that churns nodes without saving money.

+--------------------+
| Candidate node(s)  |
+---------+----------+
          |
          v
+------------------------------------------+
| Simulate pods on remaining nodes         |
| (+ at most one new, cheaper node)        |
+---------+--------------------------------+
          |
          v
  +-------+-----+      no      +-------------+
  | All pods    +------------->| Keep node   |
  | still fit?  |              +-------------+
  +-------+-----+                     ^
          | yes                       | no
          v                           |
  +-------+-----+                     |
  | Costs less? +---------------------+
  +-------+-----+
          | yes
       +--+-------------+-------------------+
       v                v                   v
+-------------+ +---------------+ +--------------------+
| Empty node: | | Multi-node:   | | Single-node: launch|
| delete      | | delete group  | | cheaper, drain old |
+-------------+ +---------------+ +--------------------+
Divergence: webhook mutation or low daemonset estimate -> node too small
-> pods Pending -> scale up -> consolidate again -> replacement loop

Karpenter removes or replaces a node only if the simulation shows every pod still fits and the result costs less; pod changes the simulation cannot see can cause replacement loops.

How do you configure NodePool disruption settings for consolidation?

Most of that churn is controlled by the disruption block of a NodePool and by how much freedom its requirements give the scheduler.

The consolidation policy sets which nodes Karpenter may act on. In the current API the values are WhenEmpty and WhenEmptyOrUnderutilized. Under WhenEmpty, Karpenter removes only nodes that run no workload pods, not counting DaemonSets. Under WhenEmptyOrUnderutilized, it also deletes or replaces nodes whose pods fit elsewhere or on a cheaper node. consolidateAfter sets how long a node must stay a candidate before Karpenter acts. A delay of a few minutes means a brief traffic dip or a rolling deploy does not trigger a repack that you reverse shortly afterward.

The field names and allowed values have changed across Karpenter API versions. Older releases used different policy names and limited where consolidateAfter could be set. Check every field against the documentation for the version you run.

Requirements matter as much as the policy. If you pin a NodePool to one instance type, every replacement node is the same shape. Karpenter can only remove whole nodes and cannot pick a smaller one, which brings back the fragmentation of a fixed node group. Allow several instance families and sizes, and both spot and on-demand capacity types if your workloads tolerate interruption. Use requirements to exclude only what you must, such as an architecture your images don't support.

Start with WhenEmpty if your resource requests are untrusted or your PodDisruptionBudgets are incomplete. Switch to WhenEmptyOrUnderutilized once both are in place.

# Karpenter v1 API (karpenter.sh/v1). Field names and allowed values differ in
# older releases, so check them against the docs for the version you run.
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-purpose
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws      # provider-specific; use your provider's NodeClass
        kind: EC2NodeClass
        name: default
      requirements:
        # Several families give consolidation more shapes to repack onto
        - key: karpenter.k8s.aws/instance-family
          operator: In
          values: ["c6i", "m6i", "r6i", "c7i", "m7i"]
        # Allow many sizes so Karpenter can replace a node with a smaller one
        - key: karpenter.k8s.aws/instance-size
          operator: NotIn
          values: ["metal"]
        # Spot plus on-demand, only if workloads tolerate interruption
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        # Exclude only what you must, e.g. an architecture your images lack
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
  disruption:
    # Start with WhenEmpty until requests and PDBs are trustworthy,
    # then switch to WhenEmptyOrUnderutilized
    consolidationPolicy: WhenEmptyOrUnderutilized
    # Node must stay a candidate this long, so short dips don't trigger a repack
    consolidateAfter: 5m
  limits:
    cpu: "1000"

A Karpenter v1 NodePool that allows several instance families, sizes and capacity types and sets consolidationPolicy and a five-minute consolidateAfter delay so consolidation can repack nodes without reacting to brief dips.

Why do accurate resource requests matter for bin packing?

Karpenter's scheduling simulation only sees what pods request. A node running pods that request 3.5 of 4 vCPUs looks nearly full to the consolidation controller, even if those pods actually use 0.4 vCPU. Inflated requests make idle capacity invisible. Consolidation cannot remove a node it believes is busy, so the waste persists no matter how you tune the disruption policy.

The opposite error is just as costly. Pods with low or missing requests let Karpenter pack more of them onto a node than it can sustain. CPU contention then shows up as throttling and latency spikes. Memory overcommit leads to node pressure, kubelet evictions and OOM kills. Consolidation makes this worse, because every repack is another chance to place under-requested pods tightly together.

Before you switch to WhenEmptyOrUnderutilized, compare requests with observed usage:

  • Allocated requests: kubectl describe node <name> shows the "Allocated resources" section, which lists summed requests and limits as a percentage of allocatable.
  • Live usage: kubectl top node and kubectl top pod --all-namespaces show current usage. Both require metrics-server.

A consistent gap, such as 80% requested and 20% used, means you should fix requests before tuning consolidation. Base new requests on usage over a representative period, not on a single snapshot.

Be careful with the Vertical Pod Autoscaler in update mode. When VPA changes requests, it evicts and recreates pods. The new pod sizes change how pods fit on nodes, which can trigger another consolidation pass. Running both can cause repeated repacking. Use VPA in recommendation-only mode to get request suggestions, then apply them deliberately.

#!/usr/bin/env bash
# Compare requested resources with observed usage on one node
# before enabling WhenEmptyOrUnderutilized consolidation.
# Requires metrics-server for the 'kubectl top' commands.
set -euo pipefail

NODE="${1:?usage: $0 <node-name>}"

# List Karpenter-managed nodes so you can pick candidates to inspect.
kubectl get nodes -l karpenter.sh/nodepool

# What the scheduler and Karpenter see: summed requests and limits
# as a percentage of the node's allocatable capacity.
echo "== Requested (scheduler view) =="
kubectl describe node "$NODE" | grep -A 8 "Allocated resources"

# What the node is actually using right now.
echo "== Live usage (point-in-time snapshot) =="
kubectl top node "$NODE"

# Per-pod requests on this node. Empty fields mean missing requests,
# which lets pods be overpacked.
echo "== Pod requests on $NODE =="
kubectl get pods --all-namespaces \
  --field-selector spec.nodeName="$NODE" \
  -o custom-columns='NS:.metadata.namespace,POD:.metadata.name,CPU_REQ:.spec.containers[*].resources.requests.cpu,MEM_REQ:.spec.containers[*].resources.requests.memory'

# Per-container live usage across the cluster, highest CPU first.
# Match these against the requests above to find inflated or missing values.
echo "== Top pods by CPU usage =="
kubectl top pod --all-namespaces --containers --sort-by=cpu | head -n 20

# A single snapshot is not enough to size requests. Repeat over a
# representative period, or use your metrics history, before changing them.

A shell script that puts a node's summed requests next to its live usage and per-pod requests, so you can spot inflated or missing requests before you tune Karpenter consolidation.

How do you protect workloads while Karpenter repacks nodes?

Protection controls decide which nodes Karpenter is allowed to drain, and how fast. Each one is easy to misconfigure into a permanent block.

PodDisruptionBudgets belong on every replicated service. Karpenter evicts pods through the eviction API, so it honors PDBs. That lets consolidation drain a node without taking a service below the replica count it needs. The common mistake is a PDB that permits zero evictions, such as minAvailable equal to the replica count, or a single-replica Deployment with minAvailable: 1. Karpenter cannot drain any node hosting that pod, so it skips the node.

NodePool disruption budgets limit how many nodes Karpenter disrupts at once, as a count or a percentage. They can also be tied to a schedule and duration, for example to block consolidation during business hours. A budget of zero nodes, or a window that never opens in practice, stops consolidation for the whole NodePool.

The karpenter.sh/do-not-disrupt annotation protects long-running batch jobs that would lose hours of work if evicted. Remove it, or let the pods finish, so these nodes do not stay pinned forever.

Control Scope Silent failure mode
PodDisruptionBudget Per workload Allows zero evictions, so the node is skipped
NodePool budget Per NodePool Zero budget or closed window blocks everything
do-not-disrupt Pod or node Stale annotation pins the node

None of these produce an error. Check events on the node and NodeClaim to see why a node was skipped.

On spot capacity, interruptions and consolidation both disrupt pods, so your PDBs must tolerate both happening together. Whether Karpenter replaces spot nodes with other spot nodes depends on version and configuration, so verify that behavior in the docs for the version you run.

# 1) PDB that permits voluntary evictions. maxUnavailable: 1 lets Karpenter drain
#    one replica at a time; minAvailable == replicas would block every drain.
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: checkout-api
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app: checkout-api
---
# 2) NodePool disruption budgets (karpenter.sh/v1). spec.template, including
#    nodeClassRef and requirements, is omitted for brevity.
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 5m
    budgets:
      - nodes: "10%"               # never disrupt more than 10% of nodes at once
      - nodes: "0"                 # no consolidation of underutilized nodes...
        schedule: "0 9 * * mon-fri" # ...starting 09:00 UTC on weekdays
        duration: 8h               # ...for 8 hours; outside it, the 10% cap applies
        reasons: ["Underutilized"]
---
# 3) Batch Job pods that Karpenter must not evict voluntarily. The annotation is
#    on the pod template, so it goes away when the pods finish.
apiVersion: batch/v1
kind: Job
metadata:
  name: nightly-reindex
spec:
  template:
    metadata:
      annotations:
        karpenter.sh/do-not-disrupt: "true"
    spec:
      restartPolicy: Never
      containers:
        - name: reindex
          image: registry.example.com/reindex:1.0.0

A PDB that still allows evictions, a NodePool budget that caps concurrent disruptions and pauses underutilization consolidation during weekday hours, and a batch Job whose pods carry the do-not-disrupt annotation.

Why is Karpenter not consolidating a node?

Karpenter records why it skipped a node, so read that before you change any settings. Start with kubectl describe node <name> and kubectl describe nodeclaim <name>. Karpenter posts events on both objects when disruption is blocked, and the message usually names the pod or policy responsible. If the events have aged out, search the controller logs for the node name with kubectl logs -n <karpenter-namespace> deploy/karpenter | grep <node-name>. Event reasons and log wording vary across releases, so search by node name and pod name, not by a specific reason string.

Blocker Why Karpenter won't move it Typical fix
Pod with no controller A bare pod would not be recreated after eviction Wrap it in a Deployment or Job
PDB with zero allowed disruptions Eviction would violate the budget Add replicas or relax minAvailable
Local PV or hostname affinity The pod can only run on this specific node Use network-attached storage or drop the hostname pin
Unsatisfiable topology spread No cheaper node layout keeps the skew within limits Raise maxSkew or use ScheduleAnyway
karpenter.sh/do-not-disrupt on the pod or node An explicit opt-out Remove it once the protected work is finished

One blocker can hide another. A node held only by a PDB may also host a pinned pod that blocks it after you fix the budget. Clear every reason listed in the events, then check the node again.

This takes minutes. The alternative is noticing that the bill didn't drop, inferring that consolidation isn't working, and then tuning consolidateAfter or instance families, none of which affects a node that a single uncontrolled pod is holding in place.

When should you avoid aggressive consolidation, and what are the alternatives?

Consolidation trades stability for density, and some workloads pay more for the churn than the cluster saves. Each repack evicts pods. A JVM service with a long warm-up, a cache that rebuilds from a cold start, or a pod holding long-lived WebSocket connections loses real capacity during every move, and new nodes still have to pull images before anything runs. Dense packing also removes slack. When a spike arrives, pods sit Pending until a node launches and becomes ready, which can take minutes. Consolidating onto larger instances means one node failure or interruption takes out more replicas at once. If Cluster Autoscaler also manages the same pools, or VPA keeps resizing requests while Karpenter repacks, the controllers can undo each other's work. Each pool or workload needs exactly one owner for node lifecycle.

Approach Idle reduction Churn Operational effort
Karpenter consolidation High High Ongoing tuning of budgets and PDBs
Cluster Autoscaler, tuned scale-down Moderate Moderate Threshold and delay tuning per node group
Fixed, right-sized node groups Low to moderate Minimal Periodic manual review
Low-priority placeholder pods Negative (headroom costs money) Low Sizing the buffer
Request right-sizing alone Moderate Rolling restarts only Recurring request review

These options can be combined. Placeholder pods can run under consolidation to keep a known amount of headroom. Before you invest in tuning, put a rough number on it. If an engineer spends several days each quarter on disruption budgets and incident follow-ups, the savings from removing a few idle nodes may not cover that time.

How do you measure whether consolidation actually cut idle capacity?

Track four signals over at least a few weeks, sampled at comparable traffic points so daily peaks don't distort the comparison:

  • Allocatable-to-requested ratio. Divide the cluster's total allocatable CPU (and, separately, memory) by the sum of pod requests. A value near 1.0 means nodes are tightly packed. A falling ratio is the most direct sign that consolidation is working.
  • Node count. This should drop or flatten while requests stay steady. If requests changed during the same window, normalize for that before giving consolidation credit.
  • Node churn. Count nodes created and terminated per day. High churn with a flat ratio means Karpenter is repacking without reducing waste.
  • Workload SLOs. Watch eviction-related errors, container restart rates, and latency during disruption windows. Savings that cost you error budget are not savings.

Worked example (illustrative estimate)

Assume identical nodes with 8 allocatable vCPU, and assume total requests hold at 96 vCPU before and after:

Metric Before After
Nodes 20 14
Allocatable vCPU 160 112
Requested vCPU 96 96
Idle requested capacity (allocatable − requested) 64 vCPU (40%) 16 vCPU (~14%)
Allocatable-to-requested ratio ~1.67 ~1.17

To apply this to your own cluster, substitute your node shapes and request totals, then run the same calculation for memory. Memory is often what blocks further packing. If the idle percentage falls but restart rates rise, loosen the disruption settings before you take the savings.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to karpenter consolidation, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.