What is Karpenter consolidation and how does it differ from Cluster Autoscaler scale-down?

Karpenter consolidation is the process by which Karpenter continuously looks for nodes it can delete, or replace with a cheaper instance, while still fitting every running pod's resource requests somewhere in the cluster. It actively repacks the cluster after it has scaled up.

Cluster Autoscaler also scales down. It removes a node when the node's requested capacity is low and its pods can be rescheduled onto other existing nodes. The limit is structural. Cluster Autoscaler works through node groups, and each group has a fixed instance shape. It can shrink a group, but it cannot turn three half-empty m-class nodes into one right-sized node of a different family. Karpenter picks instance types from a broad allowed set defined in the NodePool. That lets it bin-pack across shapes. It can delete a node outright, or provision a smaller or cheaper replacement and drain the old one onto it.

Cluster Autoscaler scale-down Karpenter consolidation
Unit of change Node within a fixed node group Individual node, any allowed type
Can switch instance type No Yes, replaces with a cheaper fit
Packing scope Within existing group shapes Across instance families and sizes
Tuning burden Node group design NodePool requirements and disruption rules

There are two alternatives. You can run Cluster Autoscaler with carefully sized node groups, or you can periodically right-size node groups by hand. Both work when workload shapes are stable and predictable.

Consolidation pays off most for teams with mixed pod shapes, load that rises and falls, and workloads that tolerate being moved. Enabling consolidation does not save money by itself. Savings depend on accurate resource requests, a wide enough instance set, and pods that are allowed to move.

How does Karpenter consolidation decide which nodes to remove or replace?

Karpenter evaluates candidate nodes by simulating where their pods would go. It considers four actions:

  • Empty node deletion: a node running only DaemonSet pods is removed.
  • Single-node deletion: an underutilized node is removed when all its pods fit into spare capacity on other nodes.
  • Single-node replacement: a node is swapped for one cheaper instance that still fits its pods.
  • Multi-node consolidation: several nodes are evaluated together and replaced by fewer nodes, or by one cheaper node. Freeing three half-empty nodes at once often saves more than removing any one of them.

The simulation uses pod resource requests, not observed usage. A pod that requests 2 vCPU and uses 200m still reserves 2 vCPU in the math, so inflated requests block packing even on idle-looking nodes. Requests set too low cause the opposite problem. Karpenter packs nodes densely on paper, and real usage then causes memory pressure, evictions, or CPU throttling.

The instance set determines how many replacement options exist. A NodePool that allows several families, many sizes, both arm64 and amd64, and both spot and on-demand gives the simulator many shapes to try. A NodePool restricted to one instance type leaves Karpenter able to delete nodes but rarely able to replace them, which is close to node-group autoscaling.

The table traces an illustrative example: three nodes of the same type, each about 40% requested, before and after a multi-node consolidation.

Before Pods (requested vCPU) After
Node A (8 vCPU) api ×2 (3) Node X (8 vCPU): api ×2, worker ×2 (6)
Node B (8 vCPU) worker ×2 (3) Node Y (4 vCPU, cheaper): cron, cache (3.5)
Node C (8 vCPU) cron, cache (3.5) removed
 BEFORE: 3 nodes x 8 vCPU = 24 vCPU provisioned

+------------------+  +------------------+  +------------------+
| Node A  8 vCPU   |  | Node B  8 vCPU   |  | Node C  8 vCPU   |
| api x2      3.0  |  | worker x2   3.0  |  | cron,cache  3.5  |
| 37.5% requested  |  | 37.5% requested  |  | 44% requested    |
+--------+---------+  +--------+---------+  +--------+---------+
         |                     |                     |
         +---------------------+---------------------+
                               |
                               v
     +--------------------------------------------------+
     | Multi-node consolidation simulation              |
     | - sums pod resource REQUESTS, not observed usage |
     | - tries shapes allowed by NodePool requirements  |
     | - acts only if every pod fits and cost goes down |
     +------------+-------------------------+-----------+
                  |                         |
                  v                         v
      +-----------------------+ +-----------------------+
      | Node X  8 vCPU        | | Node Y 4 vCPU, cheaper|
      | api x2, worker x2  6.0| | cron, cache        3.5|
      | 75% requested         | | 87.5% requested       |
      +-----------------------+ +-----------------------+

 AFTER: 2 nodes, 12 vCPU provisioned; Node C drained and removed

Illustrative multi-node consolidation: Karpenter sums the pod requests on three lightly requested 8 vCPU nodes and repacks them onto one 8 vCPU node and one cheaper 4 vCPU node.

How do you configure consolidation policy, delays and disruption budgets in a NodePool?

All of this lives in the NodePool's disruption block. Field names and allowed values have changed across Karpenter releases. Older APIs used different policy names and placed the delay setting elsewhere, so check every field below against the docs for your installed version before you apply anything.

Policy

consolidationPolicy controls which nodes are eligible. WhenEmpty removes only nodes that run no non-daemon pods. This is safe, but it rarely produces the repacking shown in the previous table. WhenEmptyOrUnderutilized also lets Karpenter delete or replace nodes whose pods fit elsewhere, and that is where most of the savings come from.

Delay

consolidateAfter sets how long a node must stay a candidate before Karpenter acts on it. A short value captures savings quickly. It can also cause churn when traffic dips briefly and then returns, because the pods you just evicted need a new node minutes later. Set the delay to at least the length of your normal load oscillations.

Budgets

budgets limit how many nodes can be disrupted at once, expressed as a count or a percentage. You can attach a schedule and duration to a budget to define a time window. The accompanying example sets nodes: "0" during business hours, which blocks voluntary disruption while users are active and lets consolidation run overnight. Some versions also let you scope a budget to specific disruption reasons.

Spot and on-demand

If your requirements allow both capacity types, consolidation can replace on-demand nodes with cheaper Spot capacity. Replacing one Spot node with another Spot node has historically been handled more cautiously. It has been gated behind a feature flag and has required a wide spread of allowed instance types, which is one reason the example's requirements are broad. Spot interruption handling is a separate mechanism, but it also drains nodes, so account for both when you size your budgets. Verify the current Spot behavior in the docs for your version.

# Written against the karpenter.sh/v1 API. Field names and values differ in
# older releases, so check them against the docs for your installed version.
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-purpose
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default            # placeholder EC2NodeClass
      requirements:
        # Broad requirements give consolidation more replacement options
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["spot", "on-demand"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["c", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["5"]
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
  limits:
    cpu: "1000"                  # cap on total provisioned vCPU
  disruption:
    # Allow deletion and replacement of underutilized nodes, not only empty ones
    consolidationPolicy: WhenEmptyOrUnderutilized
    # A node must remain a candidate this long before Karpenter acts on it
    consolidateAfter: 10m
    budgets:
      # Outside the window below, disrupt at most 10% of nodes at a time
      - nodes: "10%"
      # Block voluntary disruption on weekdays from 09:00 for 8h (cron is UTC)
      - nodes: "0"
        schedule: "0 9 * * mon-fri"
        duration: 8h

A karpenter.sh/v1 NodePool that allows both Spot and on-demand across a broad set of instance types, consolidates empty and underutilized nodes after a 10-minute delay, and uses a weekday business-hours budget to block voluntary disruption.

How do workloads protect themselves from consolidation?

Karpenter evicts pods through the Kubernetes eviction API, so it respects PodDisruptionBudgets. Before it disrupts a node, it simulates evicting every pod on it. If any eviction would violate a PDB, that node is skipped. A budget like maxUnavailable: 1 on a selector matching your API pods lets consolidation proceed one replica at a time. minAvailable equal to the replica count blocks it entirely.

For a stronger opt-out, set the karpenter.sh/do-not-disrupt: "true" annotation. On a pod, it makes Karpenter treat the pod's node as off-limits for voluntary disruption while the pod runs. On a node, it exempts the whole node. Neither annotation prevents involuntary events such as Spot reclamation.

Strict combinations that pin nodes

Protections compound. Suppose required pod anti-affinity puts one replica per node, a topology spread constraint uses whenUnsatisfiable: DoNotSchedule, and a PDB allows zero disruptions. Karpenter may find no valid placement for the evicted pods, so it leaves those nodes in place. Each node can sit mostly empty indefinitely. No alert fires, because nothing has failed. The cost shows up only as persistently low utilization on specific nodes. Audit PDBs that allow zero disruptions as carefully as you audit resource requests.

Workloads that need explicit protection

  • Long-running batch jobs. If a pod is evicted three hours into a four-hour run, those three hours of compute are lost and must be paid for again. Annotate these pods with do-not-disrupt.
  • Stateful workloads. Rebalancing, leader election and replication catch-up after an eviction consume capacity. Use a PDB sized to your quorum.
  • Pods with local storage. Cached or scratch data on emptyDir or local volumes is lost on eviction and must be rebuilt.
# PDB: let consolidation drain API pods one replica at a time
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
  namespace: prod
spec:
  maxUnavailable: 1          # minAvailable == replicas would block consolidation entirely
  selector:
    matchLabels:
      app: api
---
# Long-running batch Job: opt its pods out of voluntary disruption
apiVersion: batch/v1
kind: Job
metadata:
  name: nightly-report
  namespace: batch
spec:
  backoffLimit: 2
  template:
    metadata:
      annotations:
        # Karpenter will not voluntarily disrupt the node while this pod runs.
        # Spot reclamation and other involuntary events still apply.
        karpenter.sh/do-not-disrupt: "true"
    spec:
      restartPolicy: Never
      containers:
        - name: report
          image: registry.example.com/reports/nightly:1.4.2
          resources:
            requests:
              cpu: "2"
              memory: 4Gi
          volumeMounts:
            - name: scratch          # emptyDir data is lost on eviction
              mountPath: /scratch
      volumes:
        - name: scratch
          emptyDir: {}

A PodDisruptionBudget that lets Karpenter consolidate API pods one replica at a time, and a batch Job whose pods carry the karpenter.sh/do-not-disrupt annotation to protect hours of in-progress work and scratch data.

How much does Karpenter consolidation actually save? A worked cost model

Illustrative estimate (assumed numbers, not a benchmark)

Assumed inputs: 40 identical nodes at $0.40/hour, 730 hours/month. Before consolidation, pods request 55% of allocatable capacity on average. After consolidation, assume packing reaches 80%. A separate right-sizing pass then cuts total requests by 25%.

Formula: nodes = ceil(total requests ÷ (allocatable per node × packing ratio)), and monthly cost = nodes × hourly price × 730.

Total requests are 40 × 0.55 = 22 node-equivalents.

Scenario Requests (node-equiv.) Packing ratio Nodes Monthly cost
Baseline 22 0.55 40 $11,680
Consolidation only 22 0.80 28 $8,176
Consolidation + right-sized requests 16.5 0.80 21 $6,132

In this model, consolidation alone saves about $3,500/month. Right-sizing saves another $2,000. These are separate levers. Karpenter packs pods by what they request, so inflated requests cap what consolidation can recover.

What it doesn't touch

Consolidation does not reduce these costs:

  • Load balancers
  • Persistent volumes and snapshots
  • NAT and cross-zone data transfer
  • Add-ons such as logging or mesh sidecars that are oversized per node

Repacking can also raise transfer costs. Chatty services that used to share a zone may land in different zones after consolidation. Check your zone-to-zone transfer line item before and after rollout.

Engineering cost

Tuning budgets, PDBs and requests takes ongoing effort. With manually sized node groups, someone has to keep revisiting group sizes and instance types as workloads drift. Karpenter re-evaluates node selection continuously. The remaining human work is keeping requests honest, and that work pays off in both columns of the table.

When should you avoid aggressive consolidation?

Tight packing has a cost that does not appear on the compute bill. Each consolidation action evicts pods, and for some workloads one eviction costs more than an idle node does.

Services that take minutes to become ready, such as JVM apps warming up, caches rebuilding from cold, or pods loading large models, lose capacity during every move. Long-lived connections like WebSockets, gRPC streams and database sessions drop and reconnect. Pods that hold in-memory state lose it. Running near full capacity also removes headroom, so a traffic spike has to wait for new nodes to provision when it could have been absorbed by spare capacity. If teams lower requests to fit more pods per node, actual usage can exceed what was reserved, which brings noisy-neighbor contention and OOM kills.

A broad instance set creates its own problems. Mixed families and generations make latency less predictable. Arm nodes require multi-arch images. Per-core licensing becomes harder to track. Capacity reservations or committed-use discounts tied to specific families can go unused while Karpenter picks cheaper options elsewhere.

Situation Better choice
Stateless, fast-starting, PDB-covered services Full consolidation
Slow startup, stateful, or connection-heavy pods Empty-only consolidation
Spend committed to fixed instance families Constrained NodePool or Cluster Autoscaler with fixed node groups
No capacity to keep requests honest Empty-only until request tuning is staffed

Treat tuning as ongoing work. Put sensitive workloads in a separate NodePool with an empty-only policy, and allow full consolidation only where evictions are cheap.

How should you roll out and measure Karpenter consolidation in production?

Roll out in stages so each step fails visibly before the next adds risk.

Staged rollout

  1. Empty-only first. Removing nodes with no workload pods evicts nothing, so you get immediate savings while you collect a baseline.
  2. Audit requests. Underutilized consolidation packs pods by their requests, not their actual usage. Inflated requests make nodes look full, and undersized requests cause OOM kills after pods are packed tighter. Compare requests with observed usage and correct the largest outliers.
  3. Add PDBs to every critical service before any of its pods can be evicted.
  4. Set conservative budgets. Cap disruptions to a small number of nodes and allow them only during business hours, when people are available to respond.
  5. Enable underutilized consolidation on one NodePool at a time. Watch for churn, where the same capacity is repeatedly replaced.

What to measure

  • Requests vs. allocatable per node, over time. Use kubectl describe node <node> and read the "Allocated resources" section for a spot check. For trends, chart the same ratio from your metrics stack.
  • Consolidation-driven disruptions. Run kubectl get events -A --sort-by=.lastTimestamp and filter for Karpenter-sourced reasons. Run kubectl get nodeclaims to see node turnover.
  • Pod restarts and scheduling latency after evictions. If evicted pods stay Pending for long, consolidation is cutting capacity too close.

Reconcile node-hours against your billing data rather than modeled prices, because discounts and Spot rates change the real numbers.

Revisit these settings after major workload changes and after every Karpenter upgrade. Disruption behavior and defaults can shift between releases.

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general-purpose
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default
  disruption:
    # Stage 1 ran this NodePool with WhenEmpty. Stage 5 switches only this
    # NodePool to underutilized consolidation, after requests and PDBs are fixed.
    consolidationPolicy: WhenEmptyOrUnderutilized
    # Wait before consolidating so short dips in load don't cause churn.
    consolidateAfter: 10m
    budgets:
      # Baseline: never disrupt more than one node at a time.
      - nodes: "1"
      # Block underutilized consolidation outside business hours.
      # Schedules are evaluated in UTC; adjust to your on-call hours.
      # Daily 17:00 for 16h blocks evenings and nights until 09:00.
      - nodes: "0"
        reasons: ["Underutilized"]
        schedule: "0 17 * * *"
        duration: 16h
      # Block all of Saturday and Sunday.
      - nodes: "0"
        reasons: ["Underutilized"]
        schedule: "0 0 * * sat"
        duration: 48h
      # Empty-node removal evicts no workload pods, so it stays allowed
      # off-hours under the one-node baseline.

A Karpenter v1 NodePool with underutilized consolidation turned on for one pool, a one-node disruption cap, and budgets that block underutilized consolidation outside weekday business hours (UTC) while still allowing empty-node removal.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to karpenter consolidation, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.