What is Kubernetes CPU overprovisioning, and why do requests matter more than usage?

Kubernetes CPU overprovisioning is the gap between the CPU a pod requests and the CPU it actually consumes. The gap matters because the scheduler never looks at usage. It places a pod only if the sum of CPU requests on a node, plus the new pod's request, fits within the node's allocatable CPU. A node can therefore be "full" for scheduling while its cores sit mostly idle. When the next pod doesn't fit anywhere, the cluster autoscaler adds a node. Requested capacity, not consumed capacity, drives node count, and node count drives the bill.

Report cluster utilization two ways:

  • Requested / allocatable shows how full the scheduler thinks the cluster is.
  • Used / allocatable shows how much work the hardware is actually doing.

The distance between the two is your idle capacity. Illustrative example with assumed numbers, for one node:

Node allocatable: 8.0 cores
|=========================================| 8.0  allocatable
|====================================|      7.2  summed requests (90%)
|=========|                                 1.6  actual usage    (20%)
          <------- idle capacity ------>    5.6  requested but unused

On this node, the scheduler rejects new pods even though 80% of the CPU is unused.

This checklist is for two groups. Backend engineers own the service manifests where requests are set. Platform engineers own the node pools, the autoscaling, and the cost reports where the gap appears as spend. Fixing overprovisioning takes both: the manifests set the requests, and the node pools show what those requests cost.

+-------------------+      +----------------------------------------------+
| New pod           |      | Scheduler                                    |
| request: 1.0 core |----->| fits if sum(requests) + new <= allocatable   |
+-------------------+      | never reads actual usage                     |
                           +----------------------------------------------+
                                                  |
                                                  v
+-----------------------------------------------------------------------+
| Node A   allocatable 8.0 cores (illustrative numbers)                 |
|  requests [====================================....] 7.2 (90%)        |
|  usage    [========................................] 1.6 (20%)        |
|                   <----- 5.6 cores idle ----->                        |
|  new pod 1.0: 7.2 + 1.0 = 8.2 > 8.0  ->  rejected, stays Pending      |
+-----------------------------------------------------------------------+
                                    |
                                    v
                      +---------------------------+
                      | Cluster autoscaler        |
                      | sees Pending pod,         |
                      | adds Node B               |
                      +---------------------------+
                                    |
                                    v
                      +---------------------------+
                      | node count +1 -> cost up  |
                      | (driven by requests)      |
                      +---------------------------+

Illustrative estimate: the scheduler admits pods by summed requests against allocatable, so a node at 90% requested but 20% used rejects a new pod and triggers the autoscaler to add a node, even though 5.6 cores sit idle.

Who should use this checklist?

Use this checklist if you run Kubernetes clusters where node count can change, either through the cluster autoscaler or through manual node pool resizing, and you suspect the cluster is paying for capacity it doesn't use. Each role uses a different part of it:

  • Backend and service engineers who write Deployment manifests or Helm values. You set the requests.cpu values the scheduler reserves, so you apply the right-sizing steps and decide where headroom is intentional.
  • Platform engineers and SREs who run node pools, the cluster autoscaler, PDB policy, and the monitoring stack. You build the per-workload request-vs-usage ranking, enforce the version-controlled change process, and verify that nodes actually leave the cluster.
  • Engineering leads and whoever reviews infrastructure spend. You use the before-and-after node counts and the payback calculation to decide whether a right-sizing pass is worth scheduling.

You will need some tooling in place first. The measurement steps assume Prometheus scraping kube-state-metrics and cAdvisor, plus metrics-server for kubectl top. Manifests should live in version control so changes can be reviewed and reverted.

The checklist pays off less in some cases. If a cluster runs at a fixed minimum node count it cannot go below, freed requests won't remove nodes. If nearly every workload genuinely needs Guaranteed QoS with integer CPU pinning, requests can't be trimmed to fractional values. If the cluster is small enough that the possible savings are less than the engineering time a review costs, skip it. The payback calculation in the final section helps you judge that last case.

How does the checklist work?

The checklist is a measure, rank, change, verify loop. It rests on one fact: the scheduler and the autoscaler act on requests, so you reduce cost only by lowering requests and then letting nodes drain.

  1. Measure the gap per workload. Pull requested cores from kube-state-metrics and usage from cAdvisor's container_cpu_usage_seconds_total. Aggregate both by owning workload over a window long enough to include real peaks.
  2. Choose the sizing statistic. Use the mean only to flag gross overprovisioning. Size requests from p90–p99 of short-window rates, and check the max to see how far the tail extends.
  3. Rank by absolute gap. Multiply (requested minus sized usage) by replica count, or by node count for DaemonSets, and sort. A few workloads usually hold most of the idle reservation.
  4. Diagnose the cause. Match each top entry against the common sources: copied values, requests pinned to limits, sidecar and DaemonSet multipliers, peak-sized replicas without autoscaling, and startup-sized runtimes.
  5. Change in small, reviewed steps. Lower requests a fraction at a time through version control. Make an explicit decision about limits and QoS class, and watch pending pods, p99 latency, and the throttling ratio after each rollout.
  6. Skip workloads where headroom is deliberate. Latency-sensitive bursters, pinned Guaranteed pods, and HPA-scaled services each need separate handling. Record the reason with an annotation.
  7. Verify node removal. Clear scale-down blockers such as PDBs, local storage, strict affinity, unmanaged pods, and pool minimums. Then compare node count and requested/allocatable before and after.

The sections below follow this order.

How do you measure the request vs usage gap per workload?

Start with a quick triage, then move to trends. kubectl describe node <name> shows an "Allocated resources" section with summed CPU requests as a percentage of allocatable. kubectl top nodes and kubectl top pods -A (which need metrics-server) show current usage. A node that reports 85% of CPU requested but 20% in use is a strong lead. These are point-in-time snapshots, though. They tell you nothing about last Tuesday's peak, so don't size anything from them.

For trends, compare requests and usage in Prometheus. Requests come from kube-state-metrics, typically kube_pod_container_resource_requests{resource="cpu"}. Metric names have changed across kube-state-metrics releases, so check what your deployment exposes. Usage comes from cAdvisor's container_cpu_usage_seconds_total. That metric is a cumulative counter, so wrap it in rate(...[5m]) to get cores in use. Filter out container="" and container="POD" series to avoid double-counting.

Aggregate per workload rather than per pod. Pods are short-lived and get renamed on every rollout, so per-pod history fragments. Join on namespace and pod to kube_pod_owner, or sum by a stable label such as app. The useful output is one row per Deployment or StatefulSet: total requested cores next to observed usage.

The query window should cover the real peaks:

  • At least one full week, to capture weekday and weekend cycles.
  • Month-end processing.
  • Scheduled batch jobs.

Thirty days is a reasonable floor for most services.

Also check your scrape interval and rate window. A 60-second scrape averaged over five minutes flattens a 10-second burst to a fraction of its height. Treat the measured peak as a lower bound on what the container actually needed.

kubectl describe node worker-1 | grep -A 8 'Allocated resources'
kubectl top nodes            # requires metrics-server
kubectl top pods -A --sort-by=cpu

PROM_URL=""

OWNER='max by (namespace, pod, owner_kind, owner_name) (kube_pod_owner)'

REQ="sum by (namespace, owner_kind, owner_name) (
  kube_pod_container_resource_requests{resource=\"cpu\"}
  * on (namespace, pod) group_left(owner_kind, owner_name) $OWNER)"

USE="sum by (namespace, owner_kind, owner_name) (
  rate(container_cpu_usage_seconds_total{container!=\"\", container!=\"POD\"}[5m])
  * on (namespace, pod) group_left(owner_kind, owner_name) $OWNER)"

PEAK="max_over_time(($USE)[30d:5m])"

for q in "$REQ" "$PEAK"; do
  curl -s "$PROM_URL/api/v1/query" --data-urlencode "query=$q"
  echo
done

Runs kubectl snapshot triage, then queries Prometheus for requested CPU cores and 30-day peak usage aggregated per owning workload via kube_pod_owner.

Which usage statistic should you size CPU requests against?

An average CPU figure tells you how much work a pod did over a period. It does not tell you how much CPU the pod needed at its busiest moment. A service that idles at 50m and spikes to 400m during request bursts can average 80m. If you size the request to that average, the pod's guaranteed share falls below what those bursts need, and on a contended node the bursts get throttled. Averages are still useful for one thing. When a pod averages 20m against a 1000m request, you know it is overprovisioned without looking further.

To set requests, use high percentiles of short-window rates, p90 to p99 depending on how sensitive the workload is to latency. Check the maximum alongside them to see how far the tail extends.

Statistic What it reveals What it hides When to use it
Mean Total work done Bursts, and how wide the bursts are Spotting gross overprovisioning
Median (p50) Typical steady state Everything above it Baselines for batch or steady workloads
p90–p99 Sustained burst demand Rare extreme spikes Setting requests
Max Worst observed peak How often that peak happens Sanity-checking headroom and limits

Illustrative estimate. Take 200 pods, each requesting 500m, with p95 usage of 150m. The unused request per pod is 350m, so the fleet has 70 cores requested but unused. Assume each node has 7.5 cores allocatable. That gap is worth about 9 nodes.

The capacity you actually free will be smaller than this. You will add headroom above p95. Bin-packing leaves fragments of unused space on each node. DaemonSets keep their reservations on every node. Memory requests may stop nodes from draining even after you lower CPU requests.

Where does CPU overprovisioning usually come from?

Most excess reservation traces back to a few habits. Audit your manifests for each one before you tune anything else.

Source How to detect Usual fix
Copy-pasted request values Many Deployments share identical CPU requests (for example, every service at 500m) but have very different usage profiles Size each workload from its own usage history instead of a team template
Requests set equal to limits by default requests.cpu matches limits.cpu on workloads that don't need Guaranteed QoS Lower requests to observed steady-state usage. Keep the limit, or remove it, as a separate decision
Generous sidecar, init, or DaemonSet requests Sum the sidecar requests across all replicas, and the DaemonSet requests across all nodes Profile the sidecar or agent once and set a smaller shared default
Replicas sized for peak with no autoscaling Fixed replica count, with usage far below requests outside a daily or weekly peak Add horizontal autoscaling and size per-pod requests for normal load
Runtimes sized for startup CPU spikes in the first minutes after a pod starts, then a long flat line well below the request Size requests for steady state. Handle startup with probes and readiness gating rather than a permanent reservation

The sidecar and DaemonSet rows tend to cost the most because one value is reserved many times over. A proxy sidecar that requests 100m reserves 20 cores across 200 pods, whatever it actually uses. A node agent's request is reserved on every node, so it grows with the cluster.

Startup-heavy runtimes such as the JVM need the same care. JIT compilation and class loading are real CPU costs, but they are temporary. If you size the request for them, you keep paying for that capacity for the whole life of the pod.

How do you right-size CPU requests without causing incidents?

Work from the ranked list built earlier, not from a service-by-service audit. Sorting by absolute gap (requested cores minus sized usage, multiplied by replicas) puts the few workloads that hold most of the idle capacity at the top. You stop when the remaining gaps are too small to be worth the effort, which saves far more engineering time than reviewing every deployment.

Right-sizing workflow

  • Pick the top entry and lower its request in small steps. Reduce by a fixed fraction, such as a quarter of the gap, rather than jumping straight to the target. If something breaks, you know which change caused it and roll back one step.
  • Make every change in version control. Edit the manifest or chart values and roll out through your normal pipeline. A kubectl edit gets reverted on the next deploy and leaves no record of why the value changed.
  • Set limit policy deliberately. If limits stay equal to the new, lower requests, the CFS quota shrinks too, and bursty pods get throttled. Decide per workload whether to raise the limit, remove it, or accept the throttling.
  • Check QoS class. If requests no longer equal limits, the pod drops from Guaranteed to Burstable. Burstable pods are evicted before Guaranteed pods under node pressure. They also lose exclusive core pinning when the static CPU manager policy is enabled.
  • Watch after each rollout. Check for pending pods, p99 latency, and the throttling ratio. The ratio is the rate of container_cpu_cfs_throttled_periods_total divided by the rate of container_cpu_cfs_periods_total, computed per container. Hold or revert if the ratio rises sharply or latency regresses.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
  namespace: checkout
spec:
  replicas: 12
  selector:
    matchLabels:
      app: checkout-api
  template:
    metadata:
      labels:
        app: checkout-api
    spec:
      containers:
        - name: api
          image: registry.example.com/checkout-api:1.4.2
          resources:
            requests:
              cpu: 1700m    # was 2000m; the scheduler reserves this per replica
              memory: 1Gi
            limits:
              cpu: 2500m    # raised deliberately so bursts are not throttled at 1700m
              memory: 1Gi

A version-controlled Deployment showing one small CPU request reduction with an explicit limit decision and its QoS consequence, plus the PromQL throttled-periods ratio to watch after rollout.

When should you avoid tightening CPU requests?

Not every gap between request and usage is waste. Some headroom is the price of predictable behavior, and removing it can cost more than it saves.

  • Latency-sensitive services on nodes where pods burst together. Requests set each container's CFS weight. If several pods on a node spike at the same moment, such as on a traffic surge or a shared upstream retry storm, a smaller request means a smaller share of contended CPU. p50 usage may look idle while p99 latency depends on that reserved share.
  • Startup spikes and rare peaks. JIT warm-up, cache priming, and month-end batch jobs may not appear in a two-week window. Sizing to the window can stretch rollouts or cause readiness probe failures during the one run that matters.
  • Guaranteed QoS or CPU pinning. Guaranteed QoS requires requests to equal limits. With the static CPU manager policy, exclusive cores also require an integer CPU count. Trimming these pods to fractional values silently drops them to Burstable and unpins them.
  • HPA scaling on CPU utilization. Utilization is measured as a percentage of the request. Lowering the request raises the percentage, so the HPA may add replicas until total reserved CPU is roughly where it started.
  • Blast radius. Tighter bin-packing puts more pods on each node. When a node fails, more workloads reschedule at once, and the cluster needs spare capacity to absorb them.
  • Monitoring cost. Per-container, high-resolution metrics across thousands of containers add real storage and query load. For low-value workloads, a coarse rule is often cheaper than precise tuning.

When you leave headroom on purpose, record the reason in an annotation so the next review doesn't remove it.

What are the alternatives to manual right-sizing, and how do they interact with HPA?

Manual tuning is not the only way to close the gap, and the options differ mainly in who changes the request and when.

Approach What it changes Operational effort Risks Best fit
Manual right-sizing Requests in your manifests, through normal review High, repeated per workload Values go stale as traffic and code change Small number of critical services
VPA, recommendation-only Nothing. It publishes suggested requests Low. Someone still applies them Recommendations reflect only the history VPA has seen Data source for manual or CI-driven changes
VPA, auto mode Requests on running pods, applied by recreating them Low once configured Restarts at unplanned times. Conflicts with HPA on CPU Stateless workloads that tolerate restarts and are not scaled by HPA on CPU
HPA Replica count Moderate. Needs sane requests and min/max bounds Scales on a percentage of a request that may be wrong Load that varies and parallelizes across pods

Do not run VPA in auto mode and HPA on the same CPU metric for one workload. Both react to the same signal, so each one's adjustment shifts what the other sees. They can then chase each other instead of settling.

The HPA interaction also matters for manual changes. An HPA CPU utilization target is a percentage of the request, not of a core. Here is an illustrative estimate. A pod uses 300m against a 1000m request, which reads as 30%. Lower the request to 400m and the same load reads as 75%. With a 60% target, the HPA adds replicas even though the load has not changed. When you cut requests on an HPA-scaled workload, revisit the utilization target at the same time.

How do you confirm freed capacity actually reduces cost?

Lowering requests only frees capacity on paper. The bill changes when nodes leave the cluster, which happens only if the cluster autoscaler (or whoever manages nodes manually) can drain and remove them. Before calling a right-sizing pass done, check what keeps underused nodes alive:

  • Pod disruption budgets. Run kubectl get pdb -A. Any PDB showing 0 in ALLOWED DISRUPTIONS blocks eviction, and so does a single-replica workload with minAvailable: 1.
  • Local storage. Depending on autoscaler configuration, pods using emptyDir or host paths may stop it from draining their node.
  • Affinity and anti-affinity. Strict anti-affinity or node selectors can leave no valid destination for a pod, so its node stays.
  • Non-evictable or unmanaged pods. Look for bare pods with no controller, pods annotated cluster-autoscaler.kubernetes.io/safe-to-evict: "false", and system pods without PDBs.
  • Node pool minimums. A pool already at its configured minimum will not shrink.

Record node count and the requested/allocatable ratio before and after each change. kubectl describe node <name> lists requests as a percentage of allocatable under "Allocated resources". If node count stays the same while that ratio falls, the savings are not real yet.

Weigh the outcome against the engineering effort it took:

Illustrative estimate (assumed numbers, not a benchmark): monthly savings = nodes removed × monthly cost per node = 3 × $200 = $600 one-time cost = engineer hours × hourly cost = 16 h × $100 = $1,600 payback = one-time cost ÷ monthly savings ≈ 2.7 months

When a blocker prevents nodes from being removed, the payback period is infinite.

#!/usr/bin/env bash
set -euo pipefail
OUT="capacity-$(date +%Y%m%d-%H%M).txt"

{
  # Node count is the number that actually drives the bill.
  echo "== Node count"
  kubectl get nodes --no-headers | wc -l

  # Requests as a percentage of allocatable, per node.
  echo "== Allocated resources per node"
  for n in $(kubectl get nodes -o name); do
    echo "--- $n"
    kubectl describe "$n" | grep -A 6 'Allocated resources' | grep -E 'cpu|memory'
  done

  # PDBs that currently allow zero evictions block node drains.
  echo "== PDBs with 0 allowed disruptions"
  kubectl get pdb -A -o json | jq -r '.items[]
    | select(.status.disruptionsAllowed == 0)
    | "\(.metadata.namespace)/\(.metadata.name)"'

  # Pods explicitly marked as not safe to evict by the cluster autoscaler.
  echo "== Pods annotated safe-to-evict=false"
  kubectl get pods -A -o json | jq -r '.items[]
    | select(.metadata.annotations["cluster-autoscaler.kubernetes.io/safe-to-evict"] == "false")
    | "\(.metadata.namespace)/\(.metadata.name) on \(.spec.nodeName)"'

  # Bare pods with no controller cannot be rescheduled after eviction.
  echo "== Pods without an owner controller"
  kubectl get pods -A -o json | jq -r '.items[]
    | select((.metadata.ownerReferences // []) | length == 0)
    | "\(.metadata.namespace)/\(.metadata.name) on \(.spec.nodeName)"'
} | tee "$OUT"

echo "Saved to $OUT. Compare runs with: diff <before-file> <after-file>"

A bash script that records node count, per-node requested/allocatable percentages, and common scale-down blockers (zero-disruption PDBs, safe-to-evict=false pods, unmanaged pods) so before-and-after snapshots can be compared.

How can this checklist save engineering time?

Right-sizing without a method tends to become an open-ended audit: someone opens every Deployment, argues about each value, and then finds the nodes never shrank. The checklist cuts that waste in several ways:

  • It focuses effort on the largest gaps. Ranking by absolute gap times replicas means you tune the handful of workloads that hold most of the idle reservation. You stop when the remaining gaps aren't worth the time.
  • It checks blockers before tuning. Running the scale-down blocker snapshot first shows whether PDBs, pool minimums, or unmanaged pods would keep nodes alive anyway. That way you don't spend days lowering requests that can't turn into removed nodes.
  • Its queries are reusable. The per-owner PromQL for requests and peak usage, and the capacity snapshot script, can be saved and rerun on a schedule. Later reviews start from a current ranked list instead of from scratch.
  • Its changes are cheap to undo. Small steps committed through the normal pipeline mean a regression has one obvious cause and a one-commit revert. You avoid a long incident investigation.
  • It records decisions. Annotations on workloads that keep deliberate headroom stop the next reviewer from rediscovering, and re-arguing, the same exceptions.
  • It lets tools gather the data. VPA in recommendation-only mode can supply suggested requests continuously. Engineers review and apply them instead of computing them by hand.

Illustrative estimate (assumed numbers, not a measured result): assume a cluster with 150 workloads, where a full audit takes about 30 minutes per workload. That comes to 75 engineer-hours. Assume instead that the ranked list shows the top 15 workloads hold most of the gap, and each takes about 2 hours, including staged rollouts and monitoring. That comes to 30 engineer-hours, plus a few hours to build the queries once. Your actual ratio depends on how concentrated the gap is in your cluster. The ranked list tells you that before you commit any time.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes cpu overprovisioning, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.