What is Kubernetes resource request waste and why does it strand capacity?

Request waste is capacity a pod reserves through resources.requests but does not use. The waste comes from how scheduling works. The scheduler places pods by adding up their requests and comparing the total against each node's allocatable CPU and memory. It does not consider actual usage. If a node's pods request 90% of its allocatable CPU but use 15%, the scheduler treats that node as nearly full. New pods stay Pending or trigger the autoscaler to add nodes, while the existing nodes are mostly idle. The unused capacity is paid for but cannot be scheduled.

Requests and limits control different things:

Setting Used by Effect when exceeded
CPU request Scheduler, CPU share weighting Not a cap. The pod can use spare CPU.
CPU limit Kernel CFS quota The container is throttled.
Memory request Scheduler, eviction ranking Not a cap, but the pod becomes an earlier eviction candidate under node pressure.
Memory limit Kernel cgroup The container is OOM-killed.

Requests also set the pod's QoS class:

  • Guaranteed: every container sets CPU and memory requests equal to its limits.
  • Burstable: at least one container sets a request or limit, but the pod does not meet the Guaranteed criteria.
  • BestEffort: no container sets any requests or limits.

When a node runs low on memory, the kubelet evicts pods whose usage exceeds their requests first. BestEffort pods have no requests, so they are evicted first. Guaranteed pods are evicted last. Inflating requests is therefore partly a way of buying protection from eviction, and that protection costs capacity.

Two groups own this problem. Platform teams pay for the stranded nodes. Backend teams write the pod specs that cause the waste.

 Backend team writes the spec         Platform team pays for nodes
            |                                        ^
            v                                        |
 +-----------------------+                           |
 | New pod               |                           |
 | requests: 2 CPU       |                           |
 +-----------------------+                           |
            |                                        |
            v                                        |
 +-----------------------+                           |
 | Scheduler             |   7.2 + 2 = 9.2 CPU > 8.0 |
 | compares requests to  |-------------+             |
 | node allocatable,     |             |             |
 | not actual usage      |             v             |
 +-----------------------+   +-------------------+   |
            |                | Pod stays Pending |   |
            v                +-------------------+   |
 +-----------------------+             |             |
 | Node A                |             v             |
 | allocatable: 8.0 CPU  |   +-------------------+   |
 | requested:   7.2 CPU  |   | Cluster autoscaler|   |
 | used:        1.2 CPU  |   | adds a node       |---+
 | stranded:    6.0 CPU  |   +-------------------+
 | (reserved, idle, paid)|
 +-----------------------+

Illustrative estimate: the scheduler compares summed CPU requests with node allocatable, not with actual usage, so a node that reserves 90% of its CPU (7.2 of 8.0) but uses only 15% (1.2) rejects a new 2-CPU pod, leaving it Pending until the autoscaler adds a node that the platform team pays for.

How do you measure requests vs usage with kube-state-metrics and cAdvisor?

The comparison has two sides, and they come from different components. kube-state-metrics watches the API server and turns object specs into series. kube_pod_container_resource_requests and kube_pod_container_resource_limits report what each container declared, with a resource label for cpu or memory. Nothing in these series measures consumption.

Actual consumption comes from the kubelet's embedded cAdvisor. container_cpu_usage_seconds_total is a counter, so you take a rate() over a window to get cores in use. For memory, use container_memory_working_set_bytes. Total usage includes page cache that the kernel can reclaim under pressure. Working set subtracts the inactive file pages, and the kubelet uses it for eviction decisions. Sizing requests against total usage would justify reservations for cache the node would drop anyway.

Conceptually, the query does three things:

  1. Aggregate usage by (namespace, pod, container).
  2. Aggregate requests the same way.
  3. Subtract or divide the two with on (namespace, pod, container). The difference is the idle reserve.

Watch for these pitfalls:

  • Pod churn. Replaced pods get new names. Instant joins miss them, so compute over time windows rather than single points.
  • Pods without requests. They produce no request series, so they drop out of an inner join silently.
  • Mismatched labels. cAdvisor emits pod-level cgroups with an empty container label, and some runtimes also emit a sandbox container. Filter both out, or usage double-counts. Extra labels on either side also break one-to-one matching unless you aggregate them away.
  • Short-lived pods. Jobs that finish between scrapes never appear in usage data, but their requests still blocked scheduling.
# Idle CPU reserve per container, in cores: declared request minus measured usage.
# Both sides are aggregated to the same three labels so the join is one-to-one.
sum by (namespace, pod, container) (
  kube_pod_container_resource_requests{resource="cpu"}
)
- on (namespace, pod, container)
sum by (namespace, pod, container) (
  # Drop the pod-level cgroup (empty container) and the sandbox ("POD") to avoid double-counting.
  rate(container_cpu_usage_seconds_total{container!="", container!="POD"}[5m])
)

# Idle memory reserve per container, in bytes, using working set rather than total usage.
# Averaged over a day via a subquery so a single scrape does not decide the answer.
sum by (namespace, pod, container) (
  kube_pod_container_resource_requests{resource="memory"}
)
- on (namespace, pod, container)
sum by (namespace, pod, container) (
  avg_over_time(container_memory_working_set_bytes{container!="", container!="POD"}[1d:5m])
)

# Containers using CPU but declaring no request: an inner join drops them silently,
# so list them explicitly with 'unless'.
sum by (namespace, pod, container) (
  rate(container_cpu_usage_seconds_total{container!="", container!="POD"}[5m])
)
unless on (namespace, pod, container)
kube_pod_container_resource_requests{resource="cpu"}

PromQL that joins kube-state-metrics requests with cAdvisor CPU usage and memory working set on namespace, pod and container, filters pod-level and sandbox series, and surfaces containers that have no requests.

How do you calculate idle capacity per workload, namespace and cluster?

Start at the container level. Waste is the request minus a high percentile of usage, measured over a window long enough to include weekly cycles such as Monday traffic, batch runs and end-of-week reports. Two weeks covers a weekly peak twice. The percentile depends on how each resource fails when it runs short:

Resource Usage statistic Reason
Memory p99 or max of working set Running out gets the container OOM-killed, so size for peaks
CPU p90–p95 Running out causes throttling, which slows the container but does not kill it, so size closer to typical load

Sum container waste up to the workload by owner, then to the namespace, then to the cluster. At the cluster level, compare two ratios: total requests to allocatable, and actual usage to allocatable. When requests are near 100% of allocatable and usage is at 30%, the scheduler sees a full cluster that is mostly idle. Pending pods and autoscaler scale-ups in that state are caused by reservations, not by load.

Rank by absolute wasted cores and GiB, not by percentage. A sidecar at 90% waste on 100m matters less than a deployment at 40% waste across 50 replicas.

Illustrative estimate (assumed inputs): 200 replicas each request 1 CPU and use 0.25 CPU at p95. That leaves 200 × 0.75 = 150 idle cores. With an assumed 16 allocatable cores per node, this is about nine nodes reserved and not used.

The window length involves a trade-off. If the window is too short, it misses monthly jobs and seasonal peaks, so it underestimates what the workload needs. If it is too long, it keeps sizing for code paths, traffic patterns or memory leaks that were removed several releases ago. Restart the window after major deployments.

# Prometheus recording rules: container waste = request - high percentile of usage.
# Assumes kube-state-metrics and cAdvisor (kubelet) metrics are scraped.
groups:
- name: request-waste
  interval: 5m
  rules:
  # Pre-aggregate CPU usage so the 14d quantile below stays cheap to evaluate.
  # Note: this series only has history from when the rule was deployed.
  - record: container:cpu_usage_cores:rate5m
    expr: sum by (namespace, pod, container) (rate(container_cpu_usage_seconds_total{container!="", container!="POD"}[5m]))

  # CPU: size to p95 over two weeks (throttling slows, does not kill).
  - record: container:cpu_waste_cores
    expr: |
      clamp_min(
        sum by (namespace, pod, container) (kube_pod_container_resource_requests{resource="cpu"})
        - on (namespace, pod, container)
        quantile_over_time(0.95, container:cpu_usage_cores:rate5m[14d]), 0)

  # Memory: size to the max working set (running short means OOM kill).
  - record: container:memory_waste_bytes
    expr: |
      clamp_min(
        sum by (namespace, pod, container) (kube_pod_container_resource_requests{resource="memory"})
        - on (namespace, pod, container)
        max by (namespace, pod, container) (max_over_time(container_memory_working_set_bytes{container!="", container!="POD"}[14d])), 0)

  # Roll up to namespace; rank these by absolute cores and bytes, not percent.
  - record: namespace:cpu_waste_cores:sum
    expr: sum by (namespace) (container:cpu_waste_cores)
  - record: namespace:memory_waste_bytes:sum
    expr: sum by (namespace) (container:memory_waste_bytes)

  # Cluster signals: reserved vs allocatable, and actually used vs allocatable.
  - record: cluster:cpu_requests_over_allocatable:ratio
    expr: sum(kube_pod_container_resource_requests{resource="cpu"}) / sum(kube_node_status_allocatable{resource="cpu"})
  - record: cluster:cpu_usage_over_allocatable:ratio
    expr: sum(container:cpu_usage_cores:rate5m) / sum(kube_node_status_allocatable{resource="cpu"})

Prometheus recording rules that compute per-container CPU waste against a 14-day p95 and memory waste against the 14-day peak working set, roll both up to namespaces, and record the cluster's requests-to-allocatable and usage-to-allocatable ratios.

When should you avoid tightening requests?

Some workloads look oversized on a usage graph and still need their reservations. The cases below are the most common.

Case Why a naive cut breaks it Safer approach
JVM and similar runtimes Heap size and thread or GC pool counts are often derived from the container's cgroup limits, so a cut changes runtime behaviour Set heap and pool sizes explicitly, then re-measure
Startup spikes JIT compilation, cache warm-up and migrations peak for seconds, between scrapes Size for startup or use probes that tolerate a slow start
Batch and cron jobs Short peaks fall between scrape intervals, so averages look idle Use max-over-time or per-job peak metrics
Init containers Effective pod request is the larger of the biggest init container and the sum of app containers Check init requests before shrinking app requests

The HPA calculates utilization as usage divided by request. If you halve a CPU request, the same load reports twice the percentage, so the HPA scales out and may run more replicas than before. Change the HPA target in the same change as the request.

Some headroom is deliberate. Capacity reserved for zone failover or known traffic surges looks idle until the day it is needed.

Tighter requests also mean denser nodes. That brings noisy neighbours, CPU contention latency and node-pressure evictions, which hurt latency-sensitive services most.

CPU limits involve their own trade-off. They isolate tenants from each other, but CFS quota throttling can add tail latency even when average usage is well below the limit.

How do you right-size requests safely in production?

Change one workload at a time, and wait at least one full traffic cycle (often a week, so weekday and weekend peaks are both covered) before moving to the next. If several services change at once and latency regresses, you cannot tell which change caused it.

Size CPU and memory by different rules because they fail differently. CPU is compressible: a pod that exceeds its share gets slower but keeps running. You can set CPU requests near p95 usage plus a modest margin. Memory is not compressible: a container that exceeds its limit is OOMKilled. Base memory requests and limits on the observed peak plus clear headroom, and account for startup spikes, cache growth and garbage-collector behavior.

In the example that follows, a deployment drops from cpu: 1000m / memory: 2Gi to cpu: 300m / memory: 1200Mi (illustrative estimate, assuming a p95 of 220m CPU and a memory peak of 950Mi). After each change, check these signals:

  • CFS throttling (container_cpu_cfs_throttled_periods_total from cAdvisor)
  • Restarts where kube-state-metrics reports the last terminated reason as OOMKilled
  • Evictions caused by node memory pressure

Freed requests only lower cost when nodes are removed. That requires a node autoscaler, and the autoscaler can only drain a node if its pods can move. Pods that use local storage, PodDisruptionBudgets that allow zero disruptions, and pods without a controller or with a single replica can all keep a mostly empty node running. Fix these blockers first. Otherwise the engineering time you spend right-sizing produces lower request numbers but the same infrastructure bill.

# BEFORE: requests set by guesswork, far above observed usage.
resources:
  requests:
    cpu: 1000m
    memory: 2Gi
  limits:
    cpu: 1000m
    memory: 2Gi
---
# AFTER (illustrative estimate, assuming observed p95 CPU = 220m
# and observed memory peak = 950Mi over a full traffic cycle).
resources:
  requests:
    # CPU is compressible. Set the request near p95 plus a modest margin.
    # Going over this value means contention, not a crash.
    cpu: 300m
    # Memory is not compressible. Set the request at the observed peak
    # plus clear headroom for startup spikes, cache growth and GC.
    memory: 1200Mi
  limits:
    # Keep the memory limit equal to the request so the scheduler
    # reserves what the container can actually use. Exceeding this
    # limit is an OOMKill, not a slowdown.
    memory: 1200Mi
    # No CPU limit here, so the container can burst into idle node CPU
    # without CFS throttling. If your policy requires a CPU limit, set
    # it well above the request and watch
    # container_cpu_cfs_throttled_periods_total after the rollout.

A container resources block before and after right-sizing. CPU is set near p95 usage, and memory is set at the observed peak plus headroom, with the memory limit equal to the request.

What are the alternatives to manual right-sizing?

Manual analysis is not the only option. There are three common alternatives, and they differ in how much toil they remove and how much risk they add.

Approach Toil Disruption risk Best fit
Manual analysis High: someone queries, reviews and opens changes Low: changes ship through normal deploys Small fleets, critical services
VPA, recommendation-only (updateMode: "Off") Medium: VPA computes targets, humans apply them None: VPA only writes recommendations Most production workloads
VPA, automatic mode Low: VPA applies new values itself High: pods are evicted and recreated to pick up new requests Stateless, replicated, restart-tolerant services
Deliberate headroom None None Bursty or latency-critical services where spare capacity is the point

Automatic VPA reduces toil, but the cost is how it applies changes. The updater evicts pods so they are recreated with new requests. It honours PodDisruptionBudgets, but each eviction is still a restart. For a database, a stateful consumer with long rebalances, or anything with a slow warm-up, an unplanned restart can cost more than the reserved capacity you get back. VPA also conflicts with a Horizontal Pod Autoscaler that scales on the same CPU or memory metric. If VPA raises requests, utilisation as a percentage of requests falls, and HPA may scale in while VPA keeps adjusting. The two controllers end up working against each other.

Use recommendation mode as a second opinion. Compare VPA's target and upper bound with your own percentile analysis. Where they agree, apply the change with confidence. Where they disagree, investigate before changing anything. Accepting headroom is also a valid outcome, as long as you record it as a decision and not as a default.

How can you keep request waste from coming back and save engineering time?

Recording decisions only helps if someone looks at the numbers again. Request waste drifts back as code changes, traffic shifts and teams copy manifests between services. The goal is to catch that drift early, while each fix is still small.

Publish requests vs usage per team

Build one dashboard or scheduled report that shows requested CPU and memory against observed usage, grouped by namespace or by the team label you already use for ownership. Rank it by absolute idle capacity, not by percentage. A small service at 10% utilization matters less than a large one at 60%. When teams can see their own gap next to everyone else's, most of them fix it without being asked.

Set namespace defaults with LimitRange

A container with no requests is scheduled as if it needs nothing, which makes the node's real load hard to predict. A LimitRange with a defaultRequest in each namespace ensures every container gets a request, even when its manifest omits one. Pick modest defaults so that a missing value starts small rather than generous.

Review requests in code review

Treat a change to requests like a change to a dependency. Ask the author to explain the new value, ideally with a link to recent usage for that workload. This catches inflated requests before they ship, when correcting them costs a comment instead of a rollout.

Together, these checks replace the quarterly cleanup project with a few minutes of attention per change. Keep in mind that every waste figure here is an estimate. Missed scrapes, short-lived pods and missing ownership labels all leave gaps, so use the numbers to set priorities, not for exact accounting.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes resource request waste, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.