Why do Kubernetes requests, not usage, drive your cluster bill?

The Kubernetes scheduler places a pod by comparing its CPU and memory requests against each node's unreserved allocatable capacity. Actual usage plays no part in that decision. If a pod requests 2 vCPU and uses 0.3, the scheduler still treats the full 2 vCPU as taken. When pending pods no longer fit, a cluster autoscaler adds nodes. You end up paying for capacity that is reserved but sits idle.

Overprovisioning is the gap between what workloads request and what they use. To measure it, sum requests per namespace or workload and compare that total against observed usage at a high percentile (p95, for example) over a representative window. A large ratio between the two is a strong sign of waste.

Illustrative estimate

All inputs below are assumed. Nodes have 8 vCPU of allocatable capacity, cost $0.40/hour, and the cluster is CPU-bound. Workloads request 160 vCPU and have a p95 usage of 64 vCPU, which is a 2.5:1 ratio. Rightsizing sets requests to usage plus 25% headroom, giving 80 vCPU.

Scenario Requested vCPU Nodes needed Monthly cost (730 h)
Current requests 160 20 $5,840
Rightsized requests 80 10 $2,920

Why the estimate overstates savings

Dividing summed requests by node size assumes perfect bin-packing. Real clusters lose capacity to several things:

  • Fragmentation
  • DaemonSets
  • Memory pressure, which strands spare CPU
  • Affinity and topology rules
  • Pod disruption budgets

Treat the per-workload sum as an upper bound. Confirm the actual savings from node counts after the change.

#!/usr/bin/env bash
# Compare summed CPU requests with p95 CPU usage per namespace via the Prometheus HTTP API.
# Assumes kube-state-metrics (v2 metric names) and cAdvisor metrics are scraped.
set -euo pipefail

PROM_URL="${PROM_URL:-"
NODE_VCPU="${NODE_VCPU:-8}"   # assumed allocatable vCPU per node

# What the scheduler reserves: summed container CPU requests.
# Note: completed pods that still exist can inflate this; filter by phase if needed.
REQ='sum by (namespace) (kube_pod_container_resource_requests{resource="cpu", unit="core"})'

# What workloads use: p95 of namespace CPU usage over 7 days, sampled every 5m.
USE='quantile_over_time(0.95, sum by (namespace) (rate(container_cpu_usage_seconds_total{container!="", container!="POD"}[5m]))[7d:5m])'

query() {
  curl -sfG "$PROM_URL/api/v1/query" --data-urlencode "query=$1" \
    | jq -r '.data.result[] | "\(.metric.namespace)\t\(.value[1])"' \
    | sort -t $'\t' -k1,1
}

query "$REQ" > /tmp/req.tsv
query "$USE" > /tmp/use.tsv

# Join on namespace, then show the request:usage ratio and the idle gap in node-equivalents.
# The node-equivalent column assumes perfect bin-packing, so treat it as an upper bound.
join -t $'\t' /tmp/req.tsv /tmp/use.tsv \
  | awk -F'\t' -v node="$NODE_VCPU" '
      BEGIN { printf "%-24s %10s %10s %7s %10s\n", "NAMESPACE", "REQ_vCPU", "P95_vCPU", "RATIO", "GAP_NODES" }
      {
        ratio = ($3 > 0) ? $2 / $3 : 0
        gap = $2 - $3; if (gap < 0) gap = 0
        printf "%-24s %10.2f %10.2f %7.2f %10.2f\n", $1, $2, $3, ratio, gap / node
        tr += $2; tu += $3
      }
      END { printf "%-24s %10.2f %10.2f %7.2f\n", "TOTAL", tr, tu, (tu > 0 ? tr / tu : 0) }'

A shell script that queries Prometheus for summed CPU requests and 7-day p95 CPU usage per namespace. It prints the request-to-usage ratio and the idle gap expressed in node-equivalents, which is an upper bound on savings.

What is the Vertical Pod Autoscaler and how does it work?

The Vertical Pod Autoscaler (VPA) is a Kubernetes add-on that observes how much CPU and memory your containers actually use and proposes request values to match. It is not part of a default cluster install. You deploy it separately, then attach a VerticalPodAutoscaler custom resource to each workload you want it to watch. Behind that resource, VPA runs as three cooperating components.

The recommender reads container usage from the metrics pipeline and builds a statistical model of each container's consumption over time. From that model it publishes recommendations into the VPA object's status. It never touches running pods.

The updater compares the requests on live pods with the current recommendation. When the two drift far enough apart, it evicts the pod so the pod can be recreated with new values. Eviction is the mechanism because, in the classic design, requests on a running pod cannot simply be edited. Each change therefore costs a restart.

The admission controller is a mutating webhook. When a pod is created, it rewrites the container requests to the recommended values before the scheduler sees them. The updater only evicts, and this webhook is what applies the new numbers.

The most useful way to treat VPA is as a measurement tool first. Run only the recommender, and you get per-container sizing data grounded in real usage without any pods being restarted. That matters for two reasons:

  • You take on extra components to deploy, monitor, and upgrade.
  • The recommender needs enough usage history, ideally covering traffic peaks and batch cycles, before its output is worth acting on.
+-------------------+        +--------------------------------------------+
| Metrics pipeline  |        | VerticalPodAutoscaler custom resource      |
| (container usage) |        | status: per-container recommendations      |
+---------+---------+        +---+-------------+-------------------+------+
          |                      ^             |                   |
          v              publish |             | read              | read
+-------------------+            |             v                   v
|    RECOMMENDER    +------------+      +-------------+    +---------------+
| model of usage    |                   |   UPDATER   |    |   ADMISSION   |
| history, publishes|                   | live pod vs |    | mutating hook |
| never edits pods  |                   | recommended |    | sets requests |
+-------------------+                   +------+------+    +-------+-------+
                                               |                   ^
                                               | evict on drift    | create
                                               v                   |
                                        +-------------+            |
                                        |  Live pod   +------------+
                                        +-------------+ recreated pod

Recommender only: sizing data, no evictions, no request changes.

The recommender turns usage history into recommendations on the VPA object, the updater evicts pods whose requests have drifted from those recommendations, and the admission webhook writes the recommended requests into each recreated pod.

How do you read VPA recommendations?

Start with a VPA object that targets your Deployment through targetRef and sets updatePolicy.updateMode to Off. In this mode the recommender computes values but nothing evicts or mutates pods, so you can observe safely. After it has collected history, run kubectl describe vpa <name> and look under Status > Recommendation > Container Recommendations. Each container gets four values for CPU and memory:

Field What it means How to use it
Target Recommended request based on observed usage Starting point for new requests
Lower Bound Minimum the recommender considers reasonable In Auto mode, pods requesting less than this are candidates for eviction
Upper Bound Maximum the recommender considers reasonable Requests above this suggest overprovisioning
Uncapped Target Target before minAllowed/maxAllowed in the resource policy are applied If it differs from Target, your policy is clamping the recommendation

Treat Target as a hypothesis, not a guarantee. It is a percentile-based estimate built from a decaying histogram of past usage, which has several weaknesses:

  • Rare spikes. A traffic burst that happens once a month carries little weight in the histogram and may not be reflected in the recommendation.
  • Seasonal or batch peaks. End-of-quarter jobs or nightly reports that fall outside the observation window are invisible.
  • Recent behavior changes. After a release that changes memory profile, older samples still pull the estimate toward the old behavior until they decay.
  • Startup-heavy runtimes. JVMs and similar runtimes use far more CPU during class loading and JIT warm-up than at steady state. A recommendation tuned to steady state can starve initialization, which slows startup and can cause startup or readiness probes to fail.
# Recommendation-only VPA: computes values, never evicts or mutates pods.
# Apply:   kubectl apply -f web-vpa.yaml
# Inspect: kubectl describe vpa web-vpa
#   (see Status > Recommendation > Container Recommendations)
# Raw:     kubectl get vpa web-vpa -o jsonpath='{.status.recommendation.containerRecommendations}'
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: web-vpa
  namespace: shop
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: web
  updatePolicy:
    # Quote it: unquoted Off can be parsed as a boolean by YAML 1.1 parsers.
    updateMode: "Off"
  resourcePolicy:
    containerPolicies:
      - containerName: web
        controlledResources: ["cpu", "memory"]
        # Floor for a JVM: steady-state usage would otherwise drive the
        # target below what class loading and JIT warm-up need.
        minAllowed:
          cpu: 500m
          memory: 512Mi
        # Ceiling to keep a single anomaly from producing a huge target.
        maxAllowed:
          cpu: "2"
          memory: 4Gi
# If Uncapped Target differs from Target in the describe output,
# minAllowed/maxAllowed above are clamping the recommendation.

A VPA in updateMode "Off" for a JVM Deployment, with a minAllowed floor to protect startup and the kubectl commands used to read its Target, Lower Bound, Upper Bound and Uncapped Target.

How should you roll out VPA rightsizing safely?

Treat rollout as three stages, and promote each workload only when its behaviour justifies it.

Mode What VPA changes When it applies Eviction risk
Off Nothing; publishes recommendations only N/A None
Initial Requests on new pods Pod creation only None from VPA
Auto Requests on running pods Whenever drift exceeds thresholds Pods evicted and recreated

Start every workload in Off and let it run through at least one full traffic cycle, including peaks, batch windows and deploys. This produces recommendations at no operational risk. Move to Initial next. New values take effect only when pods are created by a rollout, a scale-up or a node drain, so changes happen on your schedule. Reserve Auto for stateless, replicated workloads covered by a PodDisruptionBudget that can absorb restarts.

Scope VPA through resourcePolicy.containerPolicies. Set minAllowed per container so a quiet week cannot shrink requests below what startup needs. Set maxAllowed so a leak or traffic spike cannot produce a request that no node can schedule. Set mode: "Off" on sidecars such as proxies and log shippers, because their sizing is usually managed separately. Use controlledResources to have VPA manage only CPU, only memory or both.

By default, VPA preserves each container's limit-to-request ratio. A 4× ratio on a 2 GiB request becomes an 8 GiB limit, which is too large to protect the node. When VPA lowers a request, it lowers the memory limit by the same ratio, and that tighter ceiling can trigger OOM kills. Set controlledValues: RequestsOnly to keep VPA away from limits, and manage limits yourself.

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: checkout-api
  namespace: shop
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-api
  updatePolicy:
    # Stage 2: apply recommendations only when pods are created.
    # Use "Off" first to collect data; "Auto" only for eviction-tolerant workloads.
    updateMode: "Initial"
  resourcePolicy:
    containerPolicies:
      - containerName: app
        # Manage both resources; use ["memory"] or ["cpu"] to narrow scope.
        controlledResources: ["cpu", "memory"]
        # Change requests only; limits stay as written in the Deployment,
        # avoiding proportional scaling to huge or too-tight limits.
        controlledValues: RequestsOnly
        # Floor: enough for startup even after a quiet week.
        minAllowed:
          cpu: 250m
          memory: 512Mi
        # Ceiling: keep requests schedulable on the largest node type.
        maxAllowed:
          cpu: "2"
          memory: 4Gi
      - containerName: istio-proxy
        # Sidecar sizing is managed separately; VPA leaves it alone.
        mode: "Off"

A VerticalPodAutoscaler in Initial mode that bounds the app container with minAllowed and maxAllowed, changes only its requests with controlledValues: RequestsOnly, and excludes the proxy sidecar.

Why does memory need different treatment than CPU?

CPU and memory fail in different ways when a request is too low. CPU is compressible. A container that hits its CPU limit gets throttled by the kernel's CFS quota and runs slower, which shows up as higher latency. Memory is not compressible. A container that exceeds its memory limit is OOM-killed, and an in-flight request, a warm cache, or a long-running job goes with it. A CPU underestimate degrades performance. A memory underestimate causes restarts.

Tight requests also cause trouble beyond the individual pod. The scheduler places pods by requests, so trimmed requests pack more pods onto each node. If several of them grow past their requests at the same time, they compete for the node's real capacity. For CPU, that means noisy-neighbor slowdowns. For memory, it means node pressure, and the kubelet evicts pods whose usage most exceeds their requests first. The pods you rightsized most aggressively are the ones most likely to be evicted.

VPA does react to OOM kills. The recommender treats an OOM event as evidence that more memory was needed and raises its recommendation. That only corrects the number after the failure has already happened. Track OOMKilled terminations and restart counts per workload, and treat any OOM-driven bump as a sign the original estimate was wrong.

Give memory more headroom than CPU. Set a minAllowed memory floor in the resource policy for workloads with spiky allocation. Count that slack as a reliability cost you chose to pay, not as waste you missed.

When should you avoid VPA Auto mode, and what are the alternatives?

Auto mode applies new requests by evicting pods so they restart with updated values. That cost is small for stateless services with several replicas. It is a problem for three kinds of workload:

  • Single-replica deployments: every resize is an outage.
  • Stateful sets: each restart triggers cache warm-up, leader re-election or replica resync.
  • Long-running jobs: an eviction discards hours of work.

A PodDisruptionBudget limits how many pods VPA evicts at once. It does not prevent restarts. It only spaces them out.

The other trap is running VPA and the Horizontal Pod Autoscaler on the same CPU or memory metric. HPA scales on utilisation, which is usage divided by request. When VPA raises the request, utilisation drops, and HPA scales in. The two controllers then keep counteracting each other. There are two safe combinations:

  • HPA scales on custom or external metrics, such as queue depth or requests per second, while VPA sizes CPU and memory.
  • VPA is restricted through controlledResources to a resource that HPA does not read.
Approach Restarts pods Effort Best fit
VPA Auto Yes Low Stateless, multi-replica services
VPA recommendation-only, applied manually Only at your deploys Medium Stateful sets, jobs, single replicas
Manual rightsizing from dashboards Only at your deploys High Small fleets, teams without VPA installed
HPA alone No (adds or removes pods) Low Load-driven services whose per-pod size is already right
LimitRange and ResourceQuota defaults No Low Containers that ship with no requests; caps on namespace totals

LimitRange and ResourceQuota set defaults and ceilings. They do not rightsize anything, so use them alongside one of the other approaches.

How do you make rightsizing durable and verify the savings?

VPA recommendations only change what is running. The next helm upgrade or kubectl apply restores whatever requests Git holds. To keep the savings, put the target values (rounded up to a sensible step) into the Deployment manifest or Helm values file through a pull request. The service owner reviews the new numbers before they ship, and the diff records why requests changed. A scheduled job that opens these PRs from current recommendations replaces repeated tuning sessions, so nobody has to remember which services were adjusted.

Verify savings at the node level

Lower requests free scheduling space. The bill drops only when nodes are removed. With the cluster autoscaler, confirm that scale-down actually happened after the rollout. With fixed node pools, someone has to resize the pool manually. Three things can keep the bill flat even after requests fall:

  • Fixed or minimum pool sizes that prevent nodes from being removed.
  • Reserved capacity or committed-use contracts that you pay for whether or not the nodes are busy.
  • Fragmentation, where a few large pods sit on many nodes so that none of those nodes can drain.

Illustrative estimate (assumed numbers, not a benchmark)

Nodes needed = ceil(total requested vCPU ÷ (allocatable vCPU per node × target packing ratio))

Assume 7.5 allocatable vCPU per node and a 0.85 packing ratio, which gives 6.375 usable vCPU per node.

  • Before: 60 vCPU requested ÷ 6.375 = 9.4, so 10 nodes.
  • After: 40 vCPU requested ÷ 6.375 = 6.3, so 7 nodes.

At an assumed $0.40 per node-hour, removing 3 nodes saves 3 × 0.40 × 730 hours ≈ $876 per month. You only get that saving if those three nodes are actually removed.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to vertical pod autoscaler rightsizing, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.