Kubernetes Rightsizing with VPA: Cutting Overprovisioned Pod Requests

Why do resource requests, not usage, drive Kubernetes cluster cost?

The Kubernetes scheduler places pods by their resource requests, not by what they consume. When a pod requests 1 CPU, the scheduler subtracts 1 CPU from the node's allocatable capacity, even if the container uses 100m. No other pod can be scheduled into the 900m it leaves idle. The cluster autoscaler works the same way. It adds nodes when pods are pending because their requests don't fit. It does not react to real load. Inflated requests therefore turn directly into extra nodes on your bill.

This affects anyone who writes manifests or owns node pools. Backend engineers set requests. Platform engineers pay for the resulting capacity.

Measure the gap before installing anything

• Spot-check live usage. kubectl top pods shows current usage per pod, which you can compare against the requests in the pod spec. It is a point-in-time snapshot, so don't size from it alone.
• Compare requests to p95 per namespace. Sum CPU and memory requests per namespace. Then query your metrics stack for p95 usage over a representative window, such as one that includes weekly peaks.
• Find stranded capacity. kubectl describe node lists allocatable resources and the percentage already requested. Nodes that are mostly requested but lightly used hold capacity that is reserved and wasted.

Illustrative estimate (assumed numbers, not a benchmark):

stranded CPU = replicas × (request − p95 usage)

Assume 12 replicas, each requesting 1 CPU, with p95 usage of 150m:

12 × (1.0 − 0.15) = 10.2 CPU reserved but unused.

That is more than two full 4-vCPU nodes held only by request padding.

Copy#!/usr/bin/env bash
set -euo pipefail

kubectl top pods -A --sort-by=cpu | head -n 20

kubectl get pods -A --field-selector=status.phase=Running -o json | jq -r '
  def cores: if endswith("m") then (.[:-1] | tonumber / 1000) else tonumber end;
  [.items[] | {ns: .metadata.namespace,
     cpu: ([.spec.containers[].resources.requests.cpu // "0" | cores] | add)}]
  | group_by(.ns)[]
  | "\(.[0].ns)\t\([.[].cpu] | add)"' | sort -t$'\t' -k2 -nr

kubectl describe nodes | awk '
  /^Name:/                 { print "\n" $0 }
  /^Allocatable:/          { f=1 }
  /^System Info:/          { f=0 }
  /^Allocated resources:/  { f=1 }
  /^Events:/               { f=0 }
  f                        { print }'

A read-only audit that compares live pod usage, per-namespace CPU requests and 7-day p95 usage, and shows per-node allocatable against requested capacity, so you can size the overprovisioning gap before installing VPA.

How does VPA work end to end?

VPA works as a feedback loop between observed usage and pod specs. Each VerticalPodAutoscaler object points at a workload controller, such as a Deployment or StatefulSet, through targetRef. The loop then runs in four steps:

1. Observe. The recommender reads container CPU and memory usage from the cluster's metrics API. It also watches for OOM kill events, which push the memory recommendation upward even when the usage samples alone would not.
2. Aggregate. Samples go into per-container histograms whose older samples carry progressively less weight. Recent behaviour counts most, but peaks from earlier in the window are not forgotten right away.
3. Recommend. From those histograms the recommender derives a target request, plus lowerBound and upperBound. It then clamps the values to any minAllowed and maxAllowed you set and writes them to the object's status.
4. Apply, depending on updateMode.
   • Off: nothing is applied. You read the status and decide.
   • Initial: the admission controller writes the recommended requests into pods only when they are created. Running pods are left alone.
   • Recreate and Auto: the updater also evicts running pods whose requests drift far enough from the recommendation, and the admission controller sets new values when the replacement pods are created.

VPA changes requests. By default it also scales limits to keep the original request-to-limit ratio, so a container that started with a 1:2 ratio keeps it after resizing. Keep this in mind if you rely on specific limit values.

Two consequences follow from this design. First, the quality of every recommendation depends on how long and how completely the recommender has observed the workload. Second, in every mode except Off, applying a change means pod creation, which in the automatic modes means eviction. The rest of this article builds on those two facts.

How does the Vertical Pod Autoscaler work in recommendation-only mode?

The Vertical Pod Autoscaler (VPA) is a Kubernetes add-on, installed separately from the core control plane, that estimates how much CPU and memory a workload's containers actually need. It has three components:

• Recommender: watches container usage, builds decaying histograms of CPU and memory over a multi-day window, and writes target, lower-bound and upper-bound values into the status of each VerticalPodAutoscaler object.
• Updater: evicts pods whose requests are far from the recommendation, so they can be recreated with new values.
• Admission controller: a mutating webhook that rewrites resource requests on pods as they are created.

With updateMode: "Off", only the recommender does any work for that object. The updater does not evict its pods, and the admission controller does not change them. You get the analysis without any change to running workloads, which is how you should start a rightsizing effort. You can compare the recommendations against your own p95 figures before you trust them.

Data flow in Off mode:

kubelet / cAdvisor → metrics pipeline → recommender → VerticalPodAutoscaler.status.recommendation

updater (evictions): not used admission controller (pod mutation): not used

The recommender is only as good as the data it receives. If the metrics pipeline drops samples, restarts, or was installed recently, the histograms are thin. The output can then look precise but reflect a quiet afternoon rather than a month-end batch run. Check that metrics have been flowing continuously for the whole period you care about before acting on any recommendation.

Copy+------------------+     +------------------+     +---------------------+
| kubelet/cAdvisor |---->| metrics pipeline |---->|     recommender     |
| (container use)  |     | (usage samples)  |     | decaying histograms |
+------------------+     +------------------+     +----------+----------+
                                                             |
                                                             | writes target,
                                                             | lower, upper
                                                             v
                                       +------------------------------------+
                                       | VerticalPodAutoscaler              |
                                       | spec.updatePolicy.updateMode: Off  |
                                       | status.recommendation              |
                                       +------------------------------------+
                                         read via kubectl describe vpa

+--------------------------------------------------------------------------+
| NOT USED WHEN updateMode IS Off                                          |
|  +--------------------------+      +----------------------------------+  |
|  | updater                  |      | admission controller             |  |
|  | (no evictions)           |      | (no pod mutation)                |  |
|  +--------------------------+      +----------------------------------+  |
+--------------------------------------------------------------------------+
  running pods keep their current resource requests

In updateMode Off, usage metrics flow only to the recommender, which writes target and bounds into the VerticalPodAutoscaler status. The updater and admission controller stay idle, so running pods keep their current requests.

How do you configure VPA and read its recommendations?

Start with a VerticalPodAutoscaler whose targetRef points at the Deployment and whose updatePolicy.updateMode is "Off". In this mode the recommender still computes values and writes them to the object's status, but nothing is evicted or mutated. That lets you review the numbers before any pod changes.

Add a resourcePolicy with one containerPolicies entry per container:

• minAllowed and maxAllowed keep recommendations inside a range you can defend. For example, a floor stops a mostly idle service from being squeezed below what it needs to absorb a burst.
• controlledResources limits VPA to cpu, memory, or both. Leave out a resource you size by hand, such as memory for a JVM with a fixed heap.
• mode: "Off" on a specific containerName excludes that container entirely. Use it for injected sidecars whose requests another system owns.

To inspect the result, run kubectl describe vpa <name>, or kubectl get vpa <name> -o yaml and read status.recommendation.containerRecommendations.

Treat target as a starting point, not a final answer. It is derived from a usage distribution plus a safety margin, which does not account for p99 latency or CPU throttling during short spikes. Latency-sensitive services often need headroom above it.

Every value also reflects only the history the recommender has observed. If that history doesn't include a weekly peak or a release-day surge, neither do the numbers.

CopyapiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: checkout-api-vpa
  namespace: shop
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-api
  updatePolicy:
    # Recommend only: no evictions, no admission-time mutation
    updateMode: "Off"
  resourcePolicy:
    containerPolicies:
      - containerName: app
        # Memory is sized by hand (fixed JVM heap), so VPA only covers CPU
        controlledResources: ["cpu"]
        minAllowed:
          cpu: 250m   # floor so an idle period can't shrink burst headroom
        maxAllowed:
          cpu: "2"    # ceiling you can defend against node size and budget
      - containerName: istio-proxy
        # Sidecar requests are owned by the mesh injector; exclude it entirely
        mode: "Off"
---

A recommendation-only VPA that limits the app container to CPU within explicit bounds, excludes an injected sidecar, and shows the kubectl commands for reading lowerBound, target, upperBound and uncappedTarget.

How much can rightsizing with VPA actually save? An illustrative cost model

Any savings estimate assumes the recommendations are trustworthy. If the observation window missed a peak, the recommendation will undersize exactly when it matters. With that caveat, here is what the numbers can look like.

Illustrative estimate (assumed numbers, not a benchmark). Assume 8-vCPU nodes with 7.5 vCPU allocatable after system reservations. DaemonSets request 0.5 vCPU per node, leaving 7 vCPU for workloads. Each node costs an assumed $0.40/hour, and the month has 730 hours. CPU is the binding resource in this example.

Nodes = ceil(Σ(replicas × request) ÷ (allocatable − DaemonSet requests))

Before: ceil(31.0 ÷ 7) = 5 nodes, or $1,460/month. After: ceil(13.5 ÷ 7) = 2 nodes, or $584/month.

Requests fell 56%, but that formula assumes perfect packing. Real clusters fragment. A 900m worker pod cannot use 600m spread across two nodes. Spread constraints, anti-affinity, memory limits on a different node, and PodDisruptionBudgets can all block consolidation. If the scheduler needs 3 nodes, the bill is $876/month, a 40% saving.

Savings appear only when the cluster autoscaler can actually drain a node and remove it. Shaving 30% off every pod while every node keeps one unmovable pod saves nothing.

Also count the engineering cost: reviewing recommendations, changing manifests, watching for throttling and OOM kills, and rolling back mistakes. On a small cluster, that labor can exceed a year of node savings.

How can VPA save engineering time?

The node bill is only part of the case. Sizing requests by hand is repetitive work, and VPA removes the most tedious parts of it.

• No per-service sizing exercise. Without VPA, an engineer sizing a service typically pulls usage graphs, picks a window, eyeballs a percentile, adds a margin, and repeats for every container. VPA in Off mode produces the same kind of answer for every targeted workload continuously. The human job shrinks from computing the number to reviewing it.
• One method across teams. When each team sizes by its own rules of thumb, reviewers have to argue every number from scratch. A shared source of recommendations, with lowerBound and upperBound attached, gives reviewers a common reference and shortens those discussions.
• Less re-tuning as workloads drift. Usage changes as code and traffic change. Hand-set requests go stale silently. Because the recommender keeps updating, you can find stale requests by comparing current requests against target, instead of waiting for an incident or a cost review to surface them.
• Fewer memory incidents on new or changing services. The recommender raises memory recommendations after OOM kills. Reading that signal is quicker than reconstructing it from events and dashboards after the fact.
• Hands-off resizing where it is safe. For stateless, multi-replica services that tolerate restarts, Initial or Auto mode removes the manifest-edit-and-review cycle entirely, subject to the caveats in the section on automatic modes.

The time saved depends on how many workloads you run and how often they change. Illustrative estimate (assumed inputs, not a measurement): assume 40 services, each re-sized twice a year, at an assumed 2 hours of manual analysis per sizing. That is 160 engineer-hours a year. If reviewing a VPA recommendation and opening a pull request takes an assumed 30 minutes instead, the same work takes 40 hours. Substitute your own counts. VPA itself still needs installing, upgrading and monitoring, so subtract that operating time before you count the result as savings.

How do you roll out new resource requests safely?

Start by letting the recommender observe at least one full traffic cycle, including weekly peaks, month-end batch jobs and scheduled reports. A recommendation built from a quiet Tuesday will undersize a workload that spikes on Sunday night.

Then treat each change like any other config change:

1. Copy the recommended values into manifests through your normal GitOps or review flow, not auto mode. That gives you a diff, an approver and a one-commit rollback.
2. Roll out one service at a time, so a regression points to a single change.
3. After each rollout, watch CPU throttling, OOMKills, evictions and tail latency for a full cycle before moving on.

CPU and memory fail differently. Cutting CPU requests mostly affects scheduling. Throttling becomes a problem mainly when limits are also tight, so CPU tolerates aggressive cuts. Memory has no graceful degradation, because exceeding the limit kills the container. Leave a larger margin above the upper-bound recommendation. If you lower requests but keep limits, requests and limits no longer match, and the pod loses Guaranteed QoS. That makes it an earlier eviction candidate under node memory pressure. Runtimes need matching changes too. A JVM sized by -Xmx or MaxRAMPercentage, or Node.js with --max-old-space-size, must be adjusted along with the container memory.

Expect pushback from teams who padded requests after a past outage. Let service owners approve their own changes, and show them the observed usage data. Arguing from measured percentiles is faster and more convincing than the usual round of manual guesswork and load tests for each service.

CopyapiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: orders-api
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: orders-api
  updatePolicy:
    updateMode: "Off"   # read status.recommendation, apply via a reviewed PR
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  template:
    spec:
      containers:
        - name: app
          image: registry.example.com/orders-api:1.4.2
          env:
            # Heap follows the container limit, so lowering memory shrinks the heap too.
            - name: JAVA_TOOL_OPTIONS
              value: "-XX:MaxRAMPercentage=75.0"
          resources:
            requests:
              cpu: "250m"      # was 1000m; near VPA target, CPU can be cut harder
              memory: "768Mi"  # was 2Gi; margin kept above VPA upperBound
            limits:
              # No CPU limit: avoids throttling from a tight cap.
              # Requests no longer equal limits, so the pod is Burstable, not Guaranteed.
              memory: "768Mi"  # equal to request so the heap size is predictable

A VPA in recommendation-only mode plus the reviewed Deployment change it informs, with CPU cut aggressively, memory kept above the upper bound, and the JVM heap tied to the new container limit.

How should VPA be combined with the Horizontal Pod Autoscaler?

The two autoscalers conflict when they act on the same resource for one workload. HPA computes CPU or memory utilization as a percentage of the pod's request. If VPA lowers the request, the same usage looks like higher utilization, so HPA adds replicas. Load then spreads across more pods, per-pod usage drops, VPA recommends a smaller request again, and the cycle repeats. You end up with many undersized pods, and the replica count no longer reflects real demand.

Give each tool its own job. Use VPA recommendations, typically in Off mode, to set per-pod requests through your normal deploy process. Let HPA scale replica count. If you want both active at the same time, drive HPA from a signal that VPA does not change, such as requests per second, queue depth, or another custom or external metric.

What are the alternatives to VPA for rightsizing?

VPA is one way to get accurate requests, not the only one. Which alternative fits depends on how many workloads you run and how much automation you want to operate.

• Manual sizing from your metrics stack. Run percentile queries like the p95 query in the audit script above, add a margin, and edit manifests. This needs no extra cluster components and gives you full control over the percentile and window. The cost is engineer time and inconsistency between teams. It also goes stale unless someone repeats it.
• Horizontal scaling instead of vertical. For stateless services, keeping per-pod requests modest and letting HPA add replicas under load means you pay for peak capacity only during the peak. HPA does not fix oversized per-pod requests, though. It multiplies them.
• Namespace guardrails with LimitRange and ResourceQuota. A LimitRange sets default requests and limits for containers that don't specify them, and can enforce minimums and maximums. A ResourceQuota caps total requests per namespace. Neither measures usage, so they prevent the worst padding rather than rightsizing anything. They work well alongside any of the other options.
• Load testing before release. Sizing from a load test gives you numbers before production traffic exists, which VPA cannot do for a new service. The results are only as good as the test's resemblance to real traffic.
• Node-level changes. Choosing node shapes that match your workloads' CPU-to-memory ratio, and letting the cluster autoscaler remove underused nodes, reduces stranded capacity. These changes complement request tuning rather than replace it.
• In-place pod resize. Newer Kubernetes versions can change a running pod's resources without recreating it. This is a mechanism for applying new values, not a source of recommendations. Check whether your cluster version supports it before planning around it.

In practice most teams combine several of these: guardrails for defaults, VPA or manual analysis for the numbers, HPA for replica count, and node shapes chosen once requests are accurate.

When should you avoid VPA automatic modes, and when are they reasonable?

Stay in Off mode when an eviction costs more than the savings. In Recreate and Auto modes, the VPA updater applies new requests by evicting pods so they are recreated with updated values. Some Kubernetes and VPA versions can resize pods in place, but check what your cluster supports before you rely on it.

That eviction behaviour rules out several workload types:

• Single-replica workloads. Every update is an outage.
• Stateful services. Restarts trigger failover, rebalancing or replica resync.
• Slow-starting or cache-warming applications. A JVM service or a cache that takes minutes to warm runs degraded after each eviction.
• New services. They have too little history for a trustworthy recommendation.
• Seasonal or rare workloads. If month-end batch jobs or holiday peaks fall outside the observation window, the recommendation reflects the quiet period and will undersize the spike.

Initial mode is a safer middle ground. It sets requests only when a pod is created, so changes ride along with normal deploys and scale-ups. Initial mode, or even Auto, is reasonable for stateless services that meet three conditions:

• They run multiple replicas.
• They tolerate restarts.
• They have PodDisruptionBudgets. The updater evicts through the eviction API, so a budget limits how many replicas go down at once.

If you enable these modes, monitor eviction frequency, OOMKills, CPU throttling, restart counts, and how often recommendations swing between values. Steady swinging means the workload is too variable for automatic updates.

VPA also adds operational overhead. The recommender, updater and admission webhook all need upgrades, resource requests and alerting of their own. An unhealthy admission controller can quietly leave pods running with their original requests, or interfere with pod creation, depending on how the webhook is configured.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes rightsizing vpa, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.