Kubernetes Rightsizing with VPA: Cut Costs by Fixing Requests

Why cluster costs follow resource requests, not usage

Rightsizing means setting each container's CPU and memory requests (and limits) close to what it actually needs. It matters because the Kubernetes scheduler places pods based on requests, not observed usage. A pod requesting 2 CPU removes 2 CPU from the node's allocatable capacity whether the container uses 2 cores or 0.2. When requests fill a node, new pods go Pending and a cluster autoscaler adds nodes. You pay for those nodes even though the existing ones are mostly idle. The gap between reserved and used capacity is stranded capacity.

Illustrative estimate (assumed numbers, not a benchmark)

Stranded cost per month = (request − p95 usage) × replicas × price per unit-hour × hours

Assume 40 replicas, each requesting 1 vCPU and using 0.3 vCPU at p95, at $0.04 per vCPU-hour over 730 hours:

(1 − 0.3) × 40 × $0.04 × 730 ≈ $818/month

This is an upper bound. In practice you keep some headroom above p95.

CPU and memory fail differently

CPU is compressible. A container that hits its CPU limit is throttled, so it runs slower and latency rises, but it keeps running.

Memory is not compressible. A container that exceeds its memory limit is OOM-killed and restarted. A pod using more memory than it requested is also a likely eviction candidate when the node comes under memory pressure. Because of this, cutting memory requests carries more risk than cutting CPU, and rightsizing has to account for both directions.

This guide is for backend engineers who own workload manifests and platform engineers who own node pools and the bill.

Copy  Pod spec: requests cpu=1, memory=1Gi  (x 40 replicas)
           |
           v
+--------------------+
|   kube-scheduler   |  places pods by REQUESTS, not observed usage
+--------------------+
           |
           v
+--- Node A (allocatable 4 vCPU) --------------------------------------+
| [pod 1 vCPU] [pod 1 vCPU] [pod 1 vCPU] [pod 1 vCPU]  -> 4/4 reserved |
|   used 0.3     used 0.3     used 0.3     used 0.3    -> 1.2/4 used   |
|  <--------- 2.8 vCPU stranded: paid for, idle --------->             |
+----------------------------------------------------------------------+
           |
           |  next pod does not fit -> Pending
           v
+--------------------+       +--------------------------------+
| cluster autoscaler |------>| Node B added: billed per hour  |
+--------------------+       +--------------------------------+

  Runtime behaviour when usage exceeds limits or requests:
+-----------------------------+   +-----------------------------------+
| CPU (compressible)          |   | Memory (not compressible)         |
| usage > limit -> throttled  |   | usage > limit -> OOM-killed,      |
| runs slower, latency rises, |   |   container is restarted          |
| keeps running               |   | usage > request under node        |
+-----------------------------+   |   pressure -> likely evicted      |
                                  +-----------------------------------+

The scheduler reserves requested capacity rather than actual usage, so over-requested pods strand paid-for capacity and push the cluster autoscaler to add nodes, while CPU overuse causes throttling and memory overuse causes OOM kills or eviction.

How the Vertical Pod Autoscaler computes recommendations

Three components, one feedback loop

VPA runs as three separate controllers, and only one of them matters for recommendation-only rightsizing:

• Recommender: reads container usage from the metrics pipeline and keeps a decaying histogram per container, where older samples count less than recent ones. CPU recommendations come from a high percentile of observed usage. Memory recommendations come from peak usage per time window plus a safety margin, and an OOM kill pushes the memory estimate up. This asymmetry reflects the cost of being wrong: throttled CPU is slow, but exceeded memory is fatal.
• Updater: evicts pods whose current requests fall too far outside the recommended range, so they can be recreated with new values.
• Admission controller: a mutating webhook that rewrites requests on pods as they are created.

In recommendation-only mode, the updater and admission controller do nothing to your pods. The recommender writes its output to the VPA object's status, and you decide what to do with it.

Reading the status

Run kubectl describe vpa <name>, or fetch the object as YAML, and look under status.recommendation.containerRecommendations. Each container has an entry with these fields:

The bounds are wide when there is little usage history and narrow as data accumulates. If the gap between them is still large, the recommender does not yet have enough confidence in its estimate.

Copy# Recommendation-only VPA: the recommender writes status, nothing is evicted or mutated.
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: checkout-api
  namespace: shop
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-api
  updatePolicy:
    updateMode: "Off"          # updater and admission controller leave pods alone
  resourcePolicy:
    containerPolicies:
      - containerName: app
        minAllowed:
          cpu: 50m             # floor that can clamp target (compare uncappedTarget)
        maxAllowed:
          memory: 2Gi
---
# Illustrative status excerpt (values are made up), as seen with:
#   kubectl get vpa checkout-api -n shop -o yaml
# status:
#   recommendation:
#     containerRecommendations:
#       - containerName: app
#         lowerBound:          # requests below this: under-provisioned
#           cpu: 50m
#           memory: 180Mi
#         target:              # starting point for requests
#           cpu: 50m
#           memory: 256Mi
#         uncappedTarget:      # cpu 25m < target 50m: minAllowed is clamping
#           cpu: 25m
#           memory: 256Mi
#         upperBound:          # requests above this: likely waste
#           cpu: 400m
#           memory: 700Mi

A VPA in recommendation-only mode (updateMode "Off") with a resource policy, followed by an illustrative status excerpt showing how target, lowerBound, upperBound and uncappedTarget appear and how a minAllowed floor makes target differ from uncappedTarget.

Running VPA in recommendation-only mode

Create a VerticalPodAutoscaler whose targetRef points at the Deployment or StatefulSet, and set updatePolicy.updateMode: "Off". In this mode only the recommender acts on the object. It reads usage metrics, builds its histograms, and writes results to status.recommendation. The updater does not evict pods, and the admission controller does not rewrite requests when new pods are created. You can read the output with kubectl describe vpa <name>. Each container gets target, lowerBound, upperBound, and uncappedTarget values.

Use resourcePolicy.containerPolicies to constrain the output for each container:

• minAllowed / maxAllowed clamp the recommendation. For example, a JVM service needs at least enough memory for its configured heap, even if observed usage looks low during a quiet week. A ceiling keeps a single traffic spike from producing a request that no node can fit.
• controlledResources limits which resources VPA reasons about. Set it to ["memory"] if CPU is already scaled horizontally.
• mode: "Off" on a specific containerName excludes sidecars, such as proxies or log shippers, whose sizing is owned elsewhere.

Compare target against uncappedTarget. If they differ, your bounds are clamping the recommendation. That means either the bounds are wrong or the workload has changed.

Recommendation-only mode is safer than the automatic modes for two reasons:

• No surprise evictions. Recreate and Auto apply changes by evicting pods. Evictions respect PodDisruptionBudgets, but they still cause restarts at times you did not choose. They also cause cold caches and dropped long-lived connections.
• Changes stay in version control. Recommendations that are applied automatically never pass through review. The running requests then diverge from what your manifests declare. With Off, VPA supplies the numbers and a human decides whether to commit them.

CopyapiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: orders-api-vpa
  namespace: shop
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: orders-api
  updatePolicy:
    # Recommender only: no evictions, no admission-time rewrites.
    updateMode: "Off"
  resourcePolicy:
    containerPolicies:
      - containerName: orders-api
        # CPU is handled by HPA; only reason about memory here.
        controlledResources: ["memory"]
        minAllowed:
          # Floor: JVM heap (-Xmx768m) plus off-heap overhead.
          memory: 1Gi
        maxAllowed:
          # Ceiling: keep a spike from producing an unschedulable request.
          memory: 4Gi
      - containerName: envoy-proxy
        # Sidecar sizing is owned by the mesh config; exclude it.
        mode: "Off"
# Read results without touching pods:
#   kubectl describe vpa orders-api-vpa -n shop
# If target != uncappedTarget, the min/max bounds are clamping it.

A recommendation-only VPA that watches a Deployment's memory within explicit bounds and leaves a sidecar out entirely.

A practical rightsizing workflow: from recommendation to merged change

Collect over a representative window

Let the recommender observe each workload long enough to see its real peaks: a weekly batch job, month-end reporting, or a traffic spike from a regional business day. A recommendation built only from a quiet stretch will understate what the pod needs. Read the target, lowerBound and upperBound values from the VPA object's status with kubectl describe vpa, and put them next to the current requests in a short table per container.

Set requests from targets, with headroom

Use target as the baseline and add a margin sized to how costly a miss would be. Latency-sensitive services warrant more headroom than background workers. For memory, set the limit equal to or close to the request. A pod whose usage exceeds its request becomes a likely eviction candidate when the node runs short of memory, and a wide gap hides that risk. For CPU, decide deliberately whether a limit earns its place. CFS quota throttling can add tail latency even when the node has idle cores, so many teams drop CPU limits on services where predictable latency matters more than strict isolation.

Ship through review

Commit the new values to the same manifests or GitOps repository as every other change. The declared state stays authoritative, and no controller rewrites what another one applied.

This replaces arguments about guessed numbers with a diff anyone can check against observed data. Reviews become quick and defensible. Usage still drifts as code and traffic change, so schedule the review as recurring work, for example quarterly, with a named owner per service. Without an owner, rightsizing tends to happen once and then lapse.

Copy# Recommendation-only VPA: it observes usage and never evicts or rewrites pods.
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: checkout-api
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-api
  updatePolicy:
    updateMode: "Off"   # people apply changes through Git review, not the updater
---
# Requests come from the VPA target plus headroom and are merged through a PR.
# Illustrative inputs: target cpu=250m, memory=410Mi, with roughly 20% headroom.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: checkout-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: checkout-api
  template:
    metadata:
      labels:
        app: checkout-api
    spec:
      containers:
        - name: app
          image: registry.example.com/checkout-api:1.4.2
          resources:
            requests:
              cpu: 300m        # target 250m plus headroom for a latency-sensitive path
              memory: 512Mi    # target ~410Mi plus headroom
            limits:
              memory: 512Mi    # equal to the request, so no memory overcommit is hidden
              # No CPU limit: this avoids CFS throttling while node cores sit idle

A VPA in recommendation-only mode next to the Deployment whose requests were set from its target with headroom: the memory limit equals the request, the CPU limit is deliberately omitted, and the change ships through normal manifest review.

Impact on cluster costs and engineering time: an illustrative estimate

Illustrative estimate (assumed numbers, not a benchmark). Assume one service runs 40 replicas on nodes with 7 vCPU and 28 GiB allocatable, priced at $0.40 per node-hour. VPA recommends lowering CPU from 1000m to 400m and memory from 2 GiB to 1.2 GiB per replica.

Node count is set by whichever resource runs out first:

nodes = ceil(max(total CPU requests / node CPU, total memory requests / node memory))

This math is a lower bound on what the scheduler needs. It ignores DaemonSets, system reservations, other workloads and imperfect packing. The freed capacity only becomes savings if the cluster autoscaler drains and removes nodes, or if you consolidate node pools yourself. If neither happens, smaller requests just leave idle space on nodes you still pay for.

To confirm the result, do not trust the recommendation diff. Record three numbers for a week before and a week after the change: requested resources, actual usage, and node count with allocatable utilization. A real saving shows up as a lower node count with requests closer to usage. It should not come with new throttling, OOM kills or eviction events.

Subtract the operating cost too. That includes the VPA components, upgrades, alert tuning, and review time. For example, with assumed inputs of 10 services × 2 hours × 4 quarterly reviews, that is 80 engineer-hours a year. At small cluster sizes, that time can exceed the node savings.

When to avoid VPA-driven rightsizing

VPA is a poor fit for some workloads, and for others its output needs human judgment before anyone acts on it.

• Disruption-sensitive workloads. In its automatic modes, VPA applies new requests by evicting pods. A single-replica Deployment or a StatefulSet without a PodDisruptionBudget can go down when VPA evicts its only pod or its quorum member. Keep these in recommendation-only mode, or add a PDB and enough replicas first.
• Rare or periodic peaks. Recommendations come from a decaying usage history. A monthly report job, a nightly batch run, or a seasonal traffic surge barely registers against weeks of quiet baseline, so the target can fall below what the peak needs. Size these workloads from known peak behavior, not from recent averages.
• Garbage-collected runtimes. A JVM grows its heap toward its configured maximum, so observed memory reflects the heap settings, not the real need. If you cut the memory request without changing heap flags, the next full heap can trigger an OOM kill. Change the heap configuration and the request together.
• Latency-sensitive services. Requests close to median usage leave little headroom. Under bursty load or noisy neighbors, CPU throttling and contention show up first in tail latency, even when averages look fine.
• Operational surface. The recommender, updater, and admission webhook need RBAC, monitoring, and compatibility checks on every cluster upgrade. A failing webhook can also affect pod admission.
• Staleness. A recommendation describes past code and traffic. After a major release or a traffic shift, treat old targets as expired and wait for fresh data before acting on them.

Alternatives and combining VPA with HPA

VPA is one of several ways to manage resources, and each one changes a different variable.

The cluster autoscaler makes scheduling decisions from requests, not from usage. If requests are inflated, it packs them efficiently onto nodes that stay underused. Bin-packing does not replace rightsizing. It only pays off after requests are correct.

Why VPA and HPA conflict on the same metric

HPA's CPU and memory targets are utilization percentages, calculated against the pod's request. When VPA changes that request, it changes the number HPA is reacting to.

Consider rising load. HPA adds replicas, so usage per pod falls. VPA then lowers the requests, which pushes utilization back up. HPA responds by scaling out again. The two controllers keep reacting to each other, and you get replica churn and evictions without a stable state.

Choosing the dimension per workload

• Stateless services whose load tracks demand: let HPA scale replicas on a custom or external metric, such as requests per second or queue depth. Let VPA size the requests. The two now act on independent signals.
• Singletons and workloads that cannot shard: use VPA alone.
• Bursty batch jobs: set requests manually from peak usage, because short-lived pods give VPA too little history to work from.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes rightsizing vpa, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.