Kubernetes Vertical Pod Autoscaler: Right-Sizing Without Restarts

How the Vertical Pod Autoscaler works: recommender, updater and admission controller

The Vertical Pod Autoscaler (VPA) adjusts CPU and memory resource requests based on observed usage. Requests drive scheduling and node packing, and most teams set them by guesswork. Requests set too high waste capacity. Requests set too low cause CPU throttling, OOM kills and noisy-neighbor problems. VPA replaces the guess with a recommendation derived from what the workload actually consumes.

You create a VerticalPodAutoscaler object whose targetRef points at a controller such as a Deployment or StatefulSet. VPA itself runs as three independent components.

Recommender

The VPA recommender reads container usage from the Metrics API, and optionally from a metrics history store. It aggregates samples into decaying histograms, so recent behavior weighs more than old behavior. It then writes target, lower-bound and upper-bound values into the VPA object's status. It never touches pods.

Updater

The updater compares running pods against those bounds. When a pod falls outside the bounds and the update mode allows it, the updater evicts the pod through the Eviction API, which means PodDisruptionBudgets apply.

Admission controller

The admission controller is a mutating webhook. When a matching pod is created, it rewrites the pod's requests to the current recommendation and scales limits proportionally.

Data flow and failure behavior

Data moves one way: metrics feed the recommender, the recommender writes VPA status, and the admission controller applies that status to the pod spec.

Because the three parts are decoupled, each one fails differently:

• Recommender stops: recommendations go stale but are still applied.
• Updater stops: pods keep their current requests until something else recreates them.
• Admission webhook unavailable: the outcome depends on its failure policy. With Ignore, pods start with their original requests. With Fail, pod creation is blocked.

Copy +-----------------+     +-----------------+     +------------------------+
 | Metrics API     |     | Recommender     |     | VPA object             |
 | (+ optional     |---->| aggregates into |---->| spec.targetRef ->      |
 |  history store) |     | decaying        |     |   workload             |
 +-----------------+     | histograms;     |     | status: target,        |
                         | writes bounds   |     |  lower/upper bounds    |
                         +-----------------+     +-----+------------+-----+
                          +----------------------------+            |
                          | reads bounds               reads target |
                          v                                         v
              +------------------------+             +-----------------------+
              | Updater                |             | Admission controller  |
              | pod outside bounds and |             | mutating webhook;     |
              | mode allows it? ->     |             | on pod CREATE: set    |
              | evict via Eviction API |             | requests = target,    |
              | (PDBs apply)           |             | scale limits          |
              +-----------+------------+             +-----------+-----------+
                          | evicts                               ^
                          v                                      |
              +------------------------+             +-----------+-----------+
              | Pods of targetRef      |             | Deployment/StatefulSet|
              | workload (old requests)|-- evicted ->| recreates the pod     |
              +------------------------+             +-----------------------+

Recommender down: status goes stale; webhook keeps applying old target.
Updater down: no evictions; requests change only when pods are recreated.
Webhook down: failurePolicy Ignore -> pods start with unmodified requests;
              failurePolicy Fail -> matching pod creation is rejected.

Usage metrics flow one way through the VPA: the recommender writes bounds to the VPA object's status, the updater evicts pods that fall outside those bounds, and the admission webhook applies the target to each pod as it is recreated.

Reading what the VPA recommender actually produces

Start with the smallest useful object: a VerticalPodAutoscaler whose targetRef points at your Deployment and whose updatePolicy.updateMode is "Off". In this mode the recommender computes values but nothing acts on them, so you can observe without risk. After the recommender has collected some usage, run kubectl describe vpa <name> and look under Status → Recommendation → Container Recommendations. Each container reports four values:

• Target: the request the VPA would apply. This is the number most people act on.
• Lower Bound / Upper Bound: a confidence range around the target. A wide range means the recommender has little data. When the bounds converge, its estimate is more stable.
• Uncapped Target: what the target would be without your resourcePolicy. If maxAllowed or minAllowed clips the recommendation, Target and Uncapped Target differ. A large gap means your caps are overriding the data, which is worth investigating rather than ignoring.

All four values come only from observed history, and the recommender weights recent samples more heavily. A workload that has run for a few hours, a month-end spike, or a batch job that ran outside the window will produce a target that is too small. Undersized CPU requests show up as throttling where limits apply. Undersized memory requests show up as OOM kills.

Checks before trusting a number

• Confirm the VPA has observed at least one full traffic cycle, including weekly or monthly peaks.
• Compare Target against your own peak usage metrics, not just kubectl top, which shows a single moment.
• Check the pod's past restarts for OOMKilled, and check CPU throttling counters.
• Treat a large gap between Target and Uncapped Target as a reason to revisit your caps.

Copy# VPA in recommendation-only mode: the recommender computes values, nothing is applied.
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: checkout-api-vpa
  namespace: shop
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-api
  updatePolicy:
    updateMode: "Off"          # observe only; no evictions, no request changes
  resourcePolicy:
    containerPolicies:
      - containerName: api
        controlledResources: ["cpu", "memory"]
        # These caps clip Target. If usage pushes past them, Uncapped Target
        # will differ from Target, and that gap is the signal to revisit them.
        minAllowed:
          cpu: 100m
          memory: 128Mi
        maxAllowed:
          cpu: "1"
          memory: 1Gi

# After the recommender has collected usage:
#   kubectl describe vpa checkout-api-vpa -n shop
#   (read Status -> Recommendation -> Container Recommendations)
#
# Put Target and Uncapped Target side by side to spot cap clipping:
#   kubectl get vpa checkout-api-vpa -n shop -o jsonpath='{range .status.recommendation.containerRecommendations[*]}{.containerName}{"\n  target:   "}{.target}{"\n  uncapped: "}{.uncappedTarget}{"\n"}{end}'
#
# Before trusting Target, check whether any container was previously OOM-killed:
#   kubectl get pods -n shop -l app=checkout-api -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.containerStatuses[*].lastState.terminated.reason}{"\n"}{end}'

A minimal Off-mode VPA with resourcePolicy caps, plus kubectl commands to read its recommendation, compare Target with Uncapped Target, and look for past OOMKilled restarts before acting on the numbers.

Update modes and the in-place resize question

Off is where every workload should start. The recommender still computes values and writes them to the VPA object's status, but nothing changes on your pods. That lets you compare recommendations against real behaviour across a full traffic cycle before you hand any control to the autoscaler. Initial is the next step: VPA sets requests only when a pod is created, so changes arrive through your normal rollouts. The modes that act on running pods carry the most risk.

In-place pod resize changes the restart story. Where both the cluster and the VPA release support it, VPA can adjust CPU and memory on a running pod without recreating it. Where support is missing, VPA evicts the pod and lets its controller recreate it with the new requests.

In-place resize does not remove all disruption:

• A container's resizePolicy can mark a resource as RestartContainer. A resize of that resource still restarts the container.
• Lowering memory may not apply safely in place. If the process is already using more memory than the new limit allows, the change can stay pending or fail.
• If the node does not have room to grow the pod, the resize cannot proceed. VPA then falls back to eviction.

Check what your own cluster and VPA versions support before you rely on any of this.

Constraining VPA with resourcePolicy and protecting availability

A resourcePolicy sets bounds on the recommender. Under containerPolicies, each entry targets a containerName, or "*" for all containers. Each entry can set minAllowed and maxAllowed, restrict controlledResources to CPU, memory or both, and set mode: "Off" to exclude sidecars that you size by hand.

controlledValues decides what happens to limits:

• RequestsAndLimits scales limits in proportion to requests. Teams that treat limits as fixed ceilings can be surprised. A limit tuned to match a heap setting can move without anyone editing it.
• RequestsOnly leaves limits alone. As requests shrink, the gap between request and limit widens, so the node is promised less than pods can actually consume. That raises overcommit.

Memory needs a more conservative minAllowed than CPU. A pod that runs short of CPU gets throttled and slows down. A pod that exceeds its memory limit is OOM-killed. Memory usage also tends to spike at startup or during cache warm-up, and a recommendation built from quiet periods can miss those spikes.

Always set maxAllowed. Without it, a recommendation can grow larger than any node's allocatable capacity, and the pod will stay Pending after its next restart.

Smaller requests let the scheduler pack more pods onto each node. The cost shows up under memory pressure. The kubelet considers usage relative to requests when it chooses pods to evict, so pods that routinely run above their requests are evicted first.

For updates that work through eviction, run at least two replicas and add a PodDisruptionBudget so that eviction cannot drain a workload completely. A workload with only one replica will have downtime every time VPA evicts it.

CopyapiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api            # Deployment runs replicas: 3 (single replica = downtime)
  updatePolicy:
    updateMode: "Auto"   # applies recommendations by evicting pods
  resourcePolicy:
    containerPolicies:
      - containerName: app
        controlledResources: ["cpu", "memory"]
        controlledValues: RequestsOnly   # keep hand-tuned limits fixed
        minAllowed:
          cpu: 50m
          memory: 256Mi  # generous floor: startup and cache spikes risk OOM
        maxAllowed:
          cpu: "2"
          memory: 4Gi    # stay below node allocatable or pods go Pending
      - containerName: log-shipper
        mode: "Off"      # sidecar sized by hand, VPA ignores it
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: api-pdb
spec:
  minAvailable: 2        # VPA evictions cannot drop below 2 ready pods
  selector:
    matchLabels:
      app: api

A VPA that bounds the app container with minAllowed and maxAllowed, scales only requests, excludes a sidecar, and is paired with a PodDisruptionBudget so eviction-based updates never take all replicas down at once.

HPA vs VPA and other alternatives

The conflict comes from how HPA measures CPU and memory. It computes utilization as usage divided by the pod's request. If VPA manages that request, both controllers act on the same ratio. When load rises, HPA adds replicas, and per-pod usage drops. VPA then lowers requests to match the lower usage, which pushes utilization back up. HPA reads that as more load and scales out again. In the other direction, VPA can raise requests until HPA's utilization target is never reached, so HPA stops scaling out. In both cases, replica counts and request sizes drift or oscillate instead of settling.

Two combinations avoid this because the controllers no longer share a signal:

• HPA on custom or external metrics (queue depth, requests per second) controls replica count, while VPA controls per-pod requests.
• VPA in Off mode produces recommendations that engineers apply by hand during normal deploys. The VPA controllers never evict or resize pods.

VPA fits best on steady, long-running services where nobody knows the correct size. Avoid automatic modes in these cases:

• Highly bursty workloads: recommendations lag behind spikes.
• Short-lived batch jobs: pods finish before the recommender has enough data.
• Services with slow warm-up: restarts lose JIT or cache state.
• Single-replica services: any eviction means downtime.

A practical rollout workflow

Adoption works best as a sequence of reversible steps, each with a clear signal that tells you whether to continue.

1. Observe in Off mode. Create VPA objects with updateMode: "Off" so the recommender computes targets without touching pods. Leave them through at least one full traffic cycle, including weekly peaks, month-end batches and scheduled CronJobs. A recommendation built only from quiet weekdays will undersize anything that spikes later.
2. Compare three numbers per container: current requests, observed usage, and the VPA target and bounds. Large gaps in either direction are your candidates. If the target sits well below requests but the upper bound is close to them, the workload is bursty and the existing request may be justified.
3. Apply changes through version control. Copy the recommended values into manifests and merge them through normal review. This keeps an audit trail and makes rollback a git revert.
4. Enable updates selectively. Move restart-tolerant, multi-replica stateless services to Initial or an automatic mode first, with a PodDisruptionBudget in place. Leave everything else in Off and keep updating it by hand.

Operational work that comes with it

Treat the recommender, updater and admission controller as production components. Alert on their restarts and errors. Inspect the admission webhook's failurePolicy. With Ignore, an outage means pods start silently with unmodified requests. With Fail, an outage can block pod creation cluster-wide. After each change, watch CPU throttling (container_cpu_cfs_throttled_periods_total) and OOMKilled terminations for the affected workloads before moving to the next batch.

Copy# Step 1: observe only. The recommender computes targets; no pods are evicted or mutated.
kubectl apply -f - <<'EOF'
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: checkout-api
  namespace: shop
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: checkout-api
  updatePolicy:
    updateMode: "Off"
EOF

# Step 2, after a full traffic cycle: gather the three numbers per container.
# (a) Current requests from the live Deployment spec
kubectl -n shop get deploy checkout-api \
  -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"\t"}{.resources.requests}{"\n"}{end}'

# (b) Observed usage right now (a point sample; use your metrics history for peaks)
kubectl -n shop top pod -l app=checkout-api --containers

# (c) VPA lower bound, target and upper bound
kubectl -n shop get vpa checkout-api \
  -o jsonpath='{range .status.recommendation.containerRecommendations[*]}{.containerName}{"\n  lower: "}{.lowerBound}{"\n  target: "}{.target}{"\n  upper: "}{.upperBound}{"\n"}{end}'

# Operational check: what happens to pod creation if the VPA webhook is down?
# Ignore = pods start with unmodified requests; Fail = pod creation can be blocked.
kubectl get mutatingwebhookconfigurations \
  -o custom-columns=NAME:.metadata.name,POLICY:.webhooks[*].failurePolicy

Creates a VPA in Off mode, then lists current requests, observed usage and the VPA bounds for each container side by side, and reports the failurePolicy of each mutating webhook so you know how a VPA admission controller outage would affect pod creation.

Impact on cost and engineering time

The scheduler places pods by their requests, not their actual usage. Every unused millicore in a request is still reserved, so over-provisioned requests shrink schedulable capacity even when nodes look idle on utilization dashboards.

Illustrative estimate (assumed numbers, not a benchmark)

Assume a Deployment has 40 replicas. Each replica requests 1000m CPU and uses about 300m at the 95th percentile. Assume VPA recommends a 400m target, which leaves headroom above observed usage.

• Freed CPU = N × (current request − target request)
• 40 × (1000m − 400m) = 24,000m = 24 cores

On nodes with an assumed 8 allocatable cores, that is roughly three nodes' worth of reservations released. Those cores either take new pods or let the cluster autoscaler remove nodes. To estimate your own savings, plug in your replica counts, your recommendation targets and your node shape.

The engineering-time effect is less visible but often larger. Without VPA, someone estimates requests up front and revisits them after load tests or incidents. Those estimates go stale as code and traffic change. VPA replaces that with continuous recommendations. The cost is that you now operate three more components, and a target that is too low shows up as throttling or OOMKilled events that someone has to trace back to the VPA.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes vertical pod autoscaler, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.