What do Kubernetes resource requests and limits actually do?

A request is the amount of CPU and memory a container is guaranteed. A limit is the ceiling it cannot exceed. Two different components act on these values at two different times.

The scheduler only reads requests. It places a pod on a node whose allocatable capacity, minus the requests of pods already there, can fit the new pod's requests. Actual usage plays no part in this decision. After the pod is placed, the kubelet and container runtime turn both values into cgroup settings:

  • CPU requests become a relative CPU weight, which sets each container's share when the node is contended.
  • CPU limits become a quota. A container that uses its quota is throttled until the next period.
  • Memory limits are hard. A container that goes over is killed by the kernel and reported as OOMKilled.

Requests and limits together set the pod's QoS class:

QoS class How it is assigned Behaviour under node pressure
Guaranteed Every container sets CPU and memory requests equal to limits Evicted last; lowest OOM score
Burstable At least one request or limit set, but not Guaranteed Evicted when usage exceeds requests, ranked by priority and overage
BestEffort No requests or limits on any container Evicted first

Requested capacity is reserved even when the container leaves it idle. A pod that requests 2 CPUs and uses 0.2 still blocks the other 1.8 from every other workload. Over-requesting therefore inflates node count and cost, which affects platform engineers who run the cluster. Under-requesting causes throttling, evictions, and OOM kills, which affects backend engineers who own the services.

# Burstable pod: requests are lower than limits.
# The scheduler reserves only the requests (250m CPU, 256Mi) on the node.
apiVersion: v1
kind: Pod
metadata:
  name: orders-api
spec:
  containers:
    - name: app
      image: registry.example.com/orders-api:1.0.0
      resources:
        requests:
          cpu: "250m"      # cgroup CPU weight; used for placement
          memory: "256Mi"  # used for placement and eviction ranking
        limits:
          cpu: "1"         # CFS quota; throttled above 1 CPU per period
          memory: "512Mi"  # hard cap; exceeding it ends in OOMKilled
---
# Guaranteed pod: requests equal limits for every resource in every container.
apiVersion: v1
kind: Pod
metadata:
  name: payments-worker
spec:
  containers:
    - name: worker
      image: registry.example.com/payments-worker:1.0.0
      resources:
        requests:
          cpu: "500m"
          memory: "512Mi"
        limits:
          cpu: "500m"
          memory: "512Mi"
# Check the assigned class with:
#   kubectl get pod orders-api -o jsonpath='{.status.qosClass}'
# Compare reserved requests to node capacity with:
#   kubectl describe node <node-name>   (see "Allocated resources")

Two pod specs that show how requests drive scheduling and limits drive runtime enforcement, giving one Burstable pod and one Guaranteed pod, plus kubectl commands to check the QoS class and the capacity reserved on a node.

Why can't one headroom rule work for both CPU and memory?

The two resources fail in different ways, so they need different margins.

CPU is compressible. When a container reaches its CPU limit, the kernel's CFS bandwidth control throttles it until the next scheduling period. The process keeps running, only slower, so the symptom is higher tail latency rather than errors. Without a limit, a container can also use idle CPU on the node beyond its request.

Memory is not compressible. The kernel cannot slow down an allocation. When a container exceeds its memory limit, the OOM killer terminates it, and kubectl describe pod shows the last state with reason OOMKilled and exit code 137 (128 + SIGKILL). In-flight requests are lost, and the pod restarts.

CPU Memory
Behaviour at the limit Throttled (CFS quota) Killed (OOMKilled, exit 137)
Visible symptom Latency, timeouts Restarts, dropped requests
Safe basis for the request A percentile of usage Observed peak plus a margin
Cost of headroom that is too small Slower responses Crash loops

A uniform "p95 plus 30%" rule fails in both directions. For CPU, the extra 30% is reserved capacity that spiky workloads rarely use, and short bursts can borrow idle cycles anyway. For memory, p95 plus 30% can still be below a real peak, such as a cache warm-up or a large batch. The result is periodic OOM kills.

Tight requests let the scheduler fit more pods on each node but leave less buffer for spikes. Generous requests absorb spikes but hold capacity that sits idle and that no other pod can be scheduled onto.

# Illustrative values only. Assumed observed usage for this container:
#   CPU:    p90 = 250m, short bursts to ~600m
#   Memory: steady ~300Mi, peak during cache warm-up ~420Mi
apiVersion: v1
kind: Pod
metadata:
  name: api-worker
spec:
  containers:
    - name: app
      image: registry.example.com/api-worker:1.0.0
      resources:
        requests:
          # CPU is compressible. Size the request near a usage percentile,
          # not the peak. Bursts can borrow idle node CPU, and running short
          # costs latency, not errors.
          cpu: "250m"
          # Memory is not compressible. Size the request from the observed
          # PEAK plus a margin (here 420Mi + ~20%), not from p95. If you
          # size it too small, the pod restarts.
          memory: "512Mi"
        limits:
          # No CPU limit is set, so the container is not CFS-throttled and
          # can use spare cycles. If policy requires a limit, keep it well
          # above the burst level so throttling does not add tail latency.
          #
          # The memory limit equals the request. Above this the container
          # is OOMKilled (exit 137). Setting limit = request keeps the
          # scheduler's view of the pod honest.
          memory: "512Mi"
# Check for OOM kills after rollout:
#   kubectl describe pod api-worker   # Last State: Terminated, Reason: OOMKilled, Exit Code: 137
# Check for CPU throttling (cgroup v2):
#   kubectl exec api-worker -- cat /sys/fs/cgroup/cpu.stat   # nr_throttled, throttled_usec

A container spec that sizes the CPU request from a usage percentile with no CPU limit, and sizes memory from the observed peak plus a margin with the limit equal to the request, followed by the commands that reveal each resource's failure mode.

Which metrics should you collect before right sizing pods?

Sizing starts with four signals per container:

  • CPU usage as a rate. container_cpu_usage_seconds_total is a cumulative counter. Take its per-second rate over a short window to get cores in use.
  • Memory working set, not total usage. Total usage includes page cache that the kernel can reclaim, so it overstates what the process needs. container_memory_working_set_bytes is closer to what the kubelet uses for eviction and what pushes a container toward its limit.
  • CPU throttling. The cAdvisor counters container_cpu_cfs_throttled_periods_total and container_cpu_cfs_periods_total show how often a CPU limit stopped the container. Frequent throttling at modest average usage means bursts are being cut off.
  • Restarts and termination reasons. Restart counts, together with a last-terminated reason of OOMKilled, show where memory is already too tight.

Choose a window that includes a full weekly cycle, at least one deploy (startup often allocates more memory), any batch or cron jobs, and a known traffic spike. Two to four weeks is a common choice for that reason.

Three pitfalls distort the data. Dashboards that show averages smooth over short bursts, so look at high percentiles and maximums instead. A coarse scrape interval can miss a memory spike that lasts a few seconds and triggers an OOMKill, so the recorded peak ends up below the real one. If working set climbs steadily and never levels off, you are probably looking at a leak, and sizing up to its growth hides the bug.

The method fits poorly in two cases. One is when the events that matter, such as quarter-end jobs or failovers, are rare enough to fall outside the window. The other is small, cheap workloads, where a conservative fixed size costs less than the engineering time needed to analyze them.

# PromQL queries for one container over a 14-day window.
# Namespace/container names are placeholders.

# CPU cores in use: rate of the cumulative counter, then p95 and max
# across the window via a subquery (5m resolution).
quantile_over_time(0.95,
  rate(container_cpu_usage_seconds_total{namespace="shop", container="checkout"}[5m])[14d:5m])
max_over_time(
  rate(container_cpu_usage_seconds_total{namespace="shop", container="checkout"}[5m])[14d:5m])

# Memory: use working set (excludes reclaimable page cache), take the max.
# The max is only as fine-grained as the scrape interval; short spikes can be missed.
max_over_time(
  container_memory_working_set_bytes{namespace="shop", container="checkout"}[14d])

# Leak check: a positive slope that never levels off suggests a leak, not a sizing need.
deriv(
  container_memory_working_set_bytes{namespace="shop", container="checkout"}[1d])

# Throttling ratio: fraction of CFS periods in which the CPU limit stopped the container.
sum by (pod) (rate(container_cpu_cfs_throttled_periods_total{namespace="shop", container="checkout"}[5m]))
/
sum by (pod) (rate(container_cpu_cfs_periods_total{namespace="shop", container="checkout"}[5m]))

# Restarts in the window (kube-state-metrics).
increase(kube_pod_container_status_restarts_total{namespace="shop", container="checkout"}[14d])

# Containers whose last termination was an OOMKill (kube-state-metrics).
kube_pod_container_status_last_terminated_reason{namespace="shop", container="checkout", reason="OOMKilled"} == 1

PromQL queries that collect the sizing signals for one container over a two-week window: CPU rate percentiles and max, peak memory working set, a leak slope check, the CFS throttling ratio, restarts, and OOMKilled terminations.

How do you calculate CPU and memory requests from usage percentiles?

Pull at least two weeks of per-container data covering a full traffic cycle, then apply different rules to each resource. Memory is incompressible, so size it to the worst realistic case. CPU is compressible, so size it to typical busy load and let the node's spare cycles absorb bursts.

Resource Base value Headroom Limit Illustrative estimate
Memory Max of p99 working set and startup peak 15–25% Equal to the request max(410, 480) MiB × 1.2 ≈ 576 → 600Mi
CPU p90 usage Round up See next section p90 = 230m → 250m

Illustrative estimate. Assumed inputs: p99 working set 410 MiB, startup peak 480 MiB, p90 CPU 230 millicores.

In the container spec, this becomes requests.memory: 600Mi, limits.memory: 600Mi and requests.cpu: 250m. Setting the memory limit equal to the request means the container is OOMKilled at a known size, not when a crowded node starts evicting pods.

The runtime must respect that ceiling:

  • JVM: Set heap with -XX:MaxRAMPercentage, often 60–75%, to leave room for metaspace, thread stacks and direct buffers.
  • Go: Set GOMEMLIMIT a little below the container limit so the garbage collector works harder before the kernel kills the process.
  • Node.js: Set --max-old-space-size below the limit. Buffers and native memory are allocated outside the V8 heap.

Count in-process caches at their full warmed size, not the size seen just after a restart.

Finally, check the whole pod. Sidecar requests add to the pod total. The pod's effective request is the larger of the biggest init container and the sum of the app containers. Compare that total against node allocatable, not node capacity, so pods still fit after system reservations.

apiVersion: v1
kind: Pod
metadata:
  name: orders-api
spec:
  containers:
    - name: app
      image: registry.example.com/orders-api:1.0.0
      env:
        # Go runtime: soft memory limit ~90% of the container limit,
        # so the GC works harder before the kernel OOMKills the process.
        - name: GOMEMLIMIT
          value: "540MiB"
      resources:
        requests:
          cpu: 250m        # p90 CPU (230m), rounded up
          memory: 600Mi    # max(p99 410Mi, startup 480Mi) x 1.2, rounded
        limits:
          memory: 600Mi    # equal to request: OOMKill at a known size
          # CPU limit intentionally left unset here; see next section.
    - name: log-shipper    # sidecar: its requests add to the pod total
      image: registry.example.com/log-shipper:2.3.0
      resources:
        requests:
          cpu: 50m
          memory: 64Mi
        limits:
          memory: 64Mi
  # Pod effective request = max(largest init container, sum of app containers)
  # = max(100m/128Mi, 300m/664Mi) = 300m CPU, 664Mi memory.
  # Compare that against node allocatable, not node capacity.
  initContainers:
    - name: migrate
      image: registry.example.com/orders-migrate:1.0.0
      resources:
        requests:
          cpu: 100m
          memory: 128Mi
        limits:
          memory: 128Mi

A pod spec that applies the illustrative estimate (250m CPU, 600Mi memory with limit equal to request), aligns a Go service's GOMEMLIMIT with the container limit, and shows how a sidecar and an init container affect the pod's effective request.

Should you set CPU limits at all?

CPU and memory limits fail in different ways, so decide on each separately.

Setting What you gain What it costs
No CPU limit Pod bursts into idle node CPU; requests still set its fair share under contention A runaway pod can consume all idle CPU, so neighbours can only count on their own requests
CPU limit Predictable ceiling per pod CFS throttling can raise p99 latency even when average usage is low
Memory limit = request Predictable; pod won't push the node into memory pressure No burst room; any spike above the limit is an OOMKill
Memory limit > request Absorbs short spikes Under node memory pressure, the kubelet evicts pods using more than their request, sometimes before the node recovers

CPU limits throttle in short windows. The kernel enforces quota per CFS period, which is usually 100 ms. Consider a pod with a 200m limit (illustrative estimate). It gets 20 ms of CPU time per period. Four busy threads use that up in about 5 ms, then wait about 95 ms. Averaged over a minute, usage looks like 20%, yet individual requests stall for close to 100 ms.

To check whether a limit is hurting you, compare the rate of container_cpu_cfs_throttled_periods_total with the rate of container_cpu_cfs_periods_total. The ratio is the fraction of scheduling periods in which the container hit its quota. A sustained ratio of a few percent on a latency-sensitive service is worth investigating, especially if spikes in the ratio line up with p99 regressions. If they do, raise the limit or remove it, and keep the CPU request accurate. Memory has no throttling equivalent, so for memory the choice comes down to how much eviction risk you will accept.

How should you use VPA recommendations, and what are the alternatives?

Running the Vertical Pod Autoscaler with updateMode: "Off" gives you its recommender without letting it touch pods. It watches usage history and writes suggestions to the VPA object's status.recommendation, which you can read with kubectl describe vpa <name>. Each container gets a target (what VPA would set as the request), a lowerBound and an upperBound. A wide gap between the bounds usually means the recommender has little history or the workload is bursty, so treat the target with more suspicion.

These numbers are inputs for review, not settings to apply directly. They lag behind change: after a release that doubles memory use, the target catches up over days, not minutes. They also reflect what the workload did, not what it should do. A memory leak or a traffic lull ends up in the recommendation. The auto modes apply changes by evicting and recreating pods, which is a problem for workloads with slow startup or weak disruption budgets.

Approach Engineering time Risk
Manual percentile queries High, repeated per service Low if reviewed, but goes stale
VPA, Off mode Low to run, moderate to review Low; nothing changes without a person
VPA, auto modes Lowest ongoing Unplanned restarts, chasing anomalies
Fixed conservative sizing Near zero Wasted capacity, hidden OOMKilled risk if guessed low

Do not let VPA and HPA act on the same CPU or memory metric. VPA raises the request, HPA's utilization percentage drops, and the two fight.

How do you keep right-sizing ongoing alongside HPA without wasting engineering time?

HPA's CPU and memory utilization targets are a percentage of the pod's request, not of node capacity. When you change the request, you change the point at which HPA scales, even if the HPA object stays the same.

Illustrative estimate (assumed numbers, not a benchmark): scale-out point per pod = target utilization × request. With a 70% target and a 500m request, HPA adds replicas at 0.70 × 500m = 350m per pod. If you cut the request to 300m, scale-out happens at 210m. Replica counts rise and the CPU you saved is partly spent on extra pods. Review the request and the HPA target in the same change.

Run right-sizing as a loop:

Metrics → Review (VPA status, p95/p99 percentile queries) → Gradual rollout (one deployment or canary first) → Watch (OOMKilled counts, CPU throttling, p99 latency) → back to Metrics

Review on a fixed schedule, such as monthly, and after major releases. A new release can change memory footprint or CPU per request, so earlier data stops being representative.

Set namespace guardrails so new workloads never run as BestEffort. A LimitRange fills in default requests and limits for containers that declare none. A ResourceQuota on requests.cpu and requests.memory caps the namespace total and rejects pods that end up with no requests. Together they ensure every new pod has a request you can tune later.

The loop has a cost. Each pass takes engineering time, and each change can cause an OOMKilled event or a latency regression. Weigh that against the node capacity you reclaim and the incidents you avoid. Focus on the largest workloads by total requested resources and leave small, stable services on a slower review cycle.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes resource requests sizing, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.