Kubernetes Requests and Limits: Avoiding CPU Throttling and OOMKills

What requests and limits actually control

Every container in a pod can declare two numbers per resource: a request and a limit. They are enforced by different components at different times. If you run workloads on a shared cluster, as a backend engineer shipping services or a platform engineer keeping nodes healthy, you need to know which component reads which number.

Requests are read by the scheduler. When placing a pod, the scheduler adds the pod's requests to the requests of pods already on each candidate node and compares the total against that node's allocatable capacity. It does not consider limits or actual usage. A pod requesting cpu: 250m and memory: 256Mi reserves that much on paper, even if it sits idle. On the node, the CPU request also becomes the container's cgroup CPU weight, which sets its share of CPU when the node is contended.

Limits are read by the kernel. The kubelet and container runtime write limits into the container's cgroup. A cpu: 500m limit becomes a CFS bandwidth quota. A memory: 512Mi limit becomes a hard memory ceiling. The kernel enforces both while the container runs, and the scheduler never sees them.

This split explains why a node can be overcommitted on limits but never on requests. The scheduler guarantees that the sum of requests fits on the node. Nothing checks the sum of limits, so ten pods with 1 GiB limits can land on an 8 GiB node. Teams set both values to get two separate outcomes: requests give predictable placement, and limits cap how much damage a single misbehaving container can do once it is running.

CopyapiVersion: v1
kind: Pod
metadata:
  name: orders-api
spec:
  containers:
    - name: app
      image: registry.example.com/orders-api:1.4.2
      resources:
        # Requests: read by the scheduler at placement time.
        # The pod only lands on a node whose allocatable capacity
        # still has 250m CPU and 256Mi memory unreserved.
        # The CPU request also sets the container's cgroup CPU weight
        # (its share of CPU when the node is contended).
        requests:
          cpu: "250m"
          memory: "256Mi"
        # Limits: ignored by the scheduler, written into the cgroup
        # by the kubelet/runtime and enforced by the kernel at runtime.
        # cpu -> CFS bandwidth quota (excess usage is throttled)
        # memory -> hard ceiling (exceeding it triggers an OOM kill)
        limits:
          cpu: "500m"
          memory: "512Mi"

A minimal container resources spec in which the scheduler uses the requests to place the pod and the kernel enforces the limits through the container's cgroup.

Why CPU and memory fail differently

That damage looks very different depending on the resource. CPU is compressible: the kernel can hand out less of it and the process simply runs slower. Memory is non-compressible: pages already allocated cannot be taken back without reclaiming or killing something, so the kernel has to choose a victim.

A CPU limit is enforced through the Completely Fair Scheduler's bandwidth control. The kubelet converts the limit into a quota of CPU time per scheduling period, and the period is 100 ms by default. A limit of 500m becomes 50 ms of CPU time every 100 ms, shared across all threads in the container. The quota is not per thread, and this is where the trouble starts.

Consider a service that runs four busy threads during a request burst. Together they consume the 50 ms quota in about 12.5 ms of wall-clock time. The container is then throttled for the remaining ~87.5 ms of the period, and every in-flight request waits. Averaged over a minute, CPU usage may sit well under the limit, while p99 latency jumps by tens of milliseconds.

Memory works differently. When a container's usage reaches its limit and the kernel cannot reclaim enough, the cgroup OOM killer sends SIGKILL. Kubernetes reports the termination reason as OOMKilled with exit code 137 (128 + signal 9).

QoS classes, eviction order and the two kinds of memory kill

Kubernetes derives a pod's QoS class from its containers' requests and limits. You do not set it directly:

• Guaranteed: every container sets CPU and memory requests and limits, and each request equals its limit.
• BestEffort: no container sets any request or limit.
• Burstable: every other pod.

The class matters most when a node runs short of memory. The kubelet ranks pods for eviction by whether their usage exceeds their requests, then by pod priority, then by how far usage exceeds requests. BestEffort pods have no requests, so any usage puts them over, and they go first. Guaranteed pods cannot exceed their requests, so they go last.

The kubelet also sets oom_score_adj so the kernel's choice of victim follows the same order. Guaranteed pods get a score near the minimum and BestEffort pods get the maximum. Burstable pods fall in between: the larger the memory request relative to node capacity, the lower the score.

Memory problems therefore end in one of two kinds of kill, and they need different fixes.

kubectl get events shows Evicted events from the kubelet. One exception: if the node runs out of memory before the kubelet reacts, the kernel can kill a container that is under its limit. That container also shows OOMKilled, so check its usage against its limit before you raise the limit.

Copy#!/usr/bin/env bash
# Triage a memory kill: was it the container's own limit, or node pressure?
NS=shop
POD=checkout-7d9f8c6b5-x2kqp

# 1. QoS class derived from requests/limits (Guaranteed, Burstable, BestEffort)
kubectl get pod "$POD" -n "$NS" -o jsonpath='{.status.qosClass}{"\n"}'

# 2. Container OOMKill: the pod is still running, but a container's last
#    state is Terminated with reason OOMKilled and exit code 137.
kubectl get pod "$POD" -n "$NS" -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.restartCount}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"\n"}{end}'

# 3. Node-pressure eviction: the pod itself has phase Failed, reason Evicted,
#    and a message naming the resource the node ran low on.
kubectl get pod "$POD" -n "$NS" -o jsonpath='{.status.phase}{"\t"}{.status.reason}{"\t"}{.status.message}{"\n"}'

# 4. The same information in human-readable form (Last State, Reason, Exit Code)
kubectl describe pod "$POD" -n "$NS"

# 5. Kubelet eviction events across the namespace. A container OOMKill does not
#    appear here, which is one way to separate the two cases.
kubectl get events -n "$NS" --field-selector reason=Evicted

# 6. All events for this pod, oldest first
kubectl get events -n "$NS" \
  --field-selector involvedObject.name="$POD" \
  --sort-by=.lastTimestamp

# 7. If you see OOMKilled but usage was under the limit, check whether the node
#    itself was under memory pressure around that time.
NODE=$(kubectl get pod "$POD" -n "$NS" -o jsonpath='{.spec.nodeName}')
kubectl describe node "$NODE" | grep -iE 'MemoryPressure|Allocated resources' -A 6

A short kubectl triage script that reads the QoS class, the container last state and the pod status, then checks Evicted events and node conditions to tell a container OOMKill apart from a kubelet node-pressure eviction.

How to detect CPU throttling and memory risk

CPU throttling rarely shows up on utilization dashboards. The CFS quota is enforced in short periods, 100 ms by default. A container with a 1-core limit can use its full 100 ms of CPU time in the first 25 ms of a period if it runs four busy threads, and then it waits 75 ms. Averaged over a one-minute scrape window, that container looks like it is at 40% of its limit, while individual requests are stalling in every period. Latency-sensitive services hit this first: p99 rises while the CPU graph stays flat.

Measure throttling directly. cAdvisor exposes container_cpu_cfs_periods_total, container_cpu_cfs_throttled_periods_total and container_cpu_cfs_throttled_seconds_total. A useful ratio is the fraction of enforcement periods in which the container hit its quota:

rate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m])

A sustained value well above zero means the limit is shaping latency. Throttled seconds tell you how much wall time was lost. Treat both with care. The values come from different cgroup files under v1 and v2, exporters have handled the change differently, and short-lived or idle containers produce noisy ratios. Confirm which series your monitoring stack actually exposes, and what they mean, before you write alerts.

For memory, alert on container_memory_working_set_bytes, not RSS. RSS leaves out page cache and tmpfs, but the cgroup still charges both. Working set is usage minus inactive file cache, which approximates memory the kernel cannot easily reclaim. It is also the figure the kubelet uses for eviction decisions. Compare it against the container's limit.

Copy# Prometheus alerting rules using cAdvisor metrics scraped from the kubelet.
# Thresholds are starting points. Verify the series exist in your stack first.
groups:
  - name: container-resources
    rules:
      # Fraction of CFS enforcement periods in which the container hit its quota.
      - alert: ContainerCPUThrottlingHigh
        expr: |
          (
            sum by (namespace, pod, container) (rate(container_cpu_cfs_throttled_periods_total{container!="", container!="POD"}[5m]))
              /
            sum by (namespace, pod, container) (rate(container_cpu_cfs_periods_total{container!="", container!="POD"}[5m]))
          ) > 0.25
          # Skip mostly idle containers. With the default 100 ms period, a busy
          # container accrues about 10 periods/s. Fewer than 1/s gives noisy ratios.
          and on (namespace, pod, container)
          sum by (namespace, pod, container) (rate(container_cpu_cfs_periods_total{container!="", container!="POD"}[5m])) > 1
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) throttled in {{ $value | humanizePercentage }} of CFS periods"

      # Working set (usage minus inactive file cache) compared with the memory limit.
      - alert: ContainerMemoryNearLimit
        expr: |
          max by (namespace, pod, container) (container_memory_working_set_bytes{container!="", container!="POD"})
            /
          # A limit of 0 means no limit is set. Exclude those series.
          max by (namespace, pod, container) (container_spec_memory_limit_bytes{container!="", container!="POD"} > 0)
            > 0.9
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.namespace }}/{{ $labels.pod }} ({{ $labels.container }}) working set at {{ $value | humanizePercentage }} of memory limit"

These Prometheus alerting rules alert on the share of CFS periods in which a container was throttled, while skipping idle containers that produce noisy ratios. They also alert when a container's working set approaches its memory limit.

Sizing requests and limits from observed data

Collect at least a full weekly cycle of data before sizing anything. A single day often misses weekly batch jobs, Monday traffic, or a cache warm-up after a deploy. Longer windows also have a cost: they keep old behaviour from before a recent release. Re-check after any significant code change.

Each resource should be sized from the statistic that matches how it fails:

• Memory request: set it near the typical working set (p50 to p90). The scheduler reserves this amount, so it should describe steady state.
• Memory limit: set it above the observed peak (max or p99.9) with headroom. Crossing the limit kills the container, so an average is the wrong input here.
• CPU request: set it from sustained usage, such as p90 of 5-minute rates. The request determines the container's share of CPU when the node is contended.
• CPU limit: on dedicated or trusted multi-team nodes, consider omitting it. Set one only where you need to stop a tenant from consuming spare cycles, and place it well above p99.

Worked example (illustrative estimate). Assume seven days of data for one service:

• Working set: p50 410 MiB, max 620 MiB
• CPU: mean 0.2 cores, p90 0.35 cores, p99 0.9 cores

From these numbers: memory request 450Mi, memory limit about 768Mi (620 × 1.25, rounded up), CPU request 400m, and no CPU limit on a single-tenant pool. If you sized CPU from the mean, the request would be half of real sustained demand.

Vertical autoscaler recommendations are another useful input, but treat them as a second opinion. They are computed from history, so they lag behind sudden load changes. Depending on the update mode and whether in-place resize is available in your cluster, applying them may restart pods.

Copy# Values from the illustrative estimate (7 days of data, single-tenant pool)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders-api
  template:
    metadata:
      labels:
        app: orders-api
    spec:
      containers:
        - name: orders-api
          image: registry.example.com/orders-api:1.0.0
          resources:
            requests:
              memory: 450Mi   # near typical working set (p50 410 MiB)
              cpu: 400m       # sustained usage (p90 0.35 cores), not the 0.2-core mean
            limits:
              memory: 768Mi   # observed max 620 MiB x 1.25 headroom, rounded up
              # No CPU limit: dedicated pool, so let bursts to p99 use spare cycles
---
# Recommendation-only VPA: a second opinion that never evicts or resizes pods
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: orders-api
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: orders-api
  updatePolicy:
    updateMode: "Off"   # read recommendations with: kubectl describe vpa orders-api

A Deployment sized from the illustrative estimate's numbers, with a recommendation-only VerticalPodAutoscaler you can compare against without restarting pods.

Runtime-specific memory pitfalls

Every major runtime manages its own heap, and none of them knows about your pod spec unless it reads the cgroup or you tell it. The rule is to cap the heap below the container limit, leaving room for memory the heap setting does not cover.

• JVM: Prefer -XX:MaxRAMPercentage over a fixed -Xmx, so the heap scales with the limit. The heap is not the whole process. Metaspace, thread stacks, direct buffers and JIT code all sit outside it.
• Go: GOMEMLIMIT sets a soft limit that makes the GC work harder as total runtime memory approaches it. Memory allocated through cgo is not counted. GOGC still controls collection frequency below that limit.
• Node.js: --max-old-space-size caps the V8 old generation. Buffer and native addon memory live outside it.

The kernel also charges non-runtime memory to the container's memory cgroup. File reads fill the page cache, and the kernel usually reclaims that cache under pressure. A emptyDir with medium: Memory is tmpfs, and files written there count against the limit and are not reclaimed until deleted.

Illustrative estimate (assumed numbers, not a benchmark). Use this formula: max heap = limit − non-heap overhead − headroom.

• Assumed inputs: limit 2048 MiB, measured non-heap 400 MiB, headroom 200 MiB.
• Result: 2048 − 400 − 200 = 1448 MiB, or about 70% of the limit.

There is a trade-off. A tighter heap means more frequent GC and some lost throughput, but it improves stability. If you set the heap too high, the container gets OOMKilled. If you set it too low, the pod reserves memory it never uses, and you see GC thrashing.

When to avoid CPU limits or Guaranteed QoS, and the alternatives

None of the three common configurations is right everywhere. Each one trades risk in one place for risk in another.

Dropping CPU limits removes CFS quota enforcement, so a latency-sensitive service can no longer be throttled partway through a request while the node has idle cores. The cost is isolation. Under contention, the kernel divides CPU in proportion to each container's request, translated into cgroup CPU weights. If one team's pods request 100m but routinely use two cores, they take capacity that correctly sized pods were counting on. Remove limits only where requests are accurate and reviewed.

Guaranteed QoS requires requests equal to limits for CPU and memory in every container. These pods are the last candidates for node-pressure eviction, and they are required for exclusive CPU pinning under the static CPU manager policy. They cannot burst, though. You size for peak, and the gap between peak and typical usage sits reserved and idle.

Memory limits above requests let the scheduler pack more pods per node, because it only counts requests. If several pods spike at once, total usage can exceed what the node has. The kubelet then evicts pods, starting with those furthest above their memory requests.

Namespace guardrails and impact on cost and engineering time

LimitRange and ResourceQuota

Two namespace-level objects enforce sizing policy. A LimitRange is applied at admission time. It fills in defaultRequest and default (limit) values for any container that omits them, and it can reject containers outside its min/max bounds. A ResourceQuota caps the sum of requests and limits across all pods in the namespace, such as requests.cpu or limits.memory. Once a quota covers a resource, every new pod must declare it or inherit it from a LimitRange, or the pod is rejected.

The example below sets a default CPU request and limit, a default memory request and limit, and a memory ceiling per container.

That interaction is a common source of surprises. A team removes CPU limits from its Deployment, following the reasoning in the previous section, but the namespace's LimitRange quietly injects a default CPU limit, and throttling returns. Or a JVM service that never declared memory receives a small default limit and starts restarting with OOMKilled. Because defaults are applied only when a pod is created, the behavior changes on the next rollout rather than when someone edits the LimitRange. That gap makes the cause hard to trace. Running kubectl get pod -o yaml shows the effective values, which is the first thing to check.

Cost and engineering time

Oversized requests reserve node capacity that the scheduler cannot hand to anyone else, so you pay for idle cores. Undersized requests and limits shift the cost into incidents: latency spikes from throttling, restart loops from OOMKills, and the hours spent finding the cause.

Three practices reduce both costs:

• Measured sizing based on observed usage, so requests track what workloads actually consume.
• Throttling and memory-headroom alerts, so problems surface before users notice them.
• Namespace defaults set deliberately, with omitted CPU limits treated as a choice rather than a gap, so a LimitRange does not silently override that choice.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes requests and limits, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.