Kubernetes CPU Limits Throttling: Why Backend API Latency Spikes

What do Kubernetes CPU requests and limits actually do in the kernel?

A Kubernetes CPU or memory value does two jobs. The scheduler reads it when choosing a node, and the kubelet, through the container runtime, translates it into cgroup settings that the Linux kernel enforces. Requests and limits map to different kernel mechanisms, and those mechanisms behave very differently under load.

CPU requests have two effects. The scheduler places a pod only on a node whose unreserved allocatable CPU covers the request. The kubelet also converts the request into a relative cgroup weight: cpu.shares on cgroup v1, cpu.weight on v2. Weight only matters when the CPU is contended. If the node has idle cores, a container with a small request can use as much CPU as it wants. When the CPU is saturated, the scheduler divides time in proportion to weight.

CPU limits become a CFS bandwidth quota: a fixed amount of CPU time allowed per scheduling period, which defaults to 100 ms. Once a container's threads use up that quota, the kernel stops scheduling them until the next period begins. This happens even if every other core on the node is idle. That is the source of throttling.

Memory has no equivalent to throttling. A memory request affects scheduling and eviction ranking. A memory limit sets a hard cgroup ceiling, and a container that exceeds it is OOM-killed.

CopyapiVersion: v1
kind: Pod
metadata:
  name: orders-api
spec:
  containers:
    - name: api
      image: registry.example.com/orders-api:1.0.0
      resources:
        requests:
          # Scheduler: only nodes with >= 250m unreserved allocatable CPU qualify.
          # Kernel: relative weight, used only when CPU is contended.
          #   cgroup v1: cpu.shares = 250 * 1024 / 1000 = 256
          #   cgroup v2: cpu.weight (converted from the shares value by the runtime)
          cpu: "250m"
          # Scheduler and eviction ranking only; no cgroup ceiling from this value.
          memory: "128Mi"
        limits:
          # Kernel: CFS bandwidth quota, enforced even when other cores are idle.
          #   cgroup v1: cpu.cfs_quota_us = 50000, cpu.cfs_period_us = 100000
          #   cgroup v2: cpu.max = "50000 100000"
          # The container gets 50 ms of CPU time per 100 ms period across all its
          # threads. Four busy threads can use that up in about 12.5 ms of wall
          # time, and then they wait roughly 87.5 ms for the next period.
          cpu: "500m"
          # Kernel: hard ceiling. Exceeding it triggers an OOM kill, not throttling.
          #   cgroup v1: memory.limit_in_bytes = 268435456
          #   cgroup v2: memory.max = 268435456
          memory: "256Mi"

A pod spec annotated with the cgroup settings that each CPU and memory request and limit becomes, including how a 500m CPU limit turns into a 50 ms per 100 ms CFS quota that can stall a multithreaded process.

How does it work end to end, from pod spec to a slow request?

The path from a YAML value to a latency spike has six steps. Each later section of this article covers one or more of them in detail.

1. You declare resources. The pod spec sets CPU requests and, optionally, CPU limits for each container.
2. The scheduler places the pod. It considers only requests. Limits play no part in placement.
3. The kubelet and runtime write cgroup files. The request becomes a weight (cpu.shares or cpu.weight). The limit becomes a quota and period (cpu.cfs_quota_us and cpu.cfs_period_us on v1, cpu.max on v2).
4. The kernel charges CPU time against the quota. In every period, CPU time used by any thread in the cgroup is deducted from the shared budget. When the budget reaches zero, the kernel stops running all of the cgroup's threads until the period ends and the quota refills.
5. In-flight requests wait. A request whose handler thread was stopped finishes later by up to the remaining length of the period. The work it does is unchanged; only the wall time grows. That extra time lands in p99 and p999 latency, not in the average.
6. The kernel records it. cpu.stat counts the periods, the throttled periods and the stalled time. cAdvisor exports these counters, and that is how you detect and confirm throttling.

The key point is step 4. The quota is a time budget per period, shared by every thread in the container, and it is enforced whether or not the node has idle CPU.

How does CFS quota throttling work, and why do averages hide it?

The kernel enforces a CPU limit as a budget per scheduling period. With the default 100ms period, a 1 CPU limit becomes 100ms of CPU time that the container's threads can spend in each window. The budget counts time across all threads, not wall-clock time. Once the threads spend it, the scheduler stops running every thread in the cgroup until the next period begins.

Illustrative estimate (assumed inputs, not a measurement): a pod has a 1 CPU limit, and four threads become runnable together because a burst of requests arrives. On a node with spare cores, all four run in parallel and use up the 100ms budget in about 25ms of wall time. The container then stays throttled for about 75ms. Any request in flight during that gap has roughly 75ms added to its latency, even though it was doing no extra work.

This is why utilization graphs can be misleading. A per-minute average that shows 30–40% of the limit spreads CPU use evenly across 600 periods. Real backend workloads do not use CPU evenly. Several kinds of work pack CPU demand into short windows, and each one can drain a single period's budget while the average stays low:

• Bursty request handling. A traffic spike makes many handler threads runnable at once.
• Garbage collection pauses. Collectors often run with many threads in parallel.
• JIT compilation. The compiler runs on its own threads alongside application work.
• Startup work. Class loading, cache warming and connection setup all happen at the same time.

Raising the limit makes the per-period budget larger, so fewer bursts exhaust it. It does not change how the mechanism works. Any burst whose parallel demand is larger than the new budget is still throttled within that period.

Copy+----------------------------------------------------------------------+
| Illustrative estimate (assumed inputs): 1 CPU limit, 100ms period    |
| quota = 100ms of CPU time per period, summed across all threads      |
+----------------------------------------------------------------------+

            0ms       25ms                          100ms
  wall time |---------|-----------------------------|---> next period
  thread 1  [########]..............................|[##
  thread 2  [########]..............................|[##
  thread 3  [########]..............................|[##
  thread 4  [########]..............................|[##
            +--------++---- throttled, ~75ms ------+|
                v                   v               v
  4 threads x 25ms         cgroup gets no CPU   quota refills
  = 100ms budget used      until period ends    next period
                                    |
                                    v
+----------------------------------------------------------------------+
| in-flight request on any thread: ~75ms added latency, no extra work  |
+----------------------------------------------------------------------+

+-------------------------+      +-------------------------------------------+
| per-minute average:     |      | one 100ms period at a time:               |
| 30-40% of the limit     | hides| most idle, some spend the whole quota     |
| graph looks safe        |----->| bursts, GC, JIT, startup -> throttled     |
+-------------------------+      +-------------------------------------------+
  Higher limit = bigger budget per period; bursts above it still throttle.

Illustrative estimate: with a 1 CPU limit, four parallel threads spend the 100ms quota in about 25ms and the cgroup is throttled for roughly 75ms, a stall that a per-minute average of 30-40% does not show.

How do you detect CPU throttling and tie it to API latency?

The kernel already counts throttling. In cgroup v2, read cpu.stat in the container's cgroup directory. Three fields matter: nr_periods is the number of enforcement periods in which the cgroup had runnable work, nr_throttled is how many of those periods ended with the quota exhausted, and throttled_usec is the total time its threads spent stalled. In cgroup v1 the same file lives under the cpu controller and reports nr_periods, nr_throttled, and throttled_time. Note that throttled_time is in nanoseconds, not microseconds.

You rarely read these files directly. cAdvisor, through the kubelet, exports them as counters: container_cpu_cfs_periods_total and container_cpu_cfs_throttled_periods_total. To get the throttled-period ratio, take the rate of throttled periods over a window and divide it by the rate of total periods. A value of 0.25 means a quarter of active periods hit the ceiling. Raw counters only ever increase, so always work with rates.

Plot that ratio next to p99 latency, not next to average CPU usage. Average usage can sit well below the limit while short bursts get throttled. Each throttled request waits for the next period, and that wait shows up in the tail.

Be careful when you interpret the metric:

• Throttling is not always harmful. A batch worker or log shipper can be throttled heavily without any user noticing. It matters for latency-sensitive paths.
• Correlation is not cause. Tail latency also comes from noisy neighbors on the node, GC pauses, slow disk or network I/O, and slow downstream calls. If p99 rises while the throttled ratio stays flat, look at those first. This check saves you from tuning limits that were never the problem.

Copy#!/usr/bin/env bash
set -euo pipefail
NS=shop; POD=api-7d9f8c6b5-x2kqp; CTR=api; INTERVAL=30

read_stat() {
  kubectl exec -n "$NS" "$POD" -c "$CTR" -- cat /sys/fs/cgroup/cpu.stat
}
field() { awk -v k="$1" '$1==k {print $2}'; }

S1=$(read_stat); sleep "$INTERVAL"; S2=$(read_stat)

P=$(( $(field nr_periods <<<"$S2") - $(field nr_periods <<<"$S1") ))
T=$(( $(field nr_throttled <<<"$S2") - $(field nr_throttled <<<"$S1") ))
U=$(( $(field throttled_usec <<<"$S2") - $(field throttled_usec <<<"$S1") ))

echo "periods=$P throttled=$T stalled_ms=$(( U / 1000 ))"
[ "$P" -gt 0 ] && awk -v t="$T" -v p="$P" 'BEGIN {printf "throttled ratio: %.2f\n", t/p}'

A shell script that computes the throttled-period ratio from two cgroup v2 cpu.stat samples, alongside the equivalent PromQL ratio from cAdvisor counters and a p99 latency query to plot next to it.

Why do language runtimes and thread pools make throttling worse?

Once you have confirmed that throttling is a cause, the runtime is often what multiplies it. CFS quota restricts CPU time, not which cores a process can see. Inside a container, the CPU count read from the host or from the process's affinity mask is usually the node's full core count. Runtimes that size garbage-collector threads, scheduler threads or default worker pools from that number will start far more parallel work than the quota can pay for.

Illustrative estimate (assumed numbers, not a benchmark): assume a 2-CPU limit with a 100 ms period, so the quota is 200 ms of CPU time per period. Assume a 32-core node and a runtime that runs 32 busy threads.

• Time until the quota runs out = quota ÷ runnable threads = 200 ms ÷ 32 ≈ 6.25 ms.
• Throttled time = period − that = 100 − 6.25 ≈ 94 ms.

With 2 threads, the same quota lasts the whole period. Total CPU work is equal in both cases, but the 32-thread version stalls every in-flight request for most of each period. A parallel GC cycle that wakes many threads has the same effect.

To check, log the count the runtime itself reports at startup, such as Runtime.availableProcessors() in Java, runtime.GOMAXPROCS(0) in Go, or os.cpu_count() in Python. Compare it with the container's limit. Whether a runtime reads cgroup quotas depends on the runtime and its version, so verify this in your own image.

If the number is wrong, set worker, thread-pool and GC thread counts explicitly through configuration. The Kubernetes Downward API can expose limits.cpu as an environment variable, so the setting follows the pod spec and does not drift from it.

CopyapiVersion: v1
kind: Pod
metadata:
  name: orders-api
spec:
  containers:
    - name: api
      image: registry.example.com/orders-api:1.0.0
      resources:
        requests:
          cpu: "2"
        limits:
          cpu: "2"   # quota: 200ms of CPU time per 100ms period
      env:
        # Downward API: expose the container's own CPU limit as an env var.
        # With divisor "1" the value is a whole number of cores, rounded up,
        # so a fractional limit like 1500m becomes 2.
        - name: CPU_LIMIT
          valueFrom:
            resourceFieldRef:
              resource: limits.cpu
              divisor: "1"
        # Go reads GOMAXPROCS at startup. Setting it from the limit stops the
        # scheduler from running one OS thread per node core.
        - name: GOMAXPROCS
          valueFrom:
            resourceFieldRef:
              resource: limits.cpu
              divisor: "1"
        # Application-defined setting (placeholder name). Your app code must
        # read it and size its worker pool from it instead of the host core count.
        - name: APP_WORKER_THREADS
          valueFrom:
            resourceFieldRef:
              resource: limits.cpu
              divisor: "1"

A pod spec that uses the Downward API to pass the container's CPU limit to the Go scheduler and to an application-defined worker-pool setting. Thread counts then follow the pod spec instead of the node's core count.

How do QoS classes change eviction, OOM handling and throttling?

Kubernetes assigns every pod one of three QoS classes. It derives the class from the requests and limits you set, and you cannot choose it directly. The class affects the order in which pods lose resources when a node comes under pressure.

The kubelet ranks pods for eviction by usage relative to requests and by pod priority, not by QoS class itself. In practice the two orderings line up closely.

Guaranteed QoS has a cost. To get it, you must set a CPU limit equal to the CPU request, and that limit becomes a CFS quota. A Burstable API pod that occasionally used idle cores during a traffic spike now gets throttled at exactly its request.

With the CPU Manager's static policy, Guaranteed pods that request whole CPUs get exclusive cores. That removes interference from neighboring workloads. Depending on the node configuration, the quota may still apply on top of the pinning.

A practical rule: use Guaranteed only when you need exclusive cores or the strongest protection from eviction and OOM kills. Size the CPU for peak load, not the average. If you only want memory protection, Burstable with memory requests equal to memory limits and no CPU limit is usually the better fit.

When should you set CPU limits on backend APIs, and when should you avoid them?

Start with requests, because the scheduler and the kernel's CPU weighting rely on them. Set CPU requests from observed usage at a meaningful percentile, such as p95 under realistic load, and include warm-up and garbage-collection phases. Over-requesting wastes capacity: nodes look full to the scheduler while their CPUs sit idle, and you pay for reservations that nothing uses. Under-requesting leads to contention: the scheduler packs too many pods onto a node, and when they all get busy, CPU is divided by weight, so each pod gets less than it needs.

Once requests are honest, decide on limits based on what the workload needs:

Removing limits has costs:

• A runaway pod can consume all idle headroom on a node. Your requests are then the only guarantee its neighbours have.
• Load tests on quiet nodes borrow spare CPU, so they can look better than production will.
• Capacity planning must be based on requests, because actual usage can exceed them whenever the node has spare CPU.
• LimitRange defaults, admission policies or chargeback rules may inject or require CPU limits. Check them before assuming a pod runs unlimited.

Who should use this approach?

The detection and tuning steps in this article are most useful for these groups:

• Teams running latency-sensitive backend APIs on Kubernetes with CPU limits set. This includes limits that were inherited from a LimitRange default or a Helm chart that nobody revisited. If your service has a p99 or p999 latency objective, throttling is a likely contributor to tail spikes.
• SREs and on-call engineers triaging tail-latency incidents. The throttled-period ratio is a quick first check that either confirms CPU limits as the cause or rules them out.
• Platform engineers who own cluster policy. LimitRange defaults, admission rules and QoS conventions decide whether every workload in a namespace gets a CFS quota. Those choices affect every team that deploys there.
• Teams running multithreaded runtimes on large nodes. JVM, Go and other runtimes that size thread pools from the visible core count are the most exposed to the effect described in the runtime section.

It matters less for batch jobs, queue consumers without latency targets, and low-traffic single-threaded services that rarely have more than one runnable thread. Those workloads can be throttled without user-visible harm. Teams on multi-tenant clusters that must keep CPU limits for isolation or chargeback should still use the detection steps, so that they size limits for peak parallel demand rather than average usage.

What are the alternatives to tight CPU limits, and how do you roll out changes safely?

Removing or loosening limits only works if something else keeps capacity predictable. The options below do that at different layers.

Accurate requests matter most. They set each container's share when the node is contended, so a well-sized request protects latency without a hard cap. Autoscaling on request rate or in-flight requests reacts to load before saturation. CPU averages lag the bursts that cause throttling. The static CPU manager policy gives latency-critical pods dedicated cores. It also reduces the node's shared pool, so reserve it for workloads that justify it. Longer CFS periods or kernel burst allowances may help in some environments. Their availability depends on your kernel and kubelet configuration, so verify before you rely on them.

Rolling out safely

1. Baseline the throttling ratio and p99 latency per deployment.
2. Change limits on a subset of replicas, such as a canary or a single node pool.
3. Compare tail latency against the unchanged replicas. Also compare node-level CPU pressure.
4. Watch co-located pods for rising latency or throttling. Your gain should not become their starvation.
5. Expand gradually.

Measuring before you change anything ties latency incidents to a known cause. That cuts triage time. It also stops teams from using extra replicas to mask throttling.

How can this save engineering time?

Throttling-driven latency is expensive to debug because it looks like many other problems. The code is not slower, the average CPU graph looks healthy, and the spikes come and go with traffic bursts. Engineers often profile application code, blame downstream services or tune garbage collection before anyone checks cpu.stat. The practices in this article cut that time in several places:

• Faster incident triage. A dashboard panel that shows the throttled-period ratio next to p99 latency lets the first responder rule CPU limits in or out early. A flat ratio sends them to GC, I/O, node contention or downstream calls straight away. A rising ratio points to limits or thread counts.
• Fewer misdirected optimizations. Engineers do not spend days optimizing code paths whose latency comes from waiting for the next CFS period, not from doing extra work.
• Less trial-and-error tuning. Requests sized from observed percentiles, and limits chosen deliberately or omitted, replace repeated cycles of raising a limit, redeploying and watching graphs.
• No per-service thread-count drift. Passing limits.cpu through the Downward API means worker pools and GOMAXPROCS follow the pod spec automatically. Nobody has to remember to update a hard-coded value when a limit changes.
• Safer changes with less rework. The canary rollout compares changed and unchanged replicas directly, so a bad limit change is caught on a subset of traffic, not after a full rollout and a rollback.

Illustrative estimate (assumed inputs, not a measurement): assume a team handles 4 tail-latency investigations a month. Assume each one takes 3 engineer-hours of profiling and log review before anyone checks throttling, and that throttling turns out to be the cause in half of them. With the ratio already graphed beside p99, assume the throttling check takes about 15 minutes. Under these assumptions, the 2 throttling-caused investigations drop from about 6 hours to about 30 minutes in total, and the other 2 start on the right path sooner. Your own numbers will differ. Track time-to-diagnosis for your incidents before and after you add the dashboard to see the real effect.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes cpu limits throttling, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.