How do Kubernetes CPU requests and limits map to kernel mechanisms?
Kubernetes does not schedule CPU time itself. The kubelet and container runtime translate each container's resources into cgroup settings, and the Linux Completely Fair Scheduler (CFS) enforces them. Requests and limits map to two different kernel mechanisms that behave differently.
A CPU request becomes a relative weight: cpu.shares on cgroup v1 or cpu.weight on cgroup v2. The weight only matters when runnable threads compete for the same cores. On a busy node, a container with twice the weight gets roughly twice the CPU time. On an idle node, the weight has no effect, and the container can use any free cycles.
A CPU limit becomes a CFS bandwidth quota. This is an absolute cap on CPU time per scheduling period. The default period is 100 ms, so a 500m limit allows 50 ms of CPU time per 100 ms window, summed across all threads in the cgroup. Once the cgroup uses its quota, its threads cannot run until the next period starts. This happens even if the node has idle cores.
| Setting | cgroup v1 | cgroup v2 | Enforced when |
|---|---|---|---|
| CPU request | cpu.shares |
cpu.weight |
Only under contention |
| CPU limit | cpu.cfs_quota_us / cpu.cfs_period_us |
cpu.max (quota and period) |
Every period, regardless of idle CPU |
CPU is a compressible resource. A container that exceeds its limit is throttled, not killed, which is different from how memory limits work. Throttling does not appear as an error. It shows up as threads that pause for milliseconds while they hold a request. For backend and platform engineers who run latency-sensitive services, those pauses land directly in p99 latency.
How does CFS quota cause throttling below the CPU limit?
The kernel enforces a CPU limit through CFS bandwidth control, and it enforces it per period rather than as an average. With the default 100ms period, a limit of 1 CPU becomes a quota of 100ms of CPU time per 100ms window. That budget is shared by every thread in the container's cgroup. Once the threads together have consumed it, the scheduler stops running all of them until the next period begins.
Illustrative calculation
Assume a container with a 1 CPU limit receives a burst of work, and four threads run in parallel on four cores. Together they burn 4ms of CPU time per millisecond of wall-clock time, so they use up the 100ms quota in about 25ms. For the remaining ~75ms of the window, the container is throttled. A request that arrived at the start of the burst now takes at least 75ms longer than its CPU work requires, even though the container never used more than its limit.
Why averages hide it
Monitoring usually samples CPU usage over intervals of tens of seconds or minutes. Short bursts disappear in those averages. A container that averages 30% of its limit can still exhaust its quota in individual 100ms windows during request spikes, garbage collection, or JIT compilation, because all of these run many threads at the same moment. A per-minute dashboard shows plenty of headroom. The throttled windows show up in p99 latency.
Older kernels also had bugs in CFS bandwidth accounting that caused throttling beyond what the quota required. Verify how your kernel behaves rather than assuming it is correct.
+--------------------------------------------------------------+
| Container cgroup (limit: 1 CPU) |
| CFS quota = 100ms CPU time per 100ms period, ALL threads |
+--------------------------------------------------------------+
|
| one shared budget, drawn down by every thread
v
ONE 100ms PERIOD (illustrative calculation, 4 threads, 4 cores)
0ms 25ms 100ms
T1 |###############|............................................|
T2 |###############|............................................|
T3 |###############|............................................|
T4 |###############|............................................|
# = running: 4 threads x 25ms = 100ms quota used up
. = throttled: no thread runs for ~75ms
|
v
next period: quota refilled
+---------------------------+ +---------------------------------+
| Per-minute average | | Per-100ms windows |
| 30% of limit | | [ok][ok][FULL][ok][FULL][ok] |
| looks like headroom | | ^ bursts, GC, JIT |
+---------------------------+ +---------------------------------+
|
v
throttled windows -> p99 latency
Illustrative calculation: four parallel threads exhaust a 1 CPU limit's 100ms CFS quota in about 25ms, leaving the container throttled for roughly 75ms, a pattern that per-minute averages hide but p99 latency reveals.
How do you detect Kubernetes CPU throttling?
The kernel records every throttling event, so you don't have to infer it from latency graphs. Each container's cgroup has a cpu.stat file. On cgroup v2 it includes three counters that matter here. nr_periods is the number of enforcement periods in which the cgroup had runnable work. nr_throttled is how many of those periods ended with the quota exhausted. throttled_usec is the total time the cgroup's threads spent runnable but blocked by the quota. On cgroup v1 the same data is in the cpu controller's cpu.stat, but the time counter is throttled_time and it is measured in nanoseconds.
cAdvisor, which the kubelet embeds, exports these counters as container_cpu_cfs_periods_total, container_cpu_cfs_throttled_periods_total and container_cpu_cfs_throttled_seconds_total. The throttle ratio is the rate of throttled periods divided by the rate of total periods over the same window, grouped by pod and container. A ratio of 0.25 means one in four scheduling periods ended with the container's threads frozen.
These counters beat CPU utilization percent because utilization is an average over the scrape interval, usually tens of seconds. A container can average 40% of its limit and still use its whole quota in the first few milliseconds of many periods, then sit idle until the next one. The average hides that pattern. The throttle ratio tells you how often the container hits the quota. Throttled seconds tell you how much time it loses when it does. Compare both against your p99 latency. If they rise together, throttling is a likely cause of the latency. If they don't, look elsewhere.
#!/usr/bin/env bash
# Read raw CFS throttling counters for one container, then the cluster-wide ratio.
NS=checkout; POD=checkout-api-7d9f8c6b5-x2lqp; CTR=api # placeholder names
# cgroup v2: one unified file. v1: under the cpu controller (often cpu,cpuacct).
kubectl exec -n "$NS" "$POD" -c "$CTR" -- sh -c '
if [ -f /sys/fs/cgroup/cpu.stat ] && grep -q nr_periods /sys/fs/cgroup/cpu.stat; then
echo "cgroup v2 (throttled_usec = microseconds)"; cat /sys/fs/cgroup/cpu.stat
else
echo "cgroup v1 (throttled_time = nanoseconds)"; cat /sys/fs/cgroup/cpu/cpu.stat
fi'
# Look at: nr_periods, nr_throttled, throttled_usec | throttled_time.
# Run it twice a few seconds apart: the deltas matter, not the absolute totals.
PROM= # placeholder Prometheus address
# Throttle ratio: share of enforcement periods that ended with the quota exhausted.
RATIO='sum by (namespace, pod, container) (rate(container_cpu_cfs_throttled_periods_total{container!=""}[5m]))
/
sum by (namespace, pod, container) (rate(container_cpu_cfs_periods_total{container!=""}[5m]))'
# Throttled time: seconds of runnable-but-blocked time per wall-clock second.
STALL='sum by (namespace, pod, container) (rate(container_cpu_cfs_throttled_seconds_total{container!=""}[5m]))'
for q in "$RATIO" "$STALL"; do
curl -s "$PROM/api/v1/query" --data-urlencode "query=$q"; echo
done
# Plot both next to p99 latency. If they rise together, suspect the CPU limit.
Reads nr_periods, nr_throttled and throttled_usec (v2) or throttled_time (v1) from a container's cpu.stat, then queries Prometheus for the cAdvisor throttle ratio and throttled seconds per pod and container.
Why do thread pools, GC, GOMAXPROCS and JVM CPU detection make throttling worse?
Runtimes size their parallelism from the CPU count they detect, and that count often isn't your quota. A Go binary sets GOMAXPROCS from the CPUs visible to the process. Depending on the Go version, that can be every core on the node. The JVM sizes ParallelGCThreads, the common ForkJoinPool and many framework thread pools from its detected processor count. Container-aware JVMs derive that count from the limit, but they round fractional limits up, so a 1.5 CPU limit looks like 2 processors.
The CFS quota is measured in CPU time per period, not in threads. Extra runnable threads don't add capacity. They spend the budget faster and then all stall together. As an illustrative estimate, take a 4 CPU limit, which gives a 400 ms quota per 100 ms period, on a 64-core node where Go runs with GOMAXPROCS=64. A parallel GC phase that keeps 64 threads busy uses the whole quota in about 6 ms. Every thread in the container then waits roughly 94 ms for the next period, including the ones serving requests. That stall lands directly in p99 latency.
Match runtime parallelism to the limit. If there is no limit, match it to the request, because the request is the CPU share you can count on under contention. For Go, set a GOMAXPROCS environment variable from the limit. You can do this through the Downward API using resourceFieldRef with limits.cpu, which rounds up to a whole number. For the JVM, set -XX:ActiveProcessorCount explicitly, and size -XX:ParallelGCThreads and application thread pools to match it.
apiVersion: v1
kind: Pod
metadata:
name: api-example
spec:
containers:
- name: go-api
image: registry.example.com/go-api:1.0.0
resources:
requests:
cpu: "2"
limits:
cpu: "2"
env:
# Derive GOMAXPROCS from this container's CPU limit instead of the node's core count.
# With the default divisor of 1, a fractional limit is rounded up to a whole number.
- name: GOMAXPROCS
valueFrom:
resourceFieldRef:
resource: limits.cpu
- name: jvm-api
image: registry.example.com/jvm-api:1.0.0
resources:
requests:
cpu: "2"
limits:
cpu: "2"
env:
# Pin the processor count the JVM reports, then size GC threads
# and the common ForkJoinPool to that same number.
- name: JAVA_TOOL_OPTIONS
value: >-
-XX:ActiveProcessorCount=2
-XX:ParallelGCThreads=2
-Djava.util.concurrent.ForkJoinPool.common.parallelism=2
A pod spec that sets Go's GOMAXPROCS from the container's CPU limit through the Downward API, and pins the JVM's processor count, GC threads and common ForkJoinPool parallelism to the same 2-CPU limit.
When should you remove CPU limits, and when should you avoid it?
The argument for removal rests on what requests already do. A CPU request sets the cgroup's CFS weight, so when a node is contended, each pod gets CPU in proportion to its request. That share is protected without a limit. When the node has idle cycles, a pod without a limit can use them, and no one loses anything because nobody else wanted those cycles. A limit only adds a ceiling that applies even when the CPU it withholds would otherwise go unused. For latency-sensitive services with bursty request patterns, that ceiling is often the main source of throttling.
This only works if requests are accurate and the node has headroom.
| Situation | Remove limits? | Why |
|---|---|---|
| Requests match observed usage, nodes have spare CPU | Yes | Weight protects shares, and bursts use idle capacity |
| Requests are guesses or stale | No | Without a cap, an underestimated pod can take far more than it reserved |
| Shared multi-tenant cluster | Usually no | Bursts are hard to attribute to a tenant, and chargeback is usually based on limits |
ResourceQuota sets limits.cpu, or a LimitRange sets a max or default |
Not without a policy change | Pods without a limit are rejected, or the LimitRange injects a default limit |
| Node CPU already saturated | No | Throttling becomes run-queue wait, so the latency remains |
Two of these risks are easy to miss. First, latency becomes harder to predict. A pod can look fast on a quiet node and slow on a busy one. Load tests run on half-empty nodes can also hide a capacity shortfall until real traffic fills the cluster. Second, if total demand exceeds the node's cores, removing limits does not create CPU. The cost shifts from throttled periods to scheduler queueing, which also shows up in p99 latency.
What happens to QoS class and scheduling without CPU limits?
Dropping the CPU limit also changes the pod's QoS class. Guaranteed requires every container to set CPU and memory limits with requests equal to those limits. Remove the CPU limit and the pod becomes Burstable, even if its memory request and limit still match.
The kubelet does not evict pods for CPU pressure, because CPU is compressible: contention slows processes down but does not exhaust anything. Under memory or disk pressure, the kubelet first targets pods whose usage exceeds their requests, then sorts by priority and by how far usage exceeds the request. A Burstable pod that stays within its memory request is rarely chosen. The class also sets the kernel OOM score adjustment. Guaranteed pods get the most protective value, and Burstable pods get a value derived from their memory request. As a result, a Burstable pod is more likely to be killed when the node runs out of memory.
The larger cost applies to nodes running the static CPU manager policy. Exclusive cores are granted only to Guaranteed pods with integer CPU requests. A Burstable pod runs on the shared pool, so latency-sensitive workloads that depend on pinned cores should keep their limits.
Keep memory limits regardless of the CPU decision. Memory is not compressible. A container that exceeds its memory limit is OOM-killed instead of slowed down. Without a memory limit, one leaking pod can push the node into memory pressure and get its neighbours evicted.
| Spec | QoS class | Exclusive cores (static policy) | OOM kill exposure |
|---|---|---|---|
| CPU and memory requests equal to limits | Guaranteed | Eligible with integer CPU | Lowest |
| CPU request only, memory request equal to limit | Burstable | Not eligible | Moderate |
| No requests or limits | BestEffort | Not eligible | Highest |
# Guaranteed: every container sets CPU and memory limits equal to requests.
# Integer CPU makes it eligible for exclusive cores under the static CPU manager policy.
apiVersion: v1
kind: Pod
metadata:
name: api-guaranteed
spec:
containers:
- name: api
image: registry.example.com/api:1.4.2
resources:
requests:
cpu: "2"
memory: 1Gi
limits:
cpu: "2" # equal to request, whole cores
memory: 1Gi
---
# Burstable: CPU limit removed, so the pod can use idle CPU without CFS throttling.
# Memory request still equals the limit; the memory limit stays because memory is not compressible.
apiVersion: v1
kind: Pod
metadata:
name: api-burstable
spec:
containers:
- name: api
image: registry.example.com/api:1.4.2
resources:
requests:
cpu: "2" # still drives scheduling and CPU shares under contention
memory: 1Gi
limits:
memory: 1Gi # exceeding this triggers an OOM kill, not a slowdown
# Check the assigned class with:
# kubectl get pod api-burstable -o jsonpath='{.status.qosClass}'
Two otherwise identical pod specs: matching CPU and memory requests and limits give the Guaranteed QoS class, and removing only the CPU limit makes the pod Burstable while its memory limit stays in place.
How do you safely roll out CPU limit removal?
Step 1: Record a baseline
Before changing anything, capture a week of throttle ratio (throttled periods divided by total periods) and p99 latency per workload. A week covers daily and weekly traffic cycles, so a quiet Tuesday won't distort the comparison. Without this baseline, you can't tell whether a later latency change came from the limit removal or from a deploy.
Step 2: Set requests from a high percentile
Base each request on something like the p90 to p95 of observed CPU usage, not the mean. Once limits are gone, the request is the only reservation the scheduler honors. If requests are sized to the mean, nodes become overcommitted whenever pods burst together, and the latency you saved by removing throttling turns into CPU contention between neighbors.
Step 3: Canary on a subset
Remove or raise limits on a small group of pods, either a separate Deployment behind the same Service or a single node pool. Compare three things against the baseline: tail latency, node-level CPU saturation, and the run-queue pressure of co-located pods. Expand only if p99 improves and node saturation stays below the level where neighbors begin to suffer.
Step 4: Budget for maintenance
This approach shifts cost from debugging to measurement. Requests drift as code and traffic change, so someone has to review usage percentiles regularly and update them. The return on that effort is fewer latency investigations that end with "it was CFS throttling all along." Those investigations tend to be slow because the CPU graphs look healthy while the problem is happening.
What are the alternatives to removing CPU limits?
Teams that must keep limits, because of multi-tenant policy, chargeback, or a noisy-neighbor history, still have options. Each one trades something different.
| Option | What it changes | Main cost | Best fit |
|---|---|---|---|
| Raise limit relative to request | More quota per period while scheduling stays the same | Less predictable node contention | Bursty request handlers |
| Reduce thread counts | Fewer runnable threads draining the quota in parallel | Lower peak throughput | Runtimes with configurable pools |
| Shorten the CFS period | Throttle stalls end sooner | More scheduler work, node-wide effect | Latency-sensitive nodes, after testing |
| CFS burst | Unused quota carries into later periods | Kernel and runtime support varies | Short spikes after idle stretches |
| Static CPU pinning (Guaranteed QoS) | Exclusive cores with integer CPU requests | Lower bin-packing density | Dedicated, steady workloads |
Raising the limit is the cheapest change. Throttling happens when work is concentrated into short spikes, so more headroom above the request absorbs those spikes without changing how the scheduler places the pod.
Cutting thread counts attacks the cause directly. Eight threads running in parallel exhaust a two-CPU quota in a quarter of the period, so fewer threads stretch the same quota over more of the window.
Shortening the period through the kubelet, where it is supported, does not grant more CPU. It shortens how long a throttled thread waits before quota refills.
CFS burst lets a cgroup bank unused quota for later spikes. Both period tuning and burst add scheduler overhead and depend on your kernel and kubelet versions. Validate them under production-like load before treating either as a default.
Static CPU pinning through the CPU manager gives the workload whole cores, so it no longer competes with other pods for them.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to kubernetes cpu throttling, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.