What is Kubernetes CPU throttling and how does CFS quota work?

Throttling is the kernel pausing a container's threads because the container has used up its CPU allowance for the current time window. It has nothing to do with whether the node is busy. A throttled container can sit idle while other cores on the same machine have nothing to do.

The mechanism is CFS bandwidth control. Kubernetes converts a CPU limit into two cgroup values:

  • Period: commonly 100ms.
  • Quota: the CPU time the cgroup may consume within each period.

Take a container with requests.cpu: 250m and limits.cpu: 500m. The 500m limit becomes a quota of 50ms per 100ms period. Under cgroup v1 that is cpu.cfs_quota_us = 50000 and cpu.cfs_period_us = 100000. Under cgroup v2 it is cpu.max set to 50000 100000. Once the cgroup's threads have consumed 50ms of CPU time in a period, they are descheduled until the next period starts.

Requests work through a different mechanism. They tell the scheduler where the pod fits, and they set a relative weight: cpu.shares on v1, cpu.weight on v2. That weight only matters when containers are competing for CPU.

Setting Used by Kernel control Effect when node is idle
requests.cpu Scheduler, CFS weighting cpu.shares / cpu.weight None; container can use spare CPU
limits.cpu CFS bandwidth control cfs_quota_us + cfs_period_us / cpu.max Hard ceiling; throttles regardless

CFS bandwidth behavior has changed across kernel releases. Older kernels had bugs that throttled containers that never came close to their quota. Before you tune anything, check the behavior on your own nodes' kernels.

# Create a pod with a 250m request and a 500m limit, then read the CFS values the kernel enforces.
kubectl apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata:
  name: cpu-demo
spec:
  containers:
  - name: app
    image: busybox:1.36
    command: ["sleep", "3600"]
    resources:
      requests:
        cpu: 250m   # scheduling + relative weight (cpu.shares / cpu.weight)
      limits:
        cpu: 500m   # hard ceiling: 50ms of CPU time per 100ms period
EOF

kubectl wait --for=condition=Ready pod/cpu-demo --timeout=60s

# Kernel version matters: CFS bandwidth behavior differs across releases.
kubectl exec cpu-demo -- uname -r

# cgroup v2 exposes cpu.max as "<quota> <period>"; expect: 50000 100000
# cgroup v1 splits it into two files; expect: 50000 and 100000
kubectl exec cpu-demo -- sh -c '
  if [ -f /sys/fs/cgroup/cpu.max ]; then
    echo "cgroup v2"
    cat /sys/fs/cgroup/cpu.max
    cat /sys/fs/cgroup/cpu.weight
  else
    echo "cgroup v1"
    cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
    cat /sys/fs/cgroup/cpu/cpu.cfs_period_us
    cat /sys/fs/cgroup/cpu/cpu.shares
  fi'

# Throttling counters: nr_throttled counts periods where the quota ran out.
kubectl exec cpu-demo -- sh -c 'cat /sys/fs/cgroup/cpu.stat 2>/dev/null || cat /sys/fs/cgroup/cpu/cpu.stat'

Applies a pod with requests.cpu 250m and limits.cpu 500m, then reads the container's cgroup files to confirm the 50000/100000 quota and period, the request-derived weight, the node's kernel version, and the throttling counters on either cgroup v1 or v2.

Why do multi-threaded containers hit the CPU limit so fast?

A CPU limit is not a speed cap. It is a budget of CPU time per period, and the kernel deducts from it whenever any thread in the container runs on any core. A container with eight runnable threads on an eight-core node can use eight milliseconds of quota for every millisecond of wall-clock time.

A timeline of one period

Illustrative estimate. Assumed inputs: a 1-CPU limit (100ms of quota per period), the default 100ms CFS period, and 8 threads that are all runnable at the start of the period on a node with at least 8 idle cores.

  • 0ms–12.5ms: All 8 threads run in parallel. They spend 8 × 12.5ms = 100ms of quota.
  • 12.5ms–100ms: The quota is gone. Every thread in the container is descheduled for the remaining ~87.5ms, even if the node is otherwise idle.
  • 100ms: The quota refills and the cycle repeats.

Average usage over the period is exactly 1 CPU, so the limit appears to be respected. But a request that arrives at 13ms waits roughly 87ms before any of its code runs.

Runtimes that multiply parallelism

Many language runtimes size their thread pools, GC worker counts, or scheduler parallelism from the number of CPUs they detect. If a runtime reads the node's core count instead of the container's limit, a 1-CPU container on a 64-core node may start dozens of GC or worker threads. Each one burns quota in parallel, and the stall arrives sooner.

Container awareness varies by runtime and by version. Check what your runtime reports as the available CPU count from inside the container. If it reports the node's core count, set the worker and GC thread counts explicitly so they match the limit.

 +-------------------------------------------------------------------+
 | Container: 1-CPU limit = 100ms quota per 100ms CFS period         |
 | Illustrative estimate: 8 runnable threads, >= 8 idle cores        |
 +-------------------------------------------------------------------+

            0ms       12.5ms                                    100ms
  thread 1  |=========|.........................................|
  thread 2  |=========|.........................................|
  thread 3  |=========|.........................................|
     ...    |   ...   |                   ...                   |
  thread 8  |=========|.........................................|
            <-- run --><----------- throttled ~87.5ms ---------->
            8 x 12.5ms = 100ms quota spent; refill at 100ms, repeat
            request arriving at 13ms waits ~87ms before it runs

 +-----------------+   reads node core count   +----------------------+
 | Node: 64 cores  |-------------------------->| Runtime sizes GC and |
 | Container: 1 CPU|   (not container limit)   | worker thread pools  |
 +-----------------+                           +----------+-----------+
                                                          |
                                                          v
                                  dozens of threads burn quota in parallel
                                  -> quota runs out sooner, stall is longer

 Fix: check the CPU count the runtime sees inside the container; if it
      reports node cores, set worker and GC thread counts to match limit.

Illustrative estimate of one 100ms CFS period: eight parallel threads spend a 1-CPU quota in about 12.5ms and then stall for about 87.5ms. A runtime that sizes its thread pools from the node's core count adds more threads and reaches the stall sooner.

How does CPU throttling affect p99 latency when dashboards look fine?

Throttling turns CPU contention into wall-clock delay. When a container exhausts its quota partway through a period, every runnable thread in it is descheduled until the period ends. A request in flight at that moment simply stops, and so does any request that arrives during the gap.

Illustrative estimate: assume a limit of 2 CPUs (200ms of quota per 100ms period) and a burst that keeps 8 threads busy. The quota runs out after about 25ms of wall time, and the container then waits roughly 75ms for the next period. A request that needed 5ms of CPU can take 80ms to complete. A request that needs CPU across several periods can be throttled in each one, so the stalls accumulate.

Why averages hide it

Only the requests that overlap a throttled window are delayed, and they may be a small fraction of traffic. The mean barely moves because most requests finish untouched. p99 and p99.9 measure exactly that unlucky fraction, so they jump while the median stays flat.

Why CPU graphs hide it

Usage metrics are usually rates computed over scrape intervals of 15 seconds or more. A container that runs at 4x its limit for 25ms and then idles for the rest of each burst averages out to a modest number. A service shown at 40% of its limit can still hit its quota in many individual 100ms periods. The dashboard reports how much CPU the container used in total. It does not show how that usage was distributed across individual periods, and the quota is enforced per period.

How do you detect CPU throttling in Kubernetes?

The kernel already tracks throttling per cgroup. You can read the counters from inside the container. On cgroup v2, read cpu.stat at the root of the container's cgroup mount, usually /sys/fs/cgroup/cpu.stat. On cgroup v1, it lives under the cpu controller directory. Three fields matter:

  • nr_periods: the number of enforcement periods in which the cgroup had runnable tasks.
  • nr_throttled: the number of those periods in which the cgroup used up its quota and was stopped.
  • throttled_usec (v2) or throttled_time in nanoseconds (v1): the total time tasks spent waiting for the next period.

These counters are cumulative, so a single reading tells you little. Take two readings a known interval apart and compare the deltas. That shows whether throttling is happening now or happened hours ago during startup.

In a cluster, the kubelet's embedded cAdvisor exports the same data as container_cpu_cfs_periods_total and container_cpu_cfs_throttled_periods_total. Graph the throttled fraction per container, for example rate(container_cpu_cfs_throttled_periods_total[5m]) / rate(container_cpu_cfs_periods_total[5m]). Put it on the same time axis as p99 latency. If throttling spikes when latency spikes, you have found a likely cause.

Keep one limitation in mind. These metrics count periods, not the effect on requests. A 3% throttled ratio can still hurt p99 if those periods land exactly on traffic bursts, when queues are deepest. A 60% ratio on a batch job may be harmless, because nobody is waiting on any single request. Use the ratio to point you toward a cause, then confirm it against latency.

#!/usr/bin/env bash
# Sample CFS throttling counters twice inside a container and print the deltas.
# Usage: ./throttle-check.sh [interval_seconds]
set -euo pipefail
INTERVAL="${1:-10}"

# cgroup v2 exposes cgroup.controllers at the mount root; v1 does not.
if [ -f /sys/fs/cgroup/cgroup.controllers ]; then
  STAT=/sys/fs/cgroup/cpu.stat;     TFIELD=throttled_usec; TO_MS=1000      # usec -> ms
else
  STAT=/sys/fs/cgroup/cpu/cpu.stat; TFIELD=throttled_time; TO_MS=1000000   # nsec -> ms
fi

# Print one counter's value; default to 0 if the field is absent (no quota set).
get() { awk -v k="$1" '$1==k {print $2; f=1} END {if (!f) print 0}' "$STAT"; }

# Counters are cumulative since the cgroup was created, so only deltas matter.
p1=$(get nr_periods); n1=$(get nr_throttled); w1=$(get "$TFIELD")
sleep "$INTERVAL"
p2=$(get nr_periods); n2=$(get nr_throttled); w2=$(get "$TFIELD")

dp=$((p2 - p1)); dn=$((n2 - n1)); dw=$((w2 - w1))
if [ "$dp" -eq 0 ]; then
  echo "No enforcement periods in ${INTERVAL}s (idle, or no CPU limit set)."
  exit 0
fi

awk -v dp="$dp" -v dn="$dn" -v dw="$dw" -v d="$TO_MS" -v i="$INTERVAL" 'BEGIN {
  printf "last %ss: periods=%d throttled=%d (%.1f%%) throttled_time=%.1f ms\n",
         i, dp, dn, 100*dn/dp, dw/d }'

# Cluster-wide equivalent (PromQL, cAdvisor metrics from the kubelet);
# graph it on the same time axis as p99 latency:
#   sum by (namespace, pod, container) (rate(container_cpu_cfs_throttled_periods_total[5m]))
#   /
#   sum by (namespace, pod, container) (rate(container_cpu_cfs_periods_total[5m]))
# This ratio counts periods, not request impact: confirm against latency.

A shell script that reads cpu.stat twice on cgroup v1 or v2 and reports throttled periods and throttled time for the interval, plus the equivalent PromQL ratio built from cAdvisor metrics for graphing next to p99 latency.

What are the alternatives to strict CPU limits?

Each mitigation trades something away, and the options are split between the app team and the platform team.

Option What it fixes Over-provisioning / bin-packing cost Isolation impact Owner
Raise the limit Fewer periods exhausted during bursts Low if requests stay put; quota is a ceiling, not a reservation Slightly weaker; bursts can take more idle CPU App team
Remove limits, keep accurate requests Eliminates quota throttling entirely None; scheduling still uses requests Under contention, CPU is shared by request-based weights, so a noisy pod slows neighbours only proportionally App team, often gated by platform policy
Reduce in-container parallelism Fewer threads burning quota at once None Unchanged App team
Guaranteed QoS + integer CPUs + static CPU manager Exclusive cores, no sharing with other pods High; whole cores are reserved even when idle Strongest Platform enables policy; app team requests integer CPUs
Tune the CFS period Shorter periods mean a burst waits less before its quota refills None on packing; adds scheduler overhead node-wide Unchanged Platform team

Raising the limit is the quickest fix, but it only delays the point where quota runs out.

Removing limits works because requests, not limits, drive scheduling and contention weighting. It holds up only if requests reflect real usage.

Reducing parallelism targets the root cause from the previous section. Setting GOMAXPROCS or the JVM's -XX:ActiveProcessorCount to match the limit stops a runtime from sizing its thread pools to the node's core count.

Exclusive cores give predictable latency, but you pay for them in stranded capacity.

Changing the CFS period affects every pod on the node. Test it on a dedicated node pool first.

When should you avoid removing CPU limits or pinning CPUs?

Removing limits trades throttling for dependence on your neighbors. A pod without a limit can burst into any idle CPU on the node, which is the point, but a runaway pod, a GC storm, or a batch job with an unbounded worker pool can do the same. Requests still guarantee each pod its proportional share under contention, but the extra headroom you were quietly relying on disappears. Capacity planning also gets harder: observed CPU usage now reflects how much spare capacity happened to exist, not what the service needs.

This is why latency often looks excellent on lightly packed staging nodes and degrades in production. On a mostly empty node, an unlimited pod gets near-exclusive cores. In production, where bin-packing pushes nodes toward full requests, that headroom shrinks. Load-test on nodes packed the way production is.

Some constraints are not technical. Multi-tenant clusters may require limits for isolation, chargeback, or admission policy. A namespace ResourceQuota that constrains limits.cpu will reject pods that omit a CPU limit.

Shortening the CFS period reduces the length of each throttling stall but increases scheduler overhead. It is a node-wide kubelet setting, and managed node offerings may not expose it.

Static pinning removes flexibility. Pinned cores cannot absorb other pods' bursts, and fractional leftovers on the node can go unused.

A reasonable split:

  • Latency-critical services: set accurate requests, then either drop limits or set them well above p99 demand. Pin cores only when you can fill whole nodes.
  • Batch workloads: keep limits. Throttling only stretches completion time, and the limits protect everything else on the node.

A practical workflow for diagnosing kubernetes cpu throttling

Step 1: Correlate before changing anything

Plot p99 latency against the throttled-period ratio for the same pods and time window. In Prometheus, that ratio is rate(container_cpu_cfs_throttled_periods_total[5m]) divided by rate(container_cpu_cfs_periods_total[5m]). If the latency spikes line up with jumps in the ratio, throttling is a real suspect. If they don't, look elsewhere first, such as GC, downstream calls or lock contention.

Step 2: Confirm from inside the pod

Read the cgroup counters directly with kubectl exec <pod> -- cat /sys/fs/cgroup/cpu.stat. On cgroup v2, check nr_periods, nr_throttled and throttled_usec. On cgroup v1 nodes the file is under the cpu controller directory and reports throttled_time instead.

Next, count threads with kubectl exec <pod> -- grep Threads /proc/1/status. Then check what the runtime believes it has, for example GOMAXPROCS or the JVM's -XX:ActiveProcessorCount. If the runtime sees 32 node cores but the pod has a 2-CPU limit, you have found the mismatch.

Step 3: Change one variable

Adjust either the runtime's parallelism or the limit, not both. If you change two things and latency improves, you won't know which change helped.

Step 4: Re-measure under realistic load

Replay production-shaped traffic, including the bursts. A steady synthetic load produces a smooth average that hides the per-period quota exhaustion you are trying to remove.

Why this loop pays off

Blindly raising limits can hide the symptom while the thread-pool mismatch stays in place. Engineers then repeat the same exercise on the next service. Raised limits also inflate the requests that teams copy across deployments, so the cluster ends up provisioned for headroom it never uses. Fixing the actual cause keeps limits close to real demand.

#!/usr/bin/env bash
# Snapshot throttling and thread state for one pod, before and after a change.
# Usage: ./throttle-check.sh <pod> [namespace]
set -euo pipefail
POD="$1"
NS="${2:-default}"

echo "== Pod CPU limit =="
kubectl get pod "$POD" -n "$NS" \
  -o jsonpath='{.spec.containers[0].resources.limits.cpu}{"\n"}'

echo "== cgroup v2 cpu.stat (falls back to v1 path) =="
# v2: nr_periods, nr_throttled, throttled_usec
# v1: nr_periods, nr_throttled, throttled_time (nanoseconds)
kubectl exec "$POD" -n "$NS" -- sh -c \
  'cat /sys/fs/cgroup/cpu.stat 2>/dev/null || cat /sys/fs/cgroup/cpu/cpu.stat'

echo "== CFS quota/period (v2 cpu.max: '<quota> <period>' in usec) =="
kubectl exec "$POD" -n "$NS" -- sh -c \
  'cat /sys/fs/cgroup/cpu.max 2>/dev/null || echo "v1: see cpu.cfs_quota_us"'

echo "== Thread count of PID 1 =="
kubectl exec "$POD" -n "$NS" -- grep Threads /proc/1/status

echo "== CPUs visible to the container vs runtime settings =="
kubectl exec "$POD" -n "$NS" -- nproc
# Shows GOMAXPROCS or JAVA_TOOL_OPTIONS if set explicitly in the environment
kubectl exec "$POD" -n "$NS" -- sh -c \
  'env | grep -E "GOMAXPROCS|JAVA_TOOL_OPTIONS" || echo "no explicit runtime CPU setting"'

# Run this twice a few minutes apart under production-shaped load and diff
# nr_throttled / nr_periods; change only one variable between runs.

A script that snapshots a pod's CPU limit, cgroup throttling counters, thread count and runtime CPU settings so you can compare before and after a single change.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes cpu throttling, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Consumers learn about a breaking change at review time rather than from production errors.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.