What does OOMKilled actually mean, and is exit code 137 enough to prove it?

An OOMKilled event means the kernel's out-of-memory killer terminated a process in the container because the container's cgroup hit its memory limit. The kubelet sees the container exit and records the reason. Before you change any setting, confirm that this is what happened.

Confirm the termination reason

Run kubectl describe pod <pod> and look at the container's Last State block. A real OOM kill shows Reason: OOMKilled with Exit Code: 137. For a structured view, run kubectl get pod <pod> -o yaml and read status.containerStatuses[].lastState.terminated.reason. If the container has not restarted yet, the same fields appear under state.terminated instead. Also check the Events section at the bottom of the describe output. Messages such as Liveness probe failed followed by Killing point to a different cause.

Why 137 alone proves nothing

Exit code 137 is 128 + 9, meaning the process received SIGKILL. The OOM killer sends SIGKILL, but other sources do too. When a liveness probe fails, the kubelet sends SIGTERM. If the process is still running after the termination grace period, it follows with SIGKILL, and the result is 137 with a reason like Error. A manual kill -9 or a runtime-initiated stop can produce the same code. The reason field identifies the cause. The exit code does not.

Why this checklist exists

A memory leak and a limit that is too low call for opposite fixes. A low limit should be raised to match real usage. A leak needs a code fix, and raising the limit only delays the next kill while hiding the trend. Backend and platform engineers should work through this checklist before touching any container memory limit.

#!/usr/bin/env bash
set -euo pipefail

POD="${1:?usage: $0 <pod> [namespace]}"
NS="${2:-default}"

echo "== lastState.terminated (previous run) =="
kubectl get pod "$POD" -n "$NS" -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}reason={.lastState.terminated.reason}{"\t"}exitCode={.lastState.terminated.exitCode}{"\t"}restarts={.restartCount}{"\n"}{end}'

echo "== state.terminated (current run) =="
kubectl get pod "$POD" -n "$NS" -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}reason={.state.terminated.reason}{"\t"}exitCode={.state.terminated.exitCode}{"\n"}{end}'

echo "== events =="
kubectl get events -n "$NS" \
  --field-selector involvedObject.name="$POD" \
  --sort-by=.lastTimestamp

A bash script that reads the termination reason, exit code and pod events with kubectl so you can tell a real OOMKilled from another SIGKILL that also exits with 137.

Who should use this checklist?

Use this checklist if you own or operate a workload that restarts with OOMKilled and you are about to decide whether to change its memory settings. In practice that means:

  • Backend and application engineers who own the service code. They are the ones who can fix a leak, bound a cache, or tune runtime memory settings, and they need evidence before committing to a code change.
  • Platform and SRE engineers who set resource defaults, review limit changes, and respond to node memory pressure. They need to know whether a kill came from the container limit or from the node before they add capacity or adjust eviction settings.
  • On-call engineers handling a crash loop at an inconvenient hour. The early sections give them a fast way to separate a real OOM kill from a probe failure and to avoid a limit bump that hides a leak.

You need kubectl access to read pods, events and nodes in the affected namespace. The later steps assume a metrics pipeline that exposes container_memory_working_set_bytes and, for the limit ratio and termination-reason queries, kube-state-metrics. Profiling the runtime requires whoever can deploy a canary or attach profiling tools to the service.

The checklist is less useful for workloads with no memory limit set (they are killed by node pressure, not their own cgroup) and for one-off batch jobs whose input size simply exceeded what they were given. In those cases, go straight to the node-level section or size the job for its input.

How does the checklist work?

The checklist narrows the cause in a fixed order, so each step rules out a class of explanations before you spend time on the next, more expensive one:

  1. Confirm the kill was an OOM kill. Read the termination reason, not just exit code 137, and check pod events for probe-driven kills. If the reason is not OOMKilled, memory limits are probably not your problem.
  2. Identify which layer killed it. Distinguish the container hitting its own limit from kubelet eviction and from the node-level OOM killer, using pod phase, node conditions, node events and usage relative to the limit.
  3. Understand what is being counted. Check cgroup accounting so you know whether the charged memory is heap, page cache, dirty pages or tmpfs. This prevents chasing a "leak" that is really file cache or a memory-backed emptyDir.
  4. Read the memory curve over several restarts. A steady climb that repeats after each restart points to a leak. A plateau that tracks load points to a limit that is too low for real concurrency.
  5. Run a controlled test and confirm in the runtime. Raise the limit on one canary replica and see whether memory levels off or keeps climbing, then use the runtime's own profiling tools to locate the growth.
  6. Size or fix from data. Set requests and limits from observed working-set percentiles under peak load, or fix the leak, and add guardrails so the next incident leaves evidence behind.

Each step uses data that already exists in the cluster or the metrics pipeline. The first two steps need only kubectl, and the expensive steps (canaries, heap dumps, code changes) come last, after cheaper checks have narrowed the hypothesis.

Is it a container OOM kill or node-level memory pressure?

Before raising anything, establish which layer did the killing. Kubernetes nests memory accounting, and each layer has its own enforcer:

Node (allocatable memory, kubelet eviction thresholds)
+------------------------------------------------------+
|  Pod cgroup                                          |
|  +----------------------+  +----------------------+  |
|  | Container cgroup     |  | Container cgroup     |  |
|  | memory.max = limit   |  | memory.max = limit   |  |
|  |  [A] kernel OOM kill |  |                      |  |
|  +----------------------+  +----------------------+  |
+------------------------------------------------------+
 [B] kubelet eviction: node crosses an eviction threshold
 [C] node OOM killer: node runs out before kubelet reacts

[A] happens when one container's usage reaches its own limit. The rest of the node can have plenty of free memory. [B] and [C] happen because the node as a whole is short of memory. In those cases the victim may be a pod that was well within its limit.

Signal [A] Container limit [B] Kubelet eviction [C] Node OOM killer
Container lastState reason OOMKilled, exit 137 none; pod is terminated often OOMKilled, exit 137
Pod status Running, restart count rises Failed, reason Evicted Running, restart count rises
Node condition normal MemoryPressure=True may stay normal
Node events none eviction threshold met system OOM recorded
Usage vs. limit at death at the limit anywhere below the limit
Fix Fix the leak or raise the container memory limit Set accurate requests, add node capacity, reserve memory for system daemons Same as [B], and use QoS so critical pods are killed last

For [C], the Kubernetes QoS class sets each container's OOM score adjustment. BestEffort pods are killed first and Guaranteed pods last. Case [C] is easy to misread because the container also shows OOMKilled. To tell it apart from [A], compare memory usage at the time of death with the container's limit. If usage was below the limit, the node killed it.

#!/usr/bin/env bash
set -euo pipefail
NS=payments          # placeholder namespace
POD=api-7d9f8c6b5-x2k4q  # placeholder pod name

kubectl get pod "$POD" -n "$NS" \
  -o jsonpath='phase={.status.phase} reason={.status.reason} qos={.status.qosClass}{"\n"}'

kubectl get pod "$POD" -n "$NS" -o jsonpath='{range .status.containerStatuses[*]}{.name}: reason={.lastState.terminated.reason} exit={.lastState.terminated.exitCode} restarts={.restartCount}{"\n"}{end}'

kubectl get pod "$POD" -n "$NS" -o jsonpath='{range .spec.containers[*]}{.name}: limit={.resources.limits.memory} request={.resources.requests.memory}{"\n"}{end}'

NODE=$(kubectl get pod "$POD" -n "$NS" -o jsonpath='{.spec.nodeName}')
echo "node=$NODE"

kubectl get node "$NODE" \
  -o jsonpath='MemoryPressure={.status.conditions[?(@.type=="MemoryPressure")].status}{"\n"}'

kubectl get events -A \
  --field-selector involvedObject.kind=Node,involvedObject.name="$NODE" \
  --sort-by=.lastTimestamp

kubectl get events -n "$NS" --field-selector involvedObject.name="$POD"

kubectl top pod "$POD" -n "$NS" --containers

A kubectl triage script that collects the pod reason, container termination state, limits, node MemoryPressure condition, and node and pod events needed to tell a container-limit OOM kill [A] from kubelet eviction [B] or a node-level OOM kill [C].

How does the container memory limit get enforced?

The kernel, not the kubelet, enforces the limit. The container's cgroup is charged for more than the process heap: anonymous memory, page cache from files the container reads and writes, tmpfs pages (including emptyDir with medium: Memory), and some kernel allocations. When that charge reaches the limit and the kernel cannot reclaim enough, the cgroup's OOM killer terminates a process. Writing 500 MiB to a memory-backed emptyDir consumes 500 MiB of the limit, even if ps shows a small process.

This is why container_memory_working_set_bytes and RSS often disagree:

Metric What it counts Who relies on it
RSS (ps, /proc/<pid>/status) Resident pages mapped by one process, mostly heap, stack and mapped libraries Developers profiling a single process
container_memory_working_set_bytes Total cgroup usage minus inactive file cache, so it includes active page cache and tmpfs The kubelet, for eviction decisions and the usage that kubectl top reports

RSS excludes page cache and tmpfs entirely, so a container can have a stable RSS and a rising working set.

This creates a trade-off for I/O-heavy workloads. A service that streams large files keeps a lot of page cache in the active list, and the working set climbs steadily. That looks like a leak, but most of it is clean cache the kernel can drop. Reclaim is not guaranteed to happen in time, though. Dirty pages must be written back before they can be freed, and tmpfs pages cannot be dropped at all without swap. A burst of allocation can therefore hit the limit before reclaim catches up, and the container is killed even though much of its charged memory was nominally reclaimable.

#!/usr/bin/env bash
NS=default
POD=my-app-7d9f8c6b5-x2k4q

kubectl exec -n "$NS" "$POD" -- sh -c '
  CG=/sys/fs/cgroup

  # The limit the kernel enforces ("max" means no limit is set).
  echo "limit:          $(cat $CG/memory.max)"

  # RSS of PID 1 counts only pages mapped by that process.
  # It excludes page cache and tmpfs.
  grep VmRSS /proc/1/status

  # Everything charged to the cgroup: anon + file cache + shmem/tmpfs + kernel.
  USAGE=$(cat $CG/memory.current)
  echo "memory.current: $USAGE"

  # shmem includes tmpfs and emptyDir with medium: Memory, and cannot be
  # reclaimed without swap. file_dirty must be written back before it can be freed.
  grep -E "^(anon|file|shmem|active_file|inactive_file|file_dirty) " $CG/memory.stat

  # Approximate working set: usage minus inactive file cache.
  # This is roughly what container_memory_working_set_bytes reports.
  INACTIVE=$(awk "/^inactive_file /{print \$2}" $CG/memory.stat)
  echo "working set:    $((USAGE - INACTIVE))"
'

kubectl top pod "$POD" -n "$NS" --containers

Reads cgroup v2 accounting inside a pod to show why a small, stable VmRSS can sit alongside a large working set made up of active page cache, dirty pages and shmem/tmpfs.

How do you tell a memory leak from a limit that is too low?

The single most useful signal is the shape of the memory curve across several restarts, not the value at the moment of the kill. Plot working-set memory per container with something like max_over_time(container_memory_working_set_bytes{namespace="shop", container="api"}[5m]) over days, and overlay restart events.

Signal Points to a leak Points to a low limit
Curve shape Sawtooth that climbs steadily after each restart Rises at startup or with traffic, then plateaus
Relation to load Grows even when traffic is flat or idle Tracks request rate, batch runs, or queue depth
Time to kill Similar uptime before each kill Kills cluster at load peaks, any uptime
After a restart Same climb repeats Stable until the next spike

Correlate each kill with deploys, traffic spikes, scheduled batch jobs, unusually large payloads, and changes to worker or thread counts. A low limit often hides behind concurrency. Illustrative estimate: a service with a 200 MiB baseline handles requests that each allocate 40 MiB transiently. With 8 concurrent workers, the peak is 200 + 8 × 40 = 520 MiB, which exceeds a 512 MiB limit. Nothing leaks. The pod survives at 6 concurrent requests and dies at 8, and raising worker concurrency produces the same kill.

Two measurement traps matter here. A 30- or 60-second scrape interval can miss a spike that lasts a few seconds, so the graph shows a comfortable plateau right up to a kill. If OOMs happen weekly, a one-day window shows nothing useful. Widen the window until it covers several kills before you decide which pattern you are looking at.

max by (pod) (
  max_over_time(container_memory_working_set_bytes{namespace="shop", container="api"}[5m])
)

max by (pod) (container_memory_working_set_bytes{namespace="shop", container="api"})
/ on (pod)
max by (pod) (kube_pod_container_resource_limits{namespace="shop", container="api", resource="memory"})

increase(kube_pod_container_status_restarts_total{namespace="shop", container="api"}[10m]) > 0

kube_pod_container_status_last_terminated_reason{namespace="shop", container="api", reason="OOMKilled"} == 1

sum(rate(http_requests_total{namespace="shop", service="api"}[5m]))

deriv(container_memory_working_set_bytes{namespace="shop", container="api"}[1h])

A set of PromQL queries that plot peak working-set memory against the limit, restarts, OOM termination reasons, request rate and memory slope, so you can see whether the curve climbs steadily like a leak or plateaus with load like a low limit.

How can you run a controlled test and confirm the cause in the runtime?

Once you have a hypothesis, test it directly. In a staging or canary environment, raise the container memory limit for one replica and leave everything else unchanged. How much headroom to add is a judgment call. It needs to be large enough that a legitimate working set has room to settle, but small enough that a leak still reaches the new ceiling within a window you can observe.

Then watch the shape of the memory curve:

  • Memory levels off below the new limit. The old limit was too low for the real working set.
  • Memory keeps climbing until it reaches the new ceiling. You have a leak, and the extra memory only bought time.

Some runtime behavior looks like a leak but isn't one:

  • JVM. The heap maximum is only part of the process footprint. Metaspace, direct buffers, code cache, and thread stacks all sit outside the heap. If heap sizing ignores the container limit, the process can exceed the limit while the heap looks healthy.
  • Go. The garbage collector grows the heap relative to live data. It does not consider the container limit unless you configure a soft memory limit such as GOMEMLIMIT.
  • Native allocations and threads. C libraries, compression buffers, and per-thread stacks add memory that heap profilers do not show.
  • Unbounded caches. These grow with traffic diversity and behave exactly like leaks.

To confirm the cause, use the runtime's own tools, such as JVM heap dumps and Native Memory Tracking, Go's pprof heap profiles, or Python's tracemalloc. Profiling in production adds CPU and pause overhead, and heap dumps can contain tokens and customer data. A canary is safer, but it may not reproduce the traffic mix that triggers the growth.

How should you size requests and limits from observed data?

Once you know which behavior you are dealing with, size from data rather than from the last value that stopped the crashes. Use the working-set metric (container_memory_working_set_bytes), because that is close to what the kubelet uses for eviction decisions and what counts against the cgroup limit. Take a high percentile (p99 or the observed maximum) over a window that includes realistic peak load: a load test, a batch run, or a known traffic spike. Then add headroom for inputs you have not seen yet.

Illustrative estimate (assumed inputs): suppose p99 working set during a peak-load test is 600 MiB, and steady-state median is 400 MiB. With 25% headroom, the limit is 750 MiB, rounded to 768Mi. Setting the request at 448Mi covers typical use and lets the scheduler pack pods realistically. The resources snippet that follows uses these values.

Requests drive scheduling and eviction order. Limits drive the OOM kill. The relationship between the two sets the QoS class:

QoS class How it is set Eviction under node pressure Trade-off
Guaranteed Requests equal limits for CPU and memory in every container Least likely to be evicted Predictable, but reserves peak memory permanently, which wastes capacity for bursty workloads
Burstable Requests set below limits Evicted earlier when usage exceeds requests Allows bursting, but a wide gap overcommits the node and invites noisy-neighbor evictions
BestEffort No requests or limits Evicted first Not suitable for services you care about

Keep the request-to-limit gap narrow for latency-sensitive services. A wider gap only makes sense when bursts are short and rarely coincide across pods on the same node.

When should you avoid simply raising the limit, and what are the alternatives?

Raising the limit is the right fix when working set climbs to a stable plateau that sits just above the current limit, for example after a traffic increase or a new feature that legitimately holds more data. If working set keeps rising between restarts, a higher limit only lengthens the interval between kills. A leak that killed the pod every six hours might now kill it every two days, which looks like a fix until it surfaces as node memory pressure, often on a weekend.

Option Engineering cost Risk
Fix the leak High: profiling, code change, review Lowest once fixed; addresses the cause
Runtime memory tuning (heap caps, GC settings) Low to moderate Can replace OOM kills with in-process out-of-memory errors or GC thrashing that raises latency
Bound caches or concurrency Moderate Possible lower hit rates or throughput; memory becomes predictable
Remove the limit Trivial One pod can exhaust node memory and trigger eviction or kernel OOM kills of unrelated workloads

Guardrails cut debugging time because they collect evidence before it disappears. SIGKILL gives the process no chance to flush logs or write a heap dump, so after the kill there is nothing left to inspect. Alert when working set crosses a threshold such as 85–90% of the limit, which leaves time to capture a heap dump or profile from the live process. Track restart counts and the OOMKilled termination reason per workload, so a slowly rising kill rate shows up as a trend and doesn't get dismissed as a one-off. Without this evidence, engineers have to wait for the next kill to learn anything.

How can this checklist save engineering time?

Most of the time lost on OOM kills goes to fixing the wrong thing, or to waiting for another kill because nobody captured evidence the first time. The checklist targets both.

  • It stops dead-end investigations early. Checking the termination reason takes one command. If the reason is Error from a probe-driven SIGKILL, nobody spends a day profiling heap usage for a problem that lives in probe timeouts or slow startup.
  • It routes the problem to the right owner. Separating a container-limit kill from kubelet eviction or a node-level OOM kill tells you whether the service team or the platform team should act. That avoids an application team raising limits to fix what is really node overcommit, and a platform team adding nodes to absorb a leak.
  • It prevents repeat incidents from limit bumps. Raising the limit on a leaking service makes the kill rarer, not absent. Each recurrence costs another on-call page, another investigation, and often a larger blast radius when it finally shows up as node pressure. Reading the curve across several restarts first means the fix you ship is the one that ends the incident.
  • It orders work from cheap to expensive. kubectl checks and existing metrics come before canaries, heap dumps and code changes. Engineers commit profiling time only once a leak is the leading hypothesis, and they know which runtime and which component to profile.
  • It replaces guesswork with a sizing method. Sizing from working-set percentiles under peak load gives a number you can justify in review and recompute when traffic changes, instead of repeatedly nudging the limit after each crash.
  • It makes the next incident faster. Alerts on working set relative to the limit, plus tracking of OOMKilled reasons and restart counts, mean the next investigation starts with a captured profile and a trend line rather than an empty container and a restart counter.

The checklist also makes the work reusable. The triage script and PromQL panels can be kept as a shared runbook and dashboard, so the second team to hit an OOM kill does not rebuild the same queries from scratch.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kubernetes oomkilled debug, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Consumers learn about a breaking change at review time rather than from production errors.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.