What is the difference between a Java OutOfMemoryError and an OOMKilled pod?

A java.lang.OutOfMemoryError and an OOMKilled pod are different failures with different fixes. Diagnose which one occurred before you change any setting.

An OutOfMemoryError is thrown inside the JVM when one specific memory area is exhausted. Common causes include Java heap space, Metaspace, Direct buffer memory and unable to create native thread. Because it is an ordinary Throwable, the JVM can log a stack trace, write a heap dump if configured to, and keep running in a degraded state. The process may stay alive, so Kubernetes may not notice anything is wrong.

OOMKilled comes from outside the JVM. The container's cgroup exceeded its memory limit, and the kernel's OOM killer sent SIGKILL. A process cannot catch SIGKILL, so there is no stack trace, no heap dump, no shutdown hook and no final log line. The JVM's own memory areas may all have been within their limits. What crossed the limit was the total resident memory of the process.

Signal Java OutOfMemoryError OOMKilled
Who acts JVM Linux kernel (cgroup OOM killer)
Process after event May keep running Terminated immediately
Stack trace / heap dump Yes, if logging and dump options allow Never
kubectl describe pod No special reason unless the JVM exits Last State: Terminated, Reason: OOMKilled
Exit code Depends on how the app exits 137 (128 + signal 9)

To check, run kubectl describe pod and read the Last State block of the restarted container. Exit code 137 alone only tells you the container received SIGKILL. A container killed after its termination grace period also exits with 137, but its reason shows Error. Only Reason: OOMKilled confirms a cgroup kill.

Then run kubectl logs --previous and search for OutOfMemoryError. If you find it, size the specific JVM memory area that ran out. If the logs simply stop, the cause is the container limit.

#!/usr/bin/env bash
# Tell an OOMKilled container apart from a Java OutOfMemoryError.
# Placeholder names: replace with your own pod, container and namespace.
POD=orders-api-7c9d8f6b5-x2k4q
CONTAINER=app
NS=payments

# 1. Human-readable view: look for the 'Last State: Terminated' block
#    of the restarted container.
kubectl describe pod "$POD" -n "$NS" | grep -A 6 'Last State'

# 2. Machine-readable view of the same fields, one line per container.
#    Reason=OOMKilled means the cgroup limit was hit and the kernel sent SIGKILL.
#    Exit code 137 with Reason=Error is a plain SIGKILL, e.g. after the
#    termination grace period expired, not a memory kill.
kubectl get pod "$POD" -n "$NS" -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"\t"}{.restartCount}{"\n"}{end}'

# 3. Logs from the previous (killed) instance. After an OOMKill these
#    simply stop: no stack trace, no shutdown hook output.
kubectl logs "$POD" -c "$CONTAINER" -n "$NS" --previous --tail=50

# 4. Search both the previous and the current instance for a JVM-level error.
#    A match here means a specific JVM memory area (heap, metaspace,
#    direct buffers, threads) ran out, which is a different problem.
for flag in --previous ""; do
  kubectl logs "$POD" -c "$CONTAINER" -n "$NS" $flag 2>/dev/null \
    | grep -n -A 3 'java.lang.OutOfMemoryError'
done

Shell commands that read a container's last termination reason and exit code, then search previous and current logs for java.lang.OutOfMemoryError to tell which failure happened.

How does the JVM use Kubernetes memory limits to size the heap?

When a container-aware JVM starts, it reads the memory limit from the cgroup filesystem (the file differs between cgroup v1 and v2) and treats that value as the machine's physical memory. Kubernetes writes the pod's resources.limits.memory into that cgroup, so the container limit becomes the input to heap ergonomics. The JVM doesn't query the API server. It sees only what the kernel enforces.

MaxRAMPercentage and InitialRAMPercentage are percentages of that detected limit, not of node memory. With a 2 GiB limit and the default MaxRAMPercentage of 25, the maximum heap is about 512 MiB, whether the node has 16 GiB or 256 GiB. InitialRAMPercentage works the same way and sets the starting heap.

Older JVMs were not container-aware. They read the host's total memory and sized the heap from the node. Several pods on a large node could each pick a heap larger than their limit, and the kernel killed them under load.

The 25 percent default is conservative because the JVM can't know how much non-heap memory your workload needs. The default protects against OOMKills but often leaves most of the limit idle.

+------------------------- container limit (cgroup) -------------------------+
| Java heap (MaxRAMPercentage x limit) | metaspace | code cache | thread      |
|                                      |           |            | stacks      |
|                                      | direct buffers | GC structures | native |
+----------------------------------------------------------------------------+

Everything to the right of the heap still counts against the same limit, and that shared budget is where -Xmx sizing goes wrong.

# Compare the cgroup memory limit Kubernetes set with the heap the JVM derived from it.
# Placeholder pod name; the container must ship a JVM on its PATH.
POD=orders-api-7d9f6c5b8-x2k4q

# 1. The limit the kernel enforces. cgroup v2 exposes memory.max;
#    cgroup v1 exposes memory/memory.limit_in_bytes. Try both.
kubectl exec "$POD" -- sh -c '
  if [ -f /sys/fs/cgroup/memory.max ]; then
    echo "cgroup v2 limit: $(cat /sys/fs/cgroup/memory.max)"
  else
    echo "cgroup v1 limit: $(cat /sys/fs/cgroup/memory/memory.limit_in_bytes)"
  fi'

# 2. Heap ergonomics with defaults. MaxHeapSize and InitialHeapSize are in bytes
#    and should be roughly MaxRAMPercentage / InitialRAMPercentage of the limit
#    above, not of the node's memory.
kubectl exec "$POD" -- sh -c \
  'java -XX:+PrintFlagsFinal -version 2>/dev/null \
     | grep -E " (MaxRAMPercentage|InitialRAMPercentage|MaxHeapSize|InitialHeapSize) "'

# 3. Same container, raised percentages. The limit is unchanged; only the share
#    given to the heap grows, leaving less for metaspace, threads and native memory.
kubectl exec "$POD" -- sh -c \
  'java -XX:MaxRAMPercentage=75 -XX:InitialRAMPercentage=50 \
        -XX:+PrintFlagsFinal -version 2>/dev/null \
     | grep -E " (MaxHeapSize|InitialHeapSize) "'

# Human-readable summary of the max heap the JVM picked.
kubectl exec "$POD" -- java -XshowSettings:vm -version

Shell commands that read the pod's cgroup memory limit and print the JVM's derived heap flags, showing that MaxRAMPercentage and InitialRAMPercentage scale with the container limit rather than node memory.

Why does setting -Xmx equal to the container limit cause OOMKills?

Setting -Xmx to the container limit breaks the budget. -Xmx caps only the Java heap. The kernel's cgroup accounting counts every page the process touches, and a running JVM always touches memory outside the heap:

Region What drives its size Capped by -Xmx?
Metaspace Number of loaded classes, including generated proxies No
Code cache Amount of JIT-compiled code No
Thread stacks Thread count × stack depth actually used No
GC structures Heap size and collector choice (card tables, remembered sets) No
Direct/NIO buffers Network and I/O libraries using off-heap buffers No
Memory-mapped files Mapped pages that are resident No
JIT compiler arenas Compilation activity, which spikes during warm-up No
JNI / malloc Native libraries for compression, crypto or drivers No

If -Xmx equals the limit, the heap can grow to fill the entire budget, and every other region pushes the process over. The JVM never throws OutOfMemoryError, because from its point of view the heap still has room. The kernel kills the process instead.

A subtler cause is glibc malloc. In applications with many threads, glibc gives threads separate allocation arenas. Freed memory inside those arenas often goes unused but is not returned to the OS. RSS climbs slowly while heap graphs stay flat. Limiting arenas with MALLOC_ARENA_MAX, or switching to an alternative allocator, can reduce this growth. Both change performance characteristics, so test them under production-like load and compare RSS before you adopt either.

The heap is one tenant in a shared container. An OOMKill shows that the total footprint was not accounted for, not that the heap was too small.

How do you size JVM memory limits from measured RSS?

Start by running the service under realistic load: production-like concurrency, payload sizes and traffic mix, for long enough to reach steady state and go through several full GC cycles. Idle footprints say little, because thread pools, caches and JIT-compiled code grow with traffic.

Watch two numbers. The kubelet reports the working set, which is cgroup memory usage minus inactive file pages. It uses this figure for eviction and for kubectl top. The OOMKill comes from the kernel when the cgroup's charged memory reaches the limit and the kernel cannot reclaim enough of it. Page cache is charged to the container. Inactive cache is usually reclaimed first, but active file pages, such as heavily read logs or memory-mapped files, can keep the working set high.

Next, add -XX:NativeMemoryTracking=summary and run jcmd <pid> VM.native_memory summary at peak load. This breaks committed memory down into heap, metaspace, threads, code cache, GC and other internal areas. NMT has a small performance and memory cost of its own. It also cannot see memory that native libraries allocate outside the JVM's accounting, so its total will not match RSS. Treat the gap between the two as a separate budget item.

Set the limit to the observed peak working set plus headroom. Then give the heap whatever is left after the non-heap budget.

Component (illustrative estimate, assumed inputs) Budget
Container limit 2048 MiB
Heap 1229 MiB (1.2 GiB)
Metaspace 160 MiB
Code cache 80 MiB
Thread stacks (200 threads × 1 MiB) 200 MiB
Direct buffers 128 MiB
GC structures and other native 100 MiB
Headroom 151 MiB

These figures are assumptions for illustration, not measurements. Repeat the measurement whenever concurrency, dependencies or JVM flags change, because each of these can shift the non-heap footprint without touching -Xmx.

#!/usr/bin/env bash
# Sample memory metrics for one JVM pod at peak load. Run it repeatedly during a load test.
# Assumes cgroup v2, JVM running as PID 1, jcmd present in the image, and the JVM started with:
#   -XX:NativeMemoryTracking=summary
set -euo pipefail
POD="${1:?usage: $0 <pod> [namespace]}"
NS="${2:-default}"
k() { kubectl exec -n "$NS" "$POD" -- "$@"; }

# 1. What the kubelet reports: working set, used for eviction and kubectl top.
kubectl top pod "$POD" -n "$NS" --containers

# 2. Raw cgroup numbers. The kernel OOM-kills based on charged memory (memory.current),
#    which includes page cache, not on the working set.
CURRENT=$(k cat /sys/fs/cgroup/memory.current)
k cat /sys/fs/cgroup/memory.stat > /tmp/memstat.$$
INACTIVE_FILE=$(awk '$1=="inactive_file"{print $2}' /tmp/memstat.$$)
ACTIVE_FILE=$(awk '$1=="active_file"{print $2}' /tmp/memstat.$$)
ANON=$(awk '$1=="anon"{print $2}' /tmp/memstat.$$)
rm -f /tmp/memstat.$$
mib() { echo $(( $1 / 1024 / 1024 )); }
echo "memory.max:      $(k cat /sys/fs/cgroup/memory.max)"
echo "memory.current:  $(mib "$CURRENT") MiB"
echo "working set:     $(mib $(( CURRENT - INACTIVE_FILE ))) MiB  (current - inactive_file)"
echo "anon:            $(mib "$ANON") MiB"
echo "active_file:     $(mib "$ACTIVE_FILE") MiB  (page cache that is hard to reclaim)"

# 3. Process RSS, for comparison with the NMT total below.
k grep VmRSS /proc/1/status

# 4. JVM view: committed memory by area (Java Heap, Class, Thread, Code, GC, Internal...).
#    The NMT total will not equal RSS. Memory that native libraries malloc outside
#    JVM accounting is missing, so budget the gap separately.
k jcmd 1 VM.native_memory summary

# Once the peak is known, pin each budget explicitly (illustrative values matching the table):
#   -Xmx1229m -Xss1m -XX:MaxMetaspaceSize=160m
#   -XX:ReservedCodeCacheSize=80m -XX:MaxDirectMemorySize=128m
# Re-run this script after any change to concurrency, dependencies or JVM flags.

A shell script that captures the kubelet working set, raw cgroup v2 counters, process RSS and the Native Memory Tracking summary from a running JVM pod, so you can compare them at peak load before setting the limit and heap.

Should you use MaxRAMPercentage or an explicit -Xmx?

Both flags set the maximum heap. They fail in different ways. -XX:MaxRAMPercentage sizes the heap as a fraction of the container limit, so it scales when the limit changes. Non-heap memory does not scale with it. Metaspace, thread stacks, code cache and GC structures stay roughly constant for a given service, so a percentage that is safe at 2 GiB can be unsafe at 512 MiB.

Illustrative estimate: assume 300 MiB of non-heap overhead. At 75% of a 2 GiB limit, the heap is 1,536 MiB and the total is about 1,836 MiB, which fits. At 75% of 512 MiB, the heap is 384 MiB and the total is about 684 MiB, which gets the pod OOMKilled.

Approach Scales with limit Main risk
MaxRAMPercentage Yes Too high for small containers
Explicit -Xmx No Falls out of sync when the limit changes
Percentage with a minimum limit Yes Needs a documented floor

The percentage also sets a throughput trade-off. A higher value means fewer GC cycles but less room for native memory spikes. A lower value is safer but leaves memory unused that you still pay for.

Recommended convention: set MaxRAMPercentage once in the team's base image through JAVA_TOOL_OPTIONS. Choose the value from the measured overhead of your smallest supported container size, and set a minimum memory limit below which services must request an exception. Do not use -Xmx in the base image. If a service sets -Xmx, it overrides the percentage and should be treated as a reviewed exception.

With one convention in the base image, most services need no heap tuning. Resizing a service becomes a change to the limit alone. Reviewers only need to examine the exceptions.

# Team base image: every JVM service builds FROM this image.
# Placeholder registry and tag; use your organization's approved JDK image.
FROM registry.example.com/base/jdk:21

# Heap convention, set once for all services.
# 60.0 is a placeholder. Derive the real value from the non-heap overhead
# measured at the smallest supported limit (MIN_MEMORY_LIMIT below).
# The base image deliberately sets no -Xmx. A service that adds -Xmx
# overrides the percentage and needs a reviewed exception.
# ExitOnOutOfMemoryError makes a heap OOM exit the JVM so Kubernetes
# restarts the pod, instead of the JVM running on in a degraded state.
ENV JAVA_TOOL_OPTIONS="-XX:MaxRAMPercentage=60.0 -XX:+ExitOnOutOfMemoryError"

# Records the floor the percentage was validated against.
# Services that need a smaller limit must request an exception.
LABEL org.example.jvm.min-memory-limit="768Mi"

# Run as non-root by default.
USER 10001

# To check the effective heap inside a running pod:
#   kubectl exec <pod> -- java -XX:+PrintFlagsFinal -version | grep -E 'MaxHeapSize|MaxRAMPercentage'
# MaxHeapSize should be about 60% of the container memory limit.
# The JVM also prints a 'Picked up JAVA_TOOL_OPTIONS' line, which confirms
# that the base image setting reached the service.

A shared base-image Dockerfile that sets MaxRAMPercentage once through JAVA_TOOL_OPTIONS, records the minimum validated memory limit, and leaves -Xmx to reviewed per-service exceptions.

When should you cap metaspace, direct memory and thread stacks?

Heap is the only region the JVM sizes from the container limit. Metaspace has no upper bound by default. Direct memory defaults to roughly the maximum heap size. Thread stack memory grows with every thread you start. If one of these regions grows, the kernel kills the process with no Java-level signal. Capping a region changes that failure mode: the JVM throws OutOfMemoryError: Metaspace or OutOfMemoryError: Direct buffer memory instead. That error names the region, can trigger a heap dump, and appears in your logs.

Set caps from measured peaks plus headroom, not from guesses:

  • -XX:MaxMetaspaceSize fits services with a stable set of classes. A steadily rising metaspace in such a service usually means a classloader leak, and a cap exposes it early.
  • -XX:MaxDirectMemorySize matters most for NIO-heavy services. Without it, direct buffers can grow as large as the heap, outside the heap's accounting.
  • -Xss sets the per-thread stack size, but bounding thread pools is usually the better control. Total stack memory scales with thread count, so a pool that grows without limit under load outruns any per-thread setting.
  • -XX:ReservedCodeCacheSize bounds JIT-compiled code.

Avoid tight caps in the following cases:

  • Heavy class loading. Applications that generate proxies, compile scripts or redeploy plugins at runtime need more metaspace as they run. A tight cap turns that normal growth into errors.
  • Large-buffer frameworks. Networking and messaging libraries that pool direct buffers can fail under peak load if the direct memory cap is small.
  • The code cache. When it fills, the JVM stops compiling new methods. Throughput degrades gradually and no error is thrown, which makes an undersized code cache harder to spot than an exception.

How should you set memory requests, limits and QoS for JVM pods?

Memory is incompressible. The kernel can throttle a container that wants more CPU, but it cannot take back memory that is already in use. The only ways to reclaim it are to kill a process or evict a pod. When a node runs short of memory, the kubelet ranks pods for eviction first by whether their usage exceeds their request. If you set the memory request equal to the limit, the pod can never be over its request, so it is not picked for that reason. It also keeps capacity planning honest, because the scheduler reserves what the pod can actually consume. The cost is density: you pay for headroom that sits idle, whereas overcommitting packs more pods onto each node and accepts some eviction risk.

Configuration QoS class Failure mode under pressure
Requests = limits for CPU and memory in every container Guaranteed Container OOMKilled at its own limit
Any request or limit set, not all equal Burstable Container OOMKill, or eviction if usage is above the request
No requests or limits BestEffort Evicted first

Removing memory limits does not remove the failure. It moves it to the node-level OOM killer or to eviction, where neighbouring pods are affected and the cause is harder to attribute.

CPU limits matter for memory too. The JVM derives its active processor count from the CPU quota, and that count sizes GC thread pools and JIT compiler threads. With a very low count, ergonomics may select the Serial collector. You can override the count with -XX:ActiveProcessorCount.

The platform team should own the defaults and the LimitRange policies. Each service team should own its per-app values, based on the RSS it has measured.

How do you make JVM OOM events diagnosable in Kubernetes?

Two groups need different signals, because the two failure modes from section 1 leave different evidence.

Java-level OOMs: capture the heap

Add -XX:+HeapDumpOnOutOfMemoryError and point -XX:HeapDumpPath at a writable volume. The container root filesystem is often read-only, and anything written there disappears with the container. A disk-backed emptyDir survives container restarts within the same pod. A persistent volume also survives pod rescheduling. Do not use a memory-backed emptyDir, because its contents count against the container's memory and can turn a heap OOM into an OOMKill.

Heap dumps have three costs:

  • Disk space. A dump needs roughly as much space as the heap, so size the volume and any ephemeral-storage limit to match.
  • Pauses. Writing the dump can stall the process long enough to fail liveness probes, and the kubelet may restart the container before the file is complete.
  • No coverage of kernel kills. When the kernel OOM killer acts, the JVM receives SIGKILL and writes nothing.

Kernel OOMKills: alert and trend

For kernel kills, the evidence comes from Kubernetes itself. Alert when a container restarts with a last termination reason of OOMKilled. Restart counts alone mix OOMKills with crashes and probe failures, so filter on the reason.

Also graph the container's working set against its limit for each pod over days. Slow native growth from direct buffers, metaspace, or thread creation shows up as a steady upward slope while heap usage stays flat. That pattern lets you act before the kill happens.

Why this saves engineering time

With these signals in place, each incident starts with a classification instead of guesswork. A heap dump means the investigation is about application objects. An OOMKilled reason combined with a flat heap means the problem is native memory or limit sizing. Engineers can go straight to the right tools without first reproducing the failure.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to jvm kubernetes memory limits, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.