Why do Java pods get OOMKilled even with -Xmx set?

-Xmx caps the Java heap and nothing else. A Kubernetes memory limit caps the whole container. The kubelet passes the limit to the container runtime, which writes it into the container's cgroup. From then on the Linux kernel enforces it: it charges every page the process touches to that cgroup. When usage hits the limit and the kernel cannot reclaim enough memory, the kernel's OOM killer sends SIGKILL to a process in the cgroup. The JVM gets no warning and no chance to run shutdown hooks or write a final log line. It is gone.

A java.lang.OutOfMemoryError is a different failure. The JVM throws it when it cannot satisfy an allocation inside one of its own managed pools, such as the heap, metaspace, or the space for direct buffers. It is an ordinary Throwable, so it produces a stack trace and travels up the call stack. Depending on how the JVM is configured, the process either keeps running in a degraded state or exits.

Signal Kernel OOM kill java.lang.OutOfMemoryError
Who acts Linux kernel, enforcing the cgroup limit The JVM itself
Pod status Last State: Terminated, Reason: OOMKilled in kubectl describe pod Usually Error or Completed, or still Running
Exit code 137 (128 + signal 9) Depends on JVM flags and application handling
Stack trace None Present in application logs
Logs End abruptly mid-stream Show the error and the allocating thread

An OOMKilled pod with a healthy-looking heap tells you something important. The memory went to parts of the JVM that -Xmx never limited. The limit has to cover the entire JVM process, not just the heap. Size the heap last, from whatever room is left over.

#!/usr/bin/env bash
set -euo pipefail

POD="orders-api-7d9f8c6b5-x2kqp"   # placeholder pod name
NS="payments"                       # placeholder namespace
C="app"                             # container name inside the pod

REASON=$(kubectl get pod "$POD" -n "$NS" \
  -o jsonpath="{.status.containerStatuses[?(@.name==\"$C\")].lastState.terminated.reason}")
CODE=$(kubectl get pod "$POD" -n "$NS" \
  -o jsonpath="{.status.containerStatuses[?(@.name==\"$C\")].lastState.terminated.exitCode}")
echo "last termination: reason=${REASON:-none} exitCode=${CODE:-none}"

LOGS=$(kubectl logs "$POD" -n "$NS" -c "$C" --previous 2>/dev/null || true)

if [[ "$REASON" == "OOMKilled" ]]; then
  # SIGKILL from the kernel OOM killer: 137 = 128 + 9, no stack trace.
  echo "Kernel OOM kill: the whole JVM process exceeded the container limit."
  echo "Last log lines before the kill (expect an abrupt end, no error):"
  echo "$LOGS" | tail -n 5
elif echo "$LOGS" | grep -q "java.lang.OutOfMemoryError"; then
  # The JVM itself failed an allocation in one of its managed pools.
  echo "JVM OutOfMemoryError: a managed pool ran out. Pool and thread:"
  echo "$LOGS" | grep -m 3 -A 5 "java.lang.OutOfMemoryError"
else
  echo "No memory-related termination found; check other causes."
fi

A script that reads the container's last termination reason and exit code and searches the previous container's logs, separating a kernel OOMKilled (exit 137, no stack trace) from a java.lang.OutOfMemoryError thrown by the JVM.

Who should use backward heap sizing?

Backward sizing means you start from the container limit, subtract the non-heap budget, and give the heap what remains. It is meant for engineers who run JVM services on Kubernetes and own their resource settings:

  • Service owners whose pods restart with OOMKilled while heap metrics look normal. This pattern points to non-heap memory, and backward sizing addresses that directly.
  • Platform teams that maintain base images or deployment templates for many JVM services. A shared method for deriving MaxRAMPercentage or -Xmx from a limit is easier to review than settings each team tunes separately.
  • Teams running thread-heavy or I/O-heavy services, such as Netty or gRPC servers, or applications that load many classes. Their non-heap footprint is large enough that the default ratios and rules of thumb stop working.
  • Anyone shrinking pods to cut cost. As limits get smaller, the fixed non-heap overhead takes up a larger share, and a percentage that was safe at a larger size can fail.

It matters less for short-lived batch JVMs that run once with generous limits, or for workloads outside containers where no cgroup limit applies. The underlying point still holds there: -Xmx is not the size of the process.

How does the JVM detect container memory and apply MaxRAMPercentage?

A container-aware JVM reads its memory limit and CPU quota from the cgroup the container runs in, not from the host's totals. Older runtimes saw the node's full RAM and core count, so they sized the heap and thread pools for a machine the pod could never use. Do not assume a legacy base image behaves correctly. Check what it actually detects.

The percentage flags are fractions of that detected limit:

  • MaxRAMPercentage sets the maximum heap.
  • InitialRAMPercentage sets the starting heap.
  • MinRAMPercentage has a misleading name. It is not a floor. It replaces MaxRAMPercentage when total memory is very small.

An explicit -Xmx (or -Xms) sets the size directly, and the JVM ignores the matching percentage flag.

The default MaxRAMPercentage is 25%. That value made sense on a shared server running many processes. In a pod running only the JVM, a 4 GiB limit gets a 1 GiB heap, and much of the rest goes unused. Teams then raise the limit when the heap is what needs to grow.

To see what the JVM decided, run these through kubectl exec:

  • java -XX:+PrintFlagsFinal -version lists effective flags. Look for MaxHeapSize, MaxRAMPercentage and UseContainerSupport.
  • java -XshowSettings:system -version reports the detected container memory limit and CPU count on Linux.
  • jcmd <pid> VM.flags and jcmd <pid> GC.heap_info show the settings of the running process.

Compare the results with the raw cgroup value in /sys/fs/cgroup/memory.max on cgroup v2, or /sys/fs/cgroup/memory/memory.limit_in_bytes on v1.

#!/usr/bin/env bash
set -euo pipefail

NS="payments"
POD="orders-api-7d9f8b6c4-x2k4q"
CTR="app"
EXEC=(kubectl exec -n "$NS" "$POD" -c "$CTR" --)

echo "== Raw cgroup memory limit (bytes; 'max' means unlimited) =="
"${EXEC[@]}" sh -c 'cat /sys/fs/cgroup/memory.max 2>/dev/null || cat /sys/fs/cgroup/memory/memory.limit_in_bytes'

echo "== Limits as the JVM sees them (Linux only) =="
"${EXEC[@]}" java -XshowSettings:system -version 2>&1

echo "== Effective heap and container flags =="
"${EXEC[@]}" java -XX:+PrintFlagsFinal -version 2>/dev/null \
  | grep -Ew 'MaxHeapSize|InitialHeapSize|MaxRAMPercentage|InitialRAMPercentage|MinRAMPercentage|UseContainerSupport'

echo "== Settings of the running application process =="
"${EXEC[@]}" jcmd 1 VM.flags
"${EXEC[@]}" jcmd 1 GC.heap_info

A shell script that reads the pod's raw cgroup memory limit, then prints the limits the JVM detected, the effective heap flags, and the running process's actual heap settings so you can compare them.

What uses JVM memory beyond the container heap?

-Xmx caps only the Java heap. The kernel counts everything the process maps and touches, and several of those regions have no default ceiling.

Region What makes it grow Bounded by default?
Metaspace Loaded classes, proxies, generated lambdas No, unless -XX:MaxMetaspaceSize is set
Code cache JIT-compiled methods Yes, -XX:ReservedCodeCacheSize
Thread stacks Thread count × -Xss Only indirectly, by thread count
GC structures Heap size, collector choice (G1 remembered sets, card tables) Scales with heap
Direct/NIO buffers ByteBuffer.allocateDirect, network I/O Defaults to roughly the max heap size
JNI/native libraries Compression, crypto, drivers No
malloc arenas Per-thread glibc arenas, fragmentation No, tunable via MALLOC_ARENA_MAX

Thread stacks deserve arithmetic. If you assume 200 threads at a 1 MB stack each, that is up to 200 MB (illustrative estimate). Pages are committed lazily, so the real figure is usually lower, but deep call stacks push it up.

Direct memory is easy to overlook. Netty and gRPC pool off-heap buffers for I/O, and Netty can allocate outside the -XX:MaxDirectMemorySize accounting. A busy proxy can hold more off-heap than on-heap memory.

glibc creates multiple malloc arenas for concurrent native allocation. On machines with many cores, the fragmentation across those arenas shows up as RSS that no JVM metric reports.

┌──────────── container memory limit ────────────┐
│ ┌──────────── Java heap (-Xmx) ──────────────┐ │
│ └────────────────────────────────────────────┘ │
│ [metaspace] [code cache] [thread stacks]       │
│ [GC structures] [direct/NIO buffers]           │
│ [JNI/native] [malloc arenas + fragmentation]   │
│ [ safety margin ]                              │
└────────────────────────────────────────────────┘
┌────────────────────── container memory limit ───────────────────────┐
│                                                                      │
│ ┌─────────────────── Java heap: capped by -Xmx ────────────────────┐ │
│ │  live objects + garbage awaiting collection                      │ │
│ └──────────────────────────────────────────────────────────────────┘ │
│                                                                      │
│ ┌────────────────── non-heap: not capped by -Xmx ──────────────────┐ │
│ │  region              grows with            bounded by            │ │
│ │  ──────────────────────────────────────────────────────────────  │ │
│ │  metaspace           classes, lambdas      MaxMetaspaceSize      │ │
│ │  code cache          JIT-compiled methods  ReservedCodeCacheSize │ │
│ │  thread stacks       threads x -Xss        thread count only     │ │
│ │  GC structures       remset, card tables   scales with heap      │ │
│ │  direct/NIO buffers  Netty/gRPC pools, NIO MaxDirectMemorySize*  │ │
│ │  JNI/native libs     compression, crypto   none                  │ │
│ │  malloc arenas       per-thread, fragments MALLOC_ARENA_MAX      │ │
│ │  * Netty can allocate outside MaxDirectMemorySize accounting     │ │
│ └──────────────────────────────────────────────────────────────────┘ │
│                                                                      │
│ ┌───────────────────────── safety margin ──────────────────────────┐ │
│ │  absorbs spikes and RSS that no JVM metric reports               │ │
│ └──────────────────────────────────────────────────────────────────┘ │
│                                                                      │
└────────────────────── exceed this -> OOMKilled ──────────────────────┘

The container limit must hold the -Xmx heap plus every non-heap region the kernel counts, several of which have no default ceiling, plus a safety margin before the OOM killer acts.

How does backward heap sizing work?

The usual approach picks a heap size first and then sets a container limit somewhat above it. Backward sizing does the opposite. The container limit is the fixed input, because it is what the kernel enforces. The heap is the output. The process has five steps:

  1. Fix the limit. Choose the container memory limit from the capacity you can afford per pod. The kernel enforces this number, so everything else must fit inside it.
  2. Estimate each non-heap region. Use the table above: thread stacks (thread count × -Xss), metaspace, code cache, direct buffers, GC structures, plus native libraries and allocator overhead. Begin with estimates, then replace them with Native Memory Tracking measurements as described below.
  3. Reserve a safety margin. It covers spikes and memory that no JVM metric reports, such as malloc fragmentation.
  4. Give the heap the remainder. Express it as MaxRAMPercentage (remainder ÷ limit) or as a fixed -Xmx.
  5. Enforce the estimates. Cap the regions that support caps (-XX:MaxMetaspaceSize, -XX:ReservedCodeCacheSize, -XX:MaxDirectMemorySize, -Xss, bounded thread pools). If one of them overflows, you get a JVM error with a stack trace instead of a silent kernel OOM kill.

Then verify. Run a soak test, compare NMT's committed total and the container working set against the budget, and correct whichever estimate was wrong. The next section works through steps 1–4 with assumed numbers, and the YAML after it carries out step 5.

How do you size the heap from the non-heap budget?

Size from the limit downward: subtract every native consumer, then give the heap whatever is left.

heap ceiling = limit − (thread stacks + metaspace + code cache + direct buffers + GC structures + safety margin)

Illustrative estimate (assumed numbers, not a benchmark): 2 GiB limit (2048 MiB), 200 threads × 1 MiB stacks = 200 MiB, metaspace ≈ 150 MiB, code cache ≈ 50 MiB, direct buffers ≈ 128 MiB, GC structures ≈ 100 MiB (assumed), safety margin ≈ 10% of the limit ≈ 200 MiB.

Non-heap total: 200 + 150 + 50 + 128 + 100 + 200 = 828 MiB. Heap ceiling: 2048 − 828 = 1220 MiB, so MaxRAMPercentage ≈ 1220 / 2048 ≈ 59%.

A 75% setting allows a 1536 MiB heap and leaves 512 MiB for native memory. That is less than the 628 MiB this service needs before any margin. Thread stacks are counted at full size here, which is the worst case. In practice not every stack page is committed. When the heap fills under load, the container can still go over its limit.

Approach Strength Weakness
MaxRAMPercentage Scales automatically when the limit changes Hides the non-heap budget, which does not grow with the limit
Fixed -Xmx Explicit and easy to review Must be updated by hand when the limit changes

A larger heap means fewer GC cycles, but it also means less native headroom. The cost is in how the service fails. If the heap is too small, you get an OutOfMemoryError with a stack trace and, if configured, a heap dump. If native memory is too small, the kernel OOM-kills the container and you get neither.

containers:
  - name: orders-api
    image: registry.example.com/orders-api:1.4.2
    resources:
      requests:
        memory: "2Gi"
      limits:
        memory: "2Gi"          # 2048 MiB, the number everything is subtracted from
    env:
      - name: JAVA_TOOL_OPTIONS
        value: >-
          -XX:MaxRAMPercentage=59.0
          -Xss1m
          -XX:MaxMetaspaceSize=150m
          -XX:ReservedCodeCacheSize=48m
          -XX:MaxDirectMemorySize=128m
          -XX:+HeapDumpOnOutOfMemoryError
          -XX:HeapDumpPath=/dumps
          -XX:NativeMemoryTracking=summary
        # 59% of 2048 MiB is about 1208 MiB of heap, just under the 1220 MiB ceiling.
        # -Xss1m keeps 200 threads within the 200 MiB stack budget.
        # The metaspace, code cache and direct buffer caps turn the non-heap
        # estimates into enforced limits instead of hopes.
        # Exceeding one of these caps gives a JVM error with a stack trace,
        # not a kernel OOM kill.
        # Heap dumps need a writable volume.
        # NMT has a small overhead. Use it to check the budget with:
        #   jcmd 1 VM.native_memory summary
        # Fixed-heap alternative: replace MaxRAMPercentage with -Xmx1200m.
        # It is explicit, but you must recompute it whenever the limit changes.
        # If both flags are set, -Xmx takes precedence.
    volumeMounts:
      - name: dumps
        mountPath: /dumps
volumes:
  - name: dumps
    emptyDir: {}

A Kubernetes container spec that turns the backward-sized budget (2 GiB limit, about 59% heap) into JVM flags: it caps each native consumer, keeps heap dumps enabled for heap failures, and enables Native Memory Tracking so you can check the non-heap estimate in a running pod.

How can you measure real non-heap usage instead of guessing?

Native Memory Tracking (NMT) replaces guesses with numbers. Start the JVM with -XX:NativeMemoryTracking=summary, then run jcmd <pid> VM.native_memory summary inside the container. The output breaks committed memory down by category, including heap, metaspace, threads, code cache, GC, and internal structures. To find growth over time, record a baseline with VM.native_memory baseline and later run summary.diff to see which category grew.

Compare NMT's total committed memory with the container working set that Kubernetes reports (container_memory_working_set_bytes, or kubectl top pod). That working set is the figure the kubelet uses for eviction decisions. A small, steady gap between the two is normal. A gap that keeps widening means memory is being allocated outside NMT's view. Common causes are native libraries loaded through JNI and allocator behavior.

One common example is glibc malloc arenas. On machines with many cores, glibc can create many per-thread arenas, and fragmentation inside them pushes RSS well above what the JVM reports. You can cap arenas with MALLOC_ARENA_MAX or switch to a different allocator. Both changes alter how the process behaves under load, so they need their own testing before you rely on them.

NMT also costs some performance, and detail mode costs noticeably more than summary. For that reason, many teams enable it only in staging or turn it on temporarily.

Take these measurements during a soak test under realistic load, and repeat it. A single run can miss traffic spikes, large payloads, class loading during rolling deploys, and slow native leaks that only show up after hours or days. Running this loop a few times usually costs less engineering time than investigating repeated OOMKilled incidents, which leave no heap dump and often no clear cause.

How should you set Kubernetes requests and limits for JVM workloads?

Treat memory as a fixed contract. Set the memory request equal to the limit. The scheduler then reserves exactly what the JVM can use, and the pod never runs above its request, so it is not among the first candidates when the kubelet evicts pods under node memory pressure. If CPU requests also equal CPU limits for every container in the pod, Kubernetes assigns the Guaranteed QoS class, which gives the strongest protection from eviction. Size the limit from heap max plus the measured non-heap budget. A limit only slightly above -Xmx gives the JVM no headroom for metaspace, threads and direct buffers.

CPU settings matter as much as memory. The JVM reads the container's CPU quota to choose GC thread counts, and on small CPU and memory allocations it may select the Serial collector instead of G1. Under a tight CPU limit, CFS throttling can pause the process partway through a GC cycle. Pauses get longer, readiness and liveness probes time out, and the restarts look like a memory problem. If you lower the CPU limit, set -XX:ActiveProcessorCount to match it.

Strategy Benefit Cost
Requests = limits (both) Guaranteed QoS, predictable GC Lower bin-packing density, higher cluster cost
Memory request < limit Cheaper packing Node-pressure evictions
No CPU limit No throttling during GC Noisy-neighbor contention

A typical spec requests and limits 2 CPU and 2Gi of memory, and runs with -XX:MaxRAMPercentage=70 and -XX:ActiveProcessorCount=2.

When should you avoid percentage-based sizing, and what are the alternatives?

Percentage-based sizing pays off most for teams running many JVM services on Kubernetes. One flag scales the heap with whatever limit a deployment sets, so resizing a pod never requires a second edit to JVM options. Thread-heavy and Netty/gRPC services also benefit, because their large native footprint forces the team to work out a non-heap budget explicitly.

It is the wrong default in two cases:

  • Very small containers. Non-heap overhead (metaspace, code cache, thread stacks, GC structures) does not shrink in proportion to the limit. With a 512Mi limit and 70%, the heap gets about 358Mi and everything else gets about 154Mi. That is an illustrative estimate, and many services need more than 154Mi just to start.
  • Services with highly variable native usage, such as bursty direct buffers or connection-driven thread creation. A fixed ratio cannot follow a native footprint that changes with load.
Approach Best for Trade-off
MaxRAMPercentage Many uniform services Fixed ratio assumes stable non-heap
Fixed -Xmx plus explicit caps Small or native-heavy pods Every limit change needs a matching flag change
-XX:MaxDirectMemorySize, bounded thread pools Netty/gRPC, thread-heavy apps Hitting a cap surfaces as errors instead of growth

Also choose how the service fails. -XX:+ExitOnOutOfMemoryError makes the JVM exit when it throws an OutOfMemoryError, so Kubernetes restarts the pod cleanly. Without it, the process can keep running with broken threads as a degraded zombie. -XX:+HeapDumpOnOutOfMemoryError writes a heap dump for diagnosis, but only if the pod has a writable volume large enough to hold it. Neither flag helps when the kernel OOMKills the container, because the JVM never gets a chance to react.

A documented sizing budget is cheaper to run. Reviewers can check new flags against it, and on-call engineers start from known numbers instead of guessing. You still need to monitor the working set and keep margins, because dependencies, traffic and JVM upgrades change the non-heap footprint over time.

How can backward sizing save engineering time?

Most of the time saved comes from failures that stop happening and from failures that become easier to diagnose.

  • Fewer undiagnosable incidents. A kernel OOM kill leaves no stack trace and no heap dump, so the investigation starts from almost nothing. When the native regions are capped and the heap is sized from the remainder, overflows tend to show up as JVM errors that name the pool and the thread. Those take far less effort to trace than a log that simply stops.
  • Less trial-and-error on limits. Without a budget, the usual response to OOMKilled is to raise the limit, redeploy, and wait to see whether it happens again. That can take several rounds and often wastes memory. A budget shows which region is short, so a single change targets the real cause.
  • Faster triage. The termination-reason script and the NMT comparison separate a heap problem from a native problem in minutes. Without them, engineers often spend that time tuning GC when the leak is in direct buffers or thread stacks.
  • Cheaper reviews and handoffs. A written budget turns "is MaxRAMPercentage=75 safe?" into a subtraction anyone can check. New team members and reviewers do not need to rebuild the reasoning from incident history.
  • Reusable templates. Platform teams can publish the method and default caps once, and each service fills in only its measured numbers.

Illustrative estimate (assumed inputs, not measured): assume a service has two OOMKilled incidents a month. Assume each takes an engineer about four hours to investigate, because there is no heap dump, plus one hour to adjust and redeploy. That comes to roughly ten engineer-hours a month. If a one-time budgeting effort takes eight hours (estimates, soak test, NMT review) and removes most of those incidents, it pays for itself within the first month. Your own numbers will differ, so substitute your incident rate and investigation time to see whether the effort is worth it for a given service.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to jvm memory kubernetes limits, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Consumers learn about a breaking change at review time rather than from production errors.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.