JVM Memory Kubernetes Limits: Sizing Heap, Off-Heap and OOMKills

How a container memory limit kills a JVM: OOMKilled vs OutOfMemoryError

A Kubernetes memory limit is enforced by the Linux kernel, not by Kubernetes or the JVM. The kubelet passes the limit to the container runtime, which writes it into the container's cgroup (memory.max on cgroup v2, memory.limit_in_bytes on v1). The kernel charges every page the process touches against that cgroup. When the cgroup cannot reclaim enough memory to stay under the limit, the kernel's OOM killer sends SIGKILL to a process in the cgroup. SIGKILL cannot be caught, so the JVM gets no chance to throw, log, or dump anything. The container exits with code 137 (128 + signal 9), and Kubernetes records the reason as OOMKilled.

A Java OutOfMemoryError is a different failure. The JVM throws it when one of its own regions is exhausted, most often the heap. The process is still alive, so it can log a stack trace and run hooks. This is also why -XX:+HeapDumpOnOutOfMemoryError never fires for an OOMKill: no Java OOME is ever raised.

To tell which one happened, run kubectl describe pod <pod> and read Last State: Terminated, Reason and Exit Code. You can also query the fields directly with kubectl get pod <pod> -o jsonpath='{.status.containerStatuses[*].lastState.terminated}'. Then check kubectl logs <pod> --previous for an OutOfMemoryError stack trace.

Copy#!/usr/bin/env bash
POD=orders-api-7d9f8c6b5-x2k4p   # placeholder pod name
NS=payments                      # placeholder namespace

kubectl describe pod "$POD" -n "$NS" | grep -A 6 'Last State'

kubectl get pod "$POD" -n "$NS" \
  -o jsonpath='{range .status.containerStatuses[*]}{.name}{"\t"}{.lastState.terminated.reason}{"\t"}{.lastState.terminated.exitCode}{"\n"}{end}'
kubectl get events -n "$NS" \
  --field-selector involvedObject.name="$POD" \
  --sort-by=.lastTimestamp

if kubectl logs "$POD" -n "$NS" --previous | grep -q 'java.lang.OutOfMemoryError'; then
  echo "Java OutOfMemoryError: check the heap dump path if -XX:+HeapDumpOnOutOfMemoryError is set"
else
  echo "No OOME in logs: if reason is OOMKilled, the kernel killed the JVM and no heap dump exists"
fi

A shell triage script that reads the last termination reason, exit code, events and previous-container logs to tell a kernel OOMKill (exit 137, no JVM output) apart from a Java OutOfMemoryError.

How it works: the sizing model in one pass

Sizing a JVM under a Kubernetes limit comes down to reconciling two accounting systems that do not know about each other.

1. The kernel counts pages, not objects. The cgroup charges every resident page the JVM process touches, along with page cache attributed to the cgroup. It does not distinguish heap from metaspace from a JNI library's malloc. When the total cannot be reclaimed below the limit, the process is killed.
2. The JVM bounds only some of its regions. At startup a container-aware JVM reads the cgroup limit and derives the maximum heap from it, either as a percentage or from an explicit -Xmx. Some off-heap regions have their own caps (code cache, direct buffers, optionally metaspace). Others have none: thread count, GC metadata, mapped files and native allocations.
3. The gap between them is where OOMKills come from. If heap max plus the real off-heap footprint stays under the limit, the JVM fails, if it fails at all, with a named OutOfMemoryError you can diagnose. If the sum exceeds the limit, the kernel kills the process first.

The method in the rest of this article follows from that model. Measure the off-heap footprint under realistic load. Set the limit to heap plus off-heap plus headroom. Cap the regions that can be capped, so their failures surface inside the JVM. Then choose requests, QoS and scaling signals that match how the JVM actually holds memory.

What the cgroup actually counts: heap plus everything else

The heap is only one tenant of the container. The kernel enforces the limit against the whole process: resident anonymous memory plus page cache charged to the cgroup. It has no knowledge of heap occupancy. A heap that is 40% full still counts at its full committed size once those pages have been touched.

Copy+------------------- container memory limit -------------------+
| Java heap ............ bounded: -Xmx / MaxRAMPercentage      |
| Metaspace ............ unbounded unless MaxMetaspaceSize set |
| Code cache ........... bounded: ReservedCodeCacheSize        |
| Thread stacks ........ bounded per thread (-Xss), not count  |
| GC data structures ... scales with heap and collector        |
| Direct/NIO buffers ... bounded: MaxDirectMemorySize          |
|                        (default tracks max heap size)        |
| Mapped files, page cache ... unbounded                       |
| JNI / native malloc, allocator arenas ... unbounded          |
+---------------------------------------------------------------+

Some flags only look like a bound. The default direct memory ceiling roughly matches the heap, so it does little to protect a container sized just above -Xmx. Thread stacks are capped per thread, but nothing limits how many threads you create. A service with 800 threads pays for 800 stacks, plus allocator arenas that glibc may create for threads doing native allocation.

Three kinds of services are the most exposed:

• Thread-heavy servers. Each thread adds stack memory and can add allocator overhead.
• Netty-based stacks. Netty pools direct buffers. Depending on its configuration, it can allocate outside the JVM's direct-memory accounting.
• Services with native libraries. Compression, crypto, image processing, and embedded databases call malloc and map files where the JVM cannot see or limit them.

GC choice also shifts the numbers. Region-based and concurrent collectors keep remembered sets, marking bitmaps, and other metadata whose size depends on heap size and tuning. No single off-heap percentage holds across services, so you have to measure each workload.

Copy+----------------- container memory limit (cgroup) -----------------+
|  kernel charges: resident anon memory + page cache, not heap use  |
|                                                                   |
|  +-------------------+   BOUNDED BY A FLAG                        |
|  | Java heap         |   -Xmx / MaxRAMPercentage (committed size) |
|  +-------------------+                                            |
|  | Code cache        |   ReservedCodeCacheSize                    |
|  +-------------------+                                            |
|  | Direct/NIO bufs   |   MaxDirectMemorySize, default ~ max heap  |
|  +-------------------+                                            |
|  | Thread stacks     |   -Xss per thread; thread count unbounded  |
|  +-------------------+                                            |
|  | GC structures     |   no direct flag; scales with heap, GC     |
|  +-------------------+   NOT BOUNDED BY DEFAULT                   |
|  | Metaspace         |   unless MaxMetaspaceSize is set           |
|  +-------------------+                                            |
|  | Mapped files,     |   grows with files mapped and read         |
|  | page cache        |                                            |
|  +-------------------+                                            |
|  | JNI / malloc,     |   outside all JVM flags                    |
|  | allocator arenas  |                                            |
|  +-------------------+                                            |
|                                                                   |
+-------------------------------------------------------------------+
     ^                       ^                         ^
     |                       |                         |
  thread-heavy          Netty direct buffers     native libraries
  (stacks, arenas)      (may skip JVM count)     (compression, crypto)

The cgroup limit covers the heap plus every off-heap region, but JVM flags cap only some of them, and thread-heavy, Netty-based and native-library services put the most pressure on the regions with no flag.

How the JVM sizes its heap from the container: MaxRAMPercentage, InitialRAMPercentage and -Xmx

Modern JVMs are container-aware: at startup they read the cgroup memory limit and treat it as physical RAM. Older JVMs read the host's total memory instead, so a pod limited to a few gigabytes on a large node could size its heap far beyond what the cgroup allowed. If you still run legacy runtimes, verify this before trusting any percentage. Even when the JVM is container-aware, the default heap percentage is conservative. It leaves headroom on purpose, and that headroom is often more than a given service actually needs.

-XX:MaxRAMPercentage sets the maximum heap as a share of the detected limit, and -XX:InitialRAMPercentage sets the starting heap the same way. An explicit -Xmx (or -Xms) takes precedence over both. Set these in the pod's env through JAVA_TOOL_OPTIONS, which the JVM picks up without any change to the entrypoint. To confirm what took effect, run java -XX:+PrintFlagsFinal -version inside the container and check MaxHeapSize.

Raising the percentage gives you more heap but shrinks the room left for the off-heap regions described above.

CopyapiVersion: v1
kind: Pod
metadata:
  name: orders-api
spec:
  containers:
    - name: app
      image: registry.example.com/orders-api:1.0.0
      resources:
        requests:
          memory: "2Gi"
        limits:
          # The JVM reads this cgroup limit and treats it as physical RAM.
          memory: "2Gi"
      env:
        # Picked up by the JVM at startup; no entrypoint change needed.
        # Max heap = 75% of the limit, leaving the rest for metaspace,
        # thread stacks, code cache, direct buffers and native allocations.
        # Do NOT also set -Xmx here: it would override both percentages.
        - name: JAVA_TOOL_OPTIONS
          value: "-XX:MaxRAMPercentage=75.0 -XX:InitialRAMPercentage=50.0"

A pod spec that sets percentage-based heap sizing through JAVA_TOOL_OPTIONS, with a kubectl command to print the effective MaxHeapSize from inside the container.

How to implement a sizing method: measure off-heap, then derive the limit

Step 1: Measure off-heap under realistic load

Start the JVM with -XX:NativeMemoryTracking=summary and drive it with production-like traffic, including the code paths that load classes, spin up threads, or use direct buffers. After warm-up, record a baseline with jcmd <pid> VM.native_memory baseline, keep the load running, then compare with jcmd <pid> VM.native_memory summary.diff. The diff shows which regions (metaspace, threads, code cache, GC structures, internal) are still growing and which have flattened. Record both the steady-state and the peak value.

Compare NMT's committed total against the process RSS and the container working set. A gap usually comes from memory NMT cannot see, such as allocator arenas or JNI libraries calling malloc directly, so you need to budget for it. NMT also uses memory and CPU of its own. Use it for measurement runs and leave it off by default in production.

Step 2: Derive the limit

limit = target heap + measured off-heap + headroom

Then either set -Xmx to the heap value or set MaxRAMPercentage = heap / limit.

Illustrative estimate (assumed inputs, not measured): 2 GiB heap, 600 MiB off-heap, 15% headroom. (2048 + 600) × 1.15 ≈ 3,045 MiB, which rounds to a 3 GiB limit. 2048 / 3072 ≈ 66.7%, so MaxRAMPercentage would be about 66.

One load test can miss slow native leaks that only show up after days of uptime, so keep watching RSS trends after rollout. Be careful about padding the headroom to compensate. Every extra 512 MiB is multiplied by the replica count and reserved on every node.

Copy#!/usr/bin/env bash
set -euo pipefail
PID="${1:?usage: $0 <java-pid>}"

jcmd "$PID" VM.native_memory baseline

sleep 1800
jcmd "$PID" VM.native_memory summary.diff scale=MB > nmt-diff.txt

jcmd "$PID" VM.native_memory summary scale=MB | grep -m1 'Total'
grep VmRSS /proc/"$PID"/status
CUR=$(cat /sys/fs/cgroup/memory.current)
INACTIVE=$(awk '/^inactive_file /{print $2}' /sys/fs/cgroup/memory.stat)
echo "working set MiB: $(( (CUR - INACTIVE) / 1048576 ))"

HEAP_MIB=2048
OFFHEAP_MIB=600
HEADROOM=1.15
awk -v h="$HEAP_MIB" -v o="$OFFHEAP_MIB" -v r="$HEADROOM" 'BEGIN {
  raw = (h + o) * r                      # ~3045 MiB
  limit = int((raw + 255) / 256) * 256   # round up to a 256 MiB step -> 3072
  printf "raw: %.0f MiB, limit: %d MiB\n", raw, limit
  printf "MaxRAMPercentage ~ %.1f (or keep -Xmx%dm)\n", h / limit * 100, h
}'

Illustrative script that baselines and diffs NMT under load, compares NMT's committed total with RSS and the cgroup working set, then computes limit = (heap + off-heap) × headroom and the matching MaxRAMPercentage using assumed inputs.

Capping off-heap regions so failures become diagnosable

Capping off-heap regions bounds that uncertainty. The point is not to save memory. A capped region fails inside the JVM, where the error has a name and a stack trace. An uncapped region keeps growing until the kernel sends SIGKILL, and that leaves you nothing to debug.

An example flag set for a service with a 1 GiB heap might be -XX:MaxMetaspaceSize=256m -XX:MaxDirectMemorySize=256m -XX:ReservedCodeCacheSize=128m -Xss512k. Add -XX:+HeapDumpOnOutOfMemoryError with a dump path on a mounted volume. Note that the dump fires only for errors the VM raises itself, such as heap or Metaspace exhaustion. Direct buffer errors are thrown from library code and do not trigger it. -Xss caps only the stack of each thread. Total stack memory stays bounded only if thread pools are bounded too.

Tight caps turn normal load into outages. Code cache is the hardest to spot, because nothing crashes and the only symptom is gradual slowdown. Set each cap from the peak you measured in the previous step, plus a margin. Do not use a round number you guessed.

Requests, limits, QoS and autoscaling for JVM workloads

Setting the memory request equal to the limit gives a predictable pod. The scheduler reserves exactly what the container may use, so a node cannot be promised more memory than it physically has. If the CPU request and limit are also equal for every container in the pod, it gets the Guaranteed QoS class. When a node comes under memory pressure, the kubelet evicts pods that exceed their requests first, and Guaranteed pods are last in line. For a resources block, set requests.memory and limits.memory to the same value, for example 2Gi, and do the same for CPU.

A JVM container can report memory close to its limit while the live heap stays flat. Once the heap has grown, many collectors keep the freed pages committed instead of promptly returning them to the OS. This makes a memory-based HPA misleading: it scales out on memory that will never drop, and new pods do not lower usage on the existing ones. Working-set alerts fire for the same reason.

Better signals are:

• GC time as a fraction of wall time.
• Heap occupancy after a full collection.
• Request latency or CPU.

These need a metrics exporter that exposes JVM memory-pool and GC data. To scale on them, you also need a custom-metrics adapter, because the HPA cannot read these metrics on its own.

When to use measured JVM memory sizing

Use the measure-then-derive method when at least one of these is true:

• Pods restart with OOMKilled and exit code 137, especially intermittently or only after hours or days of uptime. This is the clearest sign that heap max plus off-heap exceeds the limit.
• The service has a large or unpredictable off-heap footprint. Typical cases are thread-heavy servers, Netty or other direct-buffer-heavy I/O stacks, services that load many classes or redeploy plugins, and anything using JNI libraries for compression, crypto, image processing or embedded storage.
• Per-replica memory is a real cost. If a service runs many replicas, the headroom you add is multiplied across all of them. Measuring off-heap lets you replace guessed padding with a figure grounded in the workload.
• You want Guaranteed QoS with request equal to limit. That setup reserves the full limit on the node, so an accurate limit matters more than it does for a Burstable pod.
• You change something that shifts native memory: a new garbage collector, a JDK upgrade, a new native dependency, or a large change in thread-pool sizes. Each of these invalidates the previous measurement.

When to avoid it

The full procedure is not worth the effort everywhere:

• Small, low-traffic services with generous limits and no OOMKill history. A container-aware JVM with a sensible MaxRAMPercentage is usually enough, and load testing with NMT costs more than it saves.
• Short-lived batch jobs and one-off tasks that finish before slow native growth matters. Size them from a single observed run and a retry policy instead.
• When you cannot produce realistic load. NMT numbers from an idle or synthetic run understate peak off-heap usage. A limit derived from them is falsely precise and can be worse than a deliberately generous one. Fix the load test first, or fall back to vertical right-sizing from production observation.
• As a substitute for fixing a leak. If heap occupancy after full collections keeps climbing, or one NMT category grows without bound, re-deriving the limit only delays the next kill. Find and fix the leak first.
• Leaving NMT on permanently in production. It adds its own memory and CPU overhead. Enable it for measurement runs or targeted investigations, then turn it off.
• On JVMs without HotSpot's Native Memory Tracking. The jcmd VM.native_memory steps here are HotSpot-specific. On other JVM implementations, use their own diagnostic tooling, or fall back to comparing RSS and cgroup working set over time.

Diagnosing recurring OOMKills, alternatives and impact on engineering time

Diagnosis checklist

When a pod keeps getting OOMKilled, work through these in order. Each one rules out a different region.

1. Was the heap near its maximum? Check GC logs and heap-pool metrics just before the kill. If old-generation occupancy stayed high after collections, the real problem is heap pressure. Fix that with a heap or code change, not a bigger limit.
2. Did native memory grow over time? With Native Memory Tracking enabled, take jcmd <pid> VM.native_memory baseline after warm-up. Later, run summary.diff. Steady growth in one category, such as Class, Thread or Internal, points to the culprit.
3. Did thread count or direct buffers spike? Each thread adds its own stack. Direct and mapped buffers live outside the heap and are freed only when their owning objects are collected.
4. Do NMT totals reconcile with RSS? If RSS is well above NMT's committed total, the gap is memory the JVM does not track. Common sources are JNI libraries and glibc malloc arena fragmentation. Limiting arenas with MALLOC_ARENA_MAX or switching allocators are the usual fixes.

Who needs this

Apply this method to services with intermittent OOMKills or tight per-replica memory budgets. For small, low-traffic services with generous limits, percentage-based defaults are usually fine.

Alternatives

• Fixed generous limits. Simple to apply, but they hide leaks until the leak outgrows the extra room.
• Percentage-only defaults. These ignore off-heap growth.
• Vertical right-sizing from observed usage. This works well, provided the observation window includes peak load.
• Changing the GC or allocator. Collectors differ in how much native overhead they add and how readily they return memory to the OS. Re-measure after any switch.

Cost and time

Measured sizing takes upfront work: load tests and extra instrumentation. In return, it reduces repeated incident triage. It also cuts headroom that would otherwise be over-provisioned on every replica.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to jvm memory kubernetes limits, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.