How do you know you have a memory leak and not an undersized heap?
In a JVM, a memory leak means objects stay reachable from a GC root after the application has stopped needing them. The collector cannot free them, so live data keeps growing. A leak is a different problem from a heap that is simply too small for its workload, and heap dumps are expensive to capture and analyze, so check the GC logs before taking one.
On JDK 9 and later, enable unified logging with -Xlog:gc* and send it to a file with rotation. JDK 8 uses different flags, such as -XX:+PrintGCDetails and -Xloggc. In the logs, ignore peak usage and track heap occupancy after each full GC or old-generation collection. That number approximates your live set.
- Sawtooth returning to the same floor: this is healthy. The application allocates, the collector reclaims, and the live set stays stable.
- Floor rising steadily over hours or days: something is accumulating, and that justifies a heap dump.
| Symptom in GC logs | Likely cause | Next step |
|---|---|---|
| Post-GC floor climbs steadily | Memory leak | Heap dump analysis |
| Floor flat but close to max, frequent full GCs | Undersized heap | Increase -Xmx, re-measure |
| Full GCs or allocation failures with free space reported | Fragmentation or humongous objects | Review collector choice and large allocations |
| Frequent young GCs, stable floor | High allocation rate | Allocation profiling |
| Process RSS grows, post-GC heap flat | Native memory | Native Memory Tracking |
The last row matters. When RSS grows while the post-GC heap stays flat, the growth is outside the Java heap. Likely sources are direct ByteBuffers, metaspace, thread stacks, or native libraries. A heap dump will not show this growth. Enable -XX:NativeMemoryTracking=summary and compare snapshots from jcmd <pid> VM.native_memory summary over time.
#!/usr/bin/env bash
java \
-Xms4g -Xmx4g \
-Xlog:gc*:file=/var/log/app/gc.log:time,uptime,level,tags:filecount=10,filesize=50m \
-XX:NativeMemoryTracking=summary \
-jar /opt/app/app.jar &
PID=$!
grep -hE 'Pause (Full|Young \(Mixed\))' /var/log/app/gc.log* \
| grep -oE '[0-9]+M->[0-9]+M\([0-9]+M\)' \
| sed -E 's/.*->([0-9]+M).*/post-GC: \1/'
jcmd "$PID" VM.native_memory baseline
jcmd "$PID" VM.native_memory summary.diff
ps -o pid,rss,cmd -p "$PID"
Enabling rotated unified GC logging and Native Memory Tracking, extracting post-collection heap occupancy from G1 log lines to spot a rising floor, and diffing NMT snapshots when RSS grows but the heap does not.
How do you capture a JVM heap dump safely in production?
There are two ways to get a dump: on failure, or on demand. For failures, start the JVM with -XX:+HeapDumpOnOutOfMemoryError and set -XX:HeapDumpPath to a volume with room for the dump. A dump is roughly the size of the used heap, so a 12 GB heap near exhaustion needs about 12 GB free. For on-demand dumps, use jcmd <pid> GC.heap_dump <file> or jmap -dump:live,format=b,file=<file> <pid>.
With live, the JVM runs a full GC first, so only reachable objects are written. jcmd behaves the same way by default. This is usually what you want, because a leak is memory that stays reachable, and dropping garbage gives a smaller dump with less noise. The downside is that the collection hides allocation-rate problems. If the heap grows because short-lived objects are being churned faster than the collector reclaims them, a live dump will look healthy. That is a GC tuning issue, not a leak. Your GC logs are the place to diagnose it.
Operational trade-offs
- The JVM pauses for the entire write. Health checks fail during that time, and an orchestrator may kill the process partway through. Drain traffic from the node first, or take the dump on a canary.
- Container filesystems are often ephemeral. If the dump path is not mounted to persistent storage, the file is lost when the container restarts.
- OOM dumps can fill disks and slow restart loops. A pod that crashes repeatedly writes a multi-gigabyte file on every exit. Cap the space available for dumps or clean up old files automatically.
- Dumps are sensitive. They contain credentials, tokens, session data, and personal information. Restrict who can access them, encrypt them at rest, and delete them once the analysis is done.
#!/usr/bin/env bash
set -euo pipefail
PID="${1:?usage: $0 <jvm-pid>}"
DUMP_DIR="/dumps"
HEAP_GB=12 # -Xmx; a dump is roughly the size of the used heap
DUMP="${DUMP_DIR}/heap-$(hostname)-$(date +%Y%m%dT%H%M%S).hprof"
free_gb=$(df --output=avail -BG "$DUMP_DIR" | tail -1 | tr -dc '0-9')
if (( free_gb < HEAP_GB )); then
echo "only ${free_gb}G free in ${DUMP_DIR}, need ~${HEAP_GB}G" >&2
exit 1
fi
read -r -p "Confirm $(hostname) is drained, then press Enter: "
jcmd "$PID" GC.heap_dump "$DUMP"
chmod 600 "$DUMP"
gpg --batch --yes --encrypt --recipient heap-analysis@example.com \
--output "${DUMP}.gpg" "$DUMP"
rm -f "$DUMP"
find "$DUMP_DIR" -name 'heap-*.hprof*' -mtime +7 -delete
A capture script that checks free space against heap size, waits for the node to be drained, takes a live heap dump with jcmd, then locks down, encrypts, and prunes old dumps.
How does heap dump analysis work?
A heap dump is a snapshot of the Java heap written in the HPROF binary format. It records every object in the heap along with its class, its field values, and the references it holds to other objects. Arrays are written with their contents. The dump also contains class metadata and the list of GC roots: the starting points the collector uses to decide what is live. Roots include local variables and operands on live thread stacks, classes loaded by the JVM (which anchor their static fields), JNI references, and objects used as monitors.
An analyzer such as Eclipse MAT turns that file into a queryable object graph in four steps:
- Parse and index. It reads every record and builds index files that map object IDs to classes, outbound references, and sizes. This is the slow, memory-hungry step, and it is why large dumps need a dedicated analysis machine.
- Build reverse references. For each object, it records which objects point to it. That is what lets you walk backwards from a suspect object to the GC root keeping it alive.
- Compute the dominator tree. It runs a dominator algorithm over the graph from the GC roots. For every object, it determines which single object every root path must pass through. Retained sizes come directly from this tree.
- Answer queries. Histograms group objects by class or classloader. Path to GC Roots follows reverse references. OQL runs SQL-like queries over the indexed objects. Reports such as Leak Suspects are automated combinations of these queries.
None of this tells you which objects the application still needs. The analyzer only shows what is reachable and what keeps it reachable. Deciding whether that reachability is intended takes knowledge of the code, plus the multi-dump comparison described next.
Why does a single heap dump mislead, and how does the multi-dump technique work?
A heap dump shows what is live at one moment. It cannot show intent. A ConcurrentHashMap retaining 1.2 GB might be a session cache working as designed, or it might be a listener registry that never removes entries. In a single snapshot both look identical: a large retained size under a legitimate-looking owner. What separates a leak from a big cache is growth over time under comparable load, and that requires more than one dump.
The multi-dump technique
First, confirm the trend in GC logs. Look at the heap occupancy after full or old-generation collections. If that post-GC floor keeps rising across days of similar traffic, then take two or three dumps spaced hours or days apart. Capture each one at a similar load level, such as the same time of day, so that differences reflect accumulation rather than traffic. Then compare the dumps class by class. Legitimate caches plateau at their configured bounds. Leaking structures keep growing in both instance count and retained size from one dump to the next.
post-GC heap floor
^ dump 3
| dump 2 ___/
| dump 1 ___/
| ___/‾‾‾‾‾‾/
| ___/‾‾‾/
+--|---------|---------|---------|---------> time
GC logs show T+0 T+24h T+48h
rising floor
The restart tension
Slow leaks in monoliths may take days to produce a clear signal between dumps. During that wait, the floor approaches the heap limit and the pressure to restart grows. Scheduled nightly or weekly restarts relieve that pressure, but they reset the evidence along with the heap. The leak stays in the code and becomes harder to diagnose. If you must restart, take a dump before you do.
^ heap limit (-Xmx)
| +--o
| +--o----------+
| +--o----------+
| +-------+
| +----+ post-GC heap floor
+------------------+-------------+-------------+---------> time
GC logs T+0 T+24h T+48h
floor rises | | |
v v v
+---------+ +---------+ +---------+
| dump 1 | | dump 2 | | dump 3 |
|same load| |same load| |same load|
+----+----+ +----+----+ +----+----+
| | |
+-------------+-------------+
v
+-------------------------------+
| compare dumps class by class: |
| instance count, retained size |
+---+-----------------------+---+
| |
v v
+-------------------+ +-------------------+
| plateaus at bound | | keeps growing |
| => bounded cache | | => leak suspect |
+-------------------+ +-------------------+
After GC logs show a rising post-GC floor, three dumps taken at similar load and spaced apart are compared class by class: a structure that plateaus is a bounded cache, and one that grows in every dump is a leak suspect.
What is the difference between shallow heap and retained heap?
Shallow heap is the memory an object occupies by itself: its header plus its fields. A reference field counts only as the pointer, not the object it points to. Retained heap is the memory the garbage collector would free if that object were collected, meaning the object plus everything reachable only through it.
The link between the two is the dominator relationship. Object A dominates object B if every path from a GC root to B passes through A. A's retained heap is the sum of the shallow sizes of all objects A dominates, A included.
Illustrative estimate (assumed sizes, not measurements): a HashMap.Node in a static session cache has an assumed shallow size of 32 bytes. Its value is a session object holding a list of 50,000 cached rows at an assumed 400 bytes each. If that node is the only path to the rows, it dominates about 20 MB.
| Object | Shallow heap | Retained heap | What it tells you |
|---|---|---|---|
HashMap.Node (session entry) |
~32 B | ~20 MB | Small holder keeping a large graph alive |
Pooled byte[] buffer |
~8 MB | ~8 MB | Large, but expected and bounded |
Sorting by shallow size puts the 8 MB buffer at the top. That buffer is usually not the problem. A leak grows by accumulating many small holders, each retaining a subgraph. The retaining reference, such as a static map that is never pruned, is the line of code you need to fix. Retained size identifies that holder. The largest single object rarely does.
How do you find a memory leak with Eclipse MAT?
Prepare before you open anything
Copy the dump off the production host first. Parsing it there competes with the service for memory and disk. MAT builds index files next to the dump, and those indexes can approach the dump's own size. Raise MAT's maximum heap with the -Xmx setting in its .ini file. For very large files, run MAT's bundled parse script headless on a server, then open the finished indexes from the GUI. Provision an analysis machine with more RAM than your largest production heap now, not during an incident.
Work from hypothesis to proof
- Open the dump and let indexing finish. The first parse is slow. Later opens reuse the indexes.
- Read Leak Suspects as a starting hypothesis. It flags large retainers, but large is not the same as leaking. Caches, connection pools, and framework internals such as bean registries are designed to hold a lot of memory, so they often appear here as false positives. The multi-dump comparison tells you which of them actually grow.
- Open the dominator tree and sort by retained heap to find the objects that keep the most memory alive.
- Run Path to GC Roots, excluding weak, soft, and phantom references. The GC can clear those references, so they are not what keeps the object alive. What remains is the strong reference chain you need to break.
- Use the histogram grouped by class or classloader. Grouping by classloader exposes redeploy leaks, where an old application classloader stays alive after the application was replaced.
- Use OQL for targeted questions. Examples: how many instances of your session class exist, or which keys dominate a suspect map.
Menu labels vary between releases. Look for these capabilities by name rather than following an exact click path.
#!/usr/bin/env bash
set -euo pipefail
PROD_HOST=app-prod-03.internal # placeholder host
REMOTE_DUMP=/var/tmp/orders-svc.hprof # placeholder path on prod
WORK=/data/heapdumps/orders-svc # analysis disk: room for dump + indexes
MAT_HOME=/opt/mat # unpacked Eclipse MAT install
mkdir -p "$WORK"
ssh "$PROD_HOST" "gzip -c '$REMOTE_DUMP'" > "$WORK/orders-svc.hprof.gz"
gunzip "$WORK/orders-svc.hprof.gz"
cp "$MAT_HOME/MemoryAnalyzer.ini" "$MAT_HOME/MemoryAnalyzer.ini.bak"
sed -i 's/^-Xmx.*/-Xmx24g/' "$MAT_HOME/MemoryAnalyzer.ini"
grep -- '-Xmx' "$MAT_HOME/MemoryAnalyzer.ini" # confirm the new value
"$MAT_HOME/ParseHeapDump.sh" "$WORK/orders-svc.hprof" org.eclipse.mat.api:suspects
ls -lh "$WORK"
Copies a heap dump off the production host, raises MAT's own -Xmx in MemoryAnalyzer.ini, and runs ParseHeapDump.sh headless to build the indexes and a Leak Suspects report before opening the dump in the GUI.
What do common monolith leak patterns look like in a heap dump?
Most monolith leaks follow a few recurring shapes. If you recognise the shape in the dominator tree or the path to GC roots, you usually know the fix before you open the source.
| Pattern | What MAT shows | Typical fix |
|---|---|---|
| Static collection / unbounded cache | A HashMap or ConcurrentHashMap near the top of the dominator tree, rooted in a class's static field |
Bound the size and add eviction (LRU, TTL) |
| ThreadLocal on pooled threads | Worker Thread objects retaining large threadLocals map entries |
Call ThreadLocal.remove() in a finally block |
| Unremoved listeners/callbacks | A long-lived event source's listener list holding short-lived objects | Deregister in the component's close or shutdown path |
| Classloader leak after redeploy | The same class name appearing many times under different loaders | Clear static references, stop threads, and deregister drivers on undeploy |
| Accumulated strings | Many String/byte[] instances with duplicate content, often inside a map |
Bound the source map, or stop interning unbounded input |
| Sessions/requests held too long | Session or request instance counts far above concurrent users | Shorten timeouts and stop caching request-scoped objects |
All of these leaks keep a reachable path from a GC root. For static fields, that root is the class. For pooled threads, it is the live thread itself: the pool never lets the thread die, so its ThreadLocal values are never collected. Classloader leaks cause the most damage. One stray reference to a single object pins its loader, and the loader pins every class it defined along with those classes' static state. That is why each redeploy adds another full copy.
Who should use heap dump analysis?
Heap dump analysis is for engineers who own, operate, or are on call for long-running JVM services. That typically means:
- Backend engineers on monoliths and application servers. These processes run for days or weeks, host many modules in one heap, and accumulate state from code written by many teams. A leak in one module degrades the whole process.
- SREs and on-call engineers. They are usually the first to see a rising post-GC floor or repeated OOM restarts. They need to capture a usable dump safely, even if someone else analyzes it.
- Platform and framework maintainers. Shared libraries, caching layers, and redeployable application containers are common sources of ThreadLocal and classloader leaks. These leaks show up only in the heap of a long-running consumer.
Heap dump analysis is a poor fit for short-lived processes such as batch jobs, CLI tools, or functions that exit within minutes. The heap is discarded before a slow leak can matter. It is also a poor fit for teams whose problem is off-heap growth. For that, Native Memory Tracking is the right tool.
When should you use heap dump analysis?
Take and analyze a dump when the evidence points to retained objects on the Java heap:
- The post-GC floor rises across comparable load. This is the main trigger. Confirm it in GC logs over hours or days before capturing anything.
- An
OutOfMemoryError: Java heap spaceoccurred. If-XX:+HeapDumpOnOutOfMemoryErrorwas set, the dump already exists. Analyze it before the evidence is overwritten or deleted. - Scheduled restarts are shrinking in interval. Needing to restart more often as traffic grows means something accumulates per request, per session, or per deploy.
- Repeated redeploys inflate memory. Each redeploy of a module into a running container raising the floor points to a classloader leak.
- You are verifying a fix. Dumps taken before and after a change, at similar load, show whether the suspect structure stopped growing.
Do not start with a heap dump when the GC log shows a flat floor near the heap limit. That is an undersized heap. Do not start with one when process RSS grows while the heap is flat. That is native memory. Do not start with one when the problem is pause times or allocation churn with a stable live set. That is GC tuning. The table in the first section maps each symptom to the right tool.
When is heap dump analysis worth the cost, and what are the alternatives?
Heap dump analysis pays off for engineers who own a JVM service that runs for days and whose post-GC live data keeps climbing. If the service is redeployed several times a day, a slow leak may never surface, and the dump costs more than it reveals.
| Option | Fits when | Limitation |
|---|---|---|
| Scheduled restarts | Leak is slow and the fix is low priority | Hides the bug; the restart interval shrinks as traffic grows |
| Larger heap | Live set is flat but close to the max | Only delays the OOM if something is leaking |
| GC tuning | Allocation rate or pauses are the problem, live set is flat | Does nothing about retained objects |
| Native Memory Tracking | Process RSS grows while the heap does not | Must be enabled at JVM startup; adds some overhead |
jcmd <pid> GC.class_histogram |
You need a quick check for growing instance counts | Gives no retaining paths; forces a full GC by default |
A full dump has real costs. The JVM pauses while the dump is written, and the pause grows with heap size. The file is roughly the size of the used heap. It holds credentials, session tokens and customer data, so it needs restricted storage and deletion when the analysis is done. MAT needs a machine with enough memory to index the dump. Engineer time usually costs the most.
Following the steps in order keeps that time down:
- Confirm the leak in GC logs.
- Take dumps deliberately.
- Read the retaining paths.
- Verify the fix by checking that post-GC live data stays flat over a run comparable to the one that showed the leak.
Skipping the first step leads to chasing an undersized heap. Skipping the last step means returning to scheduled restarts.
How can heap dump analysis save engineering time?
Without a dump, leak hunting usually means reading code for suspicious collections, adding logging, and trying to reproduce production traffic in a test environment. Each guess takes a deploy and days of waiting to see whether the floor still rises. A heap dump replaces that loop with direct evidence. Path to GC Roots names the exact field, collection, or thread holding the memory. That turns an open-ended search into a review of one code path.
The savings come from a few specific places:
- Fewer speculative fixes. You change the reference that actually retains memory, not the code that merely looks suspicious. Every wrong guess you skip saves a deploy and an observation period of days.
- Less time spent on restarts. Fixing the root cause removes the scheduled restarts, the restart-related incidents, and the on-call time spent watching memory graphs.
- Faster triage when it recurs. A capture script, a provisioned analysis machine, and a headless MAT parse are prepared ahead of time. When the floor starts rising, the time from alert to a readable dominator tree depends on copying and indexing the file, not on improvising tooling during an incident.
- Recognisable patterns. Most monolith leaks match the shapes in the patterns table. Once a team has seen a ThreadLocal or classloader leak in a dominator tree, the next one is usually identified on sight.
- Cheap verification. Comparing dumps at similar load, or watching the post-GC floor stay flat after a fix, confirms the result. You do not have to wait for the next OOM to find out.
The upfront work is mostly configuration: GC logging, the OOM dump flags, a persistent dump volume, and an analysis machine. All of it can be done once per service, before it is needed.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to jvm heap dump analysis, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.