What is growing: managed heap, native memory, or fragmentation?

In a monolith, memory profiling is an ownership problem: one process, one heap, dozens of modules, and no built-in record of who allocated what. Before asking which module owns the growth, establish which memory layer is growing. Teams that skip this step can spend days in heap dumps chasing growth that a heap dump cannot show, because native memory never appears in one.

Start with two time series from the same process: resident set size (RSS) from the OS, and the runtime's own heap metrics, specifically heap used after a full or major collection. Pre-GC numbers are mostly short-lived garbage, so they hide the trend. If post-GC heap rises in step with RSS, the managed heap is the growing layer and a dump will help. If post-GC heap stays flat while RSS climbs, the growth is outside the collector's view, and a dump will look healthy.

Symptom Signal Tool category
Managed-heap growth Post-GC heap floor rises; RSS tracks it GC logs, heap dumps, allocation profilers
Native/off-heap growth Post-GC heap flat; RSS rises Runtime native-memory tracking, direct-buffer pool metrics
Thread stack growth Thread count rises with RSS Thread dumps, thread-count metrics
Allocator fragmentation RSS high and not returned after load drops; live heap and native totals stable Allocator statistics, /proc/<pid>/smaps

Fragmentation is the hardest case to recognize. Nothing is leaking, but the allocator cannot return partially used pages to the OS, so RSS stays at its peak. That is a tuning problem, not a module bug.

#!/usr/bin/env bash
set -euo pipefail

PID="${1:?usage: $0 <pid> [interval_seconds]}"
INTERVAL="${2:-60}"

echo "timestamp,rss_kb,threads,old_gen_used_kb,full_gc_count"

while kill -0 "$PID" 2>/dev/null; do
  # RSS and thread count come from the kernel, so they cover heap, native and stacks.
  rss_kb=$(awk '/^VmRSS:/ {print $2}' "/proc/$PID/status")
  threads=$(awk '/^Threads:/ {print $2}' "/proc/$PID/status")

  # jstat column order differs between JDK versions, so look columns up by header name.
  # OU = old generation used (KB), FGC = full GC count.
  read -r old_used fgc < <(jstat -gc "$PID" | awk '
    NR == 1 { for (i = 1; i <= NF; i++) col[$i] = i }
    NR == 2 { print $col["OU"], $col["FGC"] }')

  echo "$(date -u +%FT%TZ),$rss_kb,$threads,$old_used,$fgc"
  sleep "$INTERVAL"
done

A shell sampler that logs process RSS and thread count from /proc next to JVM old-generation usage from jstat, so you can tell managed-heap growth apart from native or thread-stack growth before you take any heap dump.

When should you use module-level memory profiling?

Use this workflow when the managed heap is the growing layer and the question is "which part of the codebase is responsible?" rather than "is memory growing?" Typical triggers:

  • The post-GC heap floor rises across days or deploys at roughly steady traffic, rather than levelling off after warm-up.
  • Out-of-memory kills or restarts are getting more frequent, or someone has started scheduling restarts to stay ahead of them.
  • Steady-state memory changed after a release, and that release contained changes from several teams.
  • A heap dump shows only generic types at the top, such as byte[], String, HashMap$Node, or framework classes, and nobody can say which module put them there.
  • Several teams share the process, so an unattributed leak tends to be passed from team to team instead of fixed.

It is the wrong tool in some cases:

  • Latency or GC CPU problems without heap growth. That is churn. Sampled allocation profiling answers it directly, and retention analysis adds little.
  • Native, thread-stack, or fragmentation growth. The first section's signals point elsewhere, and heap dumps will look healthy.
  • A heap limit that is simply too small for the real working set. If the post-GC floor is flat but close to the limit, resize before hunting for a leak.
  • Short-lived processes, such as batch jobs that exit before growth matters.
  • Small codebases with one owning team, where reading the recent diff is often faster than building attribution tooling.

How does module-level memory profiling work?

The mechanism rests on one property of garbage-collected runtimes: the collector only frees objects it cannot reach from a GC root. A managed-heap leak is therefore a set of objects that stay reachable longer than intended. The workflow turns that reachability graph into a team name in a fixed sequence of steps:

  1. Classify the layer. Compare RSS with the post-GC heap, as above, to confirm the growth is in the managed heap.
  2. Measure retention, not allocation. Heap dumps record which objects are live and which references keep them live. Allocation profiles only show which objects were created.
  3. Compare snapshots over time. Spaced dumps at steady load separate monotonic growth from load-dependent noise.
  4. Reduce the graph to retained sizes. The dominator tree shows which objects, if released, would free the growing memory.
  5. Map classes to owners. Package or path prefixes, matched against an ownership file in the repository, turn class names into teams.
  6. Check what is stored, not only where. For shared caches and registries, the key and value types and the inserting caller name the module that fills the structure.
  7. Confirm cheaply, then guard. A toggle, a size gauge, or a canary diff confirms the suspect. Continuous metrics and soak-test diffs catch the next regression earlier.

The rest of this article covers each step in that order.

Allocation profiling vs retention analysis: which one finds the leak?

Once you know the growth is in the managed heap, choose the tool by the question it answers. Allocation profiling records who creates objects. Retention analysis records who keeps them alive. A leak is a retention problem, so allocation data alone rarely finds it.

A module can sit at the top of the allocation profile and be innocent. A JSON serializer in the request path might allocate gigabytes per minute of short-lived buffers, and almost all of them die in the young generation. That drives GC frequency and CPU cost, but it does not grow the heap. A small session cache that adds one entry per login and never evicts it barely shows up in allocation counts, yet it is the thing that eventually exhausts memory.

Sampling makes this worse. Sampling allocation profilers keep overhead low by recording roughly one event per N bytes or per N allocations. A path that allocates rarely but retains everything is exactly what sampling under-represents. That combination is the typical shape of a slow leak.

Approach Answers Good for Avoid when
Allocation profiling (sampled) Which code allocates most GC pressure, churn, latency from collection pauses Hunting slow, monotonic growth
Allocation profiling (full tracing) Every allocation site Short, controlled reproductions Production traffic, because overhead is high
Retention analysis (heap dumps) What keeps objects reachable Leaks, unbounded caches, listener buildup Diagnosing churn; dumps also pause the process and can be large

Start with retention analysis to find what is accumulating. Then use allocation data to find where those objects were created.

How does heap dump analysis work? Retained size, dominators, and paths to root

A heap dump is a snapshot of every live object and the references between them. Analysis tools reduce that graph to a few numbers. Shallow size is the memory an object occupies by itself. Retained size is the memory that would be freed if that object became unreachable, meaning everything reachable only through it. GC roots are the starting points the collector treats as always live: thread stacks, static fields, and native handles. An object dominates another if every path from a root to the second passes through the first. The dominator tree arranges the heap so that each node's subtree is exactly its retained set.

GC root (static field)
  └── billing.SessionRegistry   shallow: 48 B   retained: 1.9 GB
        └── HashMap
              ├── Session ── byte[] (4 KB)
              ├── Session ── byte[] (4 KB)
              └── ... ~400k entries

A class histogram sorted by instance count shows byte[] and Session near the top. Every service has many of both, so neither looks suspicious. Sorting the dominator tree by retained size shows one small registry that holds almost 2 GB. That registry is what you change, so it is the useful answer. Once you have it, the path to root explains why it stays alive. Here it is a static field, which points to a module-level singleton that adds entries and never evicts them.

Capturing a full dump has costs. The process pauses or slows down while the dump is written, and the file can be as large as the heap. Take it from a node pulled out of rotation or from a canary instance. Dumps also contain live data, including tokens, PII, and secrets, so store them with the same access controls as production databases.

 GC ROOTS (always live)
 +----------------+  +----------------+  +----------------+
 | thread stacks  |  | static fields  |  | native handles |
 +----------------+  +-------+--------+  +----------------+
                             |
                             |  path to root: why it stays alive
                             v
              +------------------------------+
              | billing.SessionRegistry      |  shallow:  48 B
              | (module-level singleton)     |  retained: 1.9 GB
              +--------------+---------------+
                             |
                             |  dominates: every path below runs here
                             v
              +------------------------------+
              | HashMap  (~400k entries)     |
              +---+----------+----------+----+
                  |          |          |
                  v          v          v
             +---------+ +---------+  +-----+
             | Session | | Session |  | ... |
             +----+----+ +----+----+  +-----+
                  v          v
             +---------+ +---------+
             | byte[]  | | byte[]  |
             | 4 KB    | | 4 KB    |
             +---------+ +---------+

 Histogram by count:    byte[], Session  ~400k each  -> looks normal
 Dominators by retained: SessionRegistry  1.9 GB      -> the thing to fix

A 48-byte module-level registry, held alive by a static-field GC root, dominates about 400k small Session and byte[] objects. Its retained size points to the owner of the leak, while instance counts alone look harmless (illustrative figures).

How do you diff heap snapshots to find monotonic growth?

With access controls in place, the main technique is comparison, not inspection. A single dump shows what is large, not what is growing. If it was taken during a traffic spike, in-flight requests, session state, and buffers inflate the retained sizes of whatever served that traffic, and the leak blends into normal load.

Take two or three dumps from the same instance at similar, steady load, separated by hours. Use a capture that forces a full collection first, such as jcmd <pid> GC.heap_dump <file> or jmap -dump:live,format=b,file=<file> <pid> on the JVM, or v8.writeHeapSnapshot() in Node. Garbage then does not count as growth. Compare object counts and retained sizes per class. The useful signal is a type whose numbers rise in every interval and never fall back.

Skip the first snapshot after startup. Caches warming up, lazily initialized singletons, JIT and class metadata, and a collector that grows the heap on purpose all look like growth for a while and then level off.

Illustrative estimate. Assume:

  • The live heap settles at 2.5 GB after warm-up.
  • The heap limit is 4 GB.
  • The leak adds 40 MB per hour.
  • Live-heap size varies by about ±150 MB between dumps.

Time to OOM is roughly 1,500 MB ÷ 40 MB/h, which is about 37 hours. Growth only equals the noise after about 4 hours (150 ÷ 40). To make it clearly visible, aim for a delta of three times the noise, about 450 MB. That requires spacing dumps at least 11 hours apart. You get two clean intervals before the instance is in danger.

#!/usr/bin/env bash
set -euo pipefail
PID="$1"
OUT="${2:-/var/tmp/heapdiff}"
SNAPSHOTS=3
INTERVAL_SECONDS=$((11 * 3600))  # from the illustrative estimate: ~3x noise at 40 MB/h
mkdir -p "$OUT"

for i in $(seq 1 "$SNAPSHOTS"); do
  ts=$(date +%Y%m%dT%H%M%S)
  # Per-class instance counts and shallow bytes; live objects only, so it triggers a full GC first.
  jcmd "$PID" GC.class_histogram > "$OUT/histo-$i-$ts.txt"
  # Full dump (live objects by default) for retained-size analysis in a heap analyzer.
  # This pauses the JVM and writes roughly heap-sized files, so check disk space first.
  jcmd "$PID" GC.heap_dump "$OUT/heap-$i-$ts.hprof"
  if [ "$i" -lt "$SNAPSHOTS" ]; then sleep "$INTERVAL_SECONDS"; fi
done

awk '
FNR == 1 { f++ }
$1 ~ /^[0-9]+:$/ { n[$4, f] = $2; b[$4, f] = $3; cls[$4] = 1 }
END {
  for (c in cls) {
    up = 1
    for (k = 2; k <= f; k++) if (!(n[c, k] + 0 > n[c, k - 1] + 0)) { up = 0; break }
    if (up) printf "%s\t%d -> %d instances\t%+d bytes\n", c, n[c, 1], n[c, f], b[c, f] - b[c, 1]
  }
}' "$OUT"/histo-*.txt | sort -t$'\t' -k3 -g -r | head -20

This script takes three live-object class histograms and heap dumps from one JVM, spaced 11 hours apart, and lists the classes whose instance counts rose in every interval, largest byte growth first.

How do you attribute retained memory to a module and its owning team?

A diff tells you which classes grew. Attribution turns that into a team that can act on it.

Group by prefix, not by class

Roll retained size up by package, namespace, or source directory prefix. A single cache can spread its growth across dozens of classes, such as entry types, wrapper objects, and backing arrays. One leak can look like many small ones until those classes are summed under the prefix that owns them. Pick a fixed depth, for example the first three package segments, so that totals stay comparable between dumps.

Walk the stack to the first owned frame

For allocation profiles, skip standard library, runtime, and framework frames. Charge each sample to the first frame that belongs to application code. Without this rule, most allocations land in collection classes or serializers, and that tells you nothing about who called them.

Keep the ownership map in the repository

Store a file that maps path prefixes to teams, matched by longest prefix first. Review it the same way you review code. The roll-up script takes the diff output, sums retained bytes per prefix, joins those totals against the map, and prints a ranked list of teams. Prefixes with no owner go into an explicit "unowned" bucket rather than disappearing from the report.

Where it misattributes

Prefix attribution assumes the code that defines a type is the code that keeps it alive. That assumption fails in several common cases:

  • Shared utilities and caches pin objects on behalf of their callers, so the platform team that owns the utility gets blamed.
  • Generated code often lives under a single generator package, which hides which feature produced it.
  • Reflection-heavy frameworks and dependency-injection proxies wrap application objects in synthetic classes that sit outside your prefixes.

Treat the ranked list as a lead to investigate, not a verdict.

import csv, sys
from collections import defaultdict

DEPTH = 3  # fixed depth keeps prefix totals comparable between dumps

OWNERS = {
    "com.example.billing": "payments-team",
    "com.example.billing.invoice": "invoicing-team",
    "com.example.search": "search-team",
    "com.example.common": "platform-team",  # shared utils: expect false blame
}

def package_of(cls):
    # Strip inner-class suffix (Cache$Entry -> Cache), then drop the class name.
    outer = cls.split("$")[0]
    return outer.rsplit(".", 1)[0] if "." in outer else ""

def owner_of(pkg):
    # Longest prefix wins, matched on whole package segments only.
    hits = [p for p in OWNERS if pkg == p or pkg.startswith(p + ".")]
    return OWNERS[max(hits, key=len)] if hits else "unowned"

by_prefix, by_team = defaultdict(int), defaultdict(int)
with open(sys.argv[1], newline="") as f:
    for row in csv.DictReader(f):
        pkg = package_of(row["class_name"])
        delta = int(row["retained_delta_bytes"])
        by_prefix[".".join(pkg.split(".")[:DEPTH])] += delta
        by_team[owner_of(pkg)] += delta  # unowned stays visible in the report

for label, totals in (("prefix", by_prefix), ("team", by_team)):
    print(f"== growth by {label} ==")
    for key, size in sorted(totals.items(), key=lambda kv: -kv[1]):
        print(f"{size:>14,}  {key or '(default package)'}")

A standard-library Python script that sums retained-size deltas from a heap-dump diff by fixed-depth package prefix, assigns each package to a team by longest-prefix match against an ownership map, and reports anything without an owner under an explicit "unowned" bucket.

Why does the dominator often belong to the wrong module?

A dominator tells you which object keeps memory reachable, and in a monolith that object is usually generic infrastructure: a cache, a pool, a registry. The module that owns the structure rarely chose what went into it. Module A provides the map; module B fills it with request-scoped objects and never removes them. Attributing by the dominator's package blames the plumbing, not the caller.

The fix is to inspect what is stored, not only where it is stored. Three heuristics do most of the work:

  • Key and value types. Group the entries by class. If one module's types make up most of the retained size, that module is the suspect.
  • Entry age. Look for creation timestamps, sequence IDs or generation counters on the entries. A healthy cache turns over. Entries from the first hour of uptime that are still present after a day point to missing eviction or missing removal.
  • The inserting caller. Run allocation profiling with stack traces on the structure's insert path. The stack frames name the module, which the retention graph cannot do.
Shared structure Common leak pattern What to inspect
Cache No size bound or TTL, or keys built from unbounded input such as user IDs or query strings Key cardinality, entry age, which code calls insert
Connection pool Leased objects never returned; per-connection state accumulates Lease holders, session attributes per connection
Thread-local Pooled threads keep values after the request ends Value types per worker thread
Listener/callback registry Register without unregister; closures capture large object graphs Captured fields of each callback, registration site
Static map Used as a process-wide registry that is never cleared Key prefixes, growth rate per key type
Serialization buffer Buffer grows to its largest payload and is retained Buffer capacity compared with typical payload size, which code reuses it

How can you confirm a suspect module cheaply before fixing it?

A suspect from the dominator tree is a hypothesis. Before rewriting anything, find the cheapest change that should bend the growth curve if the hypothesis is right. The test is the slope: plot heap-after-GC over hours for the treated and untreated paths and compare their rates of growth. A single snapshot can't tell you whether the leak responded.

Three tests are usually cheap enough to run first:

  • Toggle or rate-limit the feature path. If a feature flag gates the code, turn it off for one instance or one percentage of traffic. If the slope flattens on those instances only, you have strong evidence, and you also have a mitigation.
  • Replay one endpoint in staging. Isolate the traffic for a single route. This narrows the cause much more than a full-load test, where every module allocates at once.
  • Instrument the suspect structure. Export the entry count of a suspect cache or registry as a gauge, as in the example below. If the size climbs in step with the heap, the suspect is confirmed. If the size stays flat while the heap grows, the suspect is cleared.

Staging often fails to reproduce the leak. Many leaks depend on production conditions: real tenant counts, high-cardinality keys, oversized payloads, or error paths that synthetic traffic never reaches. An inconclusive staging result does not clear the suspect.

When staging is inconclusive, move the test to production in a bounded form:

  • Ship the size gauge to production, since it costs almost nothing.
  • Apply the toggle to a single canary instance.
  • Take two heap dumps from that instance a few hours apart and diff them as described earlier.

Production data then answers the question directly, and you avoid trying to rebuild production's data shapes in staging.

How can module-level memory profiling save engineering time?

Most of the time lost on a monolith memory leak goes into finding the owner and ruling out wrong theories, not into the fix itself. The workflow cuts that time at several points:

  • Layer classification avoids wasted dumps. A few hours of RSS and post-GC samples show whether a heap dump can show the growth at all. If the growth is native or fragmentation, nobody spends a day reading a healthy-looking dump.
  • Snapshot diffs replace guesswork about large objects. A ranked list of classes that grew in every interval removes the load-dependent noise that sends investigations toward whatever served the last traffic spike.
  • Ownership mapping stops the hand-off loop. Without attribution, each team checks its own recent changes, finds nothing obvious, and passes the problem on. A ranked list of teams with an explicit "unowned" bucket sends the work to one team first.
  • Content checks on shared structures protect the platform team. Looking at key and value types and insert callers keeps the owner of a generic cache from being the default suspect for every leak that lands in it.
  • Cheap confirmation prevents wrong fixes. A toggle or size gauge tests the hypothesis before anyone rewrites code, and a flag that flattens the slope is also an immediate mitigation that buys time for a proper fix.
  • Continuous checks move detection before release. A soak-test diff that fails a build points at one change set, which is much faster to review than a week of deploys from many teams.

Illustrative estimate. Assume a leak incident in a monolith shared by four teams. Without attribution, assume each team spends 6 engineer-hours checking its own recent changes before the owner is found, for 24 engineer-hours of triage. With a ranked attribution report, assume the first-ranked team is correct and spends 6 hours confirming and fixing, while one platform engineer spends 2 hours running the diff and roll-up, for 8 engineer-hours. Under those assumed inputs, the saving is about 16 engineer-hours per incident, before counting the ongoing upkeep of ownership maps and metrics described below. Your own figures depend on team count, how often leaks happen, and how accurate the ownership map is.

How do you make memory regressions a diff instead of an incident?

Incident-time debugging is expensive because the evidence is gathered under pressure, after the growth has already compounded. A better approach is to keep collecting the evidence continuously so that a regression appears as a change between two builds.

Three layers do most of the work:

  • Sampling allocation profiling in production. Sampling keeps overhead low enough to leave it on. It does not prove retention, but it shows which call sites started allocating more after a deploy.
  • Size metrics per known cache or registry. These structures are where leaks usually end up, and an entry-count gauge on each one names the owner without a dump.
  • Snapshot diffs in soak or load tests. Diff a heap snapshot taken early in the run against one taken late, then compare per-module retained size against the last accepted baseline. Fail the build when the growth exceeds a threshold.

This has an ongoing cost. Ownership maps drift as packages move between teams, and per-module metrics go stale when a cache is renamed or replaced. If nobody maintains them, attribution quietly points at the wrong team.

Approach Cost Detection speed Attribution quality
Ad-hoc incident dumps only Low upfront, high per incident After user impact Good, but slow to reach
Memory limits with restarts Low Hides the problem None
Continuous profiling, metrics, and soak diffs Moderate, ongoing upkeep Pre-release or at deploy Module-level
Split into services Very high Per service Per process

Splitting the monolith does give clean per-process attribution. But it also changes deployment, data ownership, failure modes, and team boundaries. Those are good reasons to split on their own, but a leak you can't attribute isn't one of them.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to monolith memory profiling, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.