Monolith Memory Leak by Module: Attributing Heap Growth to Code

What a per-module memory leak is, and why RSS hides it

A per-module memory leak is code in one part of a monolith that keeps references to objects it no longer needs, so the garbage collector cannot reclaim them. Every module shares one process, so the operating system reports a single resident set size (RSS) for all of them. A growing RSS tells you the process is bigger. It does not tell you which module made it bigger, or whether application objects are the cause at all.

Treat RSS as the sum of several layers. Each layer grows for different reasons:

Fragmentation is another look-alike. Freed memory can sit in partially used pages that the allocator cannot hand back to the OS, so RSS stays high even though nothing is leaking.

The first gate

Before you blame application code, confirm that the post-GC live heap trends upward across many collection cycles and does not level off. A live heap that rises and then flattens is usually a cache reaching capacity. A live heap that keeps rising at a steady rate under steady traffic is the case this technique is built for.

Defining 'module' so profiler output becomes attributable

A heap profiler reports allocation sites and object types, such as com.acme.billing.InvoiceCache or app/tenants/registry.py:88. It does not know that billing belongs to the payments team. Without an explicit mapping from code to owners, every investigation starts with someone reading stack frames and guessing. The guesses also differ between investigations, so you cannot compare one week's snapshot to the next.

Most teams already have the raw material. Package or namespace prefixes work when the codebase is organized by domain. An ownership file that maps source paths to teams can be reused if one exists for code review. A small source-path-to-team table also works. The mapping config shown below lists prefixes in order of specificity, each with a module name and an owning team. The accompanying script reads profiler rows (type or frame, retained bytes, count), matches each row to the longest prefix, and sums bytes per module. Use the longest match so that billing.reports does not get absorbed into billing.

The mapping is weakest exactly where leaks are most likely to show up:

• Shared utilities collect objects for every caller.
• Generic collections like HashMap, dict and byte[] carry no domain prefix at all.
• Framework internals own request scopes and thread pools.
• Generated code sits under build paths.

All of these fall into a common bucket, and that bucket is often the largest one.

The fallback is to attribute by retainer owner. For each object in common, walk its retainer chain to the nearest object that does match a domain prefix, and charge the bytes to that module. A HashMap held by the tenant registry then counts against the tenant registry, not against the collections library.

The platform team should own the mapping, the script, and the snapshot pipeline. Domain teams should own their prefixes and review changes to them, so that attribution keeps up with refactors.

Copyimport csv
import sys
from collections import defaultdict

# Mapping config: source prefix -> (module, owning team).
# Keep it in version control so every investigation uses the same buckets.
MAPPING = {
    "com.acme.billing.reports": ("billing-reports", "finance-analytics"),
    "com.acme.billing": ("billing", "payments"),
    "com.acme.tenants": ("tenant-registry", "platform"),
    "app/tenants/": ("tenant-registry", "platform"),
    "app/search/": ("search", "discovery"),
}

# Try longer prefixes first so billing.reports is not absorbed into billing.
PREFIXES = sorted(MAPPING, key=len, reverse=True)

def attribute(name):
    for prefix in PREFIXES:
        if name.startswith(prefix):
            return MAPPING[prefix]
    # HashMap, dict, byte[], framework and generated code end up here.
    return ("common", "unowned")

def main(path):
    totals = defaultdict(lambda: [0, 0])  # (module, team) -> [bytes, count]
    with open(path, newline="") as f:
        # Expected CSV columns exported from the profiler: name,retained_bytes,count
        for row in csv.DictReader(f):
            key = attribute(row["name"])
            totals[key][0] += int(row["retained_bytes"])
            totals[key][1] += int(row["count"])
    for (module, team), (size, count) in sorted(totals.items(), key=lambda kv: -kv[1][0]):
        print(f"{module:18} {team:18} {size:>15,} bytes {count:>10,} objects")

if __name__ == "__main__":
    main(sys.argv[1])

A prefix-to-owner mapping and a script that assigns each exported profiler row to the longest matching prefix, then sums retained bytes and object counts per module. Rows that match no prefix go to a 'common' bucket.

How it works: retained-heap snapshots versus allocation profiles

A profiler offers two views of the heap, and each answers a different question. A retained-heap snapshot answers "what is holding memory right now?" It walks the object graph from GC roots and computes dominators. An object's retained size is everything that would be freed if that object became unreachable, and the retainer path is the chain of references that keeps it alive. A sampled allocation profile answers "which code allocated this?" It records stack traces for a statistical subset of allocations.

Most leaks need both views, because the code that allocates an object and the code that keeps it alive often belong to different modules:

Copy billing.invoice.render()      <- allocation site (names the code)
        | allocates
        v
   [InvoiceContext] --> [LineItem[]] --> [byte buffers]
        ^
        | held by
   core.events.ListenerRegistry   <- retainer (names the holder)
        ^
        | static field
   GC root

Sampling has trade-offs. A slow leak that adds a few objects per request looks like noise inside a high-volume allocation stream. You only see it when you compare samples of live, surviving allocations across long windows. Rare code paths, such as a nightly job or a single tenant's feature flag, may land few or no samples. The profile can then attribute growth to a busy module that is innocent. This is why the snapshot's retainer path is used to confirm the suspect.

Copy MODULE: billing                   |   MODULE: core.events
                                   |
 +------------------------------+   |   +-----------------------------+
 | Sampled allocation profile   |   |   | Retained-heap snapshot      |
 | Q: who allocated it?         |   |   | Q: who holds it now?        |
 +------------------------------+   |   +-----------------------------+
        | stack traces               |                   | walks from roots
        v                          |                   v
 +------------------------------+   |   +-----------------------------+
 | billing.invoice.render()     |   |   | GC root (static field)      |
 | ALLOCATION SITE (the code)   |   |   |                             |
 +------------------------------+   |   +-----------------------------+
        | allocates                  |                   | static ref
        v                          |                   v
 +------------------------------+   |   +-----------------------------+
 | InvoiceContext               |<------| ListenerRegistry            |
 |                              |   |   | RETAINER (the holder)       |
 +------------------------------+   |   +-----------------------------+
        |                          |
        v                          |
 +------------------------------+   |
 | LineItem[] -> byte buffers   |   |
 | (retained size grows)        |   |
 +------------------------------+   |
                                   |
 The allocation profile names the code. The retainer path crosses the
 boundary and names the holder in another module.

Two views of one leak: the sampled allocation profile traces InvoiceContext to billing.invoice.render(), and the retained-heap snapshot follows the retainer chain from a GC root through core.events.ListenerRegistry, which is in a different module.

Collecting production evidence safely

Evidence comes in two tiers. Run sampled allocation profiling continuously on every instance, because its overhead is low enough to leave on and it records which code paths allocate while the leak is still growing. Go's net/http/pprof heap endpoint and the JVM's allocation sampling via JDK Flight Recorder are examples of this (runtime-specific). Take full heap dumps only on one instance that you have drained from the load balancer, or on a canary.

The drain matters because a dump is disruptive:

• Size and disk. A dump is roughly as large as the live heap. Write it to a volume that has room for it. Filling the root disk causes a second outage.
• Pauses. Most runtimes stop the world while they write the dump. On the JVM, jmap -dump:live also forces a full GC first (runtime-specific).
• Health checks and kills. A paused process fails liveness probes. The orchestrator can kill it mid-write, which leaves you with a truncated file. Relax the probes or detach the instance from them before you start.

Treat every dump as production data. It holds whatever was in memory, including session tokens, database credentials, API keys and customer PII. Restrict access to it, keep it encrypted at rest, and delete it on a fixed schedule.

Finally, check whether your fleet ever lets a leak become visible. Deploys, nightly restarts and autoscaling churn reset every heap before the growth stands out from normal variance. To give the leak time to surface, keep one instance alive across several deploy cycles, pinned at normal traffic.

Copy#!/usr/bin/env bash
# Take a JVM heap dump on ONE instance already drained from the load balancer
# and with liveness probes relaxed (runtime-specific: HotSpot jcmd).
set -euo pipefail

PID="${1:?usage: $0 <java-pid>}"
DUMP_DIR="/mnt/heapdumps"          # dedicated volume, never the root disk
RECIPIENT="heap-analysis@example.internal"  # placeholder GPG key for analysts
OUT="$DUMP_DIR/heap-$(hostname)-$(date +%Y%m%dT%H%M%S).hprof"

# 1. Refuse to start unless free space exceeds process RSS (a rough upper
#    bound on dump size) with 50% headroom. A full disk is a second outage.
rss_kb=$(ps -o rss= -p "$PID" | tr -d ' ')
avail_kb=$(df --output=avail -k "$DUMP_DIR" | tail -n 1 | tr -d ' ')
if (( avail_kb < rss_kb * 3 / 2 )); then
  echo "not enough space: need ~$((rss_kb * 3 / 2)) KB, have $avail_kb KB" >&2
  exit 1
fi

# 2. Keep dump files private from the moment they are created.
umask 077

# 3. Write the dump. This pauses the JVM (stop-the-world); by default
#    GC.heap_dump dumps only live objects, which triggers a full GC first.
jcmd "$PID" GC.heap_dump "$OUT"

# 4. Treat the dump as production data: it contains tokens, credentials, PII.
#    Encrypt for the analysis team and remove the plaintext copy.
gpg --batch --yes --encrypt --recipient "$RECIPIENT" --output "$OUT.gpg" "$OUT"
shred -u "$OUT"

# 5. Enforce retention: delete encrypted dumps older than 7 days.
find "$DUMP_DIR" -name 'heap-*.hprof.gpg' -mtime +7 -delete

echo "wrote $OUT.gpg"

A JVM-specific script for a drained instance: it checks free disk space against process RSS before dumping, writes the heap dump with restrictive permissions, encrypts it, removes the plaintext copy, and deletes encrypted dumps older than seven days.

Diffing over time and following retainer chains to the owner

After several deploy cycles, take two or three snapshots at fixed intervals, each at the same point in the traffic cycle. A heap captured at peak holds more in-flight request objects than one captured at trough, and that difference looks like growth. Diff consecutive snapshots by type and by allocation site, then sum retained bytes by module prefix. A leaking module's delta rises in every interval. Normal churn rises and falls.

Type totals show what is growing but not who keeps it alive. To find that, follow the retainer chain from a sample of growing objects toward a GC root. The first object on the path that belongs to your code, rather than to a framework or the runtime, usually identifies the owner. The usual holders are:

• Unbounded shared caches with no eviction policy.
• Static registries that only add entries.
• Per-request listeners and callbacks that are never unsubscribed.
• Thread-local or request context captured by pooled threads or long-lived closures.
• Session and connection maps that skip removal on error paths.
• Metric tags built from user IDs or raw URLs.

To narrow the search, line up when the growth started with deploys, feature-flag flips, config changes, and shifts in traffic mix. If one endpoint class looks suspect, route it to a subset of instances and compare their growth rate against the rest.

Staging rarely reproduces this kind of leak. Many leaks grow with the number of distinct keys (tenants, users, tag values), not with request volume. Staging replays a small, repetitive key set, so the same structure stays flat there.

Copy# Runs inside the app process (e.g. a background thread started at boot).
# Takes snapshots at fixed intervals, diffs allocation sites by file,
# and sums the growth per top-level module under the app root.
import threading, time, tracemalloc
from collections import defaultdict

APP_ROOT = "/srv/app/"   # placeholder: where your monolith's packages live
INTERVAL_S = 600          # pick an interval that lands at the same traffic phase
SAMPLES = 3

def module_of(filename):
    # Map an allocation site to a module prefix, e.g. /srv/app/billing/x.py -> billing
    if not filename.startswith(APP_ROOT):
        return "<stdlib/third-party>"
    return filename[len(APP_ROOT):].split("/", 1)[0]

def take():
    # Exclude tracemalloc's own bookkeeping from the snapshot
    snap = tracemalloc.take_snapshot()
    return snap.filter_traces([tracemalloc.Filter(False, tracemalloc.__file__)])

def module_deltas(old, new):
    totals = defaultdict(int)
    for stat in new.compare_to(old, "filename"):
        totals[module_of(stat.traceback[0].filename)] += stat.size_diff
    return totals

def run():
    snaps = [take()]
    for _ in range(SAMPLES):
        time.sleep(INTERVAL_S)
        snaps.append(take())
    deltas = [module_deltas(a, b) for a, b in zip(snaps, snaps[1:])]
    # A leak grows in every interval; normal churn goes up and down
    for mod in sorted(set().union(*deltas)):
        series = [d.get(mod, 0) for d in deltas]
        flag = "RISING" if all(x > 0 for x in series) else ""
        print(f"{mod:30} {series} {flag}")

tracemalloc.start(1)  # 1 frame is enough to attribute by file; more frames cost more
threading.Thread(target=run, daemon=True).start()

Uses Python's tracemalloc to diff allocation-site growth across consecutive snapshots, sum it by module prefix, and flag modules whose growth is positive in every interval; this shows where memory is allocated, so you still follow retainer chains in a heap dump to find which object keeps it alive.

Worked example: a tenant cache held by a shared registry

Illustrative estimate. All inputs are assumed. Two heap snapshots are taken six hours apart on one instance. Over that window, the count of distinct tenant IDs seen grows from 1,200 to 3,000. Request rate stays flat. Retained size is aggregated by package prefix:

Warm-up is ruled out on three counts. First, warm-up plateaus once the working set is loaded, and a third snapshot shows billing.cache still growing at the same per-tenant slope. Second, the growth tracks distinct tenants, not elapsed time or request volume. Third, entries for tenants that made a single request hours earlier are still present.

The retainer path explains why the cache never shrinks, even though billing.cache has its own TTL. Each entry is reachable from a listener map in core.registry, keyed by tenant ID. The billing module registers a callback when it creates an entry and never unregisters it. Expiry removes the entry from the cache's own map, but the registry still holds it.

The fix has to sever that reference. There are three options:

• Bound: cap the registry's map size. This limits damage but does not fix ownership.
• Eviction: unregister the listener when the cache evicts the entry. This is the correct fix.
• Weak references: have the registry hold listeners weakly. This works only if the registry is never the sole intended owner.

To verify the fix, repeat the diff across a comparable increase in tenant count. billing.cache should stay flat or level off at its cap. Following any surviving entry's retainer path should no longer lead through core.registry.

Alternatives, when to avoid this approach, and impact on cost and engineering time

Once the fix holds, it is worth weighing the full approach against cheaper options, because none of them is free.

Skip the full approach in two cases. If post-GC heap levels off under steady load, the growth is legitimate working set, so size the limit for it. If RSS climbs while the managed heap stays flat, the leak is native, and managed profilers cannot see it.

The full approach has four costs:

• Building and maintaining the ownership mapping.
• Profiling overhead on the sampled instance.
• Dump storage, since a dump can be as large as the heap.
• Compliance controls, because dumps contain customer data and need access restrictions, retention limits and redaction.

Some lightweight accounting pays for itself:

• Size gauges on every bounded structure (caches, registries, queues), labelled by owning module.
• Alerts on post-GC heap slope over hours, not absolute usage, so normal peaks don't page anyone.
• A soak test that snapshots before and after a sustained run and fails on unexpected per-module growth.

With these in place, the gauge names the module and the soak diff names the type. Investigation starts at one retainer path instead of a days-long hunt across the fleet.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to monolith memory leak by module, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.