Monolith Memory Profiling: Attributing Heap Usage to Modules

What monolith memory profiling is and why ownership must be defined first

Monolith memory profiling is the practice of attributing heap usage in a single large process to the logical modules and teams that make up its codebase. In a service-per-process architecture, the question "which component uses this memory?" answers itself: each process has its own heap, and the operating system reports its footprint. A monolith gives you one heap, one garbage collector, and a set of shared structures: caches, connection pools, thread pools, event buses, dependency-injection containers. A HashMap held by a framework registry may contain objects created by five different modules. No profiler can tell you which team owns it, because ownership is an organizational fact, not a runtime one.

That is why ownership must be defined explicitly before you take any measurement. Write down a mapping from code to modules and teams. Typical inputs are package or namespace prefixes (com.acme.billing.*), class name patterns, and source paths. Treat this mapping as versioned configuration. Without it, every heap dump analysis turns into an argument about whose objects those are, and results from two dumps taken a month apart cannot be compared.

Who benefits and when

This approach suits backend and platform engineers running large shared codebases. Use it when memory grows after a deploy, after an OOM incident, during capacity planning, or when you are deciding which module to extract into its own service.

When to skip it

In a small codebase, a single profiler view usually shows the culprit directly. Skip it as well when the growth is off-heap or in native memory, such as direct buffers, memory-mapped files, or native library allocations. Heap attribution cannot see that memory.

Shallow size, retained size and allocation volume: picking the right metric

Each heap metric answers a different question, and blaming the wrong module almost always comes from reading one metric as if it answered another.

Shallow size is a poor ownership signal. A HashMap entry array is small, while the strings and records it references are counted under generic JDK or runtime classes that no module owns. Retained size fixes this by charging the whole subgraph to whatever keeps it alive, so it is the metric for "who is holding memory."

Allocation volume answers a separate question. Consider a JSON serialization module that allocates heavily per request but releases everything by the end of it, alongside a pricing module that builds a few large lookup tables at startup and never frees them. An allocation profile puts serialization at the top. A heap dump taken under load puts pricing at the top. If the problem is GC pause time, serialization is the module to look at. If the problem is heap occupancy or out-of-memory errors, pricing is.

None of these metrics include direct buffers, native library allocations, or memory-mapped files. Per-module heap totals will therefore add up to less than process RSS, and the size of that gap tells you how much memory sits outside what a heap dump can attribute.

How it works: dominator trees and first-owned-frame allocation attribution

An object A dominates object B when every path from the GC roots to B passes through A. Each object has exactly one immediate dominator, so the dominators form a tree. An object's retained size is the sum of shallow sizes in its subtree: the memory that would be freed if it became unreachable. Because subtrees in a tree never overlap, the totals of sibling subtrees can be added without double-counting. To attribute memory, walk from each object up the tree to the first ancestor whose class or package maps to a module, and charge its shallow size there.

Sampled allocation profiling answers a different question. The runtime records a stack trace roughly every N allocated bytes. Walk each stack from the innermost frame outward, skip frames owned by no module, and charge the sample to the first frame that is module-owned. This attributes memory to the code that created it, not the code that holds it.

CopyGC root
 └─ platform.CacheRegistry          (no owner)
     ├─ entry ← created by billing   40 MB
     ├─ entry ← created by search    25 MB
     └─ entry ← created by auth       5 MB
 └─ billing.InvoiceIndex             30 MB

Dominator: platform 70 MB, billing 30 MB
Allocation: billing 70 MB, search 25 MB, auth 5 MB

Each method has a bias. Dominators push shared objects up to a common ancestor, which is often framework code, so infrastructure gets overcounted. Allocation sites blame whatever code called the allocator, and that can be a helper library rather than the module that asked for the work. Sampling is statistical, so small modules and rare allocations can show up as noise or not at all.

Copy                          +-----------+
                          |  GC root  |
                          +-----+-----+
               +----------------+----------------+
               |                                 |
               v                                 v
  +-------------------------+       +-------------------------+
  | platform.CacheRegistry  |       | billing.InvoiceIndex    |
  | infra; charged platform |       | module: billing   30 MB |
  +------------+------------+       +-------------------------+
               |
     +---------+---------+
     |         |         |
     v         v         v
 +-------+ +-------+ +-------+
 | entry | | entry | | entry |
 | 40 MB | | 25 MB | |  5 MB |
 +---+---+ +---+---+ +---+---+
     ^         ^         ^
     |         |         |    nearest module-owned frame
 [billing]  [search]   [auth]  on each allocation stack

 DOMINATOR (who retains it)       ALLOCATION (who created it)
 each entry walks up the tree     each sample walks its stack to
 to the first owned ancestor:     the first module-owned frame:
   platform   70 MB                 billing    70 MB  (40 + 30)
   billing    30 MB                 search     25 MB
                                    auth        5 MB

The same 100 MB heap attributed two ways: the dominator tree charges the cache entries to the shared platform registry that retains them, while allocation-site sampling charges them to the billing, search and auth code that created them.

Implementing the module mapping layer and rollup

The mapping layer turns those raw identifiers into owners. Start with a plain ordered table of rules, each pairing a prefix (a package, namespace or source path) with a module and a team. Resolve each class name or stack frame by longest match: billing.invoices.export should beat billing even if the broader rule appears first. Ordering then only breaks ties, so new rules don't silently shadow older ones. Anything that matches nothing goes into an explicit unattributed bucket. Runtime and framework internals such as collections, strings, the web server and the ORM get their own platform bucket instead of being spread across modules.

The script that applies the mapping should not depend on any particular runtime. It reads two inputs: a CSV of class name and retained bytes from a histogram or dominator export, and a stack-sample export. For each heap row it looks up the class. For each sample it walks frames from the leaf upward until the first owned frame. It then sums bytes per module and per team and reports the unattributed share as a percentage of the total.

Treat that percentage as a quality metric. If it grows, the mapping is stale, and every per-module number becomes less trustworthy. Keep the rules file in the repository next to the code, and add a check that fails when a new top-level package has no owner. A rename or extraction then shows up in review instead of quietly moving bytes into unattributed.

Run the rollup as a CI job against dumps from a fixed load test, and as an on-demand command for production captures. Use the same rules and inputs each time so that two runs can be compared.

Copy#!/usr/bin/env python3
# Usage: python3 rollup.py RULES.csv HEAP.csv STACKS.txt [MAX_UNATTRIBUTED_PCT]
# RULES.csv: prefix,module,team (kept in the repo; runtime internals map to module "platform")
# HEAP.csv: class,retained_bytes   STACKS.txt: collapsed stacks "root;...;leaf COUNT"
import csv, sys
from collections import Counter

def load_rules(path):
    with open(path, newline="") as f:
        rows = [(r["prefix"], r["module"], r["team"]) for r in csv.DictReader(f)]
    # Longest prefix wins; the stable sort keeps file order only as a tie-breaker.
    return sorted(rows, key=lambda r: -len(r[0]))

def owner(name, rules):
    return next(((m, t) for p, m, t in rules if name.startswith(p)), None)

rules = load_rules(sys.argv[1])
max_pct = float(sys.argv[4]) if len(sys.argv) > 4 else 10.0
mod_bytes, team_bytes, mod_samples = Counter(), Counter(), Counter()

with open(sys.argv[2], newline="") as f:
    for row in csv.DictReader(f):
        m, t = owner(row["class"], rules) or ("unattributed", "unowned")
        mod_bytes[m] += int(row["retained_bytes"]); team_bytes[t] += int(row["retained_bytes"])

with open(sys.argv[3]) as f:
    for line in f:
        stack, _, count = line.rstrip().rpartition(" ")
        # Walk leaf -> root; skip platform frames so allocations land on the calling module.
        hit = next((o for fr in reversed(stack.split(";")) if (o := owner(fr, rules)) and o[0] != "platform"), None)
        mod_samples[hit[0] if hit else "unattributed"] += int(count)

total = sum(mod_bytes.values()) or 1
for m, b in mod_bytes.most_common():
    print(f"{m:<24}{b:>14,} B {100 * b / total:6.1f}%  samples={mod_samples[m]}")
for t, b in team_bytes.most_common():
    print(f"team {t:<19}{b:>14,} B")
unattr = 100 * mod_bytes["unattributed"] / total
print(f"unattributed share: {unattr:.1f}%")
# Quality gate: a growing unattributed share means the rules file has drifted from the code.
sys.exit(1 if unattr > max_pct else 0)

A runtime-neutral rollup script that applies longest-prefix ownership rules to a heap CSV and collapsed stack samples, prints per-module and per-team totals, and fails (for CI) when the unattributed share exceeds a threshold.

Handling shared caches, registries and pools

A shared cache breaks dominator-based attribution. Common infrastructure allocates the cache object, and every entry is reachable only through it. The dominator tree therefore assigns the entire retained size to the infrastructure module, no matter which modules filled it. Plugin registries, object pools and interning tables behave the same way. You need a second pass that looks inside the container.

Prefer tagging at insert when you control the cache. Use key prefixes for third-party containers. Report a structure as shared only when entries are genuinely reused across modules, such as pooled buffers.

Illustrative estimate (assumed inputs, not a measurement): Assume a 4 GB live heap in which a shared cache retains 1.6 GB. Dominator analysis assigns all 1.6 GB, or 40% of the heap, to infra. Now re-attribute the entries by allocation site, assuming these shares:

• billing: 50%, or 0.8 GB
• search: 30%, or 0.48 GB
• notifications: 15%, or 0.24 GB
• unattributed: 5%, or 0.08 GB

infra drops to almost nothing, and billing becomes the largest owner. Record which method produced each number so that teams don't compare results across methods.

Diffing heap dumps and capturing them safely in production

A single dump shows what the heap holds; it can't show what is growing. Take two dumps from the same instance under steady load: one after warm-up, and one later, either after a deploy or after a fixed load phase. Then compare per-module retained size between them. Subtracting the first from the second removes baseline noise such as framework metadata, connection pools and interned strings. What remains is the change.

Timing matters here. A cache filling toward its configured bound will look like a leak if you diff a cold dump against a warm one. Take at least three dumps. Growth that levels off between the second and third is warm-up. Growth that continues at a steady rate is the likely leak.

Operational guardrails

Capturing a dump can stop or slow the process for the whole write. The output can be as large as the live heap, and it needs that much free disk. Analyzing it often takes a machine with more memory than the dump itself. To keep the impact contained:

• Capture on a canary or a replica where possible, and remove the instance from the load balancer first.
• Treat every dump as sensitive. It holds whatever was in memory, including session tokens, credentials and customer records.
• Encrypt dumps at rest and restrict access to the people investigating.
• Copy dumps off-host for analysis, and delete them on a fixed schedule.

Runtimes differ in what they give you. Some expose the full object graph, so you can do dominator analysis. Others mostly offer allocation sampling. On those, you diff allocation-site profiles rather than retained sets, and the attribution method should be labelled accordingly.

Copy#!/usr/bin/env bash
# Capture one labelled JVM heap dump on a drained canary, encrypt it, ship it off-host.
# Run once per phase (e.g. warm, post-deploy, plus-1h) on the SAME instance, then diff offline.
set -euo pipefail
umask 077                                  # dump files readable by owner only

PID="$1"                                   # JVM process id on the canary
LABEL="$2"                                 # phase label: warm | post-deploy | plus-1h
DUMP_DIR=/var/tmp/heapdumps
OUT="$DUMP_DIR/$(hostname)-$LABEL-$(date +%Y%m%dT%H%M%S).hprof"
mkdir -p "$DUMP_DIR"

# A dump can be as large as the live heap. Process RSS is a conservative upper bound.
need_kb=$(ps -o rss= -p "$PID")
free_kb=$(df --output=avail -k "$DUMP_DIR" | tail -1)
if (( free_kb < need_kb )); then echo "not enough free disk for dump" >&2; exit 1; fi

# Placeholders: replace with your load balancer's drain and re-enable steps.
drain_instance()   { echo "draining $(hostname)"; }
undrain_instance() { echo "re-enabling $(hostname)"; }

drain_instance
trap undrain_instance EXIT                 # always put the instance back, even on failure

# Dumps reachable objects only (this forces a full GC). The JVM pauses while it writes.
jcmd "$PID" GC.heap_dump "$OUT"

# The dump holds tokens, credentials and customer data. Encrypt it, then remove the plaintext.
gpg --batch --encrypt --recipient heap-analysis@example.com --output "$OUT.gpg" "$OUT"
rm -f "$OUT"

# Analyze on a separate host with more memory than the dump. Do not analyze on the canary.
scp "$OUT.gpg" analysis.internal.example:/secure/heapdumps/
rm -f "$OUT.gpg"

# Retention: clear anything left behind locally after 7 days.
find "$DUMP_DIR" -type f -mtime +7 -delete

A bash script that takes one labelled JVM heap dump with jcmd on a drained canary. It checks free disk first, encrypts the dump, copies it off-host for diffing against the other phases, and enforces local retention.

Alternatives, cost and engineering time

Each alternative trades setup cost against what it can see.

Cost. The mapping layer decays. Package renames and module moves silently push bytes into "unattributed," so keep the mapping in the main repository and fail CI on unmapped prefixes. Dumps are roughly heap-sized, so storage and analysis hosts need memory comparable to production. Dumps also hold whatever was live, including tokens, personal data and session state. That requires security review, restricted access and short retention.

Time saved. A regression routes to a team from a report instead of a dump-reading session. The same report runs every release, and extraction or capacity debates start from a measured share.

Illustrative estimate (assumed numbers, not a benchmark): hours saved per quarter = regressions × (manual triage hours − report triage hours). With 6 × (8 − 2), that is 36 hours.

Present shares rounded and approximate. Publish the method and the unattributed percentage next to every breakdown. Do not use the breakdown for chargeback or blame. Shared caches and attribution rules make it a triage aid, not an invoice.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to monolith memory profiling, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.