Is it actually a leak? Reading process.memoryUsage() and GC patterns
A memory leak in a long-running API is memory that stays reachable after the work that allocated it is done, so garbage collection can never reclaim it. Confirm the leak before you profile. Heap snapshots are expensive to capture and slow to read, and a healthy process can look alarming in the first few minutes.
Normal behaviour has two shapes. The first is a GC sawtooth: heapUsed climbs, then drops after each collection. The second is warm-up: memory rises while caches, connection pools and JIT-compiled code fill in, then plateaus. A leak looks different. The low points after each GC keep rising across thousands of requests and never level off. Compare post-GC minimums over time, not peaks.
process.memoryUsage() tells you where the growth is:
| Field | What it measures | Growth suggests |
|---|---|---|
rss |
Total resident memory of the process | Any leak; check the fields below to narrow it down |
heapTotal |
Heap space V8 has reserved | Heap expanding to fit live objects |
heapUsed |
Live JS objects in the V8 heap | JS-level leak: closures, maps, listeners |
external |
C++ memory bound to JS objects, including Buffers | Buffer or native binding retention |
arrayBuffers |
ArrayBuffer and Buffer backing stores (a subset of external) |
Unreleased binary payloads |
If rss grows while heapUsed stays flat, the leak is outside the JS heap. It could be in a native addon, an allocator, or Buffer backing stores. Heap snapshots barely show that memory, so they are the wrong tool for it. In that case, use native memory tooling instead.
This workflow is for backend and platform engineers running stateful Node processes that stay up long enough for retained objects to accumulate.
// memory-sampler.js: load with: node --expose-gc -r ./memory-sampler.js server.js
// Logs post-GC memory so you compare floors, not sawtooth peaks.
const INTERVAL_MS = 60_000; // sample once a minute; leaks show over hours
const toMB = (bytes) => +(bytes / 1024 / 1024).toFixed(1);
let baseline = null;
const timer = setInterval(() => {
// A forced full GC makes heapUsed a post-GC minimum.
// It pauses the process, so use it in staging or on a canary, not fleet-wide.
if (typeof global.gc === 'function') global.gc();
const { rss, heapTotal, heapUsed, external, arrayBuffers } = process.memoryUsage();
const now = { rss, heapTotal, heapUsed, external, arrayBuffers };
if (!baseline) baseline = now; // take the baseline after warm-up if possible
// Growth since baseline, per field, tells you where memory is going:
// heapUsed up -> JS objects retained (heap snapshot will help)
// arrayBuffers/external up -> Buffer or native binding retention
// rss up, others flat -> outside the JS heap (native tooling, not snapshots)
const delta = {};
for (const key of Object.keys(now)) {
delta[key] = toMB(now[key] - baseline[key]);
}
console.log(JSON.stringify({
ts: new Date().toISOString(),
mb: Object.fromEntries(Object.entries(now).map(([k, v]) => [k, toMB(v)])),
deltaMb: delta,
}));
}, INTERVAL_MS);
// Don't keep the process alive just for sampling.
timer.unref();
A preload script that logs process.memoryUsage() after a forced GC at a fixed interval, with per-field growth since baseline, so you can tell a rising post-GC floor in heapUsed from growth in Buffers, external memory or RSS alone.
Who should use this workflow
Use the heap snapshot workflow if all of the following are true:
- You run long-lived Node.js processes. Examples include HTTP and GraphQL APIs, WebSocket gateways, queue consumers and background workers that stay up for hours or days. Short-lived processes such as CLI tools, build scripts and most function-per-invocation deployments usually exit before retained objects matter. In those cases, a leak shows up as slow invocations or cold-start churn rather than an OOM kill.
- The sampler shows growth in
heapUsed. A rising post-GC floor inheapUsedmeans JavaScript objects are being retained, and those objects are exactly what a heap snapshot records. If onlyrssorexternalgrows, use native memory tooling instead. - You can reproduce load against a non-production instance, or you can safely drain one production instance. The method depends on sending controlled traffic and on taking snapshots that pause the process.
In practice, this covers several roles:
- Backend engineers who own the handlers and caches that usually cause the leak.
- Platform and SRE engineers who see the symptom first, as OOM kills, restart loops or memory alerts, and need evidence before they hand the problem to a service team.
- On-call engineers who have to decide whether to roll back, restart or raise limits while a root cause is found.
You do not need to know V8 internals. You need to be able to run the service locally with --inspect, open Chrome DevTools and read a retainer chain.
Reproducing the leak locally under controlled load
A leak that adds a few kilobytes per request is invisible over an hour of production traffic but obvious after tens of thousands of requests fired back to back. The goal locally is to compress days of growth into minutes by isolating the suspect endpoint and hitting it far harder than real traffic would.
Start the server with node --inspect server.js. By default the inspector binds to 127.0.0.1:9229, so only processes on your machine can attach. That is the right setting for local work, because an exposed inspector port gives remote code execution to anyone who can reach it. Chrome DevTools will list the process under chrome://inspect.
The load script should send a fixed, known number of requests to one endpoint, wait for all of them to finish, and then stop. Use the same payload shape each time and keep concurrency modest so you are measuring retention, not queueing. A known count matters for two reasons:
- You can do arithmetic on the result. Suppose each batch is 10,000 requests and 10,000 instances of some object survive (illustrative estimate). That points directly at per-request retention.
- You can rule out one-time growth. The first requests trigger lazy module loading, JIT compilation, connection pools filling and caches warming. V8 also grows its heap regions based on recent allocation, so memory can step up and stay there without anything leaking.
Make the batch large enough that per-request growth clearly outweighs these one-off costs. A batch that is too small turns warm-up noise into a false positive.
// load.js: send a fixed, known number of requests to one endpoint, then exit.
// Start the server first: node --inspect server.js (inspector on 127.0.0.1:9229)
// Run: node load.js 10000
// Requires Node 18+ for the global fetch API.
const TOTAL = Number(process.argv[2] ?? 10000); // known count, so retained objects can be compared to it
const CONCURRENCY = 10; // modest, so we measure retention rather than queueing
const URL = ''; // the suspect endpoint (placeholder)
const PAYLOAD = JSON.stringify({ customerId: 'cust_123', items: [{ sku: 'SKU-1', qty: 2 }] }); // same shape every time
let sent = 0;
let failed = 0;
async function worker() {
// Each worker claims the next request number until the batch is exhausted.
while (sent < TOTAL) {
sent++;
try {
const res = await fetch(URL, {
method: 'POST',
headers: { 'content-type': 'application/json' },
body: PAYLOAD,
});
await res.arrayBuffer(); // drain the body so the client does not hold sockets open
if (!res.ok) failed++;
} catch {
failed++;
}
}
}
const start = Date.now();
await Promise.all(Array.from({ length: CONCURRENCY }, worker));
// Failed requests may not exercise the leaking path, so report them before trusting the count.
console.log(`sent=${TOTAL} failed=${failed} elapsedMs=${Date.now() - start}`);
A minimal Node.js load script (saved as load.mjs or in an ES module project) that sends a fixed number of identical POST requests at low concurrency to one endpoint on a server started with --inspect, so heap growth can be compared against a known request count.
How it works: what a heap snapshot contains
A heap snapshot is a serialized graph of everything reachable in the V8 heap at one moment. Knowing its structure makes the DevTools views easier to read.
- Nodes are heap objects: plain objects, arrays, strings, closures, Maps, compiled code, and internal V8 structures such as
system / Context. Each node records a type, a name (usually the constructor name, such asIncomingMessageorMap), and a shallow size. - Edges are references between nodes. Examples include an object property, an array element, a closure's link to its captured Context, or an internal pointer. The Retainers pane is built by walking these edges backwards.
- GC roots are the starting points the garbage collector treats as always alive. They include the global object, module scopes, active handles such as timers and sockets, and V8 internals. An object appears in a snapshot only if a path of edges leads to it from a root.
- Retained size is calculated by DevTools from the graph. It is the memory that would be freed if that one node became unreachable, meaning the node plus everything that is reachable only through it.
- Object IDs are assigned by V8 to heap objects, and they stay stable across snapshots taken from the same process. The Comparison view uses these IDs to tell which objects are new since the baseline, which were deleted, and which survived.
The file is a .heapsnapshot JSON document. DevTools loads it, computes retained sizes, and groups nodes by constructor for the Summary and Comparison views.
The full workflow puts these pieces together:
- Confirm that
heapUsedis growing after GC. - Drive a known amount of traffic.
- Capture snapshots at fixed points.
- Diff object counts between the snapshots.
- Follow the retainer edges from a growing object back to the root that keeps it alive.
The root, or the first link you own on the path to it, is where the fix goes.
How the three-snapshot technique works
The method separates one-time allocations from growth that continues with every request. The sequence:
- Warm up. Send enough traffic to fill connection pools, populate caches, and let lazy initialization finish.
- Snapshot 1 (baseline).
- Send N requests, then let the process go idle for a few seconds.
- Snapshot 2.
- Send the same N requests again, then idle.
- Snapshot 3.
load: [warm-up] [ N requests ] [ N requests ]
████░░░░░░░░░██████████░░░░░░░░░░░██████████░░░░░
snap: S1 S2 S3
baseline after 1st batch after 2nd batch
DevTools runs a full garbage collection before it writes each snapshot. Anything that appears in a snapshot was therefore still reachable from a GC root at that moment. Garbage that has not been collected yet does not show up.
What you are looking for is an object type whose count rises by roughly the same amount between S1→S2 and between S2→S3, in proportion to N. Suppose N is 1,000 and a constructor gains about 1,000 instances (or 2,000, or 3,000) in each interval. That is one or a few retained objects per request, which is the signature of a leak.
A single snapshot tells you what is large, not what is growing. A big cache or the compiled module graph will dominate it and look suspicious even when it is stable. A diff of two snapshots is better, but one-off growth still shows up in it: a cache filling to its cap, a pool reaching its configured size, optimized code being generated. The third snapshot filters those out. Real one-off growth flattens between S2 and S3, while a leak keeps climbing at the same rate.
┌─────────┐ ┌────┐ ┌────────────┐ ┌────┐ ┌────────────┐ ┌────┐
│ warm-up │──▶│ S1 │──▶│ N requests │──▶│ S2 │──▶│ N requests │──▶│ S3 │
│ caches │ │ │ │ then idle │ │ │ │ then idle │ │ │
└─────────┘ └─┬──┘ └────────────┘ └─┬──┘ └────────────┘ └─┬──┘
│ │ │
└────────────┬────────────┴────────────┬────────────┘
▼ ▼
┌─────────────────┐ ┌─────────────────┐
│ Δ1 = S2 − S1 │ │ Δ2 = S3 − S2 │
│ after batch 1 │ │ after batch 2 │
└────────┬────────┘ └────────┬────────┘
└────────────┬────────────┘
▼
┌───────────────────────────────────────────┐
│ compare instance counts per constructor │
│ (Δ1 vs Δ2, after GC: reachable objects) │
└────────┬─────────────────────────┬────────┘
▼ ▼
┌───────────────────────────────┐ ┌─────────────────────────────┐
│ Δ1 > 0, Δ2 ≈ 0 │ │ Δ1 ≈ Δ2 ≈ k × N │
│ one-off growth that flattens │ │ climbs with every batch │
│ cache/pool at cap, JIT code │ │ retained per request: LEAK │
│ → ignore │ │ → inspect retainers │
└───────────────────────────────┘ └─────────────────────────────┘
The three-snapshot sequence: after warm-up, snapshots S1, S2 and S3 bracket two identical batches of N requests, and only constructors whose counts grow by a similar multiple of N in both deltas are treated as leaks.
Reading the Chrome DevTools Memory panel and following retainers
Load S3 and switch the view selector from Summary to Comparison, with S2 as the baseline. Summary groups live objects by constructor, which is useful but noisy. Comparison shows only what changed: # New, # Deleted, # Delta, and Size Delta per constructor. A leak appears as a constructor whose # Delta grows in step with the number of requests you sent.
Sort by retained size, not shallow size. Shallow size is the object's own memory. A Map header is tiny. Retained size is everything that would be freed if that object were collected, so a small Map holding thousands of response bodies ranks at the top.
Expand a suspicious instance and open the Retainers pane. Read it bottom-up: each row is "held by", until you reach a GC root such as a module scope or the global object. The Distance column counts hops from the root. The fix usually belongs at the first link you own.
| Leak shape | What the retainer chain shows |
|---|---|
| Module-level Map/array cache | table or elements → Map/Array → context of a module function |
| Per-request EventEmitter listener | your function → array under _events → a long-lived emitter |
| Uncleared timer or interval | _onTimeout → Timeout → Node's internal timer list |
| Closure capturing req/res | IncomingMessage → context → (closure) held by a callback |
Real chains rarely look this clean. They pass through framework routers, ORM identity maps, and connection-pool internals. Bundled or minified code turns names into single letters. As a result, the code you need to change is often several hops above the first unfamiliar frame.
Diagnosing a closure leak: a worked example
Consider a handler that wants to push a notification when a background job finishes. It registers a listener on a module-level EventEmitter named jobEvents and, inside that listener, reads req.user.id and writes to res. The listener is never removed, so every request adds one function to an emitter that lives for the whole process.
The closure captures more than req.user.id. V8 stores captured variables in a Context object shared by all closures created in the same scope. If any function in that scope references req, the whole req object stays reachable, along with its socket, headers, and any parsed body attached by middleware. In a snapshot, the listeners appear under (closure). Their captured scopes appear as system / Context entries. Sorting the comparison view by retained size usually puts IncomingMessage near the top, and its retainer chain reads roughly: jobEvents → _events → jobDone (an array of listeners) → (closure) → context (a system / Context) → req. Node's MaxListenersExceededWarning for that emitter is often the first sign of this leak, before memory growth is obvious.
The fix has two parts. First, give each listener a bounded lifetime. Call removeListener when the response emits close. Using once alone is not enough: if the job event never fires for that request, the listener stays registered. Second, copy the fields you need (for example const userId = req.user.id) into a separate variable before creating the closure, and make sure no other function in that scope references req. Once you take the next snapshot, the (closure) count should stay flat and IncomingMessage should disappear from the retained objects.
const http = require('node:http');
const { EventEmitter } = require('node:events');
// Module-level emitter: lives as long as the process.
const jobEvents = new EventEmitter();
setInterval(() => jobEvents.emit('jobDone', 'job-42'), 5000);
// LEAKY: one listener per request, never removed. The closure references
// req, so the shared Context keeps the whole IncomingMessage alive.
// Snapshot retainer chain: jobEvents -> _events -> jobDone -> (closure) -> context -> req
function leakyHandler(req, res) {
res.writeHead(200, { 'Content-Type': 'text/plain' });
jobEvents.on('jobDone', (jobId) => {
res.write(`${jobId} done for ${req.headers['x-user-id']}\n`);
});
}
// FIXED: copy the one field needed, and tie the listener's lifetime to the response.
function fixedHandler(req, res) {
const userId = req.headers['x-user-id']; // no closure below references req
res.writeHead(200, { 'Content-Type': 'text/plain' });
const onJobDone = (jobId) => {
res.write(`${jobId} done for ${userId}\n`);
};
jobEvents.on('jobDone', onJobDone);
// Remove on close, whether or not the job event ever fired for this request.
// res still references req internally, so removal is what lets both be collected.
res.on('close', () => jobEvents.removeListener('jobDone', onJobDone));
}
http
.createServer((req, res) =>
req.url === '/leaky' ? leakyHandler(req, res) : fixedHandler(req, res)
)
.listen(3000);
A plain Node.js http server shows two handlers. The leaky one adds a listener to a long-lived EventEmitter on every request and captures req. The fixed one copies only the user ID and removes its listener when the response closes.
Capturing heap snapshots in staging and production safely
Reproducing locally only gets you so far. Some leaks appear only under real traffic patterns, so at some point you need a snapshot from a live instance. Do it in a way that doesn't create a second incident.
Getting access
For interactive inspection, start the process with --inspect=127.0.0.1:9229 and never bind it to 0.0.0.0. From your workstation, open an SSH tunnel such as ssh -L 9229:127.0.0.1:9229 user@host, then attach DevTools to localhost:9229. Enable this only for the debugging session and remove it afterward. Anyone who reaches the inspector port can execute arbitrary code inside your process.
When you only need a snapshot file, skip the debugger. Call v8.writeHeapSnapshot() from an internal admin route that requires authentication; it writes a .heapsnapshot file and returns its path. Alternatively, start Node with --heapsnapshot-signal=SIGUSR2 and send that signal to the process. Avoid SIGUSR1, which Node reserves for activating the inspector.
When not to do it
- Taking a snapshot blocks the event loop. Serialization is synchronous, so requests stall until it completes. Drain the instance from the load balancer first.
- Writing needs a lot of extra memory. Plan for roughly the size of the heap itself. In a container close to its memory limit, the snapshot can trigger an OOM kill, and you lose both the process and the evidence.
Handling the files
A heap snapshot contains every live string in the process, including tokens, passwords, PII, and request bodies. Treat each file as a credential dump. Store snapshots in restricted locations, limit who can read them, and delete them as soon as the analysis is finished.
Fixing, verifying, and the cost of alternatives
Most fixes take one of four forms:
- Give caches a maximum size or a TTL, so eviction happens even when no code path calls
delete. - Remove every listener in the same lifecycle that added it.
- Clear intervals and long timeouts when their owner shuts down.
- Have closures copy out the field they need, such as an ID, so they do not hold the whole request.
To verify a fix, rerun the identical load script and the three-snapshot sequence. The fix holds when objects allocated between snapshots 2 and 3 no longer survive and heapUsed levels off after GC. Then export heapUsed as a metric and alert on sustained growth over hours, not on single spikes.
| Approach | Cost | What it hides | When acceptable |
|---|---|---|---|
| Scheduled restarts | Low, but recurring | Root cause, growth rate | Short-term, while a fix is in progress |
| Higher memory limit | Infrastructure spend | Delays OOM, doesn't stop it | Rarely, alongside a fix |
| Guessing from RSS graphs | Engineer hours | Which objects are retained | Never on its own |
| Native-memory tooling | Setup time | Nothing, for Buffer and addon leaks | When the JS heap is flat but RSS grows |
The real gain is time. A repeatable reproduce, snapshot and diff loop replaces open-ended suspect hunting with a finite list of retainers to work through.
Illustrative estimate (assumed numbers, not a benchmark): Masking per month = (pods × extra GB × cost per GB-month) + (incidents × hours per incident × hourly rate). With 20 pods, 1 GB, $5, 2 incidents, 4 hours and $100: $100 + $800 = $900/month. A one-time fix of 16 hours × $100 = $1,600, which breaks even in about two months.
How this workflow saves engineering time
Most of the time lost on memory leaks goes into searching without evidence. Engineers read diffs from recent deploys, comment out middleware, or watch RSS graphs and wait for the next OOM. Each step of this workflow removes a specific kind of wasted effort:
- Triage before profiling. The post-GC sampler shows within hours whether growth is in
heapUsed, inexternal/arrayBuffers, or in RSS alone. This stops you from spending a day reading heap snapshots for a leak that lives in native memory, or chasing warm-up growth that was never a leak. - Fast feedback locally. A fixed-count load script against one endpoint compresses days of production growth into a short local run. Each hypothesis takes minutes to test instead of a deploy-and-wait cycle.
- Arithmetic instead of intuition. Because N is known, a constructor whose count grows by roughly N per batch is a strong candidate, and a constructor whose count flattens can be dropped. The third snapshot removes most of the caches and pools that would otherwise look suspicious.
- The retainer chain points at the code. Following retainers from the growing object to the first link you own shows which module, emitter, timer or closure holds it. You no longer have to bisect the codebase.
- The same loop verifies the fix. Rerunning the identical load script and snapshot sequence gives a clear pass or fail. You do not need to wait days in production to learn whether memory has stopped climbing.
- Fewer repeat incidents. A
heapUsedmetric with an alert on sustained growth catches the next leak while it is small and the change that caused it is recent. That leak is cheaper to find than one that surfaces weeks later as restart loops.
The scripts and the procedure are reusable. Once a team has a sampler preload, a load script and a written snapshot procedure, the next leak investigation starts from step one instead of from scratch. The cost comparison in the previous section shows how that time translates into cost under the assumed inputs.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to node.js memory leak heap snapshot, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Consumers learn about a breaking change at review time rather than from production errors.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.