Node.js Memory Leak Heap Snapshot Diagnosis in Production APIs

Is it really a leak? Reading process.memoryUsage() over time

A memory leak in a Node.js API is memory that stays reachable after the work that needed it has finished. Because the garbage collector can only free unreachable objects, the leaked memory grows with traffic and never comes back. High memory use alone does not prove a leak. V8 grows the heap lazily and keeps it large when allocation pressure is high, so a busy process can look bloated while being perfectly healthy. Confirm the leak first, so you don't spend hours reading snapshots of a heap that was never broken.

Sample process.memoryUsage() every 30 to 60 seconds under steady load, and log rss, heapTotal, heapUsed, external and arrayBuffers as structured fields. The sampling snippet below does this. Watch the troughs of heapUsed, not its peaks. Each trough is roughly what survived a collection. If the troughs rise steadily, you have the leak signature: a sawtooth whose floor keeps climbing. A high but flat plateau usually means a large working set or generous heap sizing.

If rss climbs while heapUsed stays flat, the growth is outside the JavaScript heap. Likely sources are Buffers, native addons or allocator fragmentation. Heap snapshots record JS objects, so they will tell you little about this case.

Copy// memory-sampler.js: load early, e.g. require('./memory-sampler') in the API entry point
const SAMPLE_INTERVAL_MS = 30_000; // 30-60s is enough to see the trend
const toMiB = (bytes) => Math.round((bytes / 1024 / 1024) * 10) / 10;

function sampleMemory() {
  // All values are in bytes. arrayBuffers is included in external.
  const { rss, heapTotal, heapUsed, external, arrayBuffers } = process.memoryUsage();

  // One JSON line per sample keeps it easy to graph in any log pipeline.
  console.log(JSON.stringify({
    msg: 'memory_sample',
    ts: new Date().toISOString(),
    pid: process.pid,
    uptimeSec: Math.round(process.uptime()),
    rssMiB: toMiB(rss),             // whole process: heap + native + allocator overhead
    heapTotalMiB: toMiB(heapTotal), // heap V8 has reserved
    heapUsedMiB: toMiB(heapUsed),   // live JS objects plus not-yet-collected garbage
    externalMiB: toMiB(external),   // C++ memory bound to JS objects (Buffers, etc.)
    arrayBuffersMiB: toMiB(arrayBuffers),
  }));
}

const timer = setInterval(sampleMemory, SAMPLE_INTERVAL_MS);
timer.unref(); // the sampler should never keep the process alive on shutdown
sampleMemory(); // record a baseline at startup

// Reading the output: plot the troughs of heapUsedMiB over hours of steady load.
// Rising troughs point to retained JS objects. Flat heapUsedMiB with rising
// rssMiB means the growth is outside the JS heap.

A dependency-free sampler that logs process.memoryUsage() fields as structured JSON every 30 seconds, so you can graph post-GC heapUsed troughs against rss, external and arrayBuffers over time.

Who should use heap snapshot diagnosis

This workflow is for engineers who run long-lived Node.js processes: HTTP and GraphQL APIs, WebSocket servers, queue workers and other services that handle many requests in one process over hours or days. Those are the processes where small per-request retention adds up. Short-lived processes such as CLI tools, build scripts or functions that are torn down after each invocation rarely accumulate enough leaked memory to matter.

Each part of the work has a natural owner:

• Service owners (the backend engineers who wrote and maintain the API) read the snapshots and fix the code. Retainer paths name your modules, closures and classes, and only someone who knows the code can tell whether a Map with 50,000 entries is a bug or a deliberate cache.
• Platform, SRE or on-call engineers own the capture path. That means the start flags, the runbook, the copy-out procedure, the storage location and who can read the files.
• Security or compliance reviewers should sign off on how snapshots are stored and how long they are kept, because the files contain raw process memory.

You need basic familiarity with Chrome DevTools' Memory panel, which is where .heapsnapshot files are loaded and analysed. You also need shell access to the running container or host.

When to use heap snapshots, and when to reach for something else

Use heap snapshots when all of these are true:

• The sampler shows a rising heapUsed floor. Post-GC troughs climb under steady load over hours, not just during warm-up.
• The growth is in the JS heap. heapUsed is rising, not only rss. If only rss or external grows, start with the off-heap tools listed in the alternatives table at the end of this article.
• You can afford the capture. The instance has memory headroom for serialization, and you can drain it or use a canary so the event-loop pause doesn't fail live requests.
• You need to know why memory is retained, not only where it was allocated. Snapshots give retainer chains. Sampling heap profilers give allocation sites at lower cost and are a better first step when you can't yet tolerate a snapshot's pause.

Typical triggers are a memory-usage alert on a post-GC trend, OOM kills or restarts that recur on a regular cadence, gradually increasing GC pause times, and memory growth that started after a specific deploy. In the last case, snapshotting the new build and comparing it with the previous build narrows the search quickly.

Don't take a snapshot as a reflex during an active outage when the process is already near its memory limit. Restart or scale first to restore service, then reproduce the leak on a drained instance or in staging, where the capture can't make things worse.

How the diagnosis works end to end

The method rests on one property of V8's garbage collector: an object is kept only if a chain of references leads to it from a GC root. A leak is therefore always a reference that should have been dropped and wasn't. Diagnosis means finding that reference.

The workflow has five steps, each covered in its own section below:

1. Confirm the trend. Sample process.memoryUsage() and check that post-GC heapUsed rises under steady load. This rules out a large but stable working set.
2. Capture snapshots safely. Trigger heap snapshots on a drained production instance or a faithful staging reproduction, copy the files off the container, and handle them as sensitive data.
3. Diff the snapshots. Take three snapshots spaced by load, then isolate objects that were allocated after warm-up and survived later traffic. Those are the objects the application is accumulating.
4. Follow the retainers. For the growing object type, open the retainer chain and walk up toward the root until you reach the reference that shouldn't exist. Common culprits are a module-level cache, an uncleared timer, an emitter listener or a never-settling promise.
5. Fix and verify. Remove or bound that reference, then rerun the same load against both builds. The fixed build's post-GC heap should level off, and a soak test should keep it that way.

How V8 heap snapshots work and when not to take one

A heap snapshot is a serialized graph of everything V8's garbage collector can reach at one instant. Nodes are heap objects: plain objects, arrays, strings, closures, compiled code, and the hidden classes that describe object shapes. Edges are the references between them, such as named properties, array elements, variables captured in a closure's context, and internal V8 links. At the top sit the GC roots: the global object, the active stack, and handles held by the runtime. An object survives collection only if some path from a root reaches it. That is why the snapshot answers the question a leak raises: not "what is big?" but "what is still holding this?" For each object it reports shallow size, retained size (what would be freed if it vanished), and the retainer chain back to a root.

That makes it the right tool when heapUsed climbs. The engineers who own the service should read the snapshots, because retainer paths only make sense to someone who knows the code. Platform teams should own the capture path, storage, and access controls.

Avoid or delay a snapshot when its cost is the bigger risk:

• It blocks the event loop. Writing is synchronous. Probes can time out mid-write, and the orchestrator may restart the process before the file is complete.
• It needs a lot of extra memory. Serialization can need memory approaching the heap's own size. In a tight container that can trigger an OOM kill.
• It contains raw data. Tokens, credentials, and personal data are in the file. Treat it like a database dump.
• It cannot see native memory. Growth in RSS alone will not show up in a snapshot.

Drain the instance from the load balancer or use a canary, and capture early, while there is still memory headroom.

Capturing snapshots from a running production process

Capture while the process is still well below its memory limit, so there is still headroom on the host for the serialization step.

Triggering the write

Start the process with --heapsnapshot-signal=SIGUSR2, then send the signal with kill -USR2 <pid>. In a container, that is usually PID 1. Node writes a file named like Heap.<timestamp>.<pid>...heapsnapshot into the process's current working directory. That directory must be writable, which read-only root filesystems often are not, so set the working directory deliberately.

The alternative is calling v8.writeHeapSnapshot() from an admin route. It returns the generated filename, which the route can log or respond with. Bind that route to an internal-only port and require authentication. An exposed endpoint lets anyone freeze your event loop, because the write is synchronous, and lets them extract your process memory.

Getting the file out

Copy the file before anything restarts the pod, either kubectl cp (which needs tar in the image) or docker cp. A container restart can discard its writable layer, and a liveness probe that fails during the write will cause exactly that. Relax the probe before you trigger the snapshot.

CopySIGUSR2 / admin call
        │
        ▼
 drained pod ──► Heap.*.heapsnapshot (cwd)
        │
        ▼
 kubectl cp / docker cp ──► encrypted, access-restricted bucket
                                  (short retention, then delete)

Treat it as sensitive

A snapshot holds whatever was in memory, such as tokens, request bodies and personal data. Restrict access to the people debugging, store it encrypted, set a short retention period, and delete the copy left on the pod.

Copy#!/usr/bin/env bash
set -euo pipefail

NS=payments
POD=orders-api-7d9f8c6b5-x2kqp   # pod already removed from Service endpoints (drained)
DIR=/var/snapshots               # the process's working directory

kubectl exec -n "$NS" "$POD" -- kill -USR2 1

prev=-1
while true; do
  sleep 5
  size=$(kubectl exec -n "$NS" "$POD" -- sh -c "du -k $DIR/*.heapsnapshot 2>/dev/null | cut -f1 | head -n1" || true)
  [ -n "$size" ] && [ "$size" = "$prev" ] && break
  prev=$size
done

FILE=$(kubectl exec -n "$NS" "$POD" -- sh -c "ls -t $DIR/*.heapsnapshot | head -n1")

umask 077                        # local copy readable only by the current user
kubectl cp "$NS/$POD:$FILE" "./$(basename "$FILE")"

gpg --encrypt --recipient heap-debug@example.internal "./$(basename "$FILE")"

shred -u "./$(basename "$FILE")"
kubectl exec -n "$NS" "$POD" -- rm -f "$FILE"

A script that signals a drained pod to write a heap snapshot, waits for the write to finish, copies the file out with kubectl cp, encrypts it, and deletes the plaintext copies.

The three-snapshot technique and reading the results

Take three snapshots from the same process. Take the first after warm-up, once caches, connection pools and lazily loaded modules have settled. Then apply sustained or replayed load and take the second. Apply more load and take the third. Load the third snapshot and filter to objects allocated between snapshots 1 and 2. Warm-up growth happened before snapshot 1, so the filter excludes it. Short-lived request objects allocated in that window should be collected by snapshot 3. Whatever is still present after more traffic is the strongest leak candidate.

Shallow size is the memory an object holds directly. Retained size is the memory that would be freed if that object were collected. Sort by retained size, because a small Map that holds thousands of entries has a tiny shallow size and a large retained size.

Some entries are V8 groupings, not constructors you wrote:

• (closure): function instances and the context they capture.
• (string): string data.
• (array): internal backing stores, such as array elements and object property storage.
• (system): V8 internals.

Select a suspect object and open the Retainers pane. Follow the shortest path upward until you reach something that should not hold request data. Common roots are a module-level variable, an active timer, or an emitter's listener array.

Use named classes or constructors instead of anonymous object literals. A plain literal appears under the generic Object group, while class SessionEntry appears under its own name in the snapshot.

Common leak patterns in APIs: closures, caches, listeners, timers and promises

A closure keeps its whole enclosing scope reachable, not only the variables it uses. Suppose a handler starts a setInterval that polls job status and references req.body.jobId. Until the interval is cleared, the timer keeps the closure alive, and the closure keeps the entire req, its parsed body and any query results in that scope. In the Retainers pane this shows as a (closure) whose context field points to a system / Context object holding req or rows. The fix has two parts. Copy jobId into a local primitive before you create the callback, and call clearInterval when the job finishes or the request is aborted.

A WeakMap keyed by the request object, or a WeakRef to it, lets GC collect entries without explicit cleanup. The cost is that memory use and cache hits then depend on when GC runs. Tests pass under one heap size and behave differently under another. Prefer explicit lifecycle management, because it is deterministic and easy to assert in tests.

Copy// BEFORE: the interval is never cleared. Its closure keeps the handler's
// context (req, res, rows) reachable for the life of the process.
app.post('/jobs/watch', async (req, res) => {
  const rows = await db.query('SELECT * FROM job_events WHERE job_id = $1', [req.body.jobId]);
  setInterval(async () => {
    const status = await getJobStatus(req.body.jobId); // captures req
    if (status === 'done') res.json({ status, events: rows.length });
  }, 1000);
});

// AFTER: copy the primitive out, keep large values out of the callback's
// scope, and clear the timer on every exit path.
app.post('/jobs/watch', (req, res) => {
  const jobId = String(req.body.jobId); // primitive; req is no longer captured
  watchJob(jobId, res);
});

function watchJob(jobId, res) {
  const timer = setInterval(async () => {
    try {
      const status = await getJobStatus(jobId);
      if (status === 'done') { stop(); res.json({ status }); }
    } catch (err) {
      stop(); // a failed poll must not leave the interval running
      if (!res.headersSent) res.status(502).json({ error: 'status check failed' });
    }
  }, 1000);

  function stop() {
    clearInterval(timer);
    res.removeListener('close', stop);
  }
  // Client disconnect or normal completion both end the watch.
  res.once('close', stop);
}

const JOB_STATUS_BASE = process.env.JOB_STATUS_BASE; // internal job service base address

async function getJobStatus(jobId) {
  // Outbound timeout so a hung upstream cannot leave a pending promise forever.
  const r = await fetch(`${JOB_STATUS_BASE}/jobs/${encodeURIComponent(jobId)}/status`, {
    signal: AbortSignal.timeout(2000),
  });
  return (await r.json()).status;
}

An Express-style handler whose uncleared setInterval closure retains req, res and query rows, with a fixed version that captures only a primitive job ID, clears the timer when the response closes or a poll fails, and gives the outbound status call an abort timeout.

Reproducing locally and verifying the fix

Before touching code, reproduce the leak somewhere you control. Point a small load script at a staging instance and replay the request mix that correlated with growth in production: same endpoints, similar payload sizes, and similar concurrency. A loop of fetch calls with a fixed concurrency limit is usually enough. Fidelity matters more than raw volume, because a leak tied to one route or one error path will not show up under generic traffic.

Start the staging process with --expose-gc and call global.gc() immediately before each snapshot. This forces a full collection, so the snapshot holds only objects that are actually reachable. Without it, garbage that is merely waiting for collection adds noise to the diff and can hide a slow leak. Keep the gc() call in a debug-only hook or a separate harness script. A forced full collection pauses the main thread, and code that calls global.gc() unconditionally throws in any process started without the flag.

To verify the fix, run the same load for the same duration against both builds. After each fixed interval, force GC and record heapUsed. The leaking build shows a staircase that keeps rising. The fixed build should rise during warm-up while caches fill and JIT code settles, then hold flat within a narrow band. A plateau at a higher level than expected is acceptable if it stays bounded.

Then make the check permanent. A soak test that runs for an hour or more in CI or staging and fails when post-GC heapUsed keeps trending upward will catch the next regression before it reaches production.

Copy// soak.mjs: staging-only load + post-GC heap check (Node 18+, global fetch).
// Server side, debug builds only; never register this in production:
//   if (process.env.DEBUG_HEAP === '1' && typeof global.gc === 'function') {
//     app.get('/__debug/heap', (req, res) => { global.gc(); res.json(process.memoryUsage()); });
//   }
// Start the server with: DEBUG_HEAP=1 node --expose-gc server.js
const BASE = process.env.TARGET ?? '';
const CONCURRENCY = 20;
const DURATION_MS = Number(process.env.SOAK_MS ?? 60 * 60 * 1000);
const SAMPLE_MS = 60 * 1000;
// Replay the route mix that correlated with heap growth in production
const routes = ['/orders?limit=50', '/orders/12345', '/search?q=widget'];

let stop = false;
async function worker() {
  while (!stop) {
    const path = routes[Math.floor(Math.random() * routes.length)];
    try { const r = await fetch(BASE + path); await r.arrayBuffer(); } catch {}
  }
}

const samples = [];
const timer = setInterval(async () => {
  try {
    const { heapUsed } = await (await fetch(BASE + '/__debug/heap')).json();
    samples.push(heapUsed); // post-GC value, so it reflects reachable objects only
    console.log(new Date().toISOString(), (heapUsed / 1048576).toFixed(1), 'MiB');
  } catch (err) { console.error('sample failed:', err.message); }
}, SAMPLE_MS);

setTimeout(() => { stop = true; clearInterval(timer); }, DURATION_MS);
await Promise.all(Array.from({ length: CONCURRENCY }, worker));

// Drop the first quarter as warm-up, then compare early vs late steady-state averages
const steady = samples.slice(Math.floor(samples.length / 4));
const q = Math.max(1, Math.floor(steady.length / 4));
const avg = (a) => a.reduce((s, x) => s + x, 0) / a.length;
const growth = avg(steady.slice(-q)) / avg(steady.slice(0, q));
console.log('steady-state growth ratio:', growth.toFixed(3));
process.exit(growth > 1.10 ? 1 : 0); // threshold is an example; tune per service

A staging soak script that drives fixed-concurrency load, samples post-GC heapUsed through a debug-only endpoint, and exits non-zero when the heap keeps climbing after warm-up.

Alternatives, cost and making diagnosis operational

Restarts and raised limits are not wrong. They are triage. A nightly rolling restart or a larger --max-old-space-size can keep a leaking service up while someone investigates. Left in place, though, they hide the next regression behind the current one. They also reserve memory the service doesn't need, and every restart drops warm caches and JIT-optimized code. That causes latency spikes and in-flight request churn.

Illustrative estimate (assumed numbers, not a benchmark). Firefighting cost per quarter = incidents × hours per incident × engineers. One-time setup cost = setup hours.

Assume 3 incidents per quarter, 6 hours each, and 2 engineers. That is 3 × 6 × 2 = 36 hours per quarter. Assume a one-time setup of 16 hours. In that case the setup pays for itself before the end of the first quarter.

The setup covers four things:

• Ship --heapsnapshot-signal in the start command so capture never requires a redeploy.
• Write a runbook for taking a snapshot and copying it off the container before the container is replaced.
• Alert on the trend of post-GC heap usage rather than raw RSS.
• Set --max-old-space-size deliberately below the container limit. This leaves headroom for buffers, native allocations and the snapshot itself.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to node.js memory leak heap snapshot, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.