Node.js Event Loop Lag Monitoring for Blocked APIs in Production

What event loop lag measures and why one stall slows every request

Event loop lag is the gap between when a callback is due and when the loop actually runs it. Schedule a 10 ms timer, see it fire at 60 ms, and the loop lagged 50 ms. Node runs your JavaScript on one thread, so while any callback runs synchronously, nothing else does. Timers, socket reads, and resolved promises all wait.

That is why one stall slows every request on the process, not just the request that caused it. A request whose database result arrived mid-stall still sits in the queue until the stall ends. So does a new connection that needs only a few microseconds of work.

Illustrative estimate (assumed inputs, not measured):

• One process serves 100 requests per second.
• Each request normally takes 5 ms.
• A single 200 ms synchronous stall occurs once in a 10-second window, which covers 1,000 requests.

About 20 requests arrive or complete during the stall. They wait between roughly 0 and 200 ms extra, about 100 ms on average.

• Mean: rises by about 2 ms (20 × 100 ms ÷ 1,000), which looks harmless.
• p99: those 20 requests are 2% of traffic, so p99 jumps past 100 ms.
• Max: close to 205 ms.

This is why lag shows up in tail percentiles and stays hidden in averages.

Lag does not capture everything. A slow database query or a slow upstream HTTP call raises node api latency without blocking the loop, because the process is just waiting on I/O. High latency with flat lag points away from your JavaScript.

Backend and platform engineers running latency-sensitive Node.js APIs should watch both signals.

Copy  new connections     timers due       DB / HTTP results ready
        |                 |                     ^   |
        v                 v                     |   v
+-----------------------------------------------|---------+
| callback queue (everything waits its turn)    |         |
|  [req A] [req B: DB row arrived] [10ms timer] |         |
+-----------------------------------------------|---------+
                  |                             |
                  v  one callback at a time     |
+-----------------------------------------------|---------+
| single JS thread (event loop)                 |         |
|  running: req X synchronous work ... 200 ms   |         |
|  queue waits 0-200 ms extra  -->  lag rises   |         |
+-----------------------------------------------|---------+
                  |                             |
                  v                             |
+-----------------------------+                 |
| responses for A, B, timer   |                 |
| all late: p99 and max jump, |                 |
| mean barely moves           |                 |
+-----------------------------+                 |
                                                |
+-----------------------------------------------+---------+
| off-loop I/O: database, upstream HTTP                   |
|  slow query = loop idle while waiting                   |
|  lag stays flat, request latency still rises            |
+---------------------------------------------------------+

One synchronous 200 ms callback on the single JavaScript thread delays every queued callback behind it, raising event loop lag and tail latency, while slow off-loop I/O raises request latency without raising lag.

Who should use event loop lag monitoring

Lag monitoring pays off when many concurrent requests share one Node.js process and a tail-latency regression costs you something. Typical readers who should adopt it:

• Backend engineers who own HTTP or RPC APIs in Node.js. This matters most when those services parse request bodies, serialize responses, validate input with regular expressions, or transform data in memory. These are the code paths that block the loop.
• Platform and SRE teams who run Node.js fleets. They need a per-instance signal that separates "this process is stuck running JavaScript" from "a dependency is slow". They also need a signal they can feed into readiness checks and load shedding.
• On-call engineers who get p99 latency alerts without an obvious cause. Lag tells them within one chart whether to look at application code or at databases and upstream services.
• Teams whose payload sizes or tenant data grow over time. Endpoints that were fast at launch can start blocking as result sets grow. Lag surfaces this trend before users report it.

It is a weaker fit for short-lived scripts, batch jobs, and services that are almost entirely I/O-bound. The last section covers those cases.

How event loop lag monitoring works

The mechanism has four parts. Each later section covers one of them in detail.

1. Sampling. monitorEventLoopDelay from perf_hooks schedules a recurring timer at the resolution interval. Each time the timer fires, Node records how late it ran compared with when it was due. If a synchronous callback held the thread, the timer could not fire on time, and the recorded delay grows by roughly the length of the block. These delays go into a histogram inside the process, so you do not pay per-sample export cost.
2. Aggregation and export. On a fixed reporting interval, your code reads percentiles and max from the histogram, converts nanoseconds to milliseconds, and exports them as gauges labelled by instance. Then it resets the histogram so each data point describes only that window.
3. Correlation. You chart lag next to request latency using the same labels. Rising latency with rising lag means the loop was blocked. Rising latency with flat lag means requests were waiting on I/O. When lag crosses a threshold, a structured log of in-flight and recent requests narrows the stall to candidate routes.
4. Diagnosis and action. A CPU profile taken while you reproduce the stall identifies the blocking function, and GC observation rules memory pressure in or out. Separately, sustained lag can drive alerts and readiness-based load shedding, so a stuck instance stops receiving new traffic while it recovers.

The signal is indirect in one respect. Lag measures the symptom, which is the loop running late, not the cause. That is why the correlation and profiling steps exist.

How to measure lag with perf_hooks monitorEventLoopDelay

Node.js ships a direct measurement in perf_hooks. Calling monitorEventLoopDelay({ resolution: 20 }) returns a histogram. Call enable() once at startup. On a fixed interval, such as every 10 seconds, read percentile(50), percentile(99) and max, then call reset(). Without the reset, the histogram accumulates from process start, and a stall from an hour ago keeps inflating max long after the problem is gone.

All values are in nanoseconds, so divide by 1e6 before exporting. Report each value as a gauge labelled with instance or pod name. Lag is a per-process property. Averaging it across a fleet hides the one replica that is stuck.

The resolution option, in milliseconds, sets how often the timer samples. A 20 ms resolution can miss a 5 ms stall entirely. Finer resolution catches shorter blocks, but it wakes the loop more often and adds a small overhead.

The reporting interval has a similar trade-off. A 60-second window can turn a two-second burst into a single unremarkable p99. A 1-second window catches it, but it multiplies series volume and storage cost.

The histogram measures how late the loop runs, which is exactly the symptom of a blocked loop. The other signals only infer it.

Copyconst { monitorEventLoopDelay } = require('node:perf_hooks');
const os = require('node:os');

// Lag is per-process, so every series carries the instance/pod name.
const labels = { instance: process.env.POD_NAME || os.hostname() };

// Sample timer delay every 20 ms. Stalls shorter than this can be missed.
// A lower value catches shorter stalls but wakes the loop more often.
const histogram = monitorEventLoopDelay({ resolution: 20 });
histogram.enable();

const NS_PER_MS = 1e6;
const REPORT_INTERVAL_MS = 10_000; // longer hides bursts, shorter adds series volume

// Placeholder: replace with your metrics client's gauge.set(labels, value).
function reportGauge(name, value, labels) {
  console.log(JSON.stringify({ metric: name, value, ...labels }));
}

const timer = setInterval(() => {
  // Histogram values are nanoseconds; export milliseconds.
  reportGauge('event_loop_lag_p50_ms', histogram.percentile(50) / NS_PER_MS, labels);
  reportGauge('event_loop_lag_p99_ms', histogram.percentile(99) / NS_PER_MS, labels);
  reportGauge('event_loop_lag_max_ms', histogram.max / NS_PER_MS, labels);

  // Start a fresh window so an old stall does not inflate max indefinitely.
  histogram.reset();
}, REPORT_INTERVAL_MS);

// Do not keep the process alive just to report metrics.
timer.unref();

Enables a perf_hooks event loop delay histogram and exports p50, p99 and max every 10 seconds as per-instance gauges in milliseconds. It resets the histogram after each report so every report covers only that 10-second window.

Correlating lag with API latency to find the endpoints involved

A lag histogram tells you the loop was blocked, not what blocked it. Correlating lag with HTTP latency turns that metric into a lead.

Start by emitting lag with the same labels your HTTP latency metrics use: service, instance or pod name, and deployed version. Without shared labels, you are comparing a fleet-wide latency average against one process's lag, and the signal blurs. With them, you can put one instance's p99 lag and p99 request duration on the same chart. Read the histogram's p99 and max on each scrape interval, then reset it so each data point covers only that window. Remember that monitorEventLoopDelay reports nanoseconds, so convert before comparing against latency in milliseconds.

To narrow a stall to specific endpoints, keep a small in-memory record of in-flight and recently finished requests. Store the route template (not the raw URL), request payload size, start time and duration. When a lag sample crosses your threshold, log that buffer as one structured event. Every request that was in flight during the stall will look slow, because they all waited. Look for the route whose own duration spans the stall and that recurs across many lag events, especially when its payload size is large. That route is your suspect.

The opposite pattern matters too. If latency rises while lag stays flat, the loop was free and requests were waiting on something else, such as a database, an upstream API or an exhausted connection pool. Investigate those dependencies instead of profiling the process.

Copyconst { monitorEventLoopDelay, performance } = require('node:perf_hooks');
const http = require('node:http');
const os = require('node:os');

// Reuse the exact labels attached to HTTP latency metrics so per-instance charts line up.
const labels = { service: 'orders-api', instance: os.hostname(), version: process.env.APP_VERSION || 'dev' };
const LAG_THRESHOLD_MS = 100;
const inFlight = new Set();
const recent = []; // bounded buffer of finished requests

const lag = monitorEventLoopDelay({ resolution: 20 });
lag.enable();

setInterval(() => {
  const now = performance.now();
  const p99Ms = lag.percentile(99) / 1e6; // histogram values are nanoseconds
  const maxMs = lag.max / 1e6;
  lag.reset(); // each data point covers only this window
  // Emit p99Ms and maxMs to your metrics client here, tagged with `labels`.
  if (maxMs < LAG_THRESHOLD_MS) return;
  // One structured event: everything in flight or just finished when the loop stalled.
  console.log(JSON.stringify({ event: 'event_loop_lag', ...labels, p99Ms, maxMs,
    inFlight: [...inFlight].map((r) => ({ route: r.route, bytes: r.bytes, elapsedMs: now - r.start })),
    recent: recent.filter((r) => now - r.endedAt < 2000) }));
}, 1000).unref();

// Record route template (not raw URL), payload size and duration per request.
function track(req, res, route) {
  const r = { route, bytes: Number(req.headers['content-length'] || 0), start: performance.now() };
  inFlight.add(r);
  res.on('close', () => {
    inFlight.delete(r);
    const endedAt = performance.now();
    recent.push({ route, bytes: r.bytes, durationMs: endedAt - r.start, endedAt });
    if (recent.length > 200) recent.shift();
  });
}

http.createServer((req, res) => {
  const route = req.url.startsWith('/orders/') ? '/orders/:id' : req.url.split('?')[0];
  track(req, res, route);
  res.end('ok');
}).listen(3000);

Emits per-window event loop lag in milliseconds under the same service, instance and version labels as HTTP metrics, and logs in-flight and recently finished requests with route template, payload size and duration whenever lag crosses a threshold.

Common causes of a blocked event loop in API code

When lag does rise with latency, the cause is almost always code that keeps the main thread busy. Every item below does its work in one synchronous call or loop, so nothing else runs until it returns.

Garbage collection pauses also show up as lag, but they point to memory pressure, not slow code. GC-driven lag tends to correlate with heap growth and allocation rate, and it can affect every route at once instead of one endpoint. If lag spikes line up with rising heap usage or with major collections in a GC trace, look at allocation patterns and retained objects before you profile CPU hot paths.

Finding the blocking code with CPU profiles

Reproduce the stall where you control load and data first. Replay the traffic pattern that triggered the lag against a staging instance and start Node with --cpu-prof. When the process exits, it writes a .cpuprofile file. If you need to start and stop capture around a specific window, attach through the built-in inspector with --inspect, or call the inspector module's Profiler domain from inside the process.

Load the profile into a DevTools-compatible viewer and switch to the flame graph. Look for frames that are both wide and deep under a single macrotask. A function that runs for 80 ms without yielding shows up as one long uninterrupted bar. A busy but healthy service looks different: it produces many thin bars. Common culprits are a JSON serializer, a regex, a sort, or a synchronous crypto or compression call.

Not every gap in the profile belongs to your code. Register a PerformanceObserver for gc entries and log each entry's duration and kind alongside your lag samples. If lag spikes match major collections, the fix is to reduce allocation. If lag spikes match application frames, the fix is to change the code itself.

Staging is the safer default for two reasons:

• Overhead. Sampling adds CPU cost and can distort the timing you are trying to measure.
• Data exposure. Profiles and heap-adjacent artifacts can capture function arguments, URLs, and string contents, so treat them like logs that contain customer data.

Profile production only when staging cannot reproduce the stall. If you do, keep the capture to seconds, run it on one instance drained from the load balancer, and restrict who can read the output file.

Copy// Preload in staging: node --require ./diagnose.js server.js
const { PerformanceObserver, monitorEventLoopDelay, constants } = require('node:perf_hooks');
const inspector = require('node:inspector');
const fs = require('node:fs');

// Log GC pauses so they can be lined up against lag spikes.
const gcKinds = {
  [constants.NODE_PERFORMANCE_GC_MINOR]: 'minor',
  [constants.NODE_PERFORMANCE_GC_MAJOR]: 'major',
  [constants.NODE_PERFORMANCE_GC_INCREMENTAL]: 'incremental',
  [constants.NODE_PERFORMANCE_GC_WEAKCB]: 'weakcb',
};
new PerformanceObserver((list) => {
  for (const e of list.getEntries()) {
    console.log(`gc kind=${gcKinds[e.detail?.kind] ?? 'other'} ms=${e.duration.toFixed(1)}`);
  }
}).observe({ entryTypes: ['gc'] });

// Sample event loop delay each second (histogram values are in nanoseconds).
const h = monitorEventLoopDelay({ resolution: 10 });
h.enable();
setInterval(() => {
  console.log(`lag p99_ms=${(h.percentile(99) / 1e6).toFixed(1)} max_ms=${(h.max / 1e6).toFixed(1)}`);
  h.reset();
}, 1000).unref();

// Profile only a bounded window, e.g. while replaying the traffic that stalls.
function captureProfile(ms, file) {
  const session = new inspector.Session();
  session.connect();
  session.post('Profiler.enable', () => session.post('Profiler.start', () => {
    setTimeout(() => session.post('Profiler.stop', (err, res) => {
      // 0o600: the profile may contain sensitive strings, so only the owner can read it.
      if (!err) fs.writeFileSync(file, JSON.stringify(res.profile), { mode: 0o600 });
      session.disconnect();
    }), ms);
  }));
}
// SIGUSR1 is reserved for the inspector, so trigger capture with SIGUSR2.
process.on('SIGUSR2', () => captureProfile(10_000, `/tmp/stall-${process.pid}.cpuprofile`));

A preload script that logs GC pauses by kind next to event loop lag samples, and on SIGUSR2 writes a 10-second CPU profile through the inspector module to an owner-only .cpuprofile file that you can open as a flame graph.

Alert thresholds and using lag as a health and load-shedding signal

A universal number like "alert at 100 ms" fails because services differ. A JSON proxy might idle at single-digit milliseconds of p99 delay, while a report service that parses large payloads may sit much higher and still be healthy. Record the p50, p99, and max from monitorEventLoopDelay over at least a full traffic cycle, then set thresholds relative to that baseline. For example, you might alert when p99 stays above three times its normal value. That multiplier is illustrative, not a recommendation.

Alert on duration, not spikes

A single interval with high max delay is usually noise. Page only when the threshold is breached for several consecutive intervals across a window of minutes. Collect gc entries with a PerformanceObserver and attach GC pause time to each lag sample. A lag spike that matches a major collection is a memory problem, not a blocking-code problem, and the alert should say so.

Shedding load

When lag stays above the threshold, fail the readiness check or return 503 with Retry-After. The load balancer then routes traffic to other instances while this one drains its queue. Keep lag out of liveness probes. Restarting a slow instance in the middle of a surge throws away warm capacity and makes the overload worse.

There are two ways this goes wrong:

• Flapping. A tight threshold toggles the instance in and out of rotation on every sample. Use hysteresis: enter shedding at a high threshold and leave only after lag stays below a lower one for some time.
• Cascading failure. During a real traffic surge, every instance can cross the threshold at once. If all of them shed, nothing is left to serve requests. Set a minimum healthy capacity, either by capping how many instances may report unready or by configuring the load balancer to keep routing when every target fails.

How lag monitoring saves engineering time

Most of the time lost on a tail-latency incident goes to figuring out where to look. Lag monitoring cuts that search down in several concrete ways:

• One chart splits the investigation. Lag next to latency answers the first triage question right away: is the process blocked, or is it waiting on I/O? Without that answer, teams often profile application code during a database incident, or tune queries during a CPU stall. Both chase the wrong layer.
• Per-instance labels find the stuck replica. A fleet-wide p99 alert does not say which pod is affected. Per-instance lag gauges do, so you can drain or inspect one process instead of searching the whole fleet.
• Structured lag events replace guesswork about routes. The in-flight and recent-request log from the correlation section names candidate routes and payload sizes at the moment of the stall. That shrinks a search across every endpoint down to a short list before anyone opens a profiler.
• Targeted reproduction. Because you know the route and payload shape, you can replay that traffic in staging and capture a short profile. Long open-ended load tests are no longer needed.
• Fewer pages and fewer manual interventions. Baseline-relative, duration-based alerts suppress one-off spikes. Readiness-based shedding lets an overloaded instance recover without a human restarting it.
• Regressions caught earlier. Lag tagged with the deployed version shows when a release introduced a blocking path. You can roll back or fix it before the problem builds up over weeks as data grows.

Illustrative estimate (assumed inputs, not measured): assume a team handles four latency incidents a quarter. Assume that without a lag signal, each incident spends about three engineer-hours deciding whether the problem is application code or a dependency, then finding the affected instance and route. Assume that with lag charts and structured lag events, that phase drops to about one hour. The saving is roughly eight engineer-hours per quarter on triage alone. That figure excludes time saved by avoided pages and earlier regression detection. Replace these assumptions with your own incident history before drawing conclusions.

Remediation patterns, alternatives, and when to skip this approach

Once a profile has pointed at the blocking code, the fix depends on whether the work is CPU-bound, payload-bound, or caused by a synchronous call.

Workers move the stall off the main thread, but data passed between threads is copied through structured cloning unless you use transferable or shared buffers. For a task that takes a few milliseconds, the copy and coordination can cost as much as the work itself. Thread pools also add sizing, crash handling and memory limits to operate.

Lag monitoring complements other tooling rather than replacing it. APM and request tracing show which downstream call was slow. Process CPU metrics show saturation. Continuous profiling captures the offending stack without a manual reproduction.

Lag monitoring adds little in I/O-bound services, where database or upstream latency dominates. It also adds little in short-lived scripts and batch jobs, which serve no concurrent requests.

The cost is a few lines of code and a handful of extra metric series per instance. In return, engineers can quickly tell whether a latency spike is an application stall or an infrastructure problem.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to node.js event loop lag, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.