Node.js Event Loop Lag: Measuring and Fixing Blocked API Servers

What event loop lag is and why one slow handler slows every endpoint

A Node.js process runs all of your JavaScript on a single thread. The event loop takes one ready callback at a time (a timer, a finished socket read, a resolved promise) and runs it to completion before it looks at the next one. Network and disk I/O can proceed in the background, but the callbacks that handle their results all wait in line for that one thread. If a handler spends 50 ms synchronously building a response, every other callback in the queue waits 50 ms. That includes health checks, unrelated endpoints and timers.

Event loop lag is the gap between when a callback was due to run and when it actually ran. A timer scheduled for 10 ms that fires at 85 ms shows 75 ms of lag. For that whole time, the thread was busy with something else.

Lag is a more direct signal than the usual dashboards. Request latency mixes thread contention with database and network waits, so it does not tell you which one caused a slowdown. CPU graphs average over seconds and across cores. One saturated core on an eight-core host appears as about 12% usage. Lag measures exactly one thing: how long ready work waited for the thread. If you run latency-sensitive Node.js APIs, it deserves a place next to p99 latency.

Illustrative estimate. Assume each request needs 40 ms of synchronous JSON parsing and traffic arrives at 30 requests per second. That is 1.2 s of blocking work per second of wall time. The thread can complete at most 25 requests per second, so the queue grows by about five requests every second, and each queued request waits longer than the one before. The server does not slow down gradually. Latency keeps rising until clients time out and retry, and the server then spends its time on requests whose callers have already given up.

Copy  incoming requests        libuv background I/O       timers
  /orders /users /health   (sockets, disk; parallel)  setTimeout(fn, 10)
          |                          | completions          |
          v                          v                      v
  +-----------------------------------------------------------------+
  | ready callback queue (FIFO)                                     |
  | [ /orders ][ /health ][ timer due @10ms ][ /users ][ /orders ]  |
  +-----------------------------------------------------------------+
          |
          | one callback at a time, each runs to completion
          v
  +-----------------------------------------------------------------+
  | single JavaScript thread                                        |
  | now running: /orders handler, 40 ms of synchronous JSON parsing |
  | every other callback above waits until it returns               |
  +-----------------------------------------------------------------+

  timeline of the queued timer
  0 ms        10 ms (due)                              85 ms (ran)
  |-----------|----------------------------------------|
              |<------- event loop lag: 75 ms -------->|

  illustrative estimate (assumed inputs):
  30 req/s x 40 ms sync work = 1.2 s of work per 1 s of wall time
  capacity = 1000 ms / 40 ms = 25 req/s  -->  queue grows ~5 req/s

All callbacks, whether from requests, completed I/O or timers, wait in one queue for the single JavaScript thread, so a 40 ms synchronous handler delays a timer due at 10 ms until 85 ms (75 ms of lag), and at 30 req/s the queue grows without bound (illustrative estimate).

Who should use event loop lag monitoring

Lag monitoring is worth adding if you run Node.js processes that serve requests and have a latency target. That covers HTTP and gRPC APIs, GraphQL servers, backend-for-frontend layers and WebSocket servers. It matters most in these situations:

• Handlers do real CPU work. Parsing or serializing large JSON bodies, rendering templates, hashing passwords, validating schemas or transforming result sets all run on the main thread.
• Input sizes vary widely. An endpoint that is fast for typical payloads can block the process for one tenant with a much larger account or one client sending an oversized body.
• Many routes share one process. A slow report endpoint and a fast health check compete for the same thread, so one team's change can hurt another team's endpoint.
• You rely on readiness checks or autoscaling. Lag gives those systems a signal that reflects saturation of the thread itself rather than host-wide CPU.

It is less critical for thin proxies whose handlers mostly await downstream calls, although the cost of collecting it is small enough that it is still a reasonable default. Short-lived scripts, CLI tools and batch jobs without a per-request latency budget usually do not need it. For those, total runtime and a CPU profile tell you more.

How to measure it: setTimeout drift versus perf_hooks.monitorEventLoopDelay

To catch that rise before clients do, you need a number for how late the loop is running. There are two common ways to get one.

setTimeout drift

Schedule a timer every N milliseconds, record performance.now() when it fires, and subtract the expected time. Any excess is time the loop spent busy. This is easy to write but gives a coarse, noisy signal. You get one sample per interval, so a 40 ms stall and several 10 ms stalls inside the same window can produce the same reading. The callback is JavaScript running on the loop you are measuring. Timer granularity and garbage collection pauses add jitter, which makes small values hard to trust.

monitorEventLoopDelay

perf_hooks.monitorEventLoopDelay({ resolution }) returns a histogram that the runtime fills natively at the resolution you set, in milliseconds. The default is 10. Call enable() to start sampling. On each scrape, read min, max, mean, stddev and percentile(99), then call reset(). Without reset(), the histogram covers the whole life of the process, and a regression in the last minute is hidden under hours of healthy samples. All values are in nanoseconds, so divide by 1e6 before you export milliseconds.

There is a trade-off in the resolution setting. A finer resolution and more frequent scraping catch short, frequent stalls, but they add a small amount of overhead and produce more metric data. A coarse resolution hides those stalls inside a single sample.

Copyconst { monitorEventLoopDelay, performance } = require('node:perf_hooks');

// Approach 1: setTimeout drift. One coarse sample per interval, measured in JS.
const INTERVAL_MS = 500;
let expected = performance.now() + INTERVAL_MS;
function checkDrift() {
  const drift = performance.now() - expected; // ms the loop was late
  console.log(`drift_ms=${Math.max(0, drift).toFixed(1)}`);
  expected = performance.now() + INTERVAL_MS;
  setTimeout(checkDrift, INTERVAL_MS).unref();
}
setTimeout(checkDrift, INTERVAL_MS).unref();

// Approach 2: native histogram sampled every 10 ms (resolution is in ms).
const h = monitorEventLoopDelay({ resolution: 10 });
h.enable();

const ns = (v) => (v / 1e6).toFixed(2); // histogram values are nanoseconds

// Each scrape reports only the last window, then clears it.
setInterval(() => {
  console.log(
    `lag_ms min=${ns(h.min)} mean=${ns(h.mean)} stddev=${ns(h.stddev)}`,
    `p50=${ns(h.percentile(50))} p99=${ns(h.percentile(99))} max=${ns(h.max)}`
  );
  h.reset(); // without this, stats cover the whole process lifetime
}, 15_000).unref();

// Simulate a short synchronous stall so both monitors have something to see.
setInterval(() => {
  const end = Date.now() + 40;
  while (Date.now() < end) {} // blocks the loop for ~40 ms
}, 2_000);

Runs setTimeout drift and monitorEventLoopDelay side by side. The histogram is converted from nanoseconds to milliseconds and reset on each scrape, so every report covers only the latest window.

How lag measurement works

Both techniques rely on the same mechanism. The event loop cycles through phases: it runs expired timers, polls for completed I/O and runs those callbacks, runs setImmediate callbacks, and repeats. Between and after callbacks, Node drains the microtask queue (resolved promises and process.nextTick). A timer's due time only marks when it becomes eligible to run. The loop can only notice that eligibility when it returns to the timers phase, which happens after whatever callback currently holds the thread has finished.

A lag monitor exploits that. It schedules a timer at a known interval and records how late each one fires. If the thread is idle or handling short callbacks, the timer fires close to on time and the recorded delay is near zero. If a handler runs synchronously for 300 ms, the timer cannot fire until that handler returns, and the recorded delay is roughly 300 ms. The monitor never needs to know which code was running. It only observes that it was kept waiting.

The difference between the two approaches is where the observation happens:

• setTimeout drift does the bookkeeping in a JavaScript callback. That callback is itself one more item in the queue, and you get a single number per interval.
• monitorEventLoopDelay drives a timer from native code at the configured resolution and records each delay directly into a histogram without running a JavaScript callback per sample. You then read the distribution from JavaScript whenever you choose.

Because the sampler only records lateness, lag covers anything that keeps the main thread busy: your handler code, dependency code, JSON work inside the framework, and garbage collection pauses. It does not include time a request spends waiting on the database or the network, since that waiting happens off the main thread and leaves the loop free. That separation is what makes lag useful for telling "the process is blocked" apart from "a downstream dependency is slow".

Exporting lag as a metric you can alert on

A lag histogram is only useful operationally if you read it on a fixed schedule and reset it. Every 10–15 seconds, call percentile(50), percentile(99) and max on the histogram, convert from nanoseconds to milliseconds, publish the three values as gauges, then call reset() so the next window starts clean. The accompanying sample does exactly this. Without the reset, the histogram accumulates since process start and a stall from an hour ago keeps distorting today's numbers.

Do not publish the mean. A single 400 ms stall inside a 15-second window where every other sample is near 10 ms barely moves the average, yet every request in flight during that stall waited 400 ms. The p99 and max are what match the latency spikes your users report. The p50 tells you whether the baseline has drifted.

Lag alone does not explain cause, so graph it next to request latency percentiles, throughput and GC activity on the same time axis. Lag that rises with throughput suggests CPU-bound handler work. Lag that spikes in step with major GC events is a different problem. GC pauses stop the main thread and show up as lag, but rewriting a handler will not remove them. You reduce them by allocating less (fewer intermediate objects, streaming instead of buffering large payloads) or by adjusting heap sizing with flags such as --max-old-space-size.

Derive thresholds from your own latency budget. If an endpoint's p99 target is 200 ms and normal handling uses 120 ms, sustained p99 lag above roughly 50 ms leaves no headroom. A batch service with a 5-second budget can tolerate far more.

Copy// Publish event loop lag percentiles and GC pause time per fixed window.
const { monitorEventLoopDelay, PerformanceObserver } = require('node:perf_hooks');

const WINDOW_MS = 15000;      // read-and-reset interval
const NS_PER_MS = 1e6;        // histogram values are in nanoseconds

// Placeholder: replace with your metrics client's gauge call.
function publishGauge(name, value) {
  console.log(JSON.stringify({ metric: name, value, ts: Date.now() }));
}

// Sample loop delay every 20 ms; finer resolution catches shorter stalls.
const lag = monitorEventLoopDelay({ resolution: 20 });
lag.enable();

// Sum GC pause time in the same window so the two can be graphed together.
let gcPauseMs = 0;
new PerformanceObserver((list) => {
  for (const entry of list.getEntries()) gcPauseMs += entry.duration;
}).observe({ entryTypes: ['gc'] });

const timer = setInterval(() => {
  // Percentiles and max, not the mean: short stalls vanish in an average.
  publishGauge('eventloop_lag_p50_ms', lag.percentile(50) / NS_PER_MS);
  publishGauge('eventloop_lag_p99_ms', lag.percentile(99) / NS_PER_MS);
  publishGauge('eventloop_lag_max_ms', lag.max / NS_PER_MS);
  publishGauge('gc_pause_total_ms', gcPauseMs);

  // Reset so each window reflects only its own 15 seconds.
  lag.reset();
  gcPauseMs = 0;
}, WINDOW_MS);

// Do not keep the process alive just for metrics.
timer.unref();

Uses perf_hooks.monitorEventLoopDelay to publish per-window p50, p99 and max lag in milliseconds alongside total GC pause time, resetting both after each 15-second read.

Common sources of blocking code in API servers

Most lag in API servers comes from a short list of operations. They have one thing in common: work that grows with input size runs on the main thread. Nothing else can run until it finishes.

Watch the regex case in particular. It is the only one where a single crafted request can stall the process for seconds. This makes it a denial-of-service risk as well as a performance problem.

Finding the culprit with CPU profiles, load tests and lag timestamps

Lag tells you the loop was blocked for 400 ms at 14:02:17. It does not tell you which function did it. The histogram records how late the loop was, not what was on the stack while it was late. To find the culprit you need a CPU profile captured while the problem is happening.

Capture a profile under realistic load

Start the server with node --cpu-prof. When the process exits cleanly, Node writes a .cpuprofile file to the working directory. For a process that is already running, attach with --inspect and record from the inspector's profiler instead. Profiling an idle server shows nothing useful, so drive traffic at it with a load generator. Use the request mix that triggered the spikes: the same endpoints, the same payload sizes, and the same pathological inputs if you suspect any.

Read the flame graph for width, not depth

Open the profile in a DevTools-compatible viewer. Deep stacks are normal in Node. What matters is a wide frame with no idle gaps around it, because that is synchronous code holding the thread. Common examples are JSON.parse on a large body, a regex match, or a sync crypto or fs call. Walk up the stack from that frame until you reach your own handler.

Use lag spikes as timestamps

When you can't reproduce the problem, treat each lag spike as a timestamp. Look for requests that started just before the spike and finished just after it. Pull their logs and traces, and you have a short list of routes and inputs to replay against the profiler.

This is faster than working from latency dashboards. During a block, every endpoint slows down at once, so the dashboard points at all of them equally. The profile and the timestamps point at the specific route and input that caused it.

Copy// lag-timestamps.js: require this first in server.js
// Run:  node --cpu-prof --cpu-prof-dir=./profiles server.js
// Then replay the suspect request mix with a load generator and stop with Ctrl+C.
const { performance } = require('node:perf_hooks');

const INTERVAL_MS = 100;  // how often we expect the timer to fire
const THRESHOLD_MS = 50;  // lag worth recording as a spike
let expected = performance.now() + INTERVAL_MS;

const timer = setInterval(() => {
  const now = performance.now();
  const lag = now - expected; // how late this tick ran
  expected = now + INTERVAL_MS;

  if (lag > THRESHOLD_MS) {
    // Emit wall-clock bounds so the spike can be matched against
    // request logs and traces (requests spanning this window are suspects).
    const end = new Date();
    const start = new Date(end.getTime() - lag); // approximate block start
    console.warn(JSON.stringify({
      msg: 'event_loop_blocked',
      lagMs: Math.round(lag),
      start: start.toISOString(),
      end: end.toISOString(),
    }));
  }
}, INTERVAL_MS);
timer.unref(); // don't keep the process alive just for monitoring

// --cpu-prof writes the .cpuprofile only on a normal exit,
// so convert termination signals into a clean process.exit().
for (const sig of ['SIGINT', 'SIGTERM']) {
  process.on(sig, () => process.exit(0));
}

A timer-drift monitor that logs each lag spike with ISO timestamps for matching against logs and traces, run under node --cpu-prof with a clean exit so the profile is written.

How lag monitoring saves engineering time

The main saving comes from shortening the path from "the API is slow" to a specific line of code. Without a lag metric, a latency incident usually starts with a broad search: is it the database, a downstream service, the network, the host, or the application? Each hypothesis needs its own dashboards and its own expert. Lag cuts that search down early in several ways.

• It splits the problem in two immediately. If p99 latency rose and lag stayed flat, the main thread was free and the time went to waiting on something external, so you look at dependencies. If lag rose with latency, the process itself was blocked, so you skip the database investigation and go straight to profiling.
• It narrows the search to a time window. The spike timestamps from the previous section turn "sometime this afternoon" into a window of a few hundred milliseconds. Matching that window against request logs gives a short list of routes and inputs instead of the whole API surface.
• It catches regressions before production. Recording p99 and max lag during an existing load test lets you compare a change against the previous build. A new synchronous call or an unbounded payload shows up as a lag increase in review rather than as an incident later.
• It prevents fixes that do not address the cause. Adding instances or raising timeouts can mask a blocking handler for a while. A lag graph shows that each instance is still blocking, which steers the work toward the code change that removes the block instead of repeated capacity increases.
• It lets readiness and shedding handle overload without paging someone. The protection mechanisms described below act on the same signal automatically, so a transient saturation can recover on its own while engineers investigate during working hours.

The setup cost is small: one module that enables the histogram and publishes three gauges, plus one alert per service. Because the signal is the same for every Node.js service, the dashboards and alert rules can be shared across teams instead of rebuilt for each one.

Fix strategies and alternatives, ordered by cost

Once you know which code blocks, pick the cheapest fix that removes the block, since each step down the list adds development and operational cost.

Avoid the work. Paginate list endpoints, stream large responses instead of building them in memory, and cap request body and query sizes. This is often a small change, and it removes the block at its source.

Use async fs, crypto and zlib. These APIs run on the libuv threadpool, which is small by default (four threads, adjustable with UV_THREADPOOL_SIZE). Under load, the threadpool becomes the new queue. DNS lookups and file reads then wait behind your hashing.

Chunk and yield. Process a slice of a long loop, then schedule the rest with setImmediate. Other requests run between slices. The cost is a longer total task time and errors that no longer surface in one synchronous try/catch.

Offload. Move CPU-heavy work to worker_threads or a separate service. Messages are copied with structured clone, and each worker has its own heap. On small tasks, the transfer overhead can exceed the time you save.

Scale out. More instances lower the chance that a request lands on a blocked process. The blocked requests stay just as slow.

Using lag for protection: load shedding and readiness checks

Measuring lag is only half the job. Once an instance is saturated, accepting more requests only lengthens the queue that every caller is already waiting in. Rejecting early is usually cheaper for everyone.

A smoothed shedding middleware

The middleware example below does not read the histogram on each request. A timer samples monitorEventLoopDelay every second, takes the p99, folds it into an exponentially weighted moving average, and then resets the histogram. Request handlers check only a boolean flag, so the per-request cost is close to zero. When the flag is set, the server returns 503 with a Retry-After header before it parses the body or touches the database.

Smoothing prevents a single garbage-collection pause from triggering rejections. Hysteresis prevents oscillation. Set the shedding threshold above the recovery threshold. For example, start shedding when the average exceeds 200 ms and stop when it drops below 100 ms (illustrative values). With only one threshold, the instance sheds, recovers within a sample, accepts a burst, and saturates again. If the threshold is too tight, or the signal is unsmoothed, the server rejects healthy traffic during normal jitter.

Readiness versus liveness

Expose the same flag through the readiness endpoint so the orchestrator stops routing new traffic to the instance until it drains. Keep the liveness check independent of lag. If liveness fails during a transient stall, the instance restarts, its load moves to peers that are already busy, and they stall too. That cascade is a restart storm.

When not to shed on lag

Avoid lag-based shedding in these cases:

• A single instance with nowhere to redirect traffic. Rejected requests have no other destination.
• Lag from shared host contention. If the cause affects every replica, they all shed at once.
• Workloads that are CPU-bound by design. For these, a concurrency limit is the more accurate signal.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to node.js event loop lag, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.