Node.js Event Loop Lag: Measuring and Fixing API Latency Spikes

What event loop lag is and why one blocked tick slows every request

Event loop lag is the gap between when a callback is ready to run and when it actually runs. If a timer is due at time t but fires at t + 30 ms, or a socket has readable data that sits for 30 ms before its handler is invoked, the loop is lagging by 30 ms.

This happens because Node.js runs your JavaScript on a single thread. The event loop takes one ready callback at a time and runs it to completion. Nothing else on that process runs until the current synchronous frame returns. Network I/O happens off-thread, but parsing a request, running middleware, and serializing a response all take turns on the same thread. When one handler does 40 ms of synchronous work, every other in-flight request waits 40 ms, including requests that need only microseconds of CPU.

Illustrative estimate

Assumptions: one Node.js process, 200 concurrent requests, and 50 of them arrive within a few milliseconds of each other carrying a large body. Each of those 50 needs a 40 ms synchronous JSON.parse. The other 150 requests are cheap, needing about 1 ms of CPU each.

The 50 parses run one after another, for 50 × 40 ms = 2,000 ms of blocked thread. The last large request waits about 2 seconds before its parse even starts. A cheap request that arrives behind the queue also waits up to about 2 seconds, even though it needs only 1 ms of work.

The expensive endpoint has turned into p99 latency for every endpoint on that process. This matters most to backend and platform engineers running latency-sensitive Node.js APIs. On those services, tail latency, not average throughput, is what breaks SLOs and causes upstream timeouts.

Copy  200 concurrent requests -> one Node.js process
  ┌──────────────────────┐   ┌──────────────────────┐
  │ 50 large-body reqs   │   │ 150 cheap reqs       │
  │ 40 ms JSON.parse each│   │ ~1 ms CPU each       │
  └──────────┬───────────┘   └──────────┬───────────┘
             │                          │
             ▼                          ▼
  ┌──────────────────────────────────────────────────┐
  │ Network I/O (off-thread): sockets become readable│
  └────────────────────────┬─────────────────────────┘
                           │ ready callbacks
                           ▼
  ┌──────────────────────────────────────────────────┐
  │ Event loop queue (FIFO of ready callbacks)       │
  │ [P1][P2][P3] ... [P50][c1][c2] ... [c150]        │
  └────────────────────────┬─────────────────────────┘
                           │ one callback at a time
                           ▼
  ┌──────────────────────────────────────────────────┐
  │ Single JS thread (runs one frame to completion)  │
  │ [P1 40ms][P2 40ms] ... [P50 40ms][c1][c2] ...    │
  │ t=0 ms                        t≈2,000 ms: c1 runs│
  └──────────────────────────────────────────────────┘

  Blocked thread: 50 x 40 ms = ~2,000 ms of synchronous parse
  c1 needs ~1 ms of CPU but waits ~2,000 ms -> tail latency for all routes
  (illustrative estimate, assumed inputs)

Illustrative estimate: 50 queued 40 ms JSON.parse calls hold the single JS thread for about 2 seconds, so a cheap request that needs about 1 ms waits behind them.

Measuring event loop lag with monitorEventLoopDelay

Node.js ships a built-in way to quantify that tail: monitorEventLoopDelay from the perf_hooks module.

Enabling the histogram

Calling monitorEventLoopDelay({ resolution }) returns a histogram that does nothing until you call enable(). The resolution option is the sampling interval in milliseconds. Internally, a timer fires on that interval and records how late it ran. That lateness is the time the loop spent unable to service callbacks. The default is 10 ms, which suits most APIs. A smaller value catches shorter stalls but adds timer wakeups. A larger value is cheaper but blurs short blocks into noise.

Reading the numbers

The histogram exposes percentile(n), min, max, mean and stddev. All values are in nanoseconds, so divide by 1e6 before you label anything as milliseconds.

Prefer p99 and max over mean. A single 300 ms synchronous JSON.parse among thousands of idle samples barely moves the mean. Every request queued behind it still waits 300 ms. High stddev relative to the mean is a hint that this is happening.

Scrape windows

The histogram accumulates for the life of the process unless you clear it. Read the values on each metrics scrape, then call reset(). Each reading then describes the last interval, not hours of mixed history.

Export these values alongside your request latency histogram, with the same scrape interval and labels. When p99 request latency jumps, you can check whether loop delay jumped in the same window.

Limits

The histogram tells you that the loop was blocked and for how long. It does not tell you what blocked it. Identifying the code responsible needs a CPU profile.

Copyconst { monitorEventLoopDelay } = require('node:perf_hooks');
const http = require('node:http');

// Sample every 10 ms (the default). Lower catches shorter stalls but costs more timer wakeups.
const loopDelay = monitorEventLoopDelay({ resolution: 10 });
loopDelay.enable(); // the histogram records nothing until enabled

const toMs = (ns) => (ns / 1e6).toFixed(3); // all histogram values are nanoseconds

function readAndReset() {
  const snap = {
    p50: toMs(loopDelay.percentile(50)),
    p99: toMs(loopDelay.percentile(99)), // tail: what queued requests actually wait
    max: toMs(loopDelay.max),            // worst single stall in this window
    mean: toMs(loopDelay.mean),          // kept for context; hides rare long blocks
    stddev: toMs(loopDelay.stddev),      // high relative to mean hints at rare long blocks
  };
  // Clear so the next scrape covers only the next interval, not process lifetime.
  // reset() is destructive, so exactly one scraper should hit this endpoint.
  loopDelay.reset();
  return snap;
}

// Serve on the same scrape interval as your request latency histogram so spikes line up.
http.createServer((req, res) => {
  if (req.url !== '/metrics') { res.writeHead(404); return res.end(); }
  const s = readAndReset();
  const body = [
    '# TYPE nodejs_eventloop_delay_ms gauge',
    `nodejs_eventloop_delay_ms{stat="p50"} ${s.p50}`,
    `nodejs_eventloop_delay_ms{stat="p99"} ${s.p99}`,
    `nodejs_eventloop_delay_ms{stat="max"} ${s.max}`,
    `nodejs_eventloop_delay_ms{stat="mean"} ${s.mean}`,
    `nodejs_eventloop_delay_ms{stat="stddev"} ${s.stddev}`,
  ].join('\n') + '\n';
  res.writeHead(200, { 'Content-Type': 'text/plain; version=0.0.4' });
  res.end(body);
}).listen(9464);

Enables monitorEventLoopDelay at a 10 ms resolution and serves p50, p99, max, mean and stddev as milliseconds on a Prometheus-style text endpoint. It resets the histogram on every scrape so each reading covers only the most recent interval.

Common sources of event loop blocking in APIs

Every culprit below has the same shape: synchronous JavaScript or a synchronous native call that runs on the main thread until it finishes. While it runs, no timers fire, no sockets are read, and every in-flight request waits. Cost usually grows with input size, so the problem appears only for the largest payload or the most hostile input.

In review, focus on two things: any Sync suffix on a request path, and any operation whose input size is controlled by the client. The before/after example shows the swap from the synchronous call to its callback, promise or streaming equivalent, which moves the work off the main thread or breaks it into chunks.

Copyconst crypto = require('node:crypto');
const fs = require('node:fs');
const zlib = require('node:zlib');
const { promisify } = require('node:util');
const { pipeline } = require('node:stream/promises');

const pbkdf2 = promisify(crypto.pbkdf2);
const ITERATIONS = 600000; // placeholder work factor

// BEFORE: each call holds the main thread until it finishes.
function hashPasswordBlocking(password, salt) {
  // Deliberately slow key derivation running on the event loop thread.
  return crypto.pbkdf2Sync(password, salt, ITERATIONS, 64, 'sha512');
}

function sendReportBlocking(res, filePath) {
  // Reads the whole file, then compresses the whole buffer, all synchronously.
  const body = zlib.gzipSync(fs.readFileSync(filePath));
  res.setHeader('Content-Encoding', 'gzip');
  res.end(body);
}

// AFTER: same results, but the event loop keeps serving other requests.
async function hashPassword(password, salt) {
  // Async pbkdf2 runs in the libuv threadpool; the handler awaits the result.
  return pbkdf2(password, salt, ITERATIONS, 64, 'sha512');
}

async function sendReport(res, filePath) {
  res.setHeader('Content-Encoding', 'gzip');
  // Stream in chunks: file -> gzip -> response, with backpressure handled
  // and errors from any stage rejecting the promise.
  await pipeline(fs.createReadStream(filePath), zlib.createGzip(), res);
}

module.exports = { hashPassword, sendReport };

A before/after comparison for code review: pbkdf2Sync, readFileSync and gzipSync on a request path, replaced by the promisified pbkdf2 and a streamed read-gzip-response pipeline so the main thread stays free while the work runs.

Diagnosing the culprit: CPU profiles and telling lag apart from other latency

When loop delay is high, the next question is which function holds the thread. Starting the process with node --cpu-prof writes a .cpuprofile file when the process exits. You can also start with --inspect and attach Chrome DevTools to record a profile on demand. In the flame graph, look for wide frames with no async gaps beneath a request handler, such as JSON.parse, a regex, or a sort. Width is time spent on-CPU, and a wide synchronous frame is exactly the stall that every queued request waited behind.

Profiling has a cost. Sampling adds CPU overhead, and the inspector port must never be reachable from outside the host. Keep capture windows to seconds or a few minutes, ideally on a single instance pulled partly out of rotation. Reproducing the problem off-production has its own trap: synthetic load often sends small, uniform payloads, while the blocking only appears with the occasional multi-megabyte body a real client sends.

Not every latency spike is loop lag. Compare loop delay against request latency before you open a profiler:

Copy#!/usr/bin/env bash
# Capture a short CPU profile from one instance that is partly out of rotation.
set -euo pipefail

PROFILE_DIR=/tmp/profiles
mkdir -p "$PROFILE_DIR"

# Option 1: --cpu-prof writes a .cpuprofile file when the process exits.
# --cpu-prof-interval is the sampling interval in microseconds (default 1000).
# A larger interval lowers overhead at the cost of resolution.
node --cpu-prof \
     --cpu-prof-dir="$PROFILE_DIR" \
     --cpu-prof-interval=1000 \
     server.js &
NODE_PID=$!

# Keep the window short: send real traffic for ~60s, then stop.
sleep 60

# The profile is only written on a normal exit. A default SIGTERM kills the
# process without writing it, so server.js should handle SIGTERM by closing
# the server and calling process.exit().
kill -TERM "$NODE_PID"
wait "$NODE_PID" || true

ls -lh "$PROFILE_DIR"/*.cpuprofile
# Open the file in Chrome DevTools (Performance panel) and look for wide
# synchronous frames under request handlers: JSON.parse, regex, sort.

# Option 2: record on demand with DevTools instead.
# Bind the inspector to loopback only and never expose port 9229.
# Reach it from your workstation through an SSH tunnel:
#   node --inspect=127.0.0.1:9229 server.js
#   ssh -L 9229:127.0.0.1:9229 user@app-host-01
# Then open chrome://inspect locally, record for a few seconds, and stop.

# To check whether GC explains the loop delay, log collections to stderr:
#   node --trace-gc server.js

Capturing a short CPU profile with --cpu-prof, or on demand through a loopback-only --inspect port reached over an SSH tunnel, with --trace-gc as a quick check for GC pauses.

Fixing event loop blocking: strategies in order of preference

Work through these in order. Each step down the list costs more to build and run, so stop at the first one that brings lag back within budget.

Avoiding the work is the cheapest fix. A 50 MB JSON.stringify cannot block the loop if you never build that object.

Chunking lets other callbacks run between batches. The trade-off is that the total task takes longer, and tests now have to wait on async completion and cover state that changes between batches.

Workers run the work on another thread with its own event loop. Data crosses the boundary through structured cloning, unless you pass transferable buffers. Each worker also carries its own heap. For small tasks, the cost of copying data in and out can be larger than the cost of the work itself. Use a pool rather than starting a worker per request.

Moving the work out of process isolates it completely. It also adds deployment, retry and observability work.

Adding instances is not on this list because it does not shorten a blocked tick. Each process has one event loop. When that loop stalls, every request already routed to the process waits for the full stall. More replicas spread the load across more processes, but each process still has the same tail-latency spikes.

Using event loop lag operationally: alerts, load shedding and readiness

Alert on sustained percentiles, not single samples. A one-off 300 ms spike during garbage collection or startup is noise. A p99 delay that stays elevated across several consecutive windows means requests are queuing behind blocked ticks. Alert on something like "p99 above threshold for five minutes," and reset the histogram after each export so every window reflects only recent behavior.

The threshold has to come from the service's own baseline. A JSON-heavy gateway and a thin proxy have very different normal delay distributions. A number copied from a blog post or from another team will either fire constantly or never fire. Record a week of normal traffic, then set the alert well above that service's usual p99.

Load shedding and readiness

Lag can drive a readiness check or a shedding middleware. When delay crosses a limit, the instance returns 503 or reports not-ready, and traffic goes elsewhere. This has two risks:

• Flapping. Shedding traffic lowers lag, so the instance passes the check, takes traffic again, and fails again. Use separate thresholds for leaving and rejoining rotation, plus a minimum time out of rotation.
• Cascades. If the cause is a payload or route pattern, peers receive the same expensive requests and stall in turn. Cap how many instances can be unready at once.

Why autoscaling doesn't fix it

One blocked core can look like modest average CPU, so CPU-based scaling may never trigger. Even when it adds pods, average latency improves while tail latency stays bad. The instance that receives the expensive request still blocks every request queued behind it.

Alternatives, cost and engineering time

Each signal sees a different part of the problem. Most teams need two of them, not all five.

Instrumenting loop delay costs little: a histogram, a periodic export and an alert. The expensive part is the fix. Worker pools add serialization costs and new failure modes. Queues add infrastructure and change your consistency model. Continuous profiling adds runtime overhead, so many teams sample it or enable it only during incidents.

The payoff is time saved during incidents. A loop-delay spike that coincides with a latency spike rules out the network and the database in minutes, not hours. It also stops you from adding instances to fix a problem that more instances can't fix. Finally, it narrows the search to synchronous code paths, and the profiler can then name the specific function.

Skip this approach for batch-style workers, ETL jobs or queue consumers where nobody waits on an individual request. In those services, a blocked loop delays work without hurting anyone, and throughput per CPU-second is the metric that matters. ELU and throughput counters are usually enough there.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to node.js event loop lag, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Consumers learn about a breaking change at review time rather than from production errors.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.