What is the undici connection pool and how does it work?

undici builds outbound HTTP from three layers. A Client owns a single connection to one origin (scheme, host, and port). A Pool is a set of Clients to that same origin. It hands each request to an idle Client and opens new connections up to a configured limit. An Agent sits on top and routes each request to a per-origin Pool, creating that Pool the first time it sees the origin. Calling three upstreams through one Agent therefore gives you three independent pools, each with its own connections and queue.

Node's built-in fetch sends requests through a global dispatcher, which is an Agent built from these same pieces. This is why node fetch keepalive is on by default: once a response finishes, its connection goes back to the origin's Pool for the next request instead of closing. You avoid a fresh TCP and TLS handshake on every call.

Pipelining

Pipelining lets a Client write several requests onto one connection before earlier responses arrive. HTTP/1.1 requires responses to come back in request order, so a slow response holds up every request queued behind it on that socket. This is head-of-line blocking. If an upstream's latency varies, one slow call can stall otherwise fast requests and increase tail latency. A pipelining value of one, meaning one in-flight request per connection, is the safe default. If you need more concurrency, add connections rather than deepening the queue on each one.

HTTP/2 multiplexing interleaves streams on one connection and avoids response-ordering stalls. It still shares one TCP connection, so packet loss and per-connection flow control can slow every stream on it.

  fetch(url)  (Node built-in)
       |
       v
  +--------------------------------------------+
  | Agent  (global dispatcher)                 |
  | routes by origin = scheme + host + port    |
  +--------------------------------------------+
       |                 |                 |
       v                 v                 v
  +-----------+     +-----------+     +-----------+
  | Pool      |     | Pool      |     | Pool      |
  | origin A  |     | origin B  |     | origin C  |
  | queue     |     | queue     |     | queue     |
  +-----------+     +-----------+     +-----------+
    |       |         |       |             |
    v       v         v       v             v
  +----+  +----+    +----+  +----+        +----+
  | C1 |  | C2 |    | C1 |  | C2 |        | C1 |
  +----+  +----+    +----+  +----+        +----+
  Client = one keep-alive socket; it returns to its Pool after
  each response, so the next request skips the TCP/TLS handshake

  pipelining = 1:  C1 --[A]--> upstream      (B goes to C2)
  pipelining = 3:  C1 --[A][B][C]--> upstream
                   A is slow -> B and C wait (head-of-line blocking)
  HTTP/2: streams interleave on one TCP connection; loss stalls all

Node's fetch sends requests through a global Agent that keeps one Pool per origin, each Pool holds keep-alive Clients (one socket each), and pipelining more than one request per socket exposes later requests to head-of-line blocking.

Why is outbound latency usually a pool problem, not a network problem?

A connection that is already open costs nothing extra to reuse. A new one costs a DNS lookup, a TCP handshake (one round trip), and for HTTPS a TLS handshake (one or two more round trips, plus CPU for key exchange). All of this happens before the first request byte leaves the process. For an upstream 20 ms away, that is easily 60 ms or more of setup on a call the server answers in 10 ms.

The second cost hides better. When every connection to an origin is busy, undici queues the request until one frees up. That wait counts toward the call's duration, so a dashboard shows a slow upstream even though the server's own logs report fast responses. Raising the upstream's timeout or blaming the network does not help, because the time is spent inside your process.

To diagnose a slow call, split it into three intervals:

  • Queue time: from when the request is dispatched to when it is assigned a connection.
  • Connect time: DNS, TCP and TLS. This is zero on a reused socket.
  • Server time: from when the request is written to when response headers arrive.

Undici publishes request and connection lifecycle events through Node's diagnostics_channel, which lets you timestamp each boundary. If queue time grows with load while server time stays flat, the pool is undersized.

This matters most for services that make high-rate calls to a small number of upstreams, where a single origin's pool becomes the bottleneck. It also matters for platform teams that own shared HTTP client defaults, because one poorly chosen limit is copied into every service that uses those defaults.

// Split each outbound call into wait (queue or connect) and server time.
import diagnosticsChannel from 'node:diagnostics_channel';
import { performance } from 'node:perf_hooks';
import { request } from 'undici';

const started = new WeakMap();   // request -> dispatch timestamp
const sent = new WeakMap();      // request -> { at, reusedSocket }
const seenSockets = new WeakSet(); // sockets that already carried a request

// Request created and handed to the pool: start the clock.
diagnosticsChannel.channel('undici:request:create').subscribe(({ request }) => {
  started.set(request, performance.now());
});

// Headers written to a socket: the request now holds a connection.
diagnosticsChannel.channel('undici:client:sendHeaders').subscribe(({ request, socket }) => {
  const reusedSocket = seenSockets.has(socket);
  seenSockets.add(socket);
  sent.set(request, { at: performance.now(), reusedSocket });
});

// Response headers arrived: everything after sendHeaders is server time.
diagnosticsChannel.channel('undici:request:headers').subscribe(({ request }) => {
  const t0 = started.get(request);
  const s = sent.get(request);
  if (t0 === undefined || !s) return;
  const wait = s.at - t0;
  const server = performance.now() - s.at;
  // On a reused socket, wait is pure queue time.
  // On a new socket, wait also includes DNS, TCP and TLS setup.
  const kind = s.reusedSocket ? 'queue' : 'queue+connect';
  console.log(`${request.origin}${request.path} ${kind}=${wait.toFixed(1)}ms server=${server.toFixed(1)}ms`);
});

const res = await request('');
await res.body.dump();

Undici's diagnostics_channel events split each call into time before headers are written and time waiting for response headers. A reused socket marks the first interval as pure queue time, and a new socket marks it as queue plus connection setup.

How do you size an undici connection pool for each upstream?

Start from Little's law: the average number of requests in flight equals arrival rate multiplied by average time in the system. That average in-flight count is roughly how many connections the pool needs if each connection carries one request at a time.

Illustrative estimate (assumed inputs): at 500 requests/s to one upstream with a 40 ms average response time, 500 × 0.040 = 20 requests in flight. A cap of 20–30 connections gives headroom for bursts and slow responses without inviting excess.

Connection cap What happens at 500 req/s, 40 ms Main risk
2 About 18 requests wait in the pool queue at any moment Callers see high latency that the upstream's own metrics never show
20–30 Most requests get a connection immediately Little, if latency stays near the average
Unbounded New sockets open whenever the upstream slows A connection stampede that deepens the slowdown

The trade-off runs in both directions. A larger cap pushes more concurrency onto the upstream, and every open connection costs a file descriptor and buffer memory in your process and in the upstream. A low cap protects the upstream, but the load still has to go somewhere. It waits in the pool queue, where it shows up only as unexplained outbound latency.

Treat every number here as a starting point. Little's law uses averages, so a long tail at p99 needs more headroom than the mean suggests. Large payloads hold connections for longer. Load-balancer or proxy idle timeouts determine how many idle connections you can actually keep warm.

How do you configure a dedicated Agent or Pool as the dispatcher?

Create an Agent when one dispatcher should serve several origins, and a Pool when it serves exactly one. Both take a connections option. On an Agent it caps sockets per origin, so it does not bound the process total. There are two ways to apply one. You can pass it as the dispatcher option on individual fetch or request calls, which scopes the setting to those calls. Or you can call setGlobalDispatcher once at startup, which changes the default for every call that doesn't specify its own dispatcher, including calls made by libraries you don't control.

Per-upstream versus global

A single global setting is easiest to maintain, but it applies one cap to every upstream. Per-upstream pools isolate failures. Suppose a slow reporting service and a fast auth service share the event loop. If the reporting service stalls, its requests queue in its own pool and auth calls keep their connections. The cost is more to maintain: each pool needs a cap, timeouts and an owner, and every call site has to pick the right one. Use dedicated pools only for upstreams whose latency or volume differs from the rest.

Version mismatch and verification

Node's built-in fetch uses a bundled copy of undici. If you install the undici package to get Agent, its version may not match the bundled one. Objects passed between the two copies can then fail or behave differently in subtle ways. One option is to use undici's own fetch together with its dispatchers.

Don't assume the setting took effect. Set connections to 1 temporarily and send several concurrent requests to a slow endpoint. If the cap is applied, they complete one after another. You can also count established sockets to the upstream with ss -tn while the service is under load.

// Use undici's own fetch with its own dispatchers, so both come from the
// same package version. The undici bundled inside Node may differ.
import { fetch, Agent, Pool, setGlobalDispatcher } from 'undici';

// Default for every call that doesn't pass a dispatcher.
// `connections` caps sockets per origin, not across the process.
setGlobalDispatcher(new Agent({ connections: 50 }));

// Dedicated pools: a stalled reporting service queues in its own pool
// and can't take sockets away from auth.
const authPool = new Pool('', { connections: 20 });
const reportsPool = new Pool('', { connections: 4 });

export async function checkToken(token) {
  const res = await fetch('', {
    method: 'POST',
    body: JSON.stringify({ token }),
    headers: { 'content-type': 'application/json' },
    dispatcher: authPool, // scoped to this call
  });
  return res.json();
}

// Verification: temporarily cap at 1 and fire concurrent requests at a
// slow endpoint. If the cap applies, total time is about N x per-request
// latency. If it doesn't, total time is about 1 x per-request latency.
async function verifyCap() {
  const probe = new Pool('', { connections: 1 });
  const start = Date.now();
  await Promise.all(
    [1, 2, 3].map(async () => {
      const res = await fetch('', { dispatcher: probe });
      await res.arrayBuffer(); // drain the body so the socket is released
    })
  );
  console.log(`3 requests took ${Date.now() - start} ms`);
  await probe.close();
}

if (process.env.VERIFY_POOL === '1') await verifyCap();

Installs a global undici Agent, creates per-upstream Pools that isolate a slow reporting service from auth, and checks that the connection cap applies by timing concurrent requests with connections set to 1.

How should you set keep-alive and timeouts so connections stay healthy?

Every idle socket in the pool is a bet that the other side still has it open. Load balancers and servers close idle connections on their own schedule. If undici's keepAliveTimeout is longer than the upstream's idle timeout, the client will sometimes write a request onto a socket the peer has already closed. The result is an intermittent ECONNRESET or "other side closed" error that looks random and never reproduces locally. The rule is to keep the client's idle timeout a few seconds shorter than the shortest idle timeout on the path.

When the server sends a Keep-Alive: timeout=N header, undici uses that value instead of its default. It subtracts keepAliveTimeoutThreshold as a safety margin and caps the result at keepAliveMaxTimeout. Set that cap explicitly so a generous server hint cannot push sockets past your load balancer's limit.

Long keep-alive has a cost. A socket stays bound to the IP that DNS returned when it connected. After a failover, or under DNS-based balancing, existing connections keep sending traffic to the old targets until they close. Shorter idle timeouts, or recycling connections periodically, spread load more evenly.

Timeout Covers Fails when
connect.timeout TCP and TLS handshake Host is unreachable or overloaded
headersTimeout Waiting for response headers Upstream is slow to start replying
bodyTimeout Gap between body chunks Stream stalls partway through

These timeouts are per phase. None of them caps the whole request, so add an overall deadline with AbortSignal.timeout(). Tight timeouts combined with automatic retries can multiply load on an upstream that is already struggling. Retry only idempotent requests, and use exponential backoff with jitter.

import { Agent, request } from 'undici';
import { setTimeout as sleep } from 'node:timers/promises';

// Assumed: the shortest idle timeout on the path is 60 s (load balancer).
const agent = new Agent({
  keepAliveTimeout: 10_000,         // idle ceiling when the server sends no Keep-Alive hint
  keepAliveMaxTimeout: 30_000,      // cap on server hints so they can't outlive the LB
  keepAliveTimeoutThreshold: 2_000, // safety margin subtracted from the server's hint
  connect: { timeout: 5_000 },      // TCP + TLS handshake
  headersTimeout: 10_000,           // wait for response headers
  bodyTimeout: 15_000,              // max gap between body chunks
});

const IDEMPOTENT = new Set(['GET', 'HEAD', 'PUT', 'DELETE', 'OPTIONS']);

export async function call(url, { method = 'GET', retries = 2 } = {}) {
  // One overall deadline shared by every attempt, including backoff sleeps.
  const deadline = AbortSignal.timeout(20_000);

  for (let attempt = 0; ; attempt++) {
    try {
      const res = await request(url, { method, dispatcher: agent, signal: deadline });
      return { status: res.statusCode, body: await res.body.text() };
    } catch (err) {
      // Never retry non-idempotent requests or once the deadline has fired.
      const canRetry = IDEMPOTENT.has(method) && attempt < retries && !deadline.aborted;
      if (!canRetry) throw err;
      // Exponential backoff with full jitter so clients don't retry in lockstep.
      const backoffMs = Math.random() * 200 * 2 ** attempt;
      await sleep(backoffMs, undefined, { signal: deadline });
    }
  }
}

An undici Agent with its idle timeout and Keep-Alive hint cap set below an assumed 60-second load balancer limit and separate connect, headers and body timeouts, used with an overall AbortSignal deadline and retries limited to idempotent methods with jittered exponential backoff.

Why must you always consume or cancel response bodies?

Undici returns a connection to the pool only after the response body has been fully read or explicitly discarded. Until then, the socket stays assigned to that request. With HTTP/1.1, which is the common case for most upstreams, nothing else can use the connection. If you read the status code, decide you don't care about the payload, and move on, that connection stays busy. Under load, a handful of these per second fills the pool. New requests then queue, and the queue time looks like the upstream getting slower.

The leak hides in the paths nobody reads closely: a 404 you treat as "not found," a 5xx that triggers a retry, an early throw after a failed validation, a 204 handler that never touches the body. With fetch, garbage collection may eventually release the stream. It does so on its own schedule, so treat it as a symptom you might notice rather than a mechanism you can rely on.

The habit is simple: every response either gets consumed or gets cancelled, on every branch.

  • fetch: read the body with await res.json(), res.text(), or res.arrayBuffer(). If you don't need it, call await res.body?.cancel(). Put the cancel in a finally block so thrown errors are covered too.
  • undici request: consume body with body.json() or body.text(), or call await body.dump() to discard it and free the connection.

Adopt this as a code-review rule. It removes a whole class of slow, load-dependent leaks, which are some of the most time-consuming incidents to diagnose after the fact.

How do you observe pool health and confirm a tuning change?

A Pool exposes a stats object with the counters that matter: connected (open sockets), free (open sockets not serving a request), pending, queued (requests waiting for a connection), and running (requests in flight). Sample these on an interval and export them as gauges labeled by upstream origin. The signal to watch is queued staying above zero while free sits at zero. That means callers are waiting on the pool, not on the upstream.

Counters show state but not timing, so pair them with undici's diagnostics_channel events. Subscribe to undici:request:create and undici:request:headers to measure time to first byte per request. Subscribe to undici:client:connected and undici:client:connectError to count new connections and connection failures. A steady rate of new connections under constant load suggests keep-alive is not holding connections open.

Plot p50, p95, and p99 outbound latency next to queue depth on the same time axis. If p99 rises when queued rises, the pool is the bottleneck. If p99 rises while the queue stays empty, look at the upstream or the network.

Confirm every change with a load test against an upstream that behaves like production, with realistic response sizes, latency, and its own concurrency limits. A raised connections limit can move the queue into the upstream. More parallel requests can trigger rate limiting, exhaust its worker pool, or increase its tail latency. Your queue gauge improves while end-to-end p99 gets worse. Only the latency percentiles from a realistic test show which effect wins.

When should you tune undici, when should you avoid it, and what are the alternatives?

Tuning pays off when outbound calls make up a large share of request time. Typical signs are high outbound request rates, several upstreams with different latency profiles that would otherwise share one pool, or socket reset errors and queued requests showing up in metrics. Skip it for low-traffic services, where default limits are rarely reached. Skip it too when the upstream is the bottleneck. A larger pool only sends more load to a service that is already saturated.

Option What it costs When it fits
Global defaults Nothing to build. Little control or visibility per upstream. Low traffic, a single upstream
node:http with http.Agent Familiar API. A separate client stack to maintain beside fetch. Legacy code already built on http.request
HTTP/2 or pipelining Fewer connections. Head-of-line blocking at the TCP or request level, and upstream support varies. Upstreams known to handle it well
Sidecar or mesh proxy Pooling moves to the platform team. Adds another hop and more config to own. Organizations already running a mesh

Explicit pools per upstream take some setup. In exchange, you can tie a latency regression to a specific dependency without guessing, which usually saves engineering time over a service's lifetime.

HTTP agent tuning checklist

  • One dispatcher per critical upstream
  • An explicit connection cap based on measured concurrency
  • A keep-alive timeout set below the upstream's idle timeout
  • Explicit connect, headers, and body timeouts
  • Response bodies always consumed or cancelled
  • Metrics on queue depth for each pool

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to undici connection pool, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.