Node.js Graceful Shutdown: Draining Connections on SIGTERM

What graceful shutdown means and what happens on termination

A Node.js graceful shutdown is not something the process does alone. It is coordination between your process and whatever routes traffic to it, such as a load balancer, a service mesh, or Kubernetes Services. The aim is simple. Stop receiving new requests, finish the ones already in flight, release resources, and exit before something forcibly kills you.

On termination, an orchestrator sends SIGTERM and starts a timer. If the process is still running when the grace period ends, it sends SIGKILL. SIGTERM is a request that your code can intercept and act on. SIGKILL is handled by the kernel and never reaches the process, so no handler, finally block, or exit hook runs. Any work not finished by then is lost, and clients see reset connections or 502s.

In Kubernetes, terminationGracePeriodSeconds (30 seconds by default) is a single budget. If the pod defines a preStop hook, the hook runs first and SIGTERM is sent only after it completes. The clock starts at the beginning of termination, not at SIGTERM. A 20-second preStop sleep against a 30-second grace period leaves your application about 10 seconds to drain.

This matters for any team running HTTP APIs behind a load balancer or orchestrator that does rolling deploys or autoscaling. It matters most with frequent rollouts, aggressive scale-down, and spot or preemptible nodes, because each of these turns termination into routine traffic.

Heavy investment is rarely worth it for one-off scripts, batch jobs that can safely restart from scratch, or local development servers.

Copy Clients
    │
    ▼
┌──────────────────────────────┐   stop routing   ┌─────────────────────┐
│ Load balancer / mesh /       │◄─────────────────│ Orchestrator        │
│ Kubernetes Service           │                  │ (e.g. kubelet)      │
└──────────────┬───────────────┘                  └──────────┬──────────┘
               │ new + in-flight requests                    │ preStop,
               ▼                                             │ SIGTERM,
┌──────────────────────────────────────────────┐             │ SIGKILL
│ Node.js process                              │◄────────────┘
│ stop accepting · finish in-flight · release  │
│ resources · exit before SIGKILL              │
└──────────────────────────────────────────────┘

 terminationGracePeriodSeconds (default 30s): one shared budget
 ├────── preStop hook (e.g. sleep 20s) ──────┼──── app drain ~10s ────┤
 │                                           │                        │
 t=0                                      SIGTERM               SIGKILL
 preStop starts                          (catchable)       (uncatchable)

The orchestrator tells the traffic layer to stop routing and signals the Node.js process, and the preStop hook and the application's drain time both come out of the same terminationGracePeriodSeconds budget before SIGKILL.

Who should use it

Graceful shutdown belongs in any Node.js service that a platform stops as part of normal operation. That includes:

• HTTP APIs and web servers behind a load balancer, ingress controller or service mesh.
• WebSocket and server-sent-event servers that hold connections open for minutes or hours.
• Queue consumers and workers whose interrupted messages would otherwise be redelivered or partially processed.
• Services that buffer state in memory, such as batched writes or telemetry, that must be flushed before exit.

Ownership is usually split. Application engineers own the signal handler, the readiness flag and the close order. Platform or SRE engineers own the Dockerfile entrypoint, the pod spec, probe settings and the grace period. The timings in both places have to agree, so both groups need to understand the whole sequence, not just their half.

As noted above, one-off scripts, restartable batch jobs and local development servers can usually skip most of this. A basic SIGINT handler that closes the server is enough there.

When should it be used

Implement it before the service takes production traffic, not after the first incident. It applies whenever termination is routine:

• Rolling deploys. Every rollout terminates every old pod.
• Autoscaling scale-down. Pods are removed while traffic is still flowing to them.
• Node maintenance. Draining a node with kubectl drain evicts its pods.
• Spot or preemptible capacity. Nodes are reclaimed on the provider's schedule, often with short notice.

Revisit the implementation when you add long-lived connections, a new queue consumer, a new database or message client, or when you change the ingress, proxy or load balancer, because each of these changes the close order or the propagation delay.

Symptoms that it is missing or broken include 502s or connection resets that cluster around deploys, terminated containers that report exit code 137, and duplicate message processing after rollouts.

How does it work

Graceful shutdown works in three layers, and a failure in any one of them defeats the other two:

1. Signal delivery. SIGTERM has to reach the Node process. That depends on what runs as PID 1 in the container.
2. Traffic deregistration. The load balancer, kube-proxy and ingress have to stop sending new connections before the process stops accepting them. The process advertises that it is leaving by failing readiness, then waits for routing to catch up.
3. In-process drain. The process closes its listener, closes idle keep-alive sockets, lets in-flight requests finish, stops background work, closes dependencies, flushes telemetry and exits, all under a hard deadline that sits below the platform's SIGKILL.

The sections below cover each layer in that order: getting the signal delivered, closing the HTTP server correctly, handling the Kubernetes propagation race, and then putting the full sequence together with a watchdog.

Making sure the Node process actually receives SIGTERM

None of the handler code matters if the signal never reaches Node. When the container runtime stops a container, it sends SIGTERM to PID 1 inside it. If PID 1 is a shell or a package-manager wrapper, Node may never see the signal. The runtime then waits out the grace period and sends SIGKILL, which ends the process with no cleanup.

Shell-form CMD node server.js actually runs /bin/sh -c "node server.js". Depending on the shell, sh may stay as PID 1 with Node as its child, and most shells do not forward SIGTERM to their children. npm start adds more layers: npm, then a shell, then Node. npm's signal-forwarding behaviour has changed over time, so do not rely on it. Use exec form and invoke node directly.

Exec form still leaves one problem. The kernel does not apply default dispositions to PID 1 for signals that have no handler, so an unhandled SIGTERM is ignored instead of terminating the process. To fix this, either register an explicit SIGTERM handler or run a minimal init that forwards signals and reaps zombie processes.

Verification checklist

• Read /proc/1/cmdline inside the running container and confirm it shows node or your init, not sh or npm.
• Send SIGTERM to the container and confirm your shutdown log line appears.
• Check that the container exits well before the grace period ends rather than being killed at the deadline.
• Repeat the check in the cluster, because the image's entrypoint can be overridden by the pod's command.

CopyFROM node:20-alpine

WORKDIR /app
COPY package*.json ./
RUN npm ci --omit=dev
COPY . .

RUN apk add --no-cache tini

USER node

ENTRYPOINT ["/sbin/tini", "--"]
CMD ["node", "server.js"]

A Dockerfile showing exec-form CMD with tini as PID 1, with the shell-form and npm variants commented out, plus host commands to confirm which process is PID 1 and whether the container stopped on SIGTERM instead of being killed with SIGKILL.

Why server.close() alone is not enough

Once SIGTERM reaches your process, the obvious move is server.close(). It stops the listener from accepting new connections and invokes its callback only after every existing connection has ended. In-flight requests are allowed to finish, which is what you want.

The problem is HTTP keep-alive. A client or load balancer that reuses connections leaves sockets open between requests. Those sockets are idle, but they still count as existing connections, so server.close() keeps waiting for them. Your process stays up until the platform's grace period runs out and it gets SIGKILL, with nothing logged and cleanup half done.

Fixing this takes three pieces:

• Idempotent signal handlers. Register one function for both SIGTERM and SIGINT, guarded by a shuttingDown flag. On a second signal, choose a behavior and write it down: ignore it, or treat it as a request to exit immediately. Exiting immediately is convenient for Ctrl+C during local development. It is risky if your orchestrator might send the signal more than once.
• Socket tracking. Listen for the server's connection event and store each socket in a Set. Remove it on close. Mark a socket active when a request arrives on it and idle when the response finishes. At shutdown, destroy the idle sockets right away and let the active ones finish their current request.
• Connection: close during shutdown. Set this header on any response sent after shutdown begins. The client then closes the socket instead of reusing it.

Newer Node releases include built-in methods on the HTTP server for closing idle connections or all connections. Check the documentation for the Node version you actually run before relying on them. Keep the manual approach as a fallback if you are not sure.

Copyconst http = require('node:http');

// Explicit policy for a repeated signal: true = exit now (handy for Ctrl+C locally),
// false = ignore it (safer if the orchestrator may resend SIGTERM).
const FORCE_EXIT_ON_SECOND_SIGNAL = process.env.NODE_ENV !== 'production';
const GRACE_MS = 10_000; // keep below the platform's kill deadline

const sockets = new Map(); // socket -> true if idle, false if serving a request
let shuttingDown = false;

const server = http.createServer((req, res) => {
  const socket = req.socket;
  sockets.set(socket, false); // mark active
  // During shutdown, tell the client not to reuse this connection.
  if (shuttingDown) res.setHeader('Connection', 'close');
  res.on('finish', () => {
    if (!sockets.has(socket)) return; // already closed
    sockets.set(socket, true); // mark idle
    // A response that started before shutdown may have advertised keep-alive.
    if (shuttingDown) socket.end();
  });
  res.end('ok\n');
});

server.on('connection', (socket) => {
  sockets.set(socket, true); // new connections start idle
  socket.on('close', () => sockets.delete(socket));
});

function shutdown(signal) {
  if (shuttingDown) {
    if (FORCE_EXIT_ON_SECOND_SIGNAL) process.exit(1);
    return; // ignore repeated signals
  }
  shuttingDown = true;
  console.log(`${signal} received, draining connections`);
  server.close((err) => process.exit(err ? 1 : 0)); // stop accepting, wait for the rest
  // Idle keep-alive sockets would otherwise hold close() open until the deadline.
  // Newer Node versions have built-ins for this; check your version's http docs.
  for (const [socket, idle] of sockets) if (idle) socket.destroy();
  setTimeout(() => process.exit(1), GRACE_MS).unref(); // last-resort deadline
}

process.on('SIGTERM', shutdown);
process.on('SIGINT', shutdown);
server.listen(3000);

An idempotent SIGTERM/SIGINT handler that tracks sockets, destroys idle keep-alive connections at shutdown, sends 'Connection: close' on responses started during shutdown, and enforces a fallback exit deadline.

The Kubernetes endpoint-propagation race and how to close it

When a pod is deleted, the kubelet starts the termination sequence and the control plane removes the pod from the Service's endpoints. These happen at the same time, with no ordering between them. kube-proxy on every node, ingress controllers and external load balancers each learn about the removal separately. Until they do, they keep routing new connections to a process that may already have called server.close(). Those clients get connection refused or reset errors, even though your shutdown code is correct.

The fix is to keep serving for a short window after termination begins, while advertising that you are going away.

Fail readiness immediately. On SIGTERM, set a flag that makes the readiness endpoint return 503. Leave the liveness endpoint alone, because failing liveness can get the container killed early. A failing readiness probe gives load balancers that health-check the pod directly a second way to notice the pod is leaving.

Delay closing the listener. There are two ways to add the delay:

• In the app: wait a few seconds after SIGTERM, then start the drain sequence.
• With a preStop hook: the kubelet runs the hook before it sends SIGTERM. This keeps the delay out of application code, but an exec hook running sleep needs a sleep binary in the image, which minimal images often lack.

The pod spec needs three fields:

• terminationGracePeriodSeconds: the total budget. The preStop time counts against it.
• lifecycle.preStop: the delay.
• readinessProbe: a short periodSeconds so the probe notices the 503 quickly.

The grace period must cover the delay plus your drain deadline.

Any delay value is a heuristic. How long propagation takes depends on kube-proxy mode, the ingress implementation and any external load balancer. Measure it under load before you settle on a number.

CopyapiVersion: v1
kind: Pod
metadata:
  name: api
spec:
  # Total shutdown budget: preStop delay (10s) + app drain deadline (30s)
  # + headroom. The preStop time counts against this budget.
  terminationGracePeriodSeconds: 45
  containers:
    - name: api
      image: registry.example.com/api:1.0.0
      ports:
        - containerPort: 8080
      lifecycle:
        preStop:
          exec:
            # Runs before SIGTERM, so the app keeps serving while endpoint
            # removal propagates. Requires a sleep binary in the image.
            # 10s is a heuristic: validate it against your proxy and LBs.
            command: ["sleep", "10"]
      readinessProbe:
        httpGet:
          path: /readyz     # returns 503 once the app receives SIGTERM
          port: 8080
        periodSeconds: 2    # short period so the 503 is noticed quickly
        failureThreshold: 1
      livenessProbe:
        httpGet:
          path: /healthz    # keeps returning 200 during shutdown
          port: 8080
        periodSeconds: 10
        failureThreshold: 3

A pod spec fragment that uses a preStop sleep to delay SIGTERM, a fast readiness probe that the app fails during shutdown, and a grace period sized to cover both the delay and the drain deadline; the 10-second delay is a placeholder that must be validated by testing.

Implementing an ordered shutdown sequence with a hard deadline

Once you have that number, the rest of the sequence is a fixed order. Each step depends on the one before it having finished.

1. Mark not-ready. Flip a flag so the readiness probe returns a non-2xx status.
2. Wait for deregistration. Sleep for the measured propagation delay.
3. Stop accepting connections. Call server.close() and close idle keep-alive sockets.
4. Drain in-flight requests. Let active requests finish.
5. Stop background consumers and timers. Halt queue consumers, intervals and scheduled jobs.
6. Close database pools and message clients.
7. Flush logs and telemetry. Give the flush its own short timeout.
8. Exit. Use exit code 0 if every step finished, otherwise non-zero.

Dependencies close last because in-flight requests still need them. A handler that is halfway through a transaction fails if the pool closes underneath it. You then turn a clean drain into a batch of 500s.

Start a watchdog timer at SIGTERM and set it well below the grace period. When it fires, it does three things:

• Logs how many sockets and tasks are still open, so you can see what blocked shutdown.
• Destroys the leftover sockets.
• Exits non-zero, which separates a forced exit from a clean one in your metrics.

Call unref() on the timer. Without it, the timer itself keeps the event loop alive, and a shutdown that finished cleanly would sit waiting for the watchdog.

Illustrative estimate. Assume a 30 s grace period, a measured 5 s propagation delay and a 15 s maximum request duration:

The 5 s between the watchdog and SIGKILL is headroom. It lets the watchdog's own logs reach the collector before the kubelet kills the process.

Long-lived connections, queue consumers and scheduled work

HTTP requests finish on their own. WebSockets, server-sent events, long streaming responses and queue consumers do not, so each one needs an explicit stop step early in the shutdown sequence.

Tell clients to leave

server.close() does not track upgraded WebSocket sockets, so close them yourself. Send close code 1001 ("going away") so well-behaved clients reconnect with backoff, and their retries land on pods that are not draining. For SSE, end the response after optionally sending a final event that tells the client to reconnect. The browser's EventSource reconnects automatically after the stream ends. For streaming downloads, either let them finish within the deadline or end them so the client can resume or retry. Do not leave a client waiting on a stream until SIGKILL cuts it off.

Stop consuming before disconnecting

For queue consumers, the order matters. First stop fetching or cancel the subscription, so the broker stops sending you messages. Then let in-flight handlers finish and ack them. Messages you cannot finish should be nacked with requeue. On brokers that redeliver unacked messages when the connection closes, closing the connection has the same effect. If you close the connection first, prefetched messages are redelivered anyway, but the handlers still processing them may produce duplicate side effects.

Clear interval timers, and skip scheduled jobs that have not started yet.

Limits

Crashes, OOM kills and unhandled exceptions skip this entire path. You still need idempotent handlers, at-least-once processing and client retries. In-memory caches, batched writes and buffered telemetry can also be lost if flushing takes longer than the deadline or the target is unreachable.

Testing the shutdown path, alternatives and trade-offs

Treat shutdown as a code path that needs its own test. Run a steady load generator against a local process or a kind/minikube deployment. Then send kill -TERM <pid> or trigger kubectl rollout restart while traffic is flowing. Record three things: connection resets seen by the client, 5xx responses from the proxy or ingress, and the process exit code. Exit code 0 means your sequence finished. Exit code 137 means SIGKILL arrived first. Repeat the test with slow requests in flight, because the bugs only appear when the deadline is reached.

Each option involves trade-offs:

• Longer grace periods drop fewer requests, but rollouts are slower and pods hold resources longer.
• Forcing reconnects adds connection setup cost and exposes clients with broken retry logic.
• Retrying POSTs without idempotency keys can charge or enqueue the same request twice.
• Destroying remaining sockets at the deadline aborts some requests. It still beats an unlogged SIGKILL, because you can log and count what was cut off.

On cost, the implementation and its test are a modest, one-time effort. Without them, engineers repeatedly spend time chasing intermittent 502s and lost messages that only appear during deploys. If you adopt a library, read what it does. The ordering decisions are still yours to understand.

How can it save engineering time

The time savings come from removing a class of failures that are expensive to debug because they are intermittent and tied to deploys:

• Fewer deploy-time incidents to investigate. Without draining, every rollout can produce a burst of 502s or resets. These look like flaky infrastructure, and engineers often spend hours correlating proxy logs with rollout timestamps before finding the cause.
• Diagnosable shutdowns. The exit code tells you whether a shutdown was clean (0), forced by the watchdog (non-zero), or killed by the platform (137). The watchdog log line names what was still open. That replaces guesswork with a specific socket count or a stuck consumer.
• Less cleanup of partial work. Stopping consumers before disconnecting and closing pools after requests drain reduces duplicate messages and half-finished transactions. Each of those otherwise needs manual reconciliation or a one-off script.
• Deploying without ceremony. When rollouts do not drop traffic, teams do not need deploy windows, manual traffic shifting or someone watching dashboards during every release. Autoscaling and node maintenance stop being events that page people.
• A shared, testable contract. Once the timing budget is written down in the pod spec and the handler, a shutdown regression shows up in the load test described above rather than in production.

Illustrative estimate. Assume a team deploys a service 10 times a week, that one deploy a week causes errors someone has to investigate, and that each investigation takes an assumed 2 engineer-hours. That is about 100 engineer-hours a year spent on one service. Against that, assume implementing and testing the shutdown sequence takes 1 to 2 engineer-days. Under these assumed inputs, the work pays for itself within the first couple of months. Replace the assumptions with your own deploy frequency and incident history before using this figure to make a decision.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to node.js graceful shutdown, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.