NestJS Graceful Shutdown: Draining Requests and Queues on SIGTERM

What enableShutdownHooks() does and what it leaves out

A NestJS graceful shutdown means that when the process is told to stop, it finishes the work it has already accepted, releases its resources cleanly, and only then exits. Without this, a deploy or pod eviction cuts off in-flight HTTP responses, leaves queue messages half-processed, and drops database connections without closing them. Users see errors and jobs run twice, during operations you schedule on purpose.

Calling enableShutdownHooks() makes Nest listen for termination signals such as SIGTERM and SIGINT. When one arrives, Nest calls app.close(). That call runs onModuleDestroy on your providers, then beforeApplicationShutdown, then closes the HTTP server so it accepts no new connections, and finally runs onApplicationShutdown. Any provider that implements these hooks gets a chance to clean up, and the hooks can be async.

What it does not do matters as much:

• No drain delay. Nest starts closing as soon as the signal arrives. Load balancers may still be routing traffic to the instance at that point.
• No readiness change. Nothing marks the instance unhealthy, so upstream routing has no signal to stop sending requests.
• No tracking of untracked work. Promises, timers, or consumers started outside a Nest-managed provider's hooks are not awaited.

Any service behind a load balancer or consuming queues on Kubernetes needs more than the default behavior.

Call enableShutdownHooks() only in the real entrypoint. Each call adds process-level signal listeners. Creating several apps in one process, as test suites often do, can trigger Node's max-listener warnings or cause the process to exit unexpectedly when a signal arrives.

Copy// worker.service.ts: a provider that uses each shutdown hook in order
import { Injectable, OnModuleDestroy, BeforeApplicationShutdown, OnApplicationShutdown } from '@nestjs/common';

@Injectable()
export class WorkerService implements OnModuleDestroy, BeforeApplicationShutdown, OnApplicationShutdown {
  async onModuleDestroy() {
    // 1st: stop taking new background work (for example, pause a queue consumer)
    console.log('onModuleDestroy: stop pulling new jobs');
  }

  async beforeApplicationShutdown(signal?: string) {
    // 2nd: runs before the HTTP server is closed; finish jobs that are already running
    console.log(`beforeApplicationShutdown (${signal}): finishing in-flight jobs`);
  }

  async onApplicationShutdown(signal?: string) {
    // 3rd: runs after the HTTP server is closed; release connections and pools last
    console.log(`onApplicationShutdown (${signal}): closing DB pool`);
  }
}

// main.ts: the real entrypoint, and the only place hooks are enabled
import { NestFactory } from '@nestjs/core';
import { AppModule } from './app.module';

async function bootstrap() {
  const app = await NestFactory.create(AppModule);

  // Adds process-level SIGTERM/SIGINT listeners that call app.close().
  // Do not call this in test setups that create many apps in one process.
  app.enableShutdownHooks();

  // Nest closes the server as soon as the signal arrives. It adds no drain
  // delay and does not mark the instance unready.
  await app.listen(3000);
}
bootstrap();

A provider that implements the three shutdown hooks, and an entrypoint that calls enableShutdownHooks() so SIGTERM runs onModuleDestroy, beforeApplicationShutdown, the HTTP server close, and onApplicationShutdown in that order.

How SIGTERM works on Kubernetes and why requests still arrive

Deleting a pod, whether through a rollout, a scale-down, or a node drain, does not stop it right away. The API server records a deletion timestamp and the pod enters the Terminating state. Several components watch for that change, and each reacts on its own schedule:

• The kubelet on the pod's node runs any preStop hook, then sends SIGTERM to the main process in each container.
• The EndpointSlice controller removes the pod from the Service's ready endpoints.
• kube-proxy on every node, plus any ingress controller or external load balancer, sees that update and rewrites its own routing rules.

Kubernetes does not coordinate these paths. SIGTERM is a single local action on one node. Endpoint removal must pass through the API server and then reach many independent consumers, and each one applies the change when its watch fires. Your process can receive SIGTERM while other nodes still route traffic to the pod's IP.

Copyt=0     pod marked Terminating; grace period starts
 |-- kubelet --> preStop (if any) --> SIGTERM to container
 |-- endpoints controller --> EndpointSlice updated
 |       |-- kube-proxy (node A) updates rules ... later
 |       |-- kube-proxy (node B) updates rules ... later still
 |       '-- ingress / cloud LB deregisters ...... often last
 |
 |   <-- new requests can still arrive in this window -->
 |
t=terminationGracePeriodSeconds (default 30s)
 '-- SIGKILL to anything still running

If the app stops accepting connections as soon as SIGTERM arrives, requests that are still being routed to it fail with connection-refused errors. The grace period includes the preStop hook. Any time you spend waiting for routing to settle counts against the same budget you need for draining in-flight work.

Copy                 +-------------------------------+
                 | API server                    |
                 | pod gets deletionTimestamp    |
                 +-------------------------------+
                    |                            |
                    v                            v
       +------------------------+   +--------------------------+
       | kubelet (pod's node)   |   | EndpointSlice controller |
       | 1. run preStop hook    |   | drops pod from ready     |
       | 2. send SIGTERM        |   | endpoints                |
       +------------------------+   +--------------------------+
                    |                            |
                    |              +-------------+--------------+
                    |              v             v              v
                    |        +-----------+ +-----------+ +-------------+
                    |        | kube-proxy| | kube-proxy| | ingress /   |
                    |        | node A    | | node B    | | cloud LB    |
                    |        | (later)   | | (later)   | | (often last)|
                    |        +-----------+ +-----------+ +-------------+
                    |              |             |              |
                    |              +-------------+--------------+
                    |                            |  stale routes: new
                    | SIGTERM                    |  requests still sent
                    v                            v
       +--------------------------------------------------------+
       | Pod (Terminating) receives SIGTERM AND new requests    |
       | grace period = terminationGracePeriodSeconds (def. 30s)|
       | preStop time counts against it; then SIGKILL           |
       +--------------------------------------------------------+

When a pod is deleted, the kubelet's local SIGTERM path and the cluster-wide endpoint removal path run independently, so routers with stale rules can keep sending new requests to the pod after it has received SIGTERM and while its grace period counts down.

Making sure SIGTERM reaches the Node process

Kubernetes sends SIGTERM to PID 1 in each container, and only PID 1. Whichever process sits at PID 1 decides whether your NestJS app ever sees the signal.

Why wrappers swallow the signal

Linux treats PID 1 specially. The kernel does not apply default signal actions to it, so SIGTERM only has an effect if that process has explicitly installed a handler. Node does install one once enableShutdownHooks() registers listeners, but only when Node itself is PID 1.

A shell-form CMD npm run start:prod actually runs /bin/sh -c "npm run start:prod". The shell becomes PID 1, starts npm, and npm starts Node. Neither the shell nor npm reliably forwards SIGTERM to its child. Yarn and custom entrypoint.sh scripts have the same problem unless they end with exec. Node never hears the signal, the grace period runs out, and the kubelet sends SIGKILL. On your laptop, Ctrl+C sends SIGINT to the whole foreground process group, so the app shuts down cleanly there and hides the bug.

The fix

Use the exec form, CMD ["node", "dist/main.js"], so Node runs directly as PID 1. If you also need zombie reaping or you spawn child processes, put a minimal init such as tini in front of Node as the entrypoint. It forwards signals and reaps orphaned processes.

Verify inside the container

Run kubectl exec <pod> -- cat /proc/1/cmdline. The output should show node, not sh or npm. Then run kubectl exec <pod> -- kill -TERM 1 and check that your shutdown log lines appear.

Copy# Build stage: compile the NestJS app to dist/
FROM node:20-alpine AS build
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

# Runtime stage: only production deps and compiled output
FROM node:20-alpine
WORKDIR /app
ENV NODE_ENV=production

# Optional minimal init: forwards signals to its child and reaps zombies.
# Only needed if the app spawns child processes.
RUN apk add --no-cache tini

COPY package*.json ./
RUN npm ci --omit=dev
COPY --from=build /app/dist ./dist

USER node

# Exec form (JSON array): no /bin/sh -c wrapper, so no shell sits
# between the kubelet's SIGTERM and Node.
# Without tini, drop ENTRYPOINT and node becomes PID 1 itself.
ENTRYPOINT ["/sbin/tini", "--"]
CMD ["node", "dist/main.js"]

# Anti-patterns that swallow SIGTERM (sh or npm ends up as PID 1):
#   CMD npm run start:prod
#   CMD ["npm", "run", "start:prod"]

# Verify in the cluster:
#   kubectl exec <pod> -- cat /proc/1/cmdline   # node, or tini when using tini
#   kubectl exec <pod> -- ps                    # node runs directly under PID 1
#   kubectl exec <pod> -- kill -TERM 1          # shutdown logs should appear
#   kubectl logs <pod> --previous               # read them if the container already exited

A multi-stage Dockerfile that starts Node with an exec-form CMD, optionally behind tini, so SIGTERM from the kubelet reaches the NestJS process, followed by the commands that confirm the signal is delivered.

Implementing the shutdown sequence step by step

Shutdown runs in a fixed order: onModuleDestroy, then beforeApplicationShutdown, then Nest closes the HTTP server, then onApplicationShutdown. A common mistake is closing database pools in onModuleDestroy. That hook runs before the server stops, so requests still in flight fail with "pool closed" errors. Release shared resources only in the last hook.

Draining HTTP connections

server.close() stops new connections, but it does not resolve until every keep-alive socket closes. During the drain window, add Connection: close to responses so clients drop their sockets after each reply. If your Node HTTP server provides closeIdleConnections(), call it once you call close(). The two adapters handle this differently. Express exposes the raw Node server, so you manage the sockets yourself. Fastify owns its server and runs its own close lifecycle, so put drain logic in Fastify's close hooks and do not touch the socket directly.

Queue consumers

Pause or cancel the consumer so no new jobs are fetched. Let the current job finish, or checkpoint it. Ack only work that completed. Nack or requeue anything unfinished so another replica picks it up. Close the connection after that.

Hard timeout

When SIGTERM arrives, start an unref()'d timer set a few seconds shorter than terminationGracePeriodSeconds. If it fires, log the in-flight request count and any unacked job IDs, then call process.exit(1). A log entry is easier to debug than a silent SIGKILL.

Copyimport { BeforeApplicationShutdown, Injectable, OnApplicationShutdown } from '@nestjs/common';
import { setTimeout as sleep } from 'node:timers/promises';

// Placeholders for your queue client and DB pool
interface QueueWorker { pause(): Promise<void>; close(): Promise<void>; pendingJobIds(): string[] }
interface DbPool { end(): Promise<void> }

@Injectable()
export class ShutdownService implements BeforeApplicationShutdown, OnApplicationShutdown {
  shuttingDown = false; // readiness controller returns 503 while true

  constructor(private readonly worker: QueueWorker, private readonly pool: DbPool) {}

  async beforeApplicationShutdown(signal?: string) {
    this.shuttingDown = true; // first line: fail readiness immediately
    // Hard deadline, set below terminationGracePeriodSeconds (here assumed 30s)
    setTimeout(() => {
      console.error(`Forced exit after ${signal}; pending jobs:`, this.worker.pendingJobIds());
      process.exit(1);
    }, 25_000).unref();

    await sleep(10_000); // drain window: endpoints may still route traffic here
    await this.worker.pause(); // stop fetching; resolve once current job is acked or nacked
  }

  // Nest has closed the HTTP server by now, so no request still needs the pool
  async onApplicationShutdown() {
    await this.worker.close();
    await this.pool.end();
  }
}

// main.ts (Express adapter): ask clients to drop keep-alive sockets during drain
// const shutdown = app.get(ShutdownService);
// app.use((req, res, next) => {
//   if (shutdown.shuttingDown) res.setHeader('Connection', 'close');
//   next();
// });
// app.enableShutdownHooks();

A Nest service that fails readiness, waits a drain window, and pauses the queue in beforeApplicationShutdown. It closes the queue and DB pool only in onApplicationShutdown, after the HTTP server has closed, with a hard timeout that logs pending jobs.

Choosing timing values: an illustrative budget

Each value comes from a measured property of your service. Kubernetes counts terminationGracePeriodSeconds from the moment the preStop hook starts. Node only receives SIGTERM after preStop finishes, so the preStop delay uses up part of the grace period before your application code runs.

Illustrative estimate (assumed numbers, not a benchmark)

The formula, if HTTP draining and job checkpointing run one after the other:

grace = preStop + request + checkpoint + close + margin → 10 + 15 + 8 + 3 + 5 = 41 s, rounded up to 45 s.

If you drain HTTP and pause workers concurrently, replace request + checkpoint with the larger of the two.

The internal timer starts at SIGTERM, not at preStop. Set it to grace − preStop − margin → 45 − 10 − 5 = 30 s. That covers the 26 s of expected work and still fires before the kubelet sends SIGKILL.

The matching pod spec sets terminationGracePeriodSeconds: 45 and a preStop hook that sleeps for 10 seconds. The application reads its 30-second timeout from an environment variable, which keeps the two values close together in the same manifest.

Verifying shutdown under load

Values on paper only matter if the process behaves that way under real traffic, so test the sequence deliberately before trusting it.

Locally

Start the app with node directly, not through a package-manager script, so the PID you signal is the Node process. Point a load generator at an endpoint that does real work, ideally one that holds a request open for a second or two so some are in flight at the moment of shutdown. Enqueue a batch of jobs as well. Partway through, run kill -TERM <pid>. The load tool should report zero failed requests. Every job should be either completed or back in the queue, with none stuck in an active state that nobody owns. A connection-reset error means the server closed sockets before responses were written.

In a test cluster

Run kubectl rollout restart on the deployment while load is running, and repeat it several times, because timing races show up intermittently. Watch the error rate, the 5xx count at the ingress or load balancer, and the job redelivery count. A spike in 502 or 503 responses usually means traffic reached a pod after it stopped listening. Rising redeliveries mean workers were killed mid-job.

Confirming order

Log one line per step with a timestamp:

1. SIGTERM received
2. Readiness set to failing
3. Queue consumers paused
4. HTTP server closed, with the in-flight count reaching zero
5. Active jobs finished
6. Connections closed
7. Process exit

If any step is missing or appears out of order, or the pod shows exit code 137, the grace period expired and the kubelet sent SIGKILL before shutdown finished.

When to avoid it, trade-offs and alternatives

Full coordination isn't always worth it. A nightly batch job that is safe to rerun, a local dev server, or a stateless API whose clients already retry on connection errors can usually get by with enableShutdownHooks() and a short grace period. The extra machinery pays off when dropped requests are visible to users or when half-finished work causes data problems.

The coordination also has costs:

• Longer grace periods slow every pod removal. Rollouts, scale-downs, node drains and autoscaler consolidation all wait for terminating pods.
• A preStop sleep only covers typical propagation lag. It is a guess at how long endpoint removal takes to reach every proxy. A slow control plane can exceed it.
• Closing keep-alive connections needs matching settings elsewhere. Clients must retry on reset connections, and proxy idle timeouts must be shorter than the server's keep-alive timeout. Otherwise the proxy reuses sockets the server has already closed.
• WebSockets and SSE need an explicit protocol. The server should send a close or reconnect signal, and clients should reconnect with backoff, ideally to a different pod.

Some alternatives replace or complement a long grace period:

SIGKILL, OOM kills and node failures skip every hook, so idempotency and redelivery are still required. Graceful shutdown reduces how often those paths run, but it does not remove the need for them.

The implementation and load test are a modest, one-time effort. The return is fewer error spikes during deploys, fewer investigations into duplicate jobs, and less time on call tracing failures that only appear during rollouts.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to nestjs graceful shutdown, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.