What is graceful shutdown in a web application, and why does a monolith need a protocol?
A graceful shutdown in a web application is a sequence of steps that runs under a single deadline. A SIGTERM handler is only the trigger. When the process is told to stop, it does four things in order:
- It stops accepting new work.
- It waits long enough for the load balancer to stop routing to it.
- It drains in-flight HTTP requests and background jobs.
- It exits before the orchestrator escalates to SIGKILL.
The order matters. If the process closes its listener before routing has caught up, clients get connection errors. If it exits before jobs finish, those jobs are abandoned mid-write.
Monoliths need this spelled out because one process often serves HTTP, runs job workers and holds pools to the database and cache. Every one of those has to wind down within the same grace period. During a rolling deploy this happens to each instance in turn, so any shutdown bug repeats across the whole fleet.
This applies to backend and platform engineers who run a monolith behind a load balancer or orchestrator.
Graceful shutdown does not protect against crashes, OOM kills or node loss. Your system still has to be crash-safe, which means idempotent jobs and retryable requests. Graceful shutdown reduces errors and retries during planned restarts. It does not replace crash safety.
SIGTERM
|-- readiness check fails, new work refused
|-- routing propagation (LB stops sending traffic)
|-- drain in-flight HTTP requests
|-- drain in-flight jobs (finish or re-queue)
|-- cleanup: close pools, flush logs/metrics
v
exit 0 ...or... grace period expires -> SIGKILL
+---------+ +---------------+ routing +--------------------------+
| clients |------>| Load balancer |<-----------| Orchestrator |
+---------+ +---------------+ update | grace period timer |
| +--------------------------+
| new requests | ^
| (until step 2) | SIGTERM | readiness
v v | probe
+-----------------------------------------------------------------------+
| Monolith process shared shutdown deadline |
| |
| 1. readiness fails, refuse new work |
| 2. wait for routing propagation |
| |
| +----------------+ +----------------+ +----------------------+ |
| | HTTP server | | Job workers | | DB / cache pools | |
| | 3. drain | | 4. finish or | | 5. close pools, | |
| | in-flight | | re-queue | | flush logs/metrics| |
| +----------------+ +----------------+ +----------------------+ |
| |
| 6. exit 0 before deadline, else SIGKILL |
+-----------------------------------------------------------------------+
On SIGTERM the monolith fails readiness, waits for the load balancer to stop routing, drains HTTP requests and jobs, closes pools, and exits before the orchestrator's grace period runs out.
How do you budget the shutdown timeline inside the grace period?
The orchestrator's contract is simple: it sends SIGTERM, waits up to the grace period, then sends SIGKILL. SIGKILL cannot be caught, so the whole sequence above must fit inside that window, with some margin left over:
deregistration delay + request drain + job drain + cleanup + margin < grace period
Size each term from what you measure in production. Do not use defaults. The request drain only has to outlast the slowest in-flight request, so base it on p99 latency or your server's request timeout, whichever is lower. Job drain depends on whether a job can finish within the window or has to checkpoint and re-queue.
Illustrative estimate (assumed numbers, not a benchmark):
- Deregistration propagation: 5 s (assumed)
- Request drain, p99 latency of 2 s plus headroom: 3 s
- Job drain, longest job allowed to finish: 20 s
- Cleanup (pools, log and metric flush): 2 s
- Safety margin: 5 s
The total is 5 + 3 + 20 + 2 + 5 = 35 s. A 30 s grace period would SIGKILL mid-job. You can fix this in two ways. You can raise the grace period to about 40 s. Or you can cap how long a job may run during shutdown at 10 s and re-queue anything still running at that point, which brings the total to 25 s.
| Grace period | Benefit | Cost |
|---|---|---|
| Short (seconds) | Fast rolling deploys, quick node replacement | Long jobs must checkpoint or re-queue, and the risk of SIGKILL is higher |
| Long (minutes) | Most work finishes in place | Slow rollouts, stuck drains delay node drains and autoscaling |
No single value is correct. Pick the smallest grace period that fits your measured budget, and make long work interruptible so you are not forced to keep raising it.
Why does my process get SIGKILLed instantly? SIGTERM handling basics
A process that dies abruptly at the end of the grace period usually never saw SIGTERM. The signal went to something else, which ignored it until the runtime gave up and sent SIGKILL.
Check what is PID 1
Run cat /proc/1/cmdline or ps -o pid,ppid,comm inside the container. If PID 1 is sh, bash, or a wrapper script rather than your server, that is the cause. A shell-form entrypoint such as CMD node server.js runs as /bin/sh -c "node server.js". The shell becomes PID 1 and does not forward SIGTERM to its child. Wrapper scripts that run setup and then launch the app as a child behave the same way.
PID 1 has a further quirk: the kernel does not apply default signal actions to it. Even when your app is PID 1, SIGTERM does nothing unless the app installs a handler.
Use exec in entrypoint scripts
End the entrypoint script with exec "$@" or exec your-server --flags. exec replaces the shell with your process, which then becomes PID 1 and receives signals directly. Use exec-form entrypoints, or run a small init that forwards signals and reaps zombies.
Keep the handler small
A signal handler can interrupt your code at almost any point. Locks, allocators, and I/O may be in an inconsistent state when it runs. The handler should only set a shutting_down flag or wake a shutdown routine, for example by closing a channel or writing to a pipe. The main loop then stops accepting work and drains.
Confirm the stop signal
Some servers treat SIGQUIT or SIGINT as the graceful signal and SIGTERM as a fast stop. Process managers and supervisors can also be configured to send a different stop signal. Check what your runtime sends and what your server expects, and align the two.
#!/bin/sh
# --- entrypoint.sh: run setup, then hand PID 1 to the server ---
set -e
# Setup steps run in the shell; this is fine because they finish first.
if [ -z "${DATABASE_URL:-}" ]; then
echo "DATABASE_URL is not set" >&2
exit 1
fi
# exec replaces this shell with the server process. The server becomes
# PID 1 and receives SIGTERM directly. Without exec, the shell stays
# PID 1, does not forward the signal, and the app is SIGKILLed later.
exec "$@"
# --- Inspect PID 1 (run inside the running container) ---
# tr '\0' ' ' < /proc/1/cmdline; echo
# ps -o pid,ppid,comm
# If PID 1 shows sh, bash, or this script instead of your server,
# the signal is not reaching the app.
# --- worker.sh: minimal handler that only sets a flag ---
# #!/bin/sh
# shutting_down=0
# # The trap does no cleanup itself; it just records the request.
# trap 'shutting_down=1' TERM INT
#
# while [ "$shutting_down" -eq 0 ]; do
# ./process-next-job.sh # finish the current unit of work
# sleep 1
# done
#
# # Drain happens here, in normal control flow, not in the trap.
# echo "stopping: no new jobs accepted"
# exit 0
An entrypoint script that uses exec so the server becomes PID 1, commands to check which process is PID 1, and a worker whose SIGTERM trap only sets a flag so the main loop can stop and drain on its own.
How does connection draining work at the load balancer and in the HTTP server?
Draining starts at the edge, not in your process. On SIGTERM, flip a readiness flag so health checks fail, but keep serving every request that still arrives. Only after the load balancer has stopped routing to you should you close the listener. If you close it first, you hit a race: routing tables, health-check intervals and proxy caches lag by a few seconds, so traffic keeps landing on a closed port and clients see connection resets.
The common fix is a pre-stop delay, a fixed sleep between failing readiness and closing the listener. It works, but it is a guess. If it is too short, you still drop requests. If it is too long, every deploy is slower by that amount per instance. Tie it to your health-check interval and unhealthy threshold, and revisit it whenever those settings change.
Then drain in-flight HTTP work in this order:
- Stop accepting new connections.
- Let active requests finish.
- Close idle keep-alive connections, or send
Connection: closeon the next response. - Enforce a hard deadline for this phase so one slow request cannot use up the whole grace period.
WebSockets, streaming responses and long polling never finish on their own. Close them explicitly and make clients reconnect with jittered backoff. Without jitter, every client reconnects to the remaining instances at once and causes a spike. For slow uploads or exports, decide whether cutting them off at the deadline is acceptable or whether they need to be resumable. The example below shows a readiness flag combined with a server drain.
const http = require('node:http');
// Tie this to health-check interval x unhealthy threshold, plus margin.
const PRE_STOP_DELAY_MS = 15000;
// Hard cap on the in-flight drain phase so one slow request cannot eat the grace period.
const DRAIN_DEADLINE_MS = 20000;
let ready = true;
const server = http.createServer((req, res) => {
if (req.url === '/readyz') {
// The load balancer polls this; 503 tells it to stop routing new traffic here.
res.statusCode = ready ? 200 : 503;
return res.end(ready ? 'ok' : 'draining');
}
// Keep serving while draining, but ask clients not to reuse this connection.
if (!ready) res.setHeader('Connection', 'close');
res.end('hello');
});
server.listen(8080);
process.once('SIGTERM', async () => {
ready = false; // 1. Fail readiness, keep the listener open.
// 2. Wait for the load balancer to notice and stop routing (a fixed guess).
await new Promise((resolve) => setTimeout(resolve, PRE_STOP_DELAY_MS));
// 3. Stop accepting connections; the callback fires once active requests finish.
server.close(() => process.exit(0));
server.closeIdleConnections(); // Drop idle keep-alives instead of waiting on them.
// 4. Deadline: force-close whatever is left. Upgraded sockets (WebSockets)
// are not tracked by the server; close those yourself before this point.
setTimeout(() => {
server.closeAllConnections();
process.exit(1);
}, DRAIN_DEADLINE_MS).unref();
});
A Node.js HTTP server that fails its readiness check on SIGTERM, keeps serving through a pre-stop delay, then closes the listener and idle keep-alives and force-closes remaining connections after a hard drain deadline.
How do you drain in-flight jobs without losing or duplicating work?
Workers need the same three-step sequence as the HTTP server. First, stop fetching: on SIGTERM, set a shutdown flag and stop polling the queue. Second, let running jobs finish, but only until a deadline set from your timeline budget. Third, handle whatever is still running at the deadline. You can explicitly release it back to the queue, which gives the fastest retry, or you can exit and let the visibility or lease timeout return it.
Neither option gives you exactly-once processing. A job can charge a card or send an email and then get killed before it acknowledges. The queue redelivers it, and the side effect happens a second time. Shutdown makes this window more frequent, but at-least-once delivery means it was always possible.
The defence is to make jobs interruptible and idempotent:
- Derive an idempotency key from the job's identity. Record the key in the same transaction as the side effect, or pass it to any external API that accepts one, so a retry becomes a no-op.
- Split long jobs into smaller units. Persist a checkpoint after each unit so a redelivery resumes where the last run stopped instead of starting over.
- Check the shutdown flag only at safe points. A safe point is between units, never in the middle of a write.
| Policy | Best for | Development cost | Risk if misapplied |
|---|---|---|---|
| Wait to finish | Short jobs well under the deadline | Low | Hits SIGKILL when a job runs long |
| Kill and retry | Idempotent, cheap jobs | Low to moderate (idempotency keys) | Duplicate side effects |
| Checkpoint | Long batch or export jobs | High (state schema, resume logic) | Corrupt or skipped progress |
The worker loop below checks the flag between jobs and wraps each side effect in an idempotency key.
import signal
import threading
shutdown = threading.Event()
def on_sigterm(signum, frame):
# Step 1: stop fetching. The loop exits at the next safe point.
shutdown.set()
signal.signal(signal.SIGTERM, on_sigterm)
def run_worker(queue, db, payments):
while not shutdown.is_set():
# Short poll timeout so the flag is rechecked often.
job = queue.fetch(timeout_seconds=2)
if job is None:
continue
# Key comes from the job's identity, so a redelivery reuses it.
key = f"charge:{job.payload['order_id']}"
try:
with db.transaction() as tx:
if tx.idempotency_key_exists(key):
queue.ack(job) # already done on an earlier delivery
continue
# Pass the same key to the external API so it can deduplicate too.
payments.charge(job.payload['amount_cents'], idempotency_key=key)
# Record the key in the same transaction as our own writes.
tx.record_idempotency_key(key)
queue.ack(job)
except Exception:
# Return the job now instead of waiting for the visibility timeout.
queue.release(job)
raise
# Safe point reached: no job in hand, nothing half-written.
# Steps 2-3 (deadline, release leftovers) belong to the supervisor.
A worker loop that checks a shutdown flag only between jobs and uses a key derived from the order ID, so a redelivered job is skipped or deduplicated instead of charging twice.
Who receives SIGTERM in each monolith process model, and in what order should resources close?
Which process sees SIGTERM depends on how the monolith is packaged. In a container, the runtime signals PID 1 only. If PID 1 is a shell wrapper that doesn't forward signals, no part of your application ever hears it.
| Process model | Receives SIGTERM | Coordinates the drain | Typical gap |
|---|---|---|---|
| Master with forked workers | Master (PID 1) | Master forwards a graceful-stop signal to each worker, then waits on them | Master exits before its workers, orphaning their requests |
| Separate web and worker processes | Each process, in its own container or unit | Each process drains independently | Web and job deadlines tuned separately, so one overruns |
| Web and jobs in one process | That single process | An in-process coordinator | HTTP and job loops shut down unaware of each other |
Teardown order
- Close intake. Stop accepting HTTP connections and stop fetching jobs.
- Wait for in-flight work, up to the shared deadline.
- Flush logs, metrics and traces. In-flight work is still emitting telemetry until step 2 finishes.
- Close database pools and broker connections.
The common failure is closing a shared resource early. Suppose the database pool closes while a request is still running. That request fails partway through its transaction, so the client gets a 500 and partial side effects may remain. A job in the same situation can't acknowledge its message, so the broker redelivers it and the work runs twice. Flushing telemetry before work finishes has a similar effect: the logs that would explain the failure are lost.
The coordinator sketch below runs these stages in order against a single deadline. Each stage gets the time remaining from that one budget rather than its own fixed timeout. If the drain overruns, the coordinator still flushes telemetry and closes connections before the runtime sends SIGKILL.
How do you verify graceful shutdown, and when should you skip it or use alternatives?
Treat shutdown as a test you run on every deploy. Before rolling a release to staging, start a load generator that sends steady synthetic requests, including a few deliberately slow endpoints. At the same time, fill the queue with jobs that sleep longer than a typical request. Then trigger a normal deploy. Afterward, check four things: non-2xx responses, connection resets seen by the client, jobs that ran twice, and jobs still marked in progress with no live worker. Have the shutdown coordinator log each phase (signal received, readiness failed, listener closed, workers stopped, connections closed) with a timestamp. When the drain overruns its budget, the log shows which stage consumed the time.
This turns a deploy-time incident, which someone would otherwise debug from scattered logs at 2 a.m., into a failing check in CI. That check catches regressions such as a new dependency that blocks on close or a job type that ignores cancellation.
Keep the investment small when a stateless service handles short requests that clients already retry. Do the same for batch systems where every job is idempotent and redelivery costs little. In those cases, a basic SIGTERM handler is enough.
| Approach | What it covers | What it misses |
|---|---|---|
| Crash safety plus idempotent retries | Any termination, including SIGKILL | Latency spikes and wasted work |
| Client-side retries | Transient resets | Non-idempotent writes without keys |
| Resumable endpoints | Long uploads and exports | Ordinary short requests |
| Blue-green cutover | Traffic drains at the router | Jobs on the old fleet |
These approaches complement drain rather than replace it. Drain also never replaces crash tolerance. Processes still die without warning, so the system must stay correct when no shutdown handler runs at all.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to graceful shutdown web application, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Consumers learn about a breaking change at review time rather than from production errors.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.