How does an nginx graceful reload work?
A graceful reload swaps nginx's configuration while the process keeps running, so clients connected during the change are not cut off. The mechanism depends on nginx's split between a single master process and a pool of worker processes.
When you run nginx -s reload, or systemctl reload nginx with a unit file whose reload action does the same thing, the master process receives SIGHUP. The master then:
- Parses and validates the new configuration.
- Tries to apply it, which includes opening log files and binding any new listen sockets.
- If that succeeds, starts new workers using the new configuration.
- Tells the old workers to shut down gracefully. They stop accepting new connections, finish the requests they are already serving, and exit.
If parsing or applying fails, the master logs an error and keeps the old workers running the old configuration. A broken config file does not take the site down on reload. That same safety means a failed reload can go unnoticed unless something checks for it.
Old workers can stay alive for a while after a reload. Long-lived connections such as WebSockets or slow downloads keep them running, and they keep using the old configuration until those connections close.
A restart works differently. It stops the master, which closes the listen sockets and drops active connections, then starts a new one. A binary upgrade, used to swap the nginx executable itself, uses a separate signal sequence that runs a new master alongside the old one.
Edits do not all behave alike. Server-block and upstream changes usually apply cleanly on reload. Changes to listen sockets and some global settings can behave differently, for example by failing to bind or by logging that a restart is required. Check the error log after any such change.
nginx -s reload or systemctl reload nginx
|
| SIGHUP
v
+-------------------------------------------+
| master process |
| 1. parse + validate new config |
| 2. apply: open logs, bind new sockets |
+-------------------------------------------+
| |
| ok | fails
v v
+-------------------------+ +--------------------------+
| 3. start new workers | | log error, change nothing|
| with new config | | old workers keep serving |
| 4. signal old workers | | old config |
| gracefully | | |
+-------------------------+ +--------------------------+
| |
v v
+--------------+ +------------------------------+
| new workers | | old workers |
| accept new | | stop accepting new conns, |
| connections | | finish in-flight requests, |
+--------------+ | then exit; long-lived conns |
| delay exit |
+------------------------------+
On SIGHUP the nginx master validates and applies the new config, then starts new workers and drains old ones gracefully. If validation or applying fails, the master logs an error and the old workers keep serving the old config.
Why is nginx -t necessary but not sufficient?
nginx -t parses the full configuration the same way the master process would at reload. It catches syntax errors, unknown directives, missing include files, unreadable certificates or keys, and hostnames in upstream or proxy_pass that cannot be resolved at that moment. Because it touches the real filesystem and resolver, run it on the target host or inside the exact image that will serve traffic. A pass on a laptop proves little.
A passing test only means nginx can load the config. It does not mean the config does what you intended.
| Concern | Caught by nginx -t |
Why |
|---|---|---|
| Syntax, unknown directives | Yes | Fails at parse time |
| Missing includes, certs, keys | Yes | Files are opened during the test |
| Unresolvable static hostnames | Yes | Resolved when the config is loaded |
| Backend down or wrong port | No | No connection is attempted |
Wrong location wins the match |
No | Valid config, wrong precedence |
| Rewrite or header logic errors | No | Semantics are never exercised |
| Change matches the ticket's intent | No | Only a review or request test can show this |
nginx -T runs the same test and then prints every loaded file, includes expanded, each preceded by a comment naming its path. Diff that output against a render of the repo version. This catches stray files in conf.d/, hand edits, and templates that rendered differently than expected.
Watch for three-way drift. The repo, the files on disk, and the config the running master holds can all differ. If a reload fails, nginx keeps serving the previous config while the disk holds the new one. The next unrelated reload then ships that change silently. Config management runs that overwrite files without reloading cause the same split.
#!/usr/bin/env bash
# Compare the config nginx would load on this host with the one built from the repo.
set -uo pipefail
IMAGE="registry.example.internal/edge-nginx:candidate" # image built from the repo commit
OUT=$(mktemp -d)
# 1. Dump the effective on-disk config. -T tests first and exits non-zero on failure.
# The dump goes to stdout; the 'syntax is ok' lines go to stderr.
if ! sudo nginx -T > "$OUT/host.conf" 2> "$OUT/host.err"; then
cat "$OUT/host.err"; exit 1
fi
# 2. Dump the same thing from the repo-built image.
# If certs are mounted at runtime, mount them here too or the test fails.
if ! docker run --rm --entrypoint nginx "$IMAGE" -T > "$OUT/repo.conf" 2> "$OUT/repo.err"; then
cat "$OUT/repo.err"; exit 1
fi
# 3. Compare which files are loaded. This catches stray conf.d/ files and missing includes.
grep '^# configuration file' "$OUT/repo.conf" | sort > "$OUT/repo.files"
grep '^# configuration file' "$OUT/host.conf" | sort > "$OUT/host.files"
diff -u "$OUT/repo.files" "$OUT/host.files" || echo "DRIFT: loaded file set differs"
# 4. Compare the full content. This catches hand edits and templates that rendered differently.
if diff -u "$OUT/repo.conf" "$OUT/host.conf" > "$OUT/content.diff"; then
echo "disk matches repo"
else
echo "DRIFT: see $OUT/content.diff"
fi
# Note: this compares disk with repo, not with the running master.
# A failed reload leaves the master on the old config while the disk holds the new one.
# Check the error log after every reload, and do not treat a clean diff as proof
# that the live config matches.
Uses nginx -T to dump the fully expanded config on the host and in the repo-built image, then diffs the loaded file list and the content to expose drift between disk and repo.
How do you gate and verify a reload in automation?
The fix is to treat every reload as a small deployment with an explicit gate and an explicit check. The gate is nginx -t && systemctl reload nginx, run as one step in the pipeline. If the test fails, the deploy fails. The job should exit non-zero and the change should be reverted or blocked. It should not log a warning and continue.
A passing test and a zero exit from systemctl still don't prove the new config is live. The reload is a signal to the master process. With a typical unit file, systemctl reports success once the signal is delivered. The master then tries to apply the new config. If that fails, for example because it can't bind a new listen address or open a file that nginx -t didn't catch, it logs the error and keeps running the old workers. Nothing visibly breaks, and the change simply doesn't happen.
So verify after every reload:
- Tail the error log for the seconds around the reload and fail on
emergoralertlines. - Check worker processes with
ps -o pid,ppid,lstart,cmdfiltered to nginx. The master PID should be unchanged. The workers should have new PIDs and start times later than the reload. Old workers may linger as "shutting down" while they drain connections, which is expected. - Send a smoke request that exercises the changed path, such as the new location, header, or upstream, and assert on its status or response content.
These checks take seconds and catch the failure at deploy time, while the person who made the change still has the context. Without them, the same silent failure shows up days later as a puzzling incident, and someone has to work backwards from symptoms to find a config that was never applied.
#!/usr/bin/env bash
# Reload nginx as a gated deploy step, then verify the new config is actually live.
set -euo pipefail
ERROR_LOG=/var/log/nginx/error.log
PID_FILE=/run/nginx.pid # check the 'pid' directive in your nginx.conf
SMOKE_URL="" # path the change touches
# Snapshot state before the reload so we can compare afterwards.
master_before=$(cat "$PID_FILE")
workers_before=$(ps --ppid "$master_before" -o pid= | tr -d ' ' | sort)
log_offset=$(wc -l < "$ERROR_LOG")
# Gate: a failed config test is a failed deploy. Exit non-zero; don't continue.
nginx -t && systemctl reload nginx || { echo "config test or reload failed" >&2; exit 1; }
sleep 2 # reload is just a signal; give the master time to apply it
# 1. Error log: any emerg/alert since the reload means the old config is still running.
if tail -n +"$((log_offset + 1))" "$ERROR_LOG" | grep -E '\[(emerg|alert)\]'; then
echo "reload rejected by master: see error log" >&2; exit 1
fi
# 2. Master PID must be unchanged, and at least one worker must be new.
# Old workers may still be listed while they drain; that's expected.
master_after=$(cat "$PID_FILE")
[ "$master_before" = "$master_after" ] || { echo "master PID changed" >&2; exit 1; }
workers_after=$(ps --ppid "$master_after" -o pid= | tr -d ' ' | sort)
new_workers=$(comm -13 <(echo "$workers_before") <(echo "$workers_after"))
[ -n "$new_workers" ] || { echo "no new workers: config not applied" >&2; exit 1; }
ps -o pid,ppid,lstart,cmd -C nginx # record PIDs and start times in the deploy log
# 3. Smoke request that exercises the changed path, asserting on status and body.
curl -fsS --max-time 5 "$SMOKE_URL" | grep -q '"status":"ok"' \
|| { echo "smoke request failed" >&2; exit 1; }
echo "nginx reload verified"
A pipeline step that gates the reload on nginx -t, then fails the deploy if the error log shows emerg/alert lines, no new worker PIDs appeared under the unchanged master, or a smoke request to the changed path doesn't return the expected response.
How do you change nginx upstreams without dropping requests?
Order upstream edits so every intermediate config has enough healthy capacity. Add new servers in one reload, confirm they receive traffic, and only then remove old ones in a separate reload. To retire a backend, first mark it down or lower its weight and reload. New workers stop sending it new requests, while old workers finish their in-flight ones. Remove it from the config once its connection count reaches zero.
What happens when a request still reaches a dying backend depends on passive failure handling. max_fails and fail_timeout decide how many failed attempts within a window mark a server unavailable, and for how long. proxy_next_upstream decides which failures, such as connection errors, timeouts or specific status codes, cause nginx to retry the request on another server instead of returning the error to the client. Retrying is only safe for idempotent requests. nginx does not pass non-idempotent methods like POST to the next server by default, and enabling that risks duplicate writes. Passive checks also learn nothing until real requests fail, so some users will see errors before a dead backend is skipped.
Open-source nginx resolves upstream hostnames when it loads the configuration. If a backend's DNS record changes, nginx keeps sending traffic to the old IP until the next reload. You can configure runtime re-resolution with a resolver directive, using either a variable in proxy_pass or, where your build supports it, the resolve server parameter. This fixes stale IPs, but it adds a dependency: request routing now relies on your DNS resolver being available and correct.
upstream app_backend {
# Mark a server unavailable for 30s after 3 failed attempts within 30s.
server 10.0.1.11:8080 max_fails=3 fail_timeout=30s;
server 10.0.1.12:8080 max_fails=3 fail_timeout=30s;
# New server, added in an earlier reload. Confirm it gets traffic first.
server 10.0.1.13:8080 max_fails=3 fail_timeout=30s;
# Being retired: no new requests after reload. Delete this line in a
# later reload, once its in-flight connections reach zero.
server 10.0.1.10:8080 down;
}
server {
listen 80;
location / {
proxy_pass
# Retry on another server only for these failures. POST and other
# non-idempotent methods are not retried unless you add
# 'non_idempotent', which risks duplicate writes.
proxy_next_upstream error timeout http_502 http_503;
proxy_next_upstream_tries 2;
proxy_connect_timeout 2s;
}
# Runtime DNS re-resolution: a variable in proxy_pass makes nginx look
# the name up through 'resolver' at request time, not only at load.
# Trade-off: routing now depends on this resolver being up and correct.
resolver 10.0.0.2 valid=30s;
resolver_timeout 2s;
location /reports/ {
set $reports_host reports.internal.example;
proxy_pass
}
}
An nginx config that drains one backend with down, sets passive failure limits with max_fails and fail_timeout, limits retries with proxy_next_upstream, and uses resolver with a variable in proxy_pass to re-resolve a backend hostname at runtime.
When should you avoid frequent reloads, and what are the alternatives?
Each reload has a cost. The master process re-parses the entire configuration, re-opens listeners, and starts a fresh set of workers. The old workers stop accepting connections but stay alive until their in-flight work finishes. A short HTTP request finishes in milliseconds. A WebSocket, a server-sent event stream, a large download, or an idle client keepalive can hold an old worker for hours. New workers also start with empty per-worker state, such as cached keepalive connections to upstreams, so backends see a burst of new connections after every reload.
worker_shutdown_timeout puts a ceiling on how long old workers can linger. After it expires, nginx closes their remaining connections. Set it too short and every reload cuts off long-lived clients, which is the downtime you were trying to avoid. Set it too long, or leave it unset, and frequent reloads stack up generations of old workers. Each generation holds memory and file descriptors.
Reloads become a problem when they are driven by events rather than by people. Service discovery that rewrites the upstream list on every pod or instance change can trigger reloads faster than old workers can drain.
| Approach | Reloads needed | Main trade-off | Fits when |
|---|---|---|---|
| Runtime DNS re-resolution | None for IP changes | Routing depends on resolver health and TTLs | Backends change IPs often |
| Batching changes | Fewer, on a schedule | Changes apply with a delay | Discovery events are bursty |
| Full restart | N/A (stop and start) | Drops all connections | Changes a reload cannot apply |
| Binary upgrade | None (process swap) | Old and new masters run side by side for a while | Replacing the nginx binary |
Who needs a formal reload process, and what does it cost?
A formal reload process pays off when one nginx instance carries more than one team's risk. Typical cases:
- Shared edge proxies, where a mistake in one
serverblock can break unrelated services. - Multi-tenant configs assembled from templates or includes.
- Automated deploy pipelines that reload with no human watching.
In these setups, a passing nginx -t only proves the config parses. It does not prove traffic reached the intended upstream, and the first person to notice a problem is often a user.
A single site on one host, edited by one person who reloads by hand, needs much less. Run nginx -t, keep a copy of the previous config, reload, then request a known endpoint. Adding CI gates there mostly adds ceremony.
The setup cost is real. You have to write the checklist, the post-reload probes and the config versioning. You also need a rollback that restores a known-good config tree, so nobody has to hand-edit files during an incident. All of it needs maintenance as routes change. The payoff:
- Fewer silent misconfigurations.
- Faster triage, because every reload maps to a reviewed diff.
- Rollback in one step.
Illustrative estimate (assumed numbers, not a benchmark):
- Setup: 24 engineer-hours. Upkeep: 2 hours/month.
- Config-related incidents: 2 per month, each taking 6 hours to triage and fix today, reduced to 2 hours with the process.
Break-even (months) = setup hours ÷ (incidents/month × hours saved per incident − upkeep/month) = 24 ÷ (2 × 4 − 2) = 4 months
This ignores user-facing impact, so if outages are costly the real break-even comes sooner. If your incident rate is low enough that the denominator goes to zero or below, the lighter process is the right choice.
What should a copyable nginx pre- and post-reload checklist include?
Run the steps in this order. Each one either catches a different class of failure or makes the next step trustworthy.
- Commit or back up the current config before you change anything. This file is your rollback target.
- Diff the new config against what is running.
nginx -Tdumps the configuration on disk, including every included file. Save its output at each successful reload, then diff the new dump against that saved copy. A stray include or a file edited outside your change shows up here. - Run
nginx -twith the same binary and paths that production uses. - Reload with
nginx -s reloador your init system's reload action, never a restart. - Tail the error log for at least a few seconds. A failed reload leaves the old workers serving and reports the reason only in the log.
- Verify the workers. New worker PIDs should appear, and old ones should drain and exit. If the PIDs are unchanged, the reload did not take effect.
- Send a smoke request through a route that the change affects.
- Watch 5xx and upstream error rates for a few minutes. Some upstream failures only appear under real traffic.
Rollback means restoring the last known-good file, running nginx -t, and reloading. You should never debug by editing the config on a live box.
| Step | What it catches |
|---|---|
| Backup / commit | Nothing directly; it makes rollback mechanical |
nginx -T diff |
Unintended changes, drift, wrong includes |
nginx -t |
Syntax errors, unknown directives, missing files |
| Error log | Reload rejected at runtime, bind or permission errors |
| Worker check | Reload silently not applied |
| Smoke request | Routing, header, and TLS mistakes |
| Error-rate watch | Bad upstreams, timeouts, capacity problems |
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to nginx reload without downtime, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.