What is the difference between an nginx 502 and a 504 upstream error?
A 504 means nginx gave up. It reached the upstream, or was still trying to, and a configured timer expired before the expected bytes arrived. A 502 means the exchange broke: the connection was refused or reset, the upstream closed it early, or what came back was not a valid HTTP response. Both are generated by nginx, not by your application, so the application's own access logs may show nothing at all.
The distinction matters because the fixes do not overlap. A 504 is a waiting problem: slow queries, saturated workers, or a timeout shorter than legitimate work. A 502 is a connectivity or protocol problem: a dead process, a wrong port, a stale keepalive connection, or a backend crashing mid-response. Raising proxy_read_timeout does nothing for a refused connection. On a real 504 it often just makes clients wait longer for the same failure while connections pile up on the proxy.
This guide is for backend and platform engineers running nginx as a reverse proxy. The first step is always to identify which phase failed. Tuning comes after that.
| Error log message | Status | Failed phase | Likely causes |
|---|---|---|---|
upstream timed out (110: Connection timed out) while reading response header from upstream |
504 | Waiting for response headers | Slow handler, exhausted worker pool, blocking downstream call |
connect() failed (111: Connection refused) |
502 | TCP connect | Process down, wrong port, backend restarting |
upstream prematurely closed connection |
502 | Reading response | Backend crash, idle keepalive closed by upstream, upstream-side timeout |
no live upstreams |
502 | Server selection | All servers marked failed by earlier errors |
"Prematurely closed" is ambiguous. Correlate its timestamp with the backend logs before deciding which cause applies.
Who should use this guide?
This guide is for anyone who owns, or gets paged for, traffic that passes through nginx on its way to an application:
- Backend engineers whose services sit behind nginx and who see 502s or 504s that never appear in their application logs.
- Platform, SRE and on-call engineers who maintain shared nginx configurations, ingress layers or load-balancing tiers and need to decide quickly whether a failure belongs to the proxy, the network or the service.
- Teams adding long-running or streaming endpoints, such as exports, reports, long polling or server-sent events, who need to change timeouts without weakening the defaults for every other route.
- Teams enabling upstream keepalive or retries who want to avoid introducing intermittent 502s or duplicate writes.
It assumes you can edit the nginx configuration, reload it, and read both the nginx error log and the access log. If nginx is managed for you by another layer, the concepts still apply. Only the place where you set the directives changes.
How does it work: the path of a proxied request through nginx
Every 502 and 504 is produced at a specific step in the life of a proxied request. Knowing the steps tells you which directive and which log field correspond to each failure.
- Accept the client request. An nginx worker reads the request line and headers from the client. By default (
proxy_request_buffering on), it also reads the full request body before contacting the upstream, so a slow upload does not tie up a backend worker. - Select an upstream server. nginx picks a server from the
upstreamgroup using the configured balancing method. Round-robin is the default. It skips servers thatmax_failsandfail_timeouthave temporarily marked unavailable. If every server is marked unavailable, nginx logsno live upstreamsand returns a 502. - Connect. nginx either reuses a cached idle connection, if
keepaliveis configured, or opens a new TCP connection. For HTTPS upstreams it also performs the TLS handshake.proxy_connect_timeoutbounds this step. A refused connection becomes a 502. A connect timeout becomes a 504. - Send the request. nginx writes the request to the upstream.
proxy_send_timeoutlimits the gap between successive writes. - Wait for the response header. This is where most 504s happen.
proxy_read_timeoutlimits how long nginx waits between reads. A malformed or missing header becomes a 502. - Relay the body. With
proxy_buffering on, nginx reads the body at backend speed and sends it to the client at client speed. With buffering off, it passes data through as it arrives. - Retry or fail. If steps 3–5 fail and nginx has not yet sent anything to the client,
proxy_next_upstreammay send the request to another server. If no attempt succeeds, nginx generates the 502 or 504 itself and writes the reason to the error log.
Each step has its own timer, its own error log message and its own access log field. The sections below take those pieces one at a time: timeouts, logging, keepalive, retries, and when to raise limits.
How do nginx proxy timeouts actually work?
Each directive times a different phase, and only one of them covers a whole operation. proxy_connect_timeout limits TCP connection setup to the upstream. proxy_send_timeout and proxy_read_timeout measure the gap between two successive writes or reads. They are not deadlines for the full request. Any byte arriving resets the read timer, so a backend that trickles data never trips it.
Illustrative estimate (assumed numbers, not a benchmark): max response time ≈ chunks × interval. Assume proxy_read_timeout is 60s and a report endpoint sends 40 chunks, one every 50s. That gives 40 × 50s = 2,000s, roughly 33 minutes, with no 504. Nginx has no directive for a total upstream deadline, so enforce one in the application or the outermost layer.
proxy_buffering determines which side absorbs slowness. When buffering is on, nginx reads the upstream response into memory or temp files at backend speed. This frees the app worker while a slow client drains the response, which is governed by send_timeout. For streaming responses such as server-sent events, buffering delays delivery, so it is usually turned off. With buffering off, a slow client ties up the upstream connection for the whole transfer.
Every request passes through a chain of timeouts:
client → LB/CDN (65s) → nginx (60s) → app server (55s) → database (50s)
Each outer layer should wait slightly longer than the layer behind it. The formula is outer = inner + margin, with the numbers above assumed. With that ordering, the database aborts first and returns a specific error. The app can then log it and return a meaningful 503. If the order is reversed, the load balancer gives up first and returns a generic 504. The work keeps running behind it, and none of the logs show the real cause.
TIMEOUT CHAIN (assumed values): outer = inner + margin
+--------+ +-----------+ +-----------+ +-----------+ +----------+
| client |-->| LB/CDN |-->| nginx |-->| app server|-->| database |
| | | 65s | | read 60s | | 55s | | 50s |
+--------+ +-----------+ +-----------+ +-----------+ +----------+
|
correct order: innermost fires first v
client <---- 503 + logged cause <---- app maps error <---- DB aborts
reversed order: LB fires first
client <---- generic 504 (app + DB keep working, no cause logged)
NGINX -> UPSTREAM TIMERS (each byte resets the timer)
+----------------------------------------------------------------------+
| connect: |--TCP setup--| proxy_connect_timeout (one phase) |
| send: |w|--gap--|w|--gap--|w| proxy_send_timeout (per gap) |
| read: |r|--gap--|r|--gap--|r| proxy_read_timeout (per gap) |
| |
| read 60s, 40 chunks x 50s gap = 2,000s total, no 504 (illustrative) |
| no nginx directive caps the total: enforce it in app or outer layer |
+----------------------------------------------------------------------+
PROXY_BUFFERING
on : app ==fast==> [nginx memory/temp file] --slow--> client
app worker freed early; client drain limited by send_timeout
off: app <======== connection held for whole transfer ========> client
needed for streaming (e.g. server-sent events)
Request timeout chain with each outer layer waiting longer than the one behind it, nginx's per-gap upstream timers, and how proxy_buffering decides whether nginx or the app absorbs a slow client.
How do you find which phase failed using nginx logs?
The default combined format records only the final status, so a 504 tells you nothing about where the time went. Define a custom log_format that adds $request_time, $upstream_connect_time, $upstream_header_time, $upstream_response_time, $upstream_addr and $upstream_status, and reference it from the access_log directive. All timing values are in seconds with millisecond resolution.
How to read the timing fields
Each upstream timer is measured from the start of that upstream attempt, so they build on each other:
- High connect time: the backend is slow to accept connections. Look at listen backlogs, exhausted worker pools or network problems. For HTTPS upstreams, this field also includes the TLS handshake.
- Fast connect, high header time: the application accepted the request but took a long time to produce its first byte. This is the classic
proxy_read_timeoutcase. - Header and response time far apart: the body was slow to stream.
$request_timemuch larger than$upstream_response_time: the extra time was spent on the client side, either a slow upload or a slow reader.
Retries produce lists
When nginx tries more than one server, it writes one value per attempt, separated by commas, in the same order across all fields. An address of 10.0.0.5:8080, 10.0.0.6:8080 with status 504, 200 (illustrative values) means the first backend timed out and the retry succeeded. The client received a 200, but one of your servers is in trouble. A colon separates values when an internal redirect moved the request to a different upstream group.
With these fields in place, you no longer have to guess which layer stalled. You can identify the failed phase and the backend that served the request from a single log line.
http {
# Quote the upstream fields: on retries they contain comma-separated lists with spaces.
log_format upstream_timing '$remote_addr [$time_local] "$request" '
'status=$status rt=$request_time '
'uct="$upstream_connect_time" '
'uht="$upstream_header_time" '
'urt="$upstream_response_time" '
'ua="$upstream_addr" us="$upstream_status"';
upstream app_backend {
server 10.0.0.5:8080;
server 10.0.0.6:8080;
}
server {
listen 80;
server_name app.example.internal;
# Use the custom format instead of the default combined format.
access_log /var/log/nginx/app_access.log upstream_timing;
location / {
proxy_pass
proxy_connect_timeout 5s;
proxy_read_timeout 30s;
# Retry on the next server after a connect error or timeout.
proxy_next_upstream error timeout;
}
}
}
A custom log_format adds request and per-attempt upstream timings, addresses, and statuses, followed by an illustrative log line in which positional comma-separated values show that a 504 on one backend was hidden by a successful retry.
Why does upstream keepalive cause intermittent 502 errors?
Out of the box, nginx speaks HTTP/1.0 to upstreams and sends Connection: close, so every proxied request opens a new TCP connection. Reuse requires three settings together: a keepalive directive in the upstream block, which sets how many idle connections each worker process caches; proxy_http_version 1.1; and proxy_set_header Connection "" to clear the close header. If any one is missing, connections are not reused and the pool never fills.
Reuse adds a failure mode. The backend has its own idle timeout. When it expires, the backend closes the socket. If nginx picks that cached connection at the same moment and writes a request, the request lands on a socket that is already closing. The error log then shows upstream prematurely closed connection or a Connection reset by peer, and the client gets a 502. These errors appear sporadically, mostly after quiet periods, and are hard to reproduce under steady load.
The fix is to make nginx close idle connections first. Set keepalive_timeout inside the upstream block to a value below the backend's idle timeout, never above it. Nginx then discards connections before the backend can close them underneath it.
The trade-off is worth knowing. Keepalive avoids TCP and TLS handshakes on each request and reduces ephemeral port exhaustion from TIME_WAIT sockets. In exchange, a timeout mismatch produces intermittent 502s, and an oversized pool holds idle connections open. On backends with a fixed number of workers or threads, those idle connections can occupy capacity that real requests need. Size the pool to your concurrency, not to the maximum the backend can accept.
upstream app_backend {
server 10.0.0.11:8080;
server 10.0.0.12:8080;
# Max idle connections cached per nginx worker process.
# Size to real concurrency, not to the backend's maximum:
# idle connections can tie up backend workers or threads.
keepalive 16;
# Close idle upstream connections BEFORE the backend does
# (25s < 30s). This avoids writing a request onto a socket the
# backend is closing, which shows up as "upstream prematurely
# closed connection" or "Connection reset by peer" (502).
keepalive_timeout 25s;
}
server {
listen 80;
server_name app.example.internal;
location / {
proxy_pass
# Both lines are required for reuse. The defaults are
# HTTP/1.0 and "Connection: close", which open a new TCP
# connection for every request.
proxy_http_version 1.1;
proxy_set_header Connection "";
proxy_set_header Host $host;
}
}
An upstream keepalive configuration that enables connection reuse with keepalive, HTTP/1.1, and a cleared Connection header, and sets nginx's upstream keepalive_timeout below the backend's assumed 30-second idle timeout to avoid the close race.
When should you retry failed requests with proxy_next_upstream?
By default, proxy_next_upstream is set to error timeout. If nginx can't connect, or the connect, send or read phase times out, it tries the next server in the upstream group. You can widen this with conditions such as invalid_header, http_502, http_503 and http_504. Only add status codes that mean "this server is unhealthy," not "this request is bad." Retrying a 500 from a deterministic application bug just reproduces the bug on another node. A retry is only possible before nginx has sent any part of the response to the client.
POST, PATCH and LOCK requests are not retried once they have been sent upstream unless you add non_idempotent. The reason is that a timeout tells you nothing about the backend's state. A request that hit proxy_read_timeout may still finish and charge a card, insert a row or send an email. A retry would then do it twice. Enable non_idempotent only when the application deduplicates, for example with idempotency keys.
Retries are unbounded by default, so set limits explicitly:
proxy_next_upstream_triescaps the total number of attempts.proxy_next_upstream_timeoutcaps the total time spent across all attempts.
A reasonable setup for read-only traffic is proxy_next_upstream error timeout http_502 http_503; with proxy_next_upstream_tries 2; and a proxy_next_upstream_timeout a little longer than your worst-case single attempt.
Avoid retries when the whole pool is struggling. If every backend is saturated, each retry adds load to servers that are already behind, and a partial slowdown can turn into a full outage. In that case, fail fast and let the client back off.
upstream api_pool {
server 10.0.1.11:8080 max_fails=3 fail_timeout=10s;
server 10.0.1.12:8080 max_fails=3 fail_timeout=10s;
server 10.0.1.13:8080 max_fails=3 fail_timeout=10s;
}
server {
listen 80;
server_name api.example.internal;
# Per-attempt limits: worst case for one attempt is about 2s + 5s.
proxy_connect_timeout 2s;
proxy_read_timeout 5s;
# Read-only endpoints: safe to replay on another server.
location /api/catalog/ {
proxy_pass
# Retry only on signals that the server is unhealthy.
# Leave out http_500, because an app bug fails the same way everywhere.
proxy_next_upstream error timeout http_502 http_503;
# Without these, retries are unbounded. Allow at most 2 attempts in total,
# and cap the combined time just above one worst-case attempt.
proxy_next_upstream_tries 2;
proxy_next_upstream_timeout 8s;
}
# Write endpoints: a timed-out POST may still commit on the backend.
# Do not retry, and do not add non_idempotent unless the app dedupes
# requests (for example, with idempotency keys).
location /api/payments/ {
proxy_pass
proxy_next_upstream off;
}
}
Bounded retries for idempotent catalog reads, limited by attempt count and total time, while the payments path fails fast to avoid duplicate side effects.
When should you raise proxy_read_timeout, and what are the alternatives?
A long proxy_read_timeout is correct when slowness is the expected behavior of a route, not a symptom. Report generation, large exports, long polling, Server-Sent Events and WebSockets fall into this group. The timer measures the gap between two successive reads from the upstream, not the total response time. A stream that sends a heartbeat every 30 seconds therefore never needs an hour-long value.
Scope the change to the location that needs it, and keep the server-wide default short. For streaming routes, disable proxy_buffering in that location only. Events then reach the client as they are produced instead of waiting in nginx's buffers. Buffering stays on everywhere else, which protects slow clients from tying up upstream workers.
Don't raise timeouts when the backend is slow because it is stuck or saturated. A longer timeout does not make the work finish. It keeps an nginx connection and an upstream worker pinned while nginx waits. When requests arrive faster than workers free up, queues grow and every route slows down. At that point a latency problem has become a capacity outage.
| Approach | Use when | Main cost |
|---|---|---|
| Scoped long timeout | Route is legitimately long-lived | Holds connections per request |
| Async job with polling or callback | Work takes longer than a request should | Job store, status endpoint, retry logic |
| Application-level deadline | Backend should give up before nginx does | Requires propagation through every call path |
| Streaming response | Results can be sent incrementally | Client must handle partial output |
| Fix backend saturation | Timeouts track load, not request type | Capacity planning, profiling |
A global timeout increase is a one-line change that ships in minutes. It tends to cost more later, during incidents, because nothing fails quickly and the slow requests are hard to find. Async jobs take more work up front. In return, they remove the whole class of failures where a request outlives its timeout.
How do you debug an nginx 504 gateway timeout step by step?
-
Reproduce against both hops. Send the same request through nginx and directly to the upstream, using curl's
-woutput with%{time_connect},%{time_starttransfer}and%{time_total}. If the direct call is also slow, nginx is only reporting the delay, not causing it. -
Read the access log timings. Compare
$upstream_connect_time,$upstream_header_timeand$request_time. A small connect time with a header time nearproxy_read_timeoutmeans the backend accepted the request and then stalled before sending headers. A slow connect points to the network or the listen queue. -
Check saturation. Look at the backend's busy workers or threads. Run
ss -lnton the upstream host. On a listening socket, a growingRecv-Qmeans connections are waiting in the backlog, and no worker has accepted them yet.Illustrative estimate (assumed numbers, not a benchmark): queue wait ≈ (requests ahead ÷ workers) × mean service time. With 400 queued requests, 16 workers and 3 s per request, the wait is 400 ÷ 16 × 3 = 75 s. That exceeds a 60 s read timeout before any work begins on the request.
-
Verify DNS. nginx resolves a literal hostname in
proxy_passonce, when it loads the configuration. If the upstream's IPs have changed, nginx keeps sending traffic to the old addresses. Putting the hostname in a variable and adding aresolvermakes nginx re-resolve at runtime. The trade-off is that DNS becomes a runtime dependency, so a slow or failing resolver now fails requests too. -
Decide what to change:
- If requests are queueing or the backend is saturated, fix the backend.
- If 502s are intermittent, align upstream keepalive with the backend's idle timeout.
- If a few known-slow endpoints time out, raise
proxy_read_timeoutonly for those locations. - If the load balancer in front of nginx times out first, align that outer layer's timeouts with nginx.
How can this approach save engineering time?
Most of the time lost on 502s and 504s goes to finding where the failure happened, not to fixing it. Without per-phase data, engineers move between the load balancer, nginx, the application and the database, comparing timestamps by hand. The practices in this guide cut that search down in specific ways:
- Timing fields in the access log show the failed phase and the backend address in a single line. Triage starts from evidence instead of guesses, and the first person paged can usually tell whether the problem belongs to the network, the proxy or the service.
- Ordered timeouts across the chain make the innermost layer fail first with a specific, logged error. Engineers read a cause instead of reconstructing one from a generic 504.
- Aligning upstream keepalive with the backend's idle timeout removes a class of intermittent 502s that are hard to reproduce. These failures often take a long time to investigate because they disappear under steady load.
- Bounded, idempotent-only retries prevent incidents where duplicate writes have to be found and reversed by hand. They also stop retry storms from turning a partial slowdown into a full outage.
- Scoping long timeouts to specific locations keeps fast failure everywhere else. Saturation then shows up early as clear errors instead of as a slow, system-wide degradation that is hard to attribute.
Illustrative estimate (assumed numbers, not a measured result): time saved per quarter ≈ incidents × engineers involved × hours of localization avoided. Assume a team has 6 gateway-error incidents per quarter. Assume each one pulls in 2 engineers who spend 3 hours narrowing down which layer failed, and that per-phase logs and ordered timeouts cut that to 1 hour. The saving is 6 × 2 × (3 − 1) = 24 engineer-hours per quarter. Replace every input with your own incident history. The point of the formula is that the saving scales with how often you investigate these errors and how many people each investigation involves.
The one-time cost is small: one log_format definition, a review of timeout values across each layer, and a check of the backend's idle timeout before you enable keepalive.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to nginx 504 gateway timeout upstream, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.