What does proxy_read_timeout actually measure, and why does Nginx return a 504?
This guide is for backend and platform engineers who run Nginx as a reverse proxy in front of APIs and get paged when clients see a 504 Gateway Timeout.
proxy_read_timeout does not cap total response time. It limits how long Nginx waits between two successive reads from the upstream connection. The timer restarts every time bytes arrive. The default is commonly documented as 60 seconds, but check the documentation for your Nginx version before relying on that figure.
Two handlers behave very differently under the same limit:
- Streaming handler. It sends a chunk every few seconds and finishes after ten minutes. Every chunk resets the timer, so Nginx never times out.
- Silent handler. It runs a 90-second query, buffers the result, and only then writes headers and body. Nginx sees nothing for 60 seconds, logs
upstream timed out, and returns 504. The backend may still finish the query and log a success that no client ever receives.
Illustrative estimate (assumed numbers, not a benchmark)
- Formula: timeout fires if
longest silent gap > proxy_read_timeout - Streaming: gap = 5 s per chunk. 5 < 60, so the request survives the full 600 s.
- Silent: gap = 90 s of compute. 90 > 60, so the request is cut off at 60 s.
If the upstream goes silent after Nginx has already forwarded the response headers, Nginx cannot send a 504. The client gets a truncated response instead.
Treat the 504 as a symptom. It tells you the upstream stopped producing bytes for too long. It does not tell you why. The root cause sits in the backend: a slow query, lock contention, a blocked worker pool, or a design that holds all output until the work is done.
How does proxy_read_timeout work?
A proxied request passes through three phases, and each has its own timer:
- Connect. Nginx opens a TCP connection to an upstream server, or reuses one from a keepalive pool.
proxy_connect_timeoutlimits how long the handshake may take. - Send. Nginx writes the request line, headers and body to the upstream.
proxy_send_timeoutlimits the gap between two successive write operations, not the total send time. - Read. Nginx waits for the response.
proxy_read_timeoutis applied to each read operation. When the upstream sends any bytes, the timer restarts. When no bytes arrive for the full timeout, Nginx gives up on the upstream connection.
What happens on expiry depends on how far the response has progressed. If no response headers have been passed to the client yet, Nginx logs upstream timed out, may try another server according to proxy_next_upstream, and otherwise returns 504. If headers have already gone to the client, Nginx can only close the connection, and the client sees a truncated body.
The directive only governs the proxy_pass module. Other upstream protocols have their own equivalents, such as fastcgi_read_timeout, uwsgi_read_timeout and grpc_read_timeout. If your location uses one of those, changing proxy_read_timeout has no effect.
Who should use this guide and tune this setting?
- On-call and platform engineers who own the Nginx configuration and need to decide whether a 504 is a configuration problem or a backend problem.
- Backend developers whose endpoints show up in
upstream timed outlog lines and who need to decide between a scoped timeout increase and a change to how the endpoint responds. - Teams running Nginx behind other proxies or load balancers, where several timeouts stack up and the wrong one is easy to change.
It is not relevant if Nginx only serves static files, or if the slow location uses FastCGI, uwsgi or gRPC rather than proxy_pass. In those cases the corresponding module's timeout directive applies instead.
When should you change proxy_read_timeout?
Change it only when all of the following hold:
- The error log shows
upstream timed outwith the suffixwhile reading response header from upstreamorwhile reading upstream. Awhile connectingsuffix means a different directive fired. - The affected route is known, and its latency has been measured from access logs or metrics.
- The slowness is expected behaviour of bounded work, not a regression, a lock, or an exhausted worker pool.
- Every outer layer (load balancer, CDN, client) will still wait longer than the new value.
Do not raise it as a first response to an incident in which many routes suddenly time out. That pattern usually points to a backend that is overloaded or blocked, and a longer timeout keeps more requests stuck in it. If the work routinely exceeds any timeout an outer layer can tolerate, change the request's shape instead, as described in section 7.
How do you tell which Nginx timeout fired from the error log?
Before changing any directive, read the full error line. Every timeout produces the same upstream timed out (110: Connection timed out) prefix. The suffix tells you which phase stalled.
| Error log suffix | Directive that fired | What stalled |
|---|---|---|
while connecting to upstream |
proxy_connect_timeout |
TCP handshake; backend down, overloaded accept queue, or network drop |
while reading response header from upstream |
proxy_read_timeout |
Backend accepted the request but sent no headers in time |
while reading upstream |
proxy_read_timeout |
Headers arrived, then the body paused longer than the timeout between reads |
while sending request to upstream |
proxy_send_timeout |
Backend stopped accepting the request body |
The error log only records failures. To see how close successful requests come to the limit, add timing variables to a custom log_format: $request_time (total, including the client), $upstream_response_time, $upstream_header_time, and $upstream_connect_time where your build provides it. If $request_time is high while the upstream values are low, the client is slow and raising backend timeouts will not help.
Set proxy_read_timeout from the tail of $upstream_header_time for the affected location, not from a guess.
Illustrative estimate (assumed numbers, not a benchmark):
timeout = observed p99.9 × headroom factor
Assume p99.9 = 18 s and a headroom factor of 1.5. That gives 18 × 1.5 = 27 s, rounded to 30 s.
This produces one targeted change backed by data, rather than a series of trial-and-error increases.
http {
# rt = total request time, including time spent reading from / writing to the client
# uct = upstream connect, uht = upstream first header byte, urt = full upstream response
# Upstream fields log "-" if that phase was never reached, and comma-separated
# values if Nginx retried another server.
log_format upstream_timing '$remote_addr [$time_local] "$request" $status '
'rt=$request_time uct=$upstream_connect_time '
'uht=$upstream_header_time urt=$upstream_response_time '
'upstream="$upstream_addr"';
upstream api_backend {
server 10.0.1.20:8080;
}
server {
listen 80;
server_name api.example.internal;
# The access log records timings for successful requests, so you can see how close they get to the limit.
access_log /var/log/nginx/api_timing.log upstream_timing;
# The error log records the "upstream timed out ... while <phase>" lines.
error_log /var/log/nginx/api_error.log warn;
location /api/reports/ {
proxy_pass
# Keep the connect timeout short. A slow handshake means the backend is down or saturated.
proxy_connect_timeout 5s;
# Illustrative estimate: assumed p99.9 uht of 18s x 1.5 headroom = 27s, rounded to 30s.
# This also caps the gap between body reads ("while reading upstream").
proxy_read_timeout 30s;
# Applies between successive writes of the request to the backend.
proxy_send_timeout 60s;
}
}
}
A custom log_format records total, connect, header and response timings for a slow API location, with proxy_read_timeout set from an assumed p99.9 header time plus headroom instead of a guessed value.
How should you raise proxy_read_timeout without hiding regressions?
Apply that 30 s value only to the slow route. Keep proxy_read_timeout conservative at the http or server level, either at the 60 s default or lower if your normal APIs are fast. Then override it inside a dedicated location for the slow endpoint, such as /api/reports/export. Every other route keeps the outer value, so a regression in /api/orders still shows up as a 504 instead of a late success.
A scoped increase is appropriate when all of these are true:
- The endpoint is known. It is one specific route, not "the API".
- The work is bounded. For example, a report over a fixed date range, not an open-ended query.
- Slowness is occasional. It is not the normal path for most requests.
- Latency is measured. You have percentiles from logs or metrics, not a guess.
Raising the timeout globally has three costs:
- Failures turn into slow successes or silent abandonment. A slow query or deadlock now completes late, or the client disconnects first and no error is recorded.
- Alerting goes quiet. The 504s that would have fired your 5xx-based alerts no longer occur.
- Capacity is held hostage. Each waiting request holds a client connection, an upstream connection (both count toward
worker_connections), and a backend worker.
Illustrative estimate (assumed numbers, not a benchmark): in-flight requests ≈ arrival rate × seconds each request is held. Assume 50 req/s arriving while the backend has stalled:
- With a 10 s timeout: 50 × 10 = 500 requests in flight.
- With a 120 s timeout: 50 × 120 = 6,000 requests in flight.
At 6,000, most application worker pools are exhausted, and worker_connections may be as well. Short timeouts fail fast, which sheds load and lets the system recover.
http {
upstream api_backend {
server 10.0.0.12:8080; # placeholder app server
}
server {
listen 80;
server_name api.example.com;
# Conservative default for every route: a stalled backend
# surfaces as a 504 quickly instead of a late success.
proxy_connect_timeout 5s;
proxy_read_timeout 15s;
location /api/ {
# Inherits the 15s server-level read timeout,
# so a regression in /api/orders still fires 5xx alerts.
proxy_pass
}
# Known, bounded, occasionally slow route with measured latency.
# Exact match so the override cannot leak to other paths.
location = /api/reports/export {
# Sized from measured p99 plus headroom, not a guess.
proxy_read_timeout 30s;
proxy_pass
}
}
}
A server-level proxy_read_timeout keeps normal API routes failing fast, while an exact-match location raises the timeout to 30 s only for the slow export endpoint.
Why must every timeout in the request path be aligned?
Each hop in the request path runs its own clock, and the shortest clock decides which error the client sees. Nginx's setting only matters if Nginx is the first component to give up.
| Hop | Typical setting | If this hop fires first |
|---|---|---|
| Client or CDN | Client library timeout, CDN origin timeout | Client sees an error or gateway timeout from that layer. Nginx keeps working on a request nobody is waiting for |
| Load balancer | Idle timeout | Gateway timeout from the load balancer, not from Nginx |
| Nginx | proxy_read_timeout |
504 and an upstream timed out log line |
| App server | WSGI worker timeout, Node server timeouts | Worker is killed or the socket closes, so Nginx logs a reset or premature close and returns 502 |
| Database | Statement timeout | Query is cancelled, and the app usually returns a 500 |
The rule is that each outer timeout should be at least as long as the one inside it, plus a margin for connection setup and response transfer. When the layers are ordered this way, the innermost layer fails first with the most specific error, and that error travels outward intact.
Illustrative estimate (assumed numbers, not a benchmark)
Formula: T_outer ≥ T_inner + margin. This example assumes a 5 s margin per hop.
- DB statement timeout: 25 s
- App worker timeout: 25 + 5 = 30 s
proxy_read_timeout: 30 + 5 = 35 s- Load balancer idle timeout: 35 + 5 = 40 s
- Client timeout: 40 + 5 = 45 s
proxy_read_timeout measures the gap between reads, not the total request time. A streaming response can therefore outlive it.
Some outer limits cannot be raised, such as a hard cap on a managed load balancer, a CDN origin timeout, or a mobile client's fixed timeout. If the work takes longer than that ceiling, tuning will not help, and you need an architectural change like the ones covered in section 7.
Timeout chain, illustrative estimate (assumed values, 5 s margin per hop)
+--------------------------------+
| Client / CDN timeout 45 s | fires first: client/CDN error;
+----------------+---------------+ Nginx keeps working for nobody
|
v
+--------------------------------+
| LB idle timeout 40 s | fires first: gateway timeout
+----------------+---------------+ from the LB, not from Nginx
|
v
+--------------------------------+
| Nginx proxy_read_timeout 35 s | fires first: Nginx returns 504,
+----------------+---------------+ "upstream timed out" in log
|
v
+--------------------------------+
| App worker timeout 30 s | fires first: worker killed or
+----------------+---------------+ socket closed -> Nginx 502
|
v
+--------------------------------+
| DB statement timeout 25 s | fires first: query cancelled,
+--------------------------------+ app usually returns 500
Rule: T_outer >= T_inner + margin. Outer clocks run longest, so the
innermost layer fails first and its specific error travels outward.
The request-path timeout chain with an illustrative budget (assumed values), showing which error the client sees depending on which hop's clock fires first.
When are Nginx retries with proxy_next_upstream safe?
By default, proxy_next_upstream retries on error and timeout: if Nginx cannot connect to a server, or a connect, send, or read timeout fires, it passes the request to the next server in the upstream group. Retries happen only if nothing has been sent to the client yet. Once response bytes are on the wire, Nginx cannot switch servers.
Non-idempotent methods such as POST, PATCH, and LOCK are treated differently. If the request has already been sent to an upstream, Nginx will not resend it to another server unless you add the non_idempotent parameter. The reason is that a read timeout says nothing about whether the backend did the work. Enabling non_idempotent on a payment or order endpoint can execute the same operation twice.
Retries also multiply latency. Each attempt can wait the full proxy_read_timeout, so with a 60-second read timeout and three servers, the client can wait three minutes. Bound this with proxy_next_upstream_tries, which caps the number of attempts, and proxy_next_upstream_timeout, which caps the total time spent trying. Both default to 0, which means no limit.
The same problem applies to clients. When Nginx returns a 504, the upstream usually keeps running. A card may be charged or rows written after the client has already seen a failure. If the client then retries, the work is duplicated unless the endpoint is naturally idempotent or accepts an idempotency key that the backend deduplicates on.
Finally, retries add load. A backend that is timing out because it is overloaded receives two or three times the traffic when every layer retries, and that extra load can push a slowdown into an outage.
upstream api_backend {
server 10.0.0.11:8080 max_fails=3 fail_timeout=10s;
server 10.0.0.12:8080 max_fails=3 fail_timeout=10s;
server 10.0.0.13:8080 max_fails=3 fail_timeout=10s;
}
server {
listen 80;
# Read-only endpoints: retrying on another server is safe.
location /api/reports/ {
proxy_pass
proxy_connect_timeout 2s;
proxy_read_timeout 20s;
# Retry on connection errors, timeouts and gateway-style failures.
proxy_next_upstream error timeout http_502 http_503;
# Bound the retry budget. Both directives default to 0 (unlimited).
proxy_next_upstream_tries 2; # original attempt + 1 retry
proxy_next_upstream_timeout 25s; # total time across all attempts
}
# Payment endpoint: POST has side effects.
location /api/payments/ {
proxy_pass
proxy_connect_timeout 2s;
proxy_read_timeout 30s;
# Deliberately no non_idempotent here. A POST that has already
# been sent is not replayed on another server after a read
# timeout, because the first server may have charged the card.
# Requests that never reached a server (connect error) can still
# move to the next one.
proxy_next_upstream error timeout;
proxy_next_upstream_tries 2;
proxy_next_upstream_timeout 5s;
# Client headers such as Idempotency-Key are forwarded by default,
# so the backend can deduplicate client-side retries after a 504.
}
}
Shows bounded retries with proxy_next_upstream_tries and proxy_next_upstream_timeout on a read-only route, and a payment route that omits non_idempotent so a sent POST is not replayed on another server.
Does nginx keepalive upstream fix timeouts?
Upstream keepalive is often suggested as a timeout fix, but it targets a different phase of the request. By default, Nginx opens a new TCP connection to the backend for every proxied request and closes it afterward. Adding keepalive inside the upstream block keeps a pool of idle connections per worker process. Two more directives are required for the pool to work: proxy_http_version 1.1, because HTTP/1.0 closes connections by default, and proxy_set_header Connection "", which stops Nginx from forwarding a close header to the backend.
Reusing connections removes the TCP handshake (and the TLS handshake, if the upstream uses TLS) from most requests. Under load, this can reduce proxy_connect_timeout failures and ephemeral-port exhaustion. It does nothing for proxy_read_timeout, which starts counting only after the request has been sent. A handler that takes 90 seconds still takes 90 seconds on a warm connection.
Keepalive also introduces a race. If the app server closes an idle connection at the same moment Nginx picks it from the pool, the request is written to a socket the other end has already closed. Nginx logs an error such as "upstream prematurely closed connection" or a connection reset, and the client gets a 502. The fix is to make the backend's idle timeout longer than Nginx's upstream keepalive_timeout. That way Nginx always closes idle connections first.
If you get this wrong, a reproducible 504 turns into a rare 502 that appears only after quiet periods. These are much harder to trace.
What are the alternatives to raising proxy_read_timeout for long-running work?
Raising the timeout keeps a client waiting on a single long request. The alternatives change the request's shape so that no single read wait gets long.
| Option | Why it avoids the 504 | Cost | Avoid when |
|---|---|---|---|
Async job: 202 Accepted plus a status endpoint |
The request returns in milliseconds. The work runs elsewhere. | You need a queue, status storage, polling or callback logic, and new failure states (lost, stuck or duplicated jobs). | One endpoint is occasionally slow and the extra system isn't justified. |
| Streaming partial output | proxy_read_timeout measures the time between two successive reads, so each chunk resets the clock. |
You must disable buffering. | The backend can't produce output until the whole result is ready. |
| Server-sent events or WebSockets | The connection stays open and is designed to be long-lived. | You must set a high proxy_read_timeout or send application heartbeats more often than the timeout. |
Clients or intermediaries don't handle long-lived connections well. |
Streaming only helps if bytes actually reach the client as they are produced. By default Nginx buffers upstream responses, which hides the progress. Set proxy_buffering off for that location, or have the upstream send the X-Accel-Buffering: no response header so the decision stays per-response. Without buffering, Nginx no longer absorbs the response on the client's behalf. A slow client then holds an upstream worker for the full transfer, so a few slow mobile connections can exhaust a small worker pool.
WebSockets are idle most of the time. Nginx closes a quiet connection once proxy_read_timeout expires, so either raise it for that location or send pings well inside the window.
If one known endpoint occasionally takes 90 seconds, use a scoped timeout increase on that location only, with a ceiling set from measured latency. That is cheaper and fails in fewer ways than any of the structural fixes.
How do you verify a timeout fix before and after deploying it?
Verification needs an endpoint whose latency you control. Add a test route to the upstream application that sleeps for a duration passed as a query parameter, and deploy it only where it cannot be reached from outside, such as behind an internal location or in a staging environment. Real endpoints are poor test subjects because their latency varies, so a pass or fail cannot be cleanly attributed to the timeout setting.
Test before the change
Call the sleep endpoint through Nginx with two durations: one just below the current timeout and one just above it. Use curl's -w option to print %{http_code} and %{time_total}. The total time shows which layer gave up.
- A 504 that arrives close to the
proxy_read_timeoutvalue came from Nginx. - A failure at a different elapsed time points to another layer, such as a load balancer in front of Nginx or a timeout inside the application.
While the test runs, tail the error log. Confirm that upstream timed out ... while reading response header from upstream appears for the slow request. That confirms you are testing the timeout you think you are.
Apply and confirm
- Run
nginx -tbefore reloading. A syntax error, or a directive placed in a context where it is not allowed, is caught at this step. Without it, the error surfaces as a failed reload. - Reload Nginx.
- Repeat both requests. The slow request should now return 200. The error log should no longer contain the timeout line for that request.
- Request a duration just above the new ceiling. It should still return a 504 at the new limit, which proves the ceiling is enforced.
Keep the routine
Save the endpoint, the curl commands and the expected log lines as a runbook. The next time someone reports a 504, the steps become a fixed sequence: reproduce the failure, read the elapsed time, match it to a layer, and check the log line. This replaces adjusting timeouts across several layers until the errors stop.
How can this approach save engineering time?
Most of the time lost on 504s goes into guessing: raising a timeout, waiting to see whether errors stop, then raising another one in a different layer. The practices in this guide replace guesses with checks that each take minutes.
- The log suffix identifies the phase immediately. Reading
while connecting,while sendingorwhile readingtells you which directive fired, so nobody spends time tuningproxy_read_timeoutwhen the backend is refusing connections. - Timing fields answer "how much" without experiments. With
$upstream_header_timealready in the access log, the new timeout value comes from existing data rather than from repeated reloads and observation. - Scoped overrides limit the blast radius. Changing one
locationmeans a regression on another route still raises a 504 and an alert, instead of surfacing days later as a vague latency complaint that someone has to investigate from scratch. - Aligned timeouts produce specific errors. When the innermost layer fails first, the error names the real cause, such as a cancelled query or a killed worker. That shortens the search compared with a generic gateway timeout from an outer layer.
- Bounded retries and idempotency keys prevent follow-up incidents. Duplicate charges or duplicated writes caused by replayed requests are expensive to find and reconcile after the fact.
- The verification runbook is reusable. A controlled sleep endpoint and a fixed set of curl commands turn each future 504 report into a repeatable check instead of a new investigation.
Illustrative estimate (assumed numbers, not a benchmark): assume an unstructured investigation involves four trial-and-error reload cycles, each needing 30 minutes of observation, for 2 hours in total. Assume the structured path takes 10 minutes to read the log suffix and timing fields, 10 minutes to make and validate one scoped change, and 10 minutes to run the runbook, for 30 minutes in total. Under these assumed inputs, each incident saves about 90 minutes. Your actual figures depend on traffic, team practice and how often 504s occur.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to nginx proxy_read_timeout 504, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.