What is the difference between an Nginx 502 Bad Gateway and a 504 Gateway Timeout?

This checklist is for backend and platform engineers running Nginx as a reverse proxy in front of application servers. Both errors mean the same thing to the client: Nginx could not get a usable response from the upstream. Internally, they describe different failures.

504 Gateway Timeout means Nginx stopped waiting. A configured timer expired before the upstream finished its part of the exchange. That timer might be proxy_connect_timeout while establishing the connection, or proxy_read_timeout while waiting for response data. The upstream may still be working on the request. Nginx has simply given up on it.

502 Bad Gateway means the exchange failed, or produced something Nginx could not use. Typical causes:

  • The connection was refused because nothing was listening.
  • The connection was reset by the peer.
  • The upstream closed the connection before sending a complete response.
  • The upstream sent an invalid header.
  • The upstream sent response headers larger than the configured buffers allow.
  • No live upstreams remained in the group.

Treat each status code as a pointer to a specific hop and a specific moment in the request. A 504 says "this hop was too slow at this phase." A 502 says "this hop broke the protocol or the connection." Neither one tells you the root cause by itself, and the error log is what narrows it down.

That is why raising proxy_read_timeout is usually the wrong first move. It can turn a 504 into a slow success. It can also leave the slow query, saturated worker pool, or mismatched downstream timeout in place, where it fails later and harder.

How do you read the Nginx error log to find the cause?

The access log tells you that a request failed: a status code, a timestamp, maybe $upstream_response_time. It rarely tells you why. Nginx writes the reason to the error log at the moment it gives up on the upstream. That entry includes the system error number, the phase of the request (connecting, sending, reading the response header), and the upstream address it was talking to. The phase matters most. A failure while connecting points to the backend's listener or network path. A failure while reading the response header means the backend accepted the request and then stalled or died.

Match on the message, the phase and the upstream address together. Then pull the timestamp and the upstream address into your backend logs.

Error log phrase Status returned Likely cause
upstream timed out (110: ...) while reading response header 504 Backend accepted the request but didn't respond within proxy_read_timeout: slow query, blocked worker, saturated pool
connect() failed (111: Connection refused) 502 Nothing listening: process down, restarting, wrong port, or container not yet ready
upstream prematurely closed connection 502 Backend crashed or was killed mid-request, or closed a reused keepalive connection
no live upstreams 502 Every server in the group is marked unavailable after hitting max_fails
upstream sent too big header 502 Response headers exceed proxy_buffer_size, often from large cookies or auth headers

Exact wording, errno text and context fields vary by build, OS and version. Match on the stable fragments rather than whole lines when you write alerts or log queries.

#!/usr/bin/env bash
# Summarize upstream failures in the Nginx error log by stable message fragment.
# Usage: ./upstream-errors.sh [/var/log/nginx/error.log]
set -euo pipefail

LOG="${1:-/var/log/nginx/error.log}"

# Match on short, stable fragments, not whole lines. Exact wording can vary by build and version.
# Format: fragment|usual status|likely cause
while IFS='|' read -r fragment status cause; do
  count=$(grep -cF -- "$fragment" "$LOG" || true)
  [ "$count" -eq 0 ] && continue

  echo "== $fragment  ($status, $count hits)"
  echo "   likely cause: $cause"

  # First and last timestamps bound the incident window to search in backend logs.
  echo "   first: $(grep -F -- "$fragment" "$LOG" | head -1 | cut -c1-19)"
  echo "   last:  $(grep -F -- "$fragment" "$LOG" | tail -1 | cut -c1-19)"

  # Which backend addresses are involved? One bad node and a whole pool failing are different problems.
  # Some messages (e.g. no live upstreams) may not include this field.
  grep -F -- "$fragment" "$LOG" \
    | grep -oE 'upstream: "[^"]+"' \
    | sort | uniq -c | sort -rn | head -3 \
    | sed 's/^/   /' || true
  echo
done <<'EOF'
upstream timed out|504|backend accepted but did not answer within proxy_read_timeout
Connection refused|502|nothing listening: process down, restarting, or wrong port
upstream prematurely closed connection|502|backend crashed mid-request or closed a reused keepalive connection
no live upstreams|502|every server in the group marked unavailable after max_fails
upstream sent too big header|502|response headers exceed proxy_buffer_size
EOF

A shell script that counts the common upstream failure messages in the Nginx error log. For each message it prints the first and last timestamps and the busiest upstream addresses, so you can line them up with backend logs.

How do Nginx proxy timeouts actually work?

Nginx applies three separate timers to each upstream attempt. None of them limits the total request duration.

Directive What the timer measures Typical failure when it fires
proxy_connect_timeout Establishing the connection to the upstream (TCP handshake, plus TLS if proxy_ssl is used) Unreachable host, full listen backlog, dropped SYNs
proxy_send_timeout The gap between two successive writes of the request to the upstream Large upload to a backend that stopped reading
proxy_read_timeout The gap between two successive reads of the response from the upstream Slow handler that sends nothing before its first byte

The timers reset on every successful read or write. A response that streams one chunk every 20 seconds will never trip a 30-second proxy_read_timeout, even if the stream runs for an hour. An endpoint that computes for 45 seconds before sending its first byte will fail. Long-running reports and exports hit 504s for this reason, while long SSE or chunked streams do not.

Illustrative estimate (assumed numbers, not a benchmark): A request fails when its longest silent gap exceeds proxy_read_timeout. Total stream length does not matter. Assume a 30 s timeout, a 45 s wait before the first byte, and 5 s gaps between chunks after that. The largest gap is 45 s, which is more than 30 s, so the request returns a 504.

All three directives are valid in the http, server and location contexts. A lower level inherits a value unless it sets its own. A minimal setup has these parts:

  • Set conservative timeouts at the server level.
  • Override proxy_read_timeout only in the location that serves the slow endpoint.
  • Leave the other routes on the tighter values.

Defaults and upper limits can differ between versions and builds. Check them in the documentation for your installed version rather than relying on remembered values.

upstream app_backend {
    server 10.0.0.12:8080;
}

server {
    listen 80;
    server_name app.example.internal;

    # Server-level timeouts: inherited by every location below
    # unless a location sets its own value.
    proxy_connect_timeout 5s;   # TCP (and TLS) setup to the upstream
    proxy_send_timeout    30s;  # max gap between two writes of the request
    proxy_read_timeout    30s;  # max gap between two reads of the response

    location / {
        proxy_pass
    }

    # Report export computes for a long time before its first byte.
    # Only the read gap is the problem, so only that timer is raised.
    location /reports/export {
        proxy_read_timeout 120s;
        proxy_pass
    }

    # Server-Sent Events: chunks arrive regularly, so the 30s read timer
    # keeps resetting. The backend should emit a heartbeat more often
    # than proxy_read_timeout; buffering is disabled so events flush.
    location /events {
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_buffering off;
        proxy_pass
    }
}

A server block with tight inherited proxy timeouts, a per-location proxy_read_timeout override for a slow-first-byte export endpoint, and an SSE route that stays within the default 30s gap because each chunk resets the read timer.

Which layer timed out? Ordering timeouts across the request chain

A request typically crosses several hops: client, cloud load balancer, Nginx, app server, database. Each hop enforces its own timeout, and the shortest one on the path decides when the request dies. If the load balancer gives up before Nginx does, the client sees a 504 that Nginx never generated.

To find where the error page came from, look up the request in the Nginx access log. If Nginx logged a 504, the timeout happened at Nginx or behind it. If Nginx logged a 499 (client closed the connection) or a late 200, something in front of Nginx gave up first. Also compare the response body and Server header with Nginx's error page, because load balancers usually serve their own.

Order timeouts so that inner layers fail first. The layer closest to the problem has the most context to log, and it releases its resources instead of continuing work that nobody is waiting for.

Illustrative estimate (assumed numbers, not a benchmark): T_outer = T_inner + margin. Assume a 5 s margin:

  • Database statement timeout: 20 s
  • App request timeout: 25 s
  • proxy_read_timeout: 30 s
  • Load balancer idle timeout: 35 s
  • Client timeout: 40 s

Each value can be changed on its own. If you raise proxy_read_timeout to 120 s and leave the load balancer at a lower default, the change has no effect. Keep an end-to-end inventory that records, for each layer, the setting name, its current value, and who owns it.

+----------------------+
| Client               |  timeout: 40 s
+----------------------+
           |
           v
+----------------------+
| Cloud load balancer  |  idle timeout: 35 s  (may serve its own 504 page)
+----------------------+
           |
           v
+----------------------+
| Nginx                |  proxy_read_timeout: 30 s  (access log: 504 or 499)
+----------------------+
           |
           v
+----------------------+
| App server           |  request timeout: 25 s
+----------------------+
           |
           v
+----------------------+
| Database             |  statement timeout: 20 s
+----------------------+

The shortest timeout on the path wins. Outer = inner + 5 s margin, so the
innermost layer fails first. These values are illustrative, not defaults.
Each layer is set on its own, so keep an end-to-end timeout inventory.

Illustrative timeout ordering across the request chain, where each outer layer waits 5 s longer than the layer behind it, so the layer closest to the problem times out first and logs the error.

Why does upstream keepalive cause intermittent 502 errors?

The keepalive directive in an upstream block keeps a pool of idle connections to the backend in each worker process. Without it, every proxied request opens a new TCP connection (and a new TLS handshake if the upstream uses HTTPS), then closes it. That adds latency and leaves sockets in TIME_WAIT on the Nginx host. At high request rates, those sockets can exhaust the ephemeral port range. Keepalive only works if you also set proxy_http_version 1.1 and clear the hop-by-hop header with proxy_set_header Connection "". Otherwise Nginx talks HTTP/1.0 and asks the backend to close after each response.

The intermittent 502 comes from an idle-timeout race. Suppose the app server closes idle connections sooner than Nginx stops reusing them. Nginx can then pick a connection from its pool just as the backend tears it down. The request fails with "upstream prematurely closed connection" or "connection reset by peer." This needs a connection to sit idle for close to the backend's timeout and then get reused. That is why the errors appear under bursty or low, uneven traffic and vanish under load tests. The fix is to make the backend's idle timeout longer than Nginx's upstream keepalive_timeout.

A separate 502 cause logs "upstream sent too big header." Large cookies or auth headers can exceed proxy_buffer_size. Raise it, and proxy_buffers with it.

For streaming responses, proxy_buffering off sends bytes to the client immediately. The trade-off is that a slow client then holds an upstream connection open for the entire transfer.

upstream app_backend {
    server 10.0.0.11:8080;
    server 10.0.0.12:8080;

    # Idle connections cached per worker process. This is not a cap on total connections.
    keepalive 32;
    # Nginx stops reusing an idle connection after this long.
    # The backend's own idle timeout must be LONGER than this (e.g. 75s on the app),
    # or the backend may close a socket just as Nginx reuses it -> intermittent 502.
    keepalive_timeout 60s;
}

server {
    listen 80;
    server_name app.example.internal;

    # Both are required for upstream keepalive to take effect.
    proxy_http_version 1.1;          # the default, 1.0, closes after each response
    proxy_set_header Connection ""; # drop the hop-by-hop "close" header
    proxy_set_header Host $host;

    # Separate 502 cause: "upstream sent too big header".
    # The response header must fit in proxy_buffer_size. Keep buffer sizes equal
    # so the busy-buffers constraint stays valid.
    proxy_buffer_size       16k;
    proxy_buffers           8 16k;
    proxy_busy_buffers_size 32k;

    location / {
        proxy_pass
    }

    location /events/ {
        # Stream bytes to the client as they arrive (SSE, long downloads).
        # Trade-off: a slow client now holds this upstream connection for the whole transfer.
        proxy_buffering off;
        proxy_read_timeout 300s;
        proxy_pass
    }
}

This upstream config enables keepalive correctly, sets Nginx's idle timeout below the backend's to avoid the reuse race, sizes header buffers to prevent 'too big header' 502s, and turns off buffering only for a streaming location.

When should you avoid raising proxy_read_timeout or enabling retries?

A longer proxy_read_timeout has the same cost as a slow client. Every waiting request holds an Nginx connection and an upstream worker. Before you raise it, check the upstream's own limits. These include worker or thread pool size, the application's request timeout (a worker killed mid-request resets the socket, and Nginx logs that as a 502), and the listen backlog. If the app kills a request at 30 seconds, a 120-second proxy timeout fixes nothing.

Illustrative estimate (assumed numbers, not a benchmark): busy workers = request rate × latency. At 40 req/s and 0.5 s latency, you need 20 workers. If a dependency stalls and requests now wait out a 60 s timeout, you need 2,400 workers. A pool of 64 is exhausted in under 2 seconds. The client may have given up long before that, so the upstream keeps working on responses nobody will read.

Retries via proxy_next_upstream add load at the moment upstreams are least able to absorb it. A request that timed out after the upstream received it may already have executed, and retrying a non-idempotent request (for example, a payment) duplicates it. Nginx does not retry POST, PATCH or LOCK by default unless you add non_idempotent. Bound retries with proxy_next_upstream_tries and proxy_next_upstream_timeout.

Alternative Benefit Cost
Async jobs with polling or callbacks Long work leaves the request path Job queue, status API, client changes
Short fail-fast timeouts Frees workers and surfaces faults quickly More visible errors during slowdowns
Per-location timeouts Long limits only where needed Config drift as the app's routes change

Checklist: diagnosing and fixing Nginx 502 and 504 errors

Work through these in order. Each step narrows the search space for the next.

  1. Reproduce the failure with a known request and timestamp so you can find its exact log lines.
  2. Read the error log entry for that request. "upstream timed out", "connection refused" and "upstream prematurely closed connection" each point to a different cause.
  3. Identify the failing hop: load balancer, Nginx, application server or database.
  4. Compare timeouts across the chain. Each outer layer should wait longer than the layer inside it.
  5. Verify keepalive alignment. Nginx's idle upstream connections must be closed before the backend closes them.
  6. Check upstream capacity: worker pools, queue depth, CPU and connection limits. A slow upstream needs capacity, not a longer timeout.
  7. Change one setting at a time so the result can be attributed to that change.
  8. Run nginx -t to validate the configuration, then reload. A reload keeps serving traffic. A failed config test leaves the running configuration untouched.
  9. Verify the fix against the same request.

To make step 9 objective, extend your log_format with $upstream_status, $upstream_connect_time, $upstream_header_time and $upstream_response_time. These fields split each request into phases:

  • High connect time: the problem is reachability or backlog.
  • High header time with low connect time: the application is slow to start responding.
  • Header time is fine but response time is high: the body is slow to transfer.
  • Comma-separated values: Nginx tried more than one upstream server. This is how retries show up in the log.

Read these fields before and after each change. They show whether the failing phase actually moved. That replaces repeated reload-and-hope directive edits with a single measured comparison.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to nginx 504 gateway timeout, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.