Nginx Proxy Buffering and Timeouts for Slow Upstream APIs

How a proxied request moves through Nginx and which directive governs each phase

A proxied request crosses two connections: client to Nginx, and Nginx to upstream. Each phase has its own timer, and a failure points to the phase whose timer expired.

Apart from the connect and keepalive timers, these are idle timers. They measure silence between I/O operations, not total request duration. Nginx resets the timer each time bytes move.

Assume proxy_read_timeout is set to 60 seconds. An export endpoint that streams a chunk every 10 seconds can run for 20 minutes without hitting the timeout, because the gap between reads never exceeds 10 seconds. A report endpoint that computes for 90 seconds before sending its first byte fails at the 60-second mark, even though its total runtime is much shorter. Nginx sees 60 seconds of silence and gives up. Raising the timeout fixes the second case. It does nothing useful for the first.

Do not assume you know the values in effect. Defaults differ between versions, and distribution packages often ship config snippets that override them. Run nginx -T to print the fully merged configuration, including every included file, before you change anything.

Copy+----------+                +-----------+                +------------+
|  Client  |                |   Nginx   |                |  Upstream  |
+----------+                +-----------+                +------------+
     |                            |                            |
     |-- request body ----------->|                            |
     |   client_body_timeout      |                            |
     |   (gap between reads)      |                            |
     |                            |-- TCP/TLS handshake ------>|
     |                            |   proxy_connect_timeout    |
     |                            |   (handshake completes)    |
     |                            |-- forward request -------->|
     |                            |   proxy_send_timeout       |
     |                            |   (gap between writes)     |
     |                            |<-- response bytes ---------|
     |                            |   proxy_read_timeout       |
     |                            |   (gap between reads)      |
     |<-- response ---------------|                            |
     |   send_timeout             |                            |
     |   (gap between writes)     |                            |
     |-- next request? ---------->|                            |
     |   keepalive_timeout        |                            |
     |   (idle wait for request)  |                            |
     |                            |                            |

Idle timers reset whenever bytes move: client_body_timeout, send_timeout,
proxy_send_timeout, proxy_read_timeout. Connect and keepalive do not.
Check effective values with: nginx -T

A proxied request spans the client-to-Nginx and Nginx-to-upstream connections, and each phase is governed by its own timer, most of which measure idle gaps between I/O operations rather than total request duration.

Telling 502 from 504 using the error log

The error log is faster to check than the access log because each entry says which phase failed. The status code tells you something broke. The log message tells you whether to look at timeouts, the upstream process, or buffer sizes.

Raising timeouts fixes only the two 504 rows. Raising timeouts for a 502 changes nothing.

A 504 that the client sees may not come from Nginx. If the Nginx access log shows the request as 499, or the request is missing from the log entirely, then a load balancer or CDN in front of Nginx gave up first. In that case the fix belongs in that layer's idle timeout, not in Nginx. The 499 status means the client closed the connection before Nginx replied.

Matching the log message to its row first gives the on-call engineer a clear owner for the problem: network, application, or edge. That saves the time otherwise spent changing timeouts and redeploying to see whether the errors stop.

Copy#!/usr/bin/env bash
# Incident triage: group recent Nginx errors by cause so the right owner
# (network, application, or edge) gets paged.
# Usage: ./triage.sh [error_log] [access_log]
set -u
ERR=${1:-/var/log/nginx/error.log}
ACC=${2:-/var/log/nginx/access.log}

# grep -c prints 0 on no match but exits 1; '|| true' keeps the script going.
count() { grep -cF -- "$1" "$ERR" || true; }

echo "== 504: timeouts (proxy_*_timeout, network path) =="
echo "read header timeout:   $(count 'while reading response header from upstream')"
echo "connect timeout:       $(count 'while connecting to upstream')"

echo "== 502: upstream process or protocol (raising timeouts won't help) =="
echo "connection refused:    $(count 'Connection refused')"
echo "connection reset:      $(count 'Connection reset by peer')"
echo "premature close:       $(count 'upstream prematurely closed connection')"
echo "invalid header:        $(count 'upstream sent invalid header')"

echo "== 502: buffer size (check proxy_buffer_size) =="
echo "too big header:        $(count 'upstream sent too big header')"

# Edge check: clients see 504 but Nginx logs 499 (client closed first)
# or has no entry at all -> a load balancer/CDN idle timeout fired.
# Assumes the default 'combined' log format, where status is field 9.
echo "== Access log status counts =="
awk '$9 ~ /^(499|502|504)$/ { n[$9]++ } END { for (s in n) print s, n[s] }' "$ACC"

A bash triage script that counts each error_log message pattern by the cause it points to, then checks the access log for 499s that suggest a load balancer or CDN timed out before Nginx.

How response buffering works and why it is on by default

With proxy_buffering on, Nginx reads the upstream response as fast as the upstream can send it, independent of how fast the client reads. The response headers and the first part of the body go into a single buffer sized by proxy_buffer_size. The rest of the body fills the pool defined by proxy_buffers, which sets a count and a size per connection. proxy_busy_buffers_size caps how much of that pool can be locked sending data to the client while Nginx is still reading from upstream. When the pool is full, Nginx writes the overflow to a temporary file, up to proxy_max_temp_file_size.

Buffering is the default because the alternative ties an upstream worker to the slowest client. A Gunicorn or Puma process that could finish in 50 ms would otherwise stay occupied for seconds while a mobile client on a weak connection drains the response. With buffering, the upstream hands off the response and moves on, and Nginx absorbs the slow delivery.

The trade-off is memory against disk. Larger buffers keep more responses in RAM and avoid temp-file writes, but the allocation applies to every concurrent proxied connection.

Illustrative estimate

Assume this configuration: proxy_buffer_size 16k for headers and the first chunk, and proxy_buffers 16 32k for the body pool.

• Worst case per connection: 16 KB + (16 × 32 KB) = 528 KB.
• At 2,000 concurrent connections that each fill their buffers: 2,000 × 528 KB ≈ 1.03 GB.

Nginx allocates buffers as data arrives, so small responses use far less than this. Size the configuration for your largest common response, not for the worst case.

Copyupstream app_backend {
    server 127.0.0.1:8000;  # e.g. Gunicorn or Puma
}

server {
    listen 80;
    server_name api.example.internal;

    location /api/ {
        proxy_pass

        # Default is on. Read the upstream response at upstream speed so the
        # app worker is released quickly, regardless of client speed.
        proxy_buffering on;

        # One buffer for response headers and the first part of the body.
        proxy_buffer_size 16k;

        # Per-connection body pool: 16 buffers of 32k = 512k.
        # Worst case with proxy_buffer_size: 16k + 512k = 528k per connection.
        # Illustrative estimate: 2,000 connections x 528k is about 1.03 GB.
        proxy_buffers 16 32k;

        # Max portion of the pool that can be busy sending to the client
        # while Nginx is still reading from upstream.
        proxy_busy_buffers_size 64k;

        # Overflow beyond the in-memory pool spills to a temp file.
        # Cap its size; 0 disables temp files, so once the pool is full
        # Nginx reads from upstream only as fast as the client drains it.
        proxy_max_temp_file_size 256m;

        # How much is written to the temp file in one operation.
        proxy_temp_file_write_size 64k;
    }
}

An annotated location block that sets the header buffer, body buffer pool, busy-buffer limit and temp-file spill limit, with comments tracing the 528 KB per-connection worst case used in the illustrative estimate.

When to turn buffering off, and when to avoid it

Buffering works against you when the client needs bytes as soon as the upstream produces them. Server-sent events, long-polling endpoints and streamed or chunked responses (progress output, token-by-token generation) fall into this group. With proxy_buffering off, Nginx relays data to the client as it arrives from the upstream. With buffering on, events can sit in a buffer until it fills or the response ends.

You don't have to disable buffering for a whole location. The upstream can send the X-Accel-Buffering: no response header to switch it off for that one response. The application then decides per endpoint, and everything else keeps the default.

Avoid disabling buffering broadly. Without buffers, the upstream finishes at the speed of the client's download. A client on a slow link, or one that reads deliberately slowly, keeps an application worker busy for the whole transfer. A modest number of such clients can tie up a fixed-size worker pool.

Request bodies are a separate concern. client_max_body_size caps the upload size, and Nginx rejects larger bodies with 413. proxy_request_buffering off streams the body to the upstream instead of collecting it first.

Copyupstream app_backend {
    server 10.0.0.11:8080;
    server 10.0.0.12:8080;
    keepalive 16;
}

server {
    listen 80;
    server_name api.example.internal;

    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_set_header Host $host;

    # Default: buffering on. Nginx absorbs the response so the app worker
    # is freed quickly, even for slow clients. The upstream can still send
    # "X-Accel-Buffering: no" to disable buffering for a single response.
    location /api/ {
        proxy_pass
    }

    # Server-sent events: relay each event as soon as it arrives.
    # Scoped to this location only, because unbuffered responses tie
    # an app worker to the client's download speed.
    location /api/events {
        proxy_buffering off;
        proxy_read_timeout 1h;   # allow long gaps between events
        proxy_pass
    }

    # Large uploads: allow bigger bodies and stream them to the upstream.
    # Trade-off: no retry to another upstream once sending has started,
    # and a slow upload holds the upstream connection open throughout.
    location /api/uploads {
        client_max_body_size 500m;     # larger bodies get 413
        proxy_request_buffering off;
        proxy_pass
    }
}

Buffering stays on by default and is disabled only where needed: response buffering for an SSE endpoint, and request buffering for a large-upload endpoint with a raised client_max_body_size.

Implementing timeouts per location instead of globally

Start from conservative defaults at the server level and make exceptions explicit:

1. Set tight server-wide values. Keep proxy_connect_timeout short (a few seconds, since a healthy upstream accepts connections almost immediately), and keep proxy_read_timeout and proxy_send_timeout close to the slowest normal API call.
2. Isolate known slow endpoints. Add a dedicated location for paths like /reports/ or /exports/ and raise proxy_read_timeout only there. This directive measures the gap between two successive reads, not the total response time. An export that streams output periodically can run longer than the value you set.
3. Bound retries. Set proxy_next_upstream to the conditions you actually want to retry, such as error timeout. Also set proxy_next_upstream_tries and proxy_next_upstream_timeout. Both default to unlimited.

A global 10-minute timeout makes a hung upstream look the same as a slow one. A request stuck on a deadlock waits the full ten minutes before anyone sees an error. During an incident, those stuck connections pile up and consume Nginx connections and upstream workers.

Short timeouts carry the opposite risk. They cut off valid requests while the upstream keeps working and finishes a job nobody receives. If the client then retries, side effects such as charges or emails can happen twice.

By default, Nginx does not pass non-idempotent requests (POST, LOCK, PATCH) to the next server. You have to add the non_idempotent parameter to enable that, and you should add it only when the endpoint is safe to repeat.

Retries also multiply load. Illustrative estimate (assumed numbers, not a benchmark): worst-case upstream attempts = client requests × tries. With an assumed 200 req/s and tries 3, a pool that is timing out can receive up to 600 attempts/s, which is the moment it can least absorb them.

Aligning the timeout chain across load balancer, Nginx, app server and database

Each layer should give up before the layer outside it. If the database query is the slow part, the database cancels it, the application gets a clear error and returns a meaningful status, and the outer layers relay that response instead of timing out themselves. When the order is inverted, the outermost timer fires first. A load balancer returns a 504 while the query keeps running, and the error points at the wrong component.

Raising proxy_read_timeout does nothing if a load balancer or CDN in front of Nginx has a shorter idle timeout. That layer closes the connection first, so you must change it there.

Illustrative estimate (assumed numbers, not a benchmark): outer = inner + margin. With an assumed 20 s query timeout and 5 s margin per layer, you get app 25 s, Nginx 30 s and LB 35 s. Note that proxy_read_timeout measures the gap between two reads, not the total response time. A streaming upstream can therefore run longer than this value.

Checklist:

• List every hop, including CDNs, service meshes and sidecars.
• Record each hop's timeout, and confirm it in the running config.
• Verify the values strictly decrease from the outermost layer to the innermost.
• Find which layer logged the error first, then fix the timeout at that layer.

Alternatives to longer timeouts and the impact on cost and engineering time

When a request genuinely needs tens of seconds, raising timeouts only delays the failure. The more durable fix is the asynchronous job pattern. The API validates the request, enqueues a job, and returns 202 Accepted with a status URL. The client then polls that URL or receives a webhook or push message when the job finishes. Every HTTP exchange stays short, so the default timeouts across the chain remain valid.

Illustrative estimate (assumed numbers, not a benchmark). Little's law gives the number of requests in flight:

concurrent = arrival rate × time held

Assume 20 req/s, a 45 s job, a 50 ms enqueue, and clients polling every 5 s with 20 ms per poll.

• Synchronous: 20 × 45 = 900 connections and request workers held open at every hop.
• Async: 20 × 0.05 = 1 for enqueues. Each job is polled about 9 times, so polling adds (20 × 9) × 0.02 ≈ 3.6. The total is about 5 concurrent requests.

The 45 s of work still runs. It moves to queue workers that you size for throughput, and the queue absorbs bursts instead of Nginx.

Use async for the few endpoints that are truly slow. A short list of intentional per-location exceptions is easier to reason about than a global timeout raised to fit the worst-case endpoint.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to nginx proxy buffering timeouts, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.