What does Nginx do by default when proxying to an upstream?

By default, Nginx proxies to an upstream over HTTP/1.0 and sets Connection: close on the upstream request. The backend sends its response and the TCP connection is torn down. The next request, even to the same server from the same worker, starts from scratch: a TCP three-way handshake and, for an HTTPS upstream, a full TLS handshake before any request bytes are sent.

That per-request setup has several costs:

  • Latency. Each request pays at least one round trip for TCP, plus one or two more for TLS. Inside one data center this adds a small amount per request. Across zones or regions it can exceed the time the backend spends handling the request.
  • CPU. TLS key exchange is the expensive part of a session. Repeating it for every request costs CPU on both Nginx and the backend.
  • Ephemeral ports. Each new connection uses a local source port. A port in TIME_WAIT cannot be reused for the same backend address and port until the timer expires. At high request rates to a few backends, the ephemeral range can run out, and new connections then fail.
  • TIME_WAIT buildup. Whichever side closes first holds the socket in TIME_WAIT. At thousands of requests per second, these sockets pile up on the proxy, the backend, or both.

Upstream keepalive fixes this by reusing connections. Each worker keeps a small pool of idle connections to the backends and sends new requests over them instead of opening fresh ones.

Default (connect per request)
  worker ──SYN/TLS──▶ backend   req 1 ──▶ close
  worker ──SYN/TLS──▶ backend   req 2 ──▶ close
  worker ──SYN/TLS──▶ backend   req 3 ──▶ close

Keepalive (pooled, reused)
  worker ══[idle pool]══▶ backend
           req 1 → req 2 → req 3 ... (one handshake)
DEFAULT  HTTP/1.0 + Connection: close (new connection per request)

┌──────────────┐                          ┌──────────────┐
│ Nginx worker │ ─TCP/TLS─ req 1 ─ close ▶│   backend    │
│              │ ─TCP/TLS─ req 2 ─ close ▶│              │
│ TIME_WAIT +  │ ─TCP/TLS─ req 3 ─ close ▶│              │
│ new src port │ ─TCP/TLS─ req N ─ close ▶│              │
└──────────────┘                          └──────────────┘
  one handshake + one ephemeral port + one TIME_WAIT per request

KEEPALIVE  HTTP/1.1, idle connections pooled per worker

┌──────────────┐                          ┌──────────────┐
│ Nginx worker │ ─TCP/TLS once──────────▶ │   backend    │
│ ┌──────────┐ │ ════ conn 1 (reused) ══▶ │              │
│ │idle pool │ │   req 1 → req 2 → req 3  │              │
│ └──────────┘ │ ════ conn 2 (reused) ══▶ │              │
└──────────────┘                          └──────────────┘
  handshake once per connection, then many requests per connection

By default each proxied request opens and closes its own TCP (and TLS) connection, while upstream keepalive lets each worker reuse pooled idle connections for many requests.

How does upstream keepalive work?

The keepalive directive turns on a connection cache that sits between the load-balancing method and the network. Each worker process has its own cache for each upstream group. Workers do not share idle connections with each other.

A request goes through these steps:

  1. Peer selection. The configured balancer (round robin by default, or least_conn, hash, ip_hash) picks a backend server for the request, exactly as it would without keepalive.
  2. Cache lookup. The keepalive layer then looks in the worker's cache for an idle connection to that same server address. If it finds one, Nginx sends the request over it and skips the TCP and TLS handshakes. If not, Nginx opens a new connection as usual.
  3. Response handling. Nginx reads the full response from the backend. A connection can only be reused if the exchange left it in a clean state: HTTP/1.1 was used, neither side asked to close it, and the response body was read to the end.
  4. Return to the cache. If those conditions hold, the connection goes back into the worker's idle cache instead of being closed. If the cache already holds keepalive N idle connections, the least recently used one is closed to make room.
  5. Idle housekeeping. While a connection sits in the cache, Nginx watches it. If the backend closes it during that time, Nginx notices and drops it from the cache. If it stays idle longer than the upstream keepalive_timeout, or has served keepalive_requests requests, Nginx retires it.

Two consequences follow from this design. First, the balancer still decides which server gets each request, so keepalive only changes how the request gets there, not where it goes. Second, step 5 cannot catch a close that is still in flight when Nginx reuses the connection. That gap causes the 502s covered later in this article.

How do you enable nginx upstream keepalive correctly?

Turning on reuse takes three directives in two places. Each one fixes a different default, so all three must be present.

  1. keepalive N in the upstream block. This creates a per-worker cache of idle connections. Without it, Nginx has nowhere to park a connection after a response completes, so it closes it.
  2. proxy_http_version 1.1 in the location. Nginx speaks HTTP/1.0 to upstreams by default. HTTP/1.0 does not keep connections open unless both sides negotiate it, so the backend closes after each response.
  3. proxy_set_header Connection "" in the location. By default, Nginx sends Connection: close upstream. Clearing the header removes that explicit request to tear down the connection.
Missing piece What happens Logged?
keepalive N Nginx closes the connection after each response No
proxy_http_version 1.1 HTTP/1.0 semantics; the backend closes after each response No
Connection "" Connection: close sent; the backend closes No

In every case, traffic still flows and returns 200s. The only symptom is the extra handshake latency you were trying to remove.

Ordering matters. If you use least_conn, hash, or ip_hash, place that directive above keepalive in the upstream block. The keepalive cache wraps whichever balancer is active when it is declared.

Don't hardcode the header if you proxy WebSockets. Use a map on $http_upgrade that outputs upgrade by default and an empty string when $http_upgrade is empty. Then set Connection $connection_upgrade along with Upgrade $http_upgrade. Upgrade requests get Connection: upgrade, and ordinary requests get an empty value, so they stay eligible for the pool.

http {
    # WebSocket upgrades get "Connection: upgrade"; all other requests
    # get an empty Connection header so they can reuse pooled connections.
    map $http_upgrade $connection_upgrade {
        default upgrade;
        ''      "";
    }

    upstream app_backend {
        # Load-balancing method goes first, before keepalive.
        least_conn;

        server 10.0.0.11:8080;
        server 10.0.0.12:8080;

        # Max idle connections cached per worker process.
        keepalive 32;
    }

    server {
        listen 80;
        server_name app.example.internal;

        location / {
            proxy_pass

            # Default upstream protocol is HTTP/1.0, which closes after each response.
            proxy_http_version 1.1;

            # Replaces the default "Connection: close" sent upstream.
            proxy_set_header Connection $connection_upgrade;
            proxy_set_header Upgrade    $http_upgrade;
            proxy_set_header Host       $host;
        }
    }
}

A minimal upstream keepalive configuration: least_conn is declared before keepalive, the location uses HTTP/1.1, and a map on $http_upgrade sends Connection: upgrade for WebSocket requests and an empty Connection header for all other requests.

How many keepalive connections should you configure?

The keepalive value is the maximum number of idle connections that each worker process caches for an upstream group. It does not cap total connections to the backend. Under load, Nginx opens as many connections as concurrent requests require. When a request finishes and the worker's idle cache is already full, the least recently used connection is closed.

Illustrative estimate

All inputs are assumed:

  • 3 Nginx instances, each with worker_processes 4
  • keepalive 32 on an upstream group of 4 backend servers
  • Each backend runs a thread-per-connection server with a pool of 64 threads

Idle connections across the fleet can reach 3 × 4 × 32 = 384. If they spread evenly, that is about 96 per backend, which already exceeds the 64-thread pool before any traffic arrives. Idle sockets would hold every thread, and new connections would queue or be refused.

Trade-offs

  • Too high: Idle connections consume file descriptors and memory on both sides. Beyond the number needed for steady-state concurrency, they add no latency benefit.
  • Backend limits: Backends with small connection or thread pools can be starved by connections that are doing nothing.
  • Too low: Under bursty load, connections opened beyond the idle limit are closed after one use. You pay TCP (and TLS) setup again on the next burst, which is the cost keepalive was meant to remove.

Start from the per-worker concurrency you actually observe, multiply it out across workers and instances, and check the total against each backend's limits. Then adjust based on measured new-connection rates rather than a number copied from another config.

Why does upstream keepalive cause 502 errors, and how do you fix them?

The failure is a race on an idle pooled connection. Nginx holds a connection open after a response. The backend's own keep-alive timer expires and it starts closing the socket. In that same window, Nginx picks the connection from its pool and writes a new request. The request arrives at a closed socket, and Nginx logs upstream prematurely closed connection while reading response header from upstream or recv() failed (104: Connection reset by peer), then returns a 502.

The error is intermittent because it only happens when a connection sits idle for roughly the backend's timeout. Under steady load, connections are reused long before that. During quiet overnight traffic, they sit idle long enough to hit the race.

The core rule is that Nginx must always close first. Set the backend's keep-alive timeout longer than Nginx's upstream idle timeout. Builds that support them provide two upstream-context directives for this. keepalive_timeout caps idle time, and keepalive_requests caps how many requests one connection serves before Nginx retires it. Every middlebox in the path also drops idle flows silently, including cloud load balancers, NAT gateways and firewalls. Nginx's timeout must be shorter than all of them.

Layer (Nginx → backend) Timeout rule Illustrative values
Nginx upstream keepalive_timeout Shortest 50s
Load balancer / NAT / firewall idle timeout Longer than Nginx 60s+
Backend server keep-alive timeout Longer than Nginx and every middlebox 75s

The values in the table are an illustrative example, not measured recommendations. Raising the backend's timeout above a middlebox does no harm, because the middlebox never sees the connection go idle for that long.

proxy_next_upstream retries can hide the race for idempotent requests. POST, PATCH and other non-idempotent methods are not retried unless you add non_idempotent, so those requests reach clients as 502s.

upstream app_backend {
    server 10.0.0.11:8080;
    server 10.0.0.12:8080;

    # Idle connections cached per worker process.
    keepalive 32;

    # Nginx must close idle connections first: keep this shorter than
    # every LB/NAT/firewall idle timeout and the backend's keep-alive
    # timeout (e.g. backend at 75s in this illustrative setup).
    keepalive_timeout 50s;

    # Retire a connection after this many requests.
    keepalive_requests 1000;
}

server {
    listen 80;

    location / {
        proxy_pass

        # Required for upstream keepalive: HTTP/1.1 and no "Connection: close".
        proxy_http_version 1.1;
        proxy_set_header Connection "";

        # Retry on another server if the connection fails before a response.
        # Nginx does not retry non-idempotent requests (POST, PATCH, LOCK)
        # on another server once the request was sent, so they surface as 502s.
        proxy_next_upstream error timeout;
        proxy_next_upstream_tries 2;

        # Only add non_idempotent if the backend safely handles duplicate POSTs:
        # proxy_next_upstream error timeout non_idempotent;
    }
}

An Nginx upstream that keeps its idle timeout below the backend and middlebox timeouts, so Nginx closes pooled connections first, with retries limited to idempotent requests by default.

How can you verify that Nginx is reusing upstream connections?

Don't trust the config file to tell you reuse is happening. A typo in the Connection header handling or a missing proxy_http_version 1.1 line still produces valid config that silently opens a new connection per request. Measure before the change, apply it, and measure again.

Log connection timing separately

Add $upstream_connect_time and $upstream_response_time to a dedicated log_format, alongside $upstream_addr and $request_time. A fresh connection shows a connect time that reflects the TCP handshake, plus TLS if you proxy over HTTPS. A reused connection shows zero or very close to it, because no handshake happened. After enabling keepalive, most log lines for a busy upstream should show near-zero connect time. If they don't, reuse isn't working.

The two variables also split latency into two parts. When a request is slow, you can see whether the time went into setting up the connection or into the backend processing the request. That tells you which team or host to look at, without packet captures.

Count sockets with ss

On the Nginx host, ss -tn state established dst <backend-ip> lists open connections to a backend. With keepalive working, this count stays roughly stable under steady load. Without keepalive, it churns as connections open and close.

Then compare ss -tn state time-wait counts before and after the change. TIME_WAIT sockets pile up on whichever side closes the connection first, so check both the proxy and the backend. A large drop is strong evidence that connections are being reused instead of torn down.

#!/usr/bin/env bash
set -euo pipefail

BACKEND="${1:-10.0.1.20}"
LOG="${2:-/var/log/nginx/upstream_timing.log}"
SAMPLE="${3:-5000}"

echo "established to $BACKEND: $(ss -Htn state established dst "$BACKEND" | wc -l)"
echo "time-wait to $BACKEND:   $(ss -Htn state time-wait dst "$BACKEND" | wc -l)"

tail -n "$SAMPLE" "$LOG" | awk '
  match($0, /uct=[^ ]+/) {
    v = substr($0, RSTART + 4, RLENGTH - 4)
    split(v, parts, ",")    # retries log a list; keep the first attempt
    v = parts[1]
    if (v == "-") next       # no upstream connection for this request
    total++
    if (v + 0 < 0.001) reused++
  }
  END {
    if (total == 0) { print "no upstream lines found"; exit }
    printf "near-zero connect time: %d/%d (%.1f%%)\n", reused, total, 100 * reused / total
  }'

A before-and-after check that counts established and TIME_WAIT sockets to a backend with ss, then reports the share of logged requests whose $upstream_connect_time is near zero, which indicates a reused connection.

Who should use upstream keepalive?

Upstream keepalive is aimed at teams that run Nginx as a reverse proxy, API gateway or ingress layer in front of HTTP application servers, and who own the configuration of both the proxy and the backends' timeouts. In practice that means platform, SRE and infrastructure engineers, plus application teams that ship their own Nginx config alongside a service.

It is most worth the effort if one or more of these apply to you:

  • You proxy to backends over TLS. Every avoided handshake saves CPU on both sides as well as round trips.
  • Your backends are not on the same host. Cross-zone, cross-region, or otherwise higher-latency links make each saved round trip more valuable.
  • You serve a sustained, high request rate to a small set of backends. This is where ephemeral port exhaustion and TIME_WAIT buildup appear, and reuse removes most of the new connections that cause them.
  • You care about tail latency. Connection setup adds to every request that lands on a fresh connection, which shows up in high percentiles.

It is a lower priority for a low-traffic site proxying to an app on the same machine over plain HTTP. It also needs extra care if you cannot control or even see the idle timeouts of the backends and middleboxes in the path. The next section covers those trade-offs in more detail.

When should you use upstream keepalive, and when should you avoid it?

Keepalive saves the cost of opening a connection, so the benefit scales with how expensive that setup is and how often it happens. A TLS upstream pays for a TCP handshake plus a TLS handshake on every new connection. Each handshake round trip costs more when the backend sits in another zone or region. At high request rates those costs add up to real CPU on both sides and measurable tail latency. To a backend on the same host over plain HTTP, setup takes a fraction of a millisecond, and at low volume the gain is hard to measure.

Keepalive hides connection overhead. It does not fix a slow backend. If a request spends 300 ms in application code and 2 ms on connection setup, reuse saves 2 ms.

Scenario Expected benefit Main risk
TLS to backends, high request rate High Few; size the pool to concurrency
Cross-zone or higher-RTT backends High Stale connections after backend changes
Plain HTTP, same host, low volume Low Little to gain for the added config
Behind an L4 load balancer Moderate Connections pinned to whichever backends they first reached
Hostnames in the upstream block Moderate Traffic to stale IPs until reload
Backends with tight connection limits Varies Idle pooled connections consume backend slots

Be careful in three situations:

  • Scale-out. Long-lived connections slow rebalancing, so new backends stay underused. It is worse behind an L4 load balancer, where Nginx sees a single address and cannot spread the load.
  • Hostnames. Hostnames in the upstream block are resolved at config load, so persistent connections can keep reaching old IPs after DNS changes until you reload.
  • Connection limits. On backends with strict connection caps, idle connections held by every Nginx worker can exhaust the backend's slots.

How can upstream keepalive save engineering time?

The direct gains are latency and CPU. The engineering time saved comes from the problems that stop happening, plus the problems that become easier to diagnose.

  • Fewer connection-exhaustion incidents. Ephemeral port exhaustion and TIME_WAIT buildup tend to appear only at peak traffic, and they produce confusing connect failures. With most requests running over reused connections, the rate of new connections falls, so these incidents become much less likely. Engineers spend less time tuning kernel port ranges and TIME_WAIT settings to work around a problem the proxy creates.
  • Faster latency triage. Logging $upstream_connect_time next to $upstream_response_time shows at a glance whether a slow request was spent on connection setup or inside the backend. That points an investigation at the network path or the application without packet captures or guesswork.
  • Less capacity work for handshake CPU. On TLS upstreams, CPU that would go to repeated handshakes stays available for real work. That can postpone scaling decisions that would otherwise need planning and rollout time.
  • Predictable 502 debugging. Once the "Nginx closes first" timeout rule is written down and checked, the classic intermittent 502 on idle connections has a known cause and a known fix. Without that rule, the same symptom can take days to trace because it is hard to reproduce.
  • A reusable template. The three required directives, the WebSocket-safe map, and the timeout ordering form a single pattern. Teams can copy it into every new upstream instead of rediscovering it service by service.

The work involved is mostly up front: aligning timeouts across layers and adding the verification checks shown earlier. The next section includes an illustrative estimate of how that work compares with the time saved.

What are the alternatives to Nginx upstream keepalive, and what does it cost to adopt?

Each option removes a different part of the per-request connection cost, and each fails in its own way.

Option Latency benefit Complexity Typical failure modes
Per-request connections (default) None. Every request pays a TCP handshake, plus TLS if the upstream is encrypted Lowest Ephemeral port exhaustion and TIME_WAIT buildup under high request rates
Upstream keepalive Removes handshakes for reused connections Low config effort, but timeouts must be coordinated Intermittent 502s when the backend closes an idle connection first
Unix socket to a same-host backend Avoids the network stack, but there is still a connect per request unless you also enable keepalive Low, but it ties Nginx and the backend to one host Socket permission errors, a stale socket file after restarts, no horizontal scaling
gRPC/HTTP/2 proxying Persistent connections that match the backend protocol Medium. The backend must support the protocol The same idle-timeout races, plus protocol-level errors that are harder to read in logs

Enabling keepalive takes only a few lines of config. The real cost is coordination. The backend's idle timeout must be longer than Nginx's upstream keepalive timeout. Every layer in between needs the same rule, including internal load balancers and service proxies, and those layers are often owned by different teams. Write that rule down and add checks for 502 rate and upstream connection counts before you roll out. These two steps are the cheapest part of the work.

Illustrative estimate (assumed numbers, not a benchmark):

Time saved = (incidents avoided × engineer-hours per incident) − upfront hours

Assumed inputs:

  • 3 intermittent-502 investigations avoided per year
  • 6 engineer-hours per investigation
  • 4 hours to document timeout alignment and add the checks

Result: (3 × 6) − 4 = 14 engineer-hours saved per year.

Intermittent 502s are hard to reproduce, so the hours spent per incident tend to exceed what anyone plans for.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to nginx upstream keepalive, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.