Nginx WebSocket Proxy: Fixing Headers, Timeouts and Load Balancing

How an nginx WebSocket proxy works and why it breaks

A WebSocket session starts as an ordinary HTTP/1.1 GET carrying Upgrade: websocket and Connection: Upgrade. If the backend agrees, it answers 101 Switching Protocols, and the same TCP connection then carries bidirectional frames for as long as the session lives. That could be minutes or days. When nginx sits in the middle, it is not passing individual requests. It holds two sockets per session, one facing the client and one facing the upstream, and copies bytes between them. Every open chat tab or live dashboard uses a pair of sockets on an nginx worker.

Teams put nginx in front of WebSocket backends for the same reasons they do for HTTP. It terminates TLS so application servers don't handle certificates, it gives clients a single hostname that can route by path, and it spreads connections across backend instances. This section applies to backend and platform engineers running chat, notifications, live updates or collaborative editing.

Most nginx defaults assume short request/response cycles, and long-lived upgraded connections break that assumption in three predictable places:

1. Hop-by-hop headers are not forwarded. By default nginx does not pass Upgrade and Connection upstream, so the handshake never completes.
2. Idle timeouts close quiet connections. Both nginx and any load balancer in front of it can drop a connection that carries no traffic for a while.
3. Load balancing ignores connection state. Backends that keep per-session state can receive reconnects they cannot serve, and long-lived connections can pile up unevenly across instances.

Copy Client ──TLS──> [ Cloud/L4 LB ] ──> [ nginx worker ] ──> [ upstream A ]
                       │                   │    │      └─> [ upstream B ]
                       │                   │    │
                  (2) LB idle        (1) Upgrade/   (3) balancing
                      timeout            Connection      ignores session
                                         dropped         state / skews load
                                    (2) proxy_read_timeout

Copy +----------+  GET + Upgrade: websocket, Connection: Upgrade
 |  Client  |  <- 101 Switching Protocols, then frames both ways
 +----------+
      |  TLS, one long-lived TCP connection per session
      v
 +-----------------+
 |  Cloud / L4 LB  |<-- [2] LB idle timeout drops quiet connections
 +-----------------+
      |
      v
 +-------------------------------+
 |  nginx worker                 |
 |  +--------+       +--------+  |
 |  | client |<----->|upstream|  |<-- [2] nginx idle timeout closes
 |  | socket |       | socket |  |        sessions with no traffic
 |  +--------+       +--------+  |
 |  one socket pair per session  |
 +-------------------------------+
      |<-- [1] Upgrade / Connection not forwarded by default,
      |        so the handshake never reaches 101
      |<-- [3] balancer picks a backend without session state
      +----------------+
      |                |
      v                v
 +-------------+  +-------------+
 | upstream A  |  | upstream B  |
 | many old    |  | few new     |
 | connections |  | connections |
 +-------------+  +-------------+

A WebSocket session passes from the client through an optional load balancer to an nginx worker, which holds one client socket and one upstream socket for that session; the markers show where dropped hop-by-hop headers [1], idle timeouts [2] and state-unaware load balancing [3] cause failures.

Forwarding the Upgrade and Connection headers

Point (1) in the diagram is the most common failure. Upgrade and Connection are hop-by-hop headers. HTTP/1.1 defines them as applying only to a single connection, so a compliant proxy consumes them and does not forward them. nginx follows this rule. The upstream receives a plain GET, answers with a normal 200 or 400 instead of 101 Switching Protocols, and the client's handshake fails.

The second problem is protocol version. By default nginx talks to upstreams over HTTP/1.0, which has no upgrade mechanism. proxy_http_version 1.1 is therefore required.

Minimal location block

A minimal working location needs three directives:

• proxy_pass to the upstream
• proxy_http_version 1.1
• proxy_set_header Upgrade $http_upgrade, plus proxy_set_header Connection "upgrade"

Serving WebSocket and HTTP from one location

Hardcoding Connection: upgrade breaks ordinary requests that share the same location. The fix is a map in the http context:

• When $http_upgrade is non-empty, $connection_upgrade resolves to upgrade.
• When $http_upgrade is empty, it resolves to close.

Then set proxy_set_header Connection $connection_upgrade. This keeps nginx's default close behaviour for normal HTTP traffic.

If the upstream uses keepalive connection pooling, map the empty case to "" instead of close. Otherwise every non-WebSocket request discards its pooled connection.

Headers the backend needs

Add these headers so the backend sees the original request:

• Host $host: nginx otherwise sends the upstream name. A backend that checks Origin against Host to block cross-site WebSocket hijacking will then reject legitimate clients.
• X-Forwarded-Proto $scheme: this lets the application generate wss:// URLs rather than ws://.
• X-Forwarded-For $proxy_add_x_forwarded_for: this preserves the client IP for logging and rate limiting.

Copy# http context: derive the Connection header from the client's Upgrade header
map $http_upgrade $connection_upgrade {
    default upgrade;   # client asked to upgrade -> forward Connection: upgrade
    ''      close;     # plain HTTP -> keep nginx's default close behaviour
    # with upstream keepalive pooling, use '' instead of close here
}

upstream ws_backend {
    server 10.0.0.11:8080;
    server 10.0.0.12:8080;
}

server {
    listen 443 ssl;
    server_name app.example.com;
    ssl_certificate     /etc/nginx/tls/app.example.com.crt;
    ssl_certificate_key /etc/nginx/tls/app.example.com.key;

    location / {
        proxy_pass

        # HTTP/1.0 (nginx's upstream default) has no upgrade mechanism
        proxy_http_version 1.1;

        # Hop-by-hop headers are not forwarded by default; set them explicitly
        proxy_set_header Upgrade    $http_upgrade;
        proxy_set_header Connection $connection_upgrade;

        # Preserve the original request so the backend can check Origin and build URLs
        proxy_set_header Host              $host;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
    }
}

A single nginx location that proxies both WebSocket upgrades and ordinary HTTP requests, using a map on $http_upgrade to set Connection and forwarding Host, X-Forwarded-For and X-Forwarded-Proto to the backend.

Idle timeouts: proxy_read_timeout, proxy_send_timeout and heartbeats

Both directives default to 60 seconds, and neither limits the total length of a connection. proxy_read_timeout limits the time between two successive reads from the upstream, and proxy_send_timeout limits the time between two successive writes to it. A WebSocket that carries no frames for a minute looks the same to nginx as a stalled backend, so nginx closes it. This is why quiet dashboards or chat tabs disconnect after about a minute even though nothing failed.

There are two ways to fix this. The first is to raise proxy_read_timeout to several hours. This removes the disconnects, but it also stops nginx from noticing connections that are actually dead. A laptop that sleeps or a phone that changes networks often leaves without sending a TCP close, and nginx keeps that socket and its upstream connection open until the timeout expires.

Illustrative estimate (assumed numbers, not a benchmark): lingering dead connections ≈ abandonment rate × timeout. Assume 2,000 abandoned connections per hour and a 6-hour timeout: 2,000 × 6 = up to 12,000 dead connections held at once.

The second is to have the application send ping/pong frames every N seconds. Each ping costs a few bytes, the timer resets whenever a frame passes through nginx, and a dead peer is detected within one or two missed intervals.

Nginx is often not the only hop. Cloud load balancers and corporate proxies in front of it apply their own idle timeouts. Set the heartbeat interval below the smallest idle timeout on the whole path, and keep proxy_read_timeout at a comfortable multiple of that interval.

Copy# Upgrade header mapping (goes in the http {} block).
map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

upstream ws_backend {
    server 10.0.0.11:8080;
    server 10.0.0.12:8080;
}

server {
    listen 80;
    server_name app.example.com;

    location /ws/ {
        proxy_pass
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $host;

        # Assumes the app sends a ping every 25s, which is below the
        # smallest idle timeout on the path (for example a 60s cloud LB).
        # 75s allows about two missed pings before nginx gives up, so
        # dead peers are still released quickly.
        proxy_read_timeout 75s;
        proxy_send_timeout 75s;

        # Avoid this: hours-long timeouts hide dead sockets.
        # proxy_read_timeout 6h;
    }
}

A WebSocket location that keeps proxy_read_timeout and proxy_send_timeout at a small multiple of an assumed 25-second application heartbeat instead of raising them to hours.

Load balancing long-lived connections: round-robin, least_conn and sticky sessions

Once nginx completes the upgrade, the TCP connection to the chosen backend stays open for the life of the socket. Balancing decisions therefore apply only to new connections. If you add a backend, it receives no existing traffic. If one backend restarts, its clients reconnect and spread across the remaining servers, and they stay there. Load stays uneven until clients reconnect for their own reasons.

Round-robin spreads connection attempts evenly, but it ignores how long each connection lives. A backend that has been up for hours accumulates far more open sockets than a fresh one. least_conn sends each new connection to the server with the fewest active ones, so it steers new clients toward an under-filled node. This closes the gap gradually.

Affinity is needed in two cases. The first is a client library that falls back to HTTP long-polling, where successive requests must reach the same process. The second is a server that keeps session state in memory across reconnects. Open-source nginx offers ip_hash, or hash with a key you choose, such as a cookie or a query argument. Add consistent to the hash directive to limit remapping when servers change. ip_hash concentrates load when many users share one NAT address. Any change to the upstream list remaps some clients and breaks their affinity.

Copy# Inside the http {} block.
# Standard WebSocket upgrade handling.
map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

# Pure WebSocket clients with no affinity needs: route each new
# connection to the backend with the fewest active connections.
upstream ws_stateless {
    least_conn;
    server 10.0.1.11:8080;
    server 10.0.1.12:8080;
    server 10.0.1.13:8080;
}

# Clients that fall back to long-polling, or servers that keep session
# state in memory: pin by session cookie. If the cookie is missing, use
# the client IP so those requests don't all hash the same empty key.
map $cookie_sid $affinity_key {
    ''      $remote_addr;
    default $cookie_sid;
}

upstream ws_sticky {
    # 'consistent' (ketama) remaps only some keys when servers change.
    hash $affinity_key consistent;
    server 10.0.1.21:8080;
    server 10.0.1.22:8080;
}

server {
    listen 80;
    location /ws/      { proxy_pass  include ws_proxy.conf; }
    location /socket/  { proxy_pass     include ws_proxy.conf; }
}

# ws_proxy.conf:
#   proxy_http_version 1.1;
#   proxy_set_header Upgrade $http_upgrade;
#   proxy_set_header Connection $connection_upgrade;
#   proxy_read_timeout 3600s;

Two upstreams: one uses least_conn for stateless WebSocket traffic, and one uses a consistent hash on a session cookie, falling back to the client IP, for clients that need affinity.

When to avoid sticky sessions and nginx-level affinity, and what to use instead

If you need ip_hash or a consistent hash just to make reconnects work, the backend is probably holding state that exists in only one process: room membership, unacknowledged messages, a user's subscription list. Affinity doesn't remove that dependency. It hides it until the node restarts, gets scaled in, or fails, and then every client pinned to it loses its context at once.

Make any backend able to serve any reconnect

The durable fix is to move that state out of the process:

• Shared store. Keep session data, cursors and presence in an external key-value store or database. On reconnect, the backend loads the state by session ID instead of expecting it in memory.
• Pub/sub layer. Publish messages to a topic, and have each backend subscribe on behalf of its connected clients. A message for user A reaches A regardless of which node holds the socket.
• Stateless reconnect handshake. The client sends a signed token and its last-seen message ID. The server verifies the token, then replays anything missed from the store. Neither step needs prior knowledge of the client.

The trade is real. You add a broker that must be highly available itself, and a network hop on hot paths. For a small internal tool, accepting dropped state on deploy may be cheaper.

When nginx isn't the right layer

If you don't need HTTP routing or header manipulation, a dedicated L4 TCP proxy forwards bytes with less per-connection overhead. If connection counts are large and spiky, a managed WebSocket gateway can hold the sockets and call your stateless backends over plain HTTP. This removes long-lived connections from your fleet entirely.

Reloads, deploys and capacity planning

Reloads leave old workers running

When you reload nginx, the master process starts new workers with the new configuration and tells the old workers to shut down gracefully. A graceful shutdown means each old worker stops accepting new connections and exits only after its existing connections close. Short HTTP requests drain in seconds. A WebSocket can stay open for hours. Until those sockets close, the old workers keep holding memory, and those clients keep running under the old configuration. Several reloads in a row can leave multiple generations of workers running side by side. worker_shutdown_timeout caps this: once the timeout passes, nginx closes any connections the old workers still hold. That cap also means every reload eventually disconnects the long-lived clients on those workers.

Backend deploys cause reconnect storms

Restarting a backend drops every socket it holds, and those clients all try to reconnect at about the same moment. If clients retry right away or on a fixed interval, they hit the new instances in synchronized waves. Clients should use exponential backoff with random jitter so their reconnects spread out over time.

Size for concurrent sockets

Each proxied WebSocket uses two connections in nginx: one to the client and one to the upstream. Both count against worker_connections and against the file descriptor limit.

Illustrative estimate. Assume 40,000 concurrent clients and 4 workers. That is 80,000 connections in total, or 20,000 per worker if the load spreads evenly. Workers rarely share load evenly, so allow headroom of roughly 1.5×: set worker_connections to about 30,000 and raise the per-worker file descriptor limit (through worker_rlimit_nofile and the OS limit) to at least that number.

Verifying the setup and the impact on engineering time

Start with the handshake itself. Send a request with curl -i -N and the headers Connection: Upgrade, Upgrade: websocket, Sec-WebSocket-Version: 13 and a base64 Sec-WebSocket-Key. A working proxy returns HTTP/1.1 101 Switching Protocols. Any other status means the upgrade failed before the backend saw a WebSocket request. Then connect with a real WebSocket client, stay idle for longer than proxy_read_timeout, and confirm that heartbeats keep the socket open.

To make the logs useful, add $upstream_addr, $upstream_connect_time, $request_time and $http_upgrade to the log_format. nginx writes the access log entry for a WebSocket only when the connection closes, so $request_time is the connection's lifetime. That makes idle cutoffs easy to spot, because many connections will end at almost exactly your timeout value.

Keep a check for each row in a checklist and run it after every config change. This replaces repeated trial-and-error timeout tuning during incidents.

Illustrative estimate (assumed numbers, not a benchmark):

hours saved = incidents × engineers × (ad hoc hours − checklist hours)

Assume 6 incidents a quarter and 2 engineers per incident, with ad hoc debugging taking 4 hours and the checklist taking 1 hour.

6 × 2 × (4 − 1) = 36 engineer-hours saved per quarter.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to nginx websocket proxy, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.