Why do gRPC and WebSockets break behind a default Nginx proxy?
If you already run Nginx at the edge for REST traffic, adding gRPC and WebSocket services to the same listener looks easy. It usually fails in one of three ways: gRPC calls return protocol errors, WebSocket handshakes return a plain 200 or 400 instead of 101 Switching Protocols, or streams that work at first drop after about a minute.
All three come from the same cause. Nginx's default proxying assumes short request/response exchanges. gRPC and WebSockets keep one connection open for minutes or hours.
Default proxy_pass behaviour |
gRPC needs | WebSocket needs | Directive that changes it |
|---|---|---|---|
| Speaks HTTP/1.0 to upstreams | HTTP/2 end to end | HTTP/1.1 | grpc_pass / proxy_http_version |
Drops hop-by-hop headers (Upgrade, Connection) |
Not applicable | Forward both headers | proxy_set_header |
| 60s read/send timeouts | Longer than the gap between stream messages | Longer than the gap between frames | grpc_read_timeout, proxy_read_timeout, and their _send_ equivalents |
These timeouts measure the idle time between two successive reads or writes, not the total length of the connection. A stream that sends a message every 30 seconds survives indefinitely. A stream that goes quiet for 61 seconds is closed, even if it has only been open for two minutes.
This setup suits teams that already terminate TLS in Nginx and serve mixed REST, gRPC, and WebSocket traffic on one port. A single config gives you one place for access logs, timeouts, and TLS settings. When a stream drops, you check a short list of known directives instead of tracing the connection through several proxies.
How does Nginx proxy a gRPC request end to end?
Both protocols arrive on the same TLS listener, and Nginx handles them differently from that point on.
gRPC path. The client opens a TLS connection and negotiates HTTP/2 through ALPN. Nginx terminates TLS and decodes the HTTP/2 frames. It then opens a separate HTTP/2 connection to the backend through grpc_pass. A separate upstream module is needed because proxy_pass only speaks HTTP/1.x to upstreams. gRPC depends on HTTP/2 streams and on trailers, which carry grpc-status, so it cannot use proxy_pass. The upstream scheme controls encryption on the second hop. grpc:// sends plaintext HTTP/2, which suits a trusted private network. grpcs:// re-encrypts the traffic, which you need when the backend expects TLS or mTLS.
WebSocket path. The client sends an ordinary HTTP/1.1 GET with Upgrade: websocket and Connection: Upgrade. These are hop-by-hop headers, so Nginx drops them unless you forward them explicitly. Once the backend answers 101 Switching Protocols, Nginx stops parsing HTTP and relays bytes in both directions until either side closes the connection.
| Hop | gRPC | WebSocket |
|---|---|---|
| Client → Nginx :443 | TLS + HTTP/2 (TLS terminates here) | TLS + HTTP/1.1 GET with Upgrade (TLS terminates here) |
| Inside Nginx | grpc_pass, frames decoded per stream |
proxy_pass, 101 response, then a raw relay |
| Nginx → backend | grpc:// plaintext or grpcs:// TLS |
or |
| Idle timeout applied | grpc_read_timeout / grpc_send_timeout |
proxy_read_timeout / proxy_send_timeout |
In both paths the timeout measures the gap between two successive reads or writes. It does not measure the total lifetime of the connection. That difference decides which settings matter for long-lived streams.
+---------------------------+ +-------------------------------+
| gRPC client | | WebSocket client |
| TLS + HTTP/2 (ALPN h2) | | TLS + HTTP/1.1 GET |
| | | Upgrade: websocket |
+-------------+-------------+ +---------------+---------------+
| |
v v
+--------------------------------------------------------------------------+
| Nginx :443 listener (TLS terminates here for both paths) |
| +---------------------------+ +-------------------------------+ |
| | gRPC location | | WebSocket location | |
| | grpc_pass | | proxy_pass | |
| | HTTP/2 frames decoded | | forward Upgrade + Connection | |
| | per stream, trailers kept | | 101, then raw byte relay | |
| | [T] grpc_read_timeout | | [T] proxy_read_timeout | |
| | grpc_send_timeout | | proxy_send_timeout | |
| +-------------+-------------+ +---------------+---------------+ |
+----------------+--------------------------------------+------------------+
| |
| grpc:// plaintext HTTP/2 | http:// or https://
| grpcs:// TLS re-encrypted | plaintext or TLS
v v
+-----------------+ +-----------------+
| gRPC backend | | WebSocket app |
| HTTP/2 server | | HTTP/1.1 server |
+-----------------+ +-----------------+
[T] = idle timeout: gap between successive reads/writes, not total lifetime
Both protocols terminate TLS on one Nginx :443 listener. gRPC continues to the backend over grpc_pass using grpc:// or grpcs://, and WebSocket continues over proxy_pass with a 101 upgrade followed by a raw relay, and each path applies its own idle timeouts.
How do you configure Nginx for gRPC with grpc_pass?
Start with the listener. gRPC requires HTTP/2, so enable it on the listen socket. Older releases use the http2 parameter on listen, while newer ones use a separate http2 directive, so check which form your build accepts. Browsers and most gRPC clients negotiate HTTP/2 through TLS ALPN, which makes terminating TLS at Nginx the usual setup.
Upstreams and routing
Define an upstream block for the gRPC backends and add keepalive. Without keepalive, Nginx opens a new connection to the backend for each client stream. With it, Nginx reuses idle HTTP/2 connections and avoids repeated handshakes.
Every gRPC method maps to the path /package.Service/Method. A prefix location such as /payments.v1.Ledger/ can therefore route one service to its own upstream through grpc_pass. REST locations can sit in the same server block alongside it.
Metadata, errors and timeouts
grpc_set_header works like proxy_set_header. Use it to forward values like X-Real-IP $remote_addr, which the backend then reads as gRPC metadata.
When the upstream is down, Nginx returns its own 502, 503 or 504 response. A gRPC client sees that as an opaque protocol error. To avoid this, point those codes at an internal location with error_page. That location sets grpc-status (14, UNAVAILABLE, is the usual choice), a grpc-message, and Content-Type: application/grpc. Clients can then handle the failure with their normal retry logic.
Set grpc_read_timeout and grpc_send_timeout for streaming routes. Both default to 60 seconds, which is too short for long-lived streams.
The internal hop
grpc:// sends plaintext HTTP/2 to the backend. grpcs:// re-encrypts the connection, and you configure it with the grpc_ssl_* directives. Use plaintext only when the network segment between Nginx and the backend is trusted.
upstream ledger_grpc {
server 10.0.1.10:50051;
server 10.0.1.11:50051;
keepalive 32; # idle backend connections cached per worker, reused across streams
}
server {
listen 443 ssl;
http2 on; # nginx 1.25.1+; older builds use: listen 443 ssl http2;
server_name api.example.internal;
ssl_certificate /etc/nginx/tls/api.crt;
ssl_certificate_key /etc/nginx/tls/api.key;
# Every method of payments.v1.Ledger arrives as /payments.v1.Ledger/<Method>
location /payments.v1.Ledger/ {
grpc_pass grpc://ledger_grpc; # plaintext h2 internal hop; use grpcs:// to re-encrypt
grpc_set_header X-Real-IP $remote_addr; # backend reads this as metadata x-real-ip
grpc_read_timeout 1h; # default 60s would cut off long-lived streams
grpc_send_timeout 1h;
error_page 502 503 504 = /grpc_unavailable;
}
# REST traffic shares the same server block
location /api/ {
proxy_pass
}
# Turn nginx-generated upstream errors into a gRPC status clients can retry on
location = /grpc_unavailable {
internal;
default_type application/grpc;
add_header grpc-status 14; # UNAVAILABLE
add_header grpc-message "upstream unavailable";
return 204;
}
}
A single TLS server block that routes one gRPC service to a keepalive upstream next to a REST location, forwards the client IP as metadata, extends stream timeouts, and maps 502/503/504 to grpc-status 14.
How do you configure Nginx proxy upgrade for WebSockets?
WebSocket handshakes fail behind a default Nginx config because Upgrade and Connection are hop-by-hop headers. They describe a single connection, not the request, so Nginx drops them instead of forwarding them upstream. The backend sees an ordinary GET and answers with a normal response rather than 101 Switching Protocols. Nginx also speaks HTTP/1.0 to upstreams by default, and HTTP/1.0 has no upgrade mechanism. The fix needs three pieces.
- Set
proxy_http_version 1.1in the location. - Add
proxy_set_header Upgrade $http_upgradeto pass the client's requested protocol through. - Set
Connectionto match the request. Amapblock on$http_upgradeworks well here: it yieldsupgradewhen the client sent anUpgradeheader and something else when it didn't. That second value is eithercloseor an empty string. With an empty string, Nginx omits the header, which lets plain HTTP requests reuse pooled upstream keepalive connections. Because the value depends on each request, one location can serve both regular API calls and WebSocket clients.
Once the upstream returns 101, Nginx stops parsing HTTP and becomes a byte relay. Frames move in both directions without response buffering. proxy_read_timeout and proxy_send_timeout still apply during the relay: each one closes the tunnel if no data moves in its direction for the configured interval. A quiet socket with no application pings will be cut at the default.
Streaming responses that never upgrade, such as server-sent events, behave differently. Nginx buffers them like any other response. Set proxy_buffering off for those paths, or have the upstream send X-Accel-Buffering: no, so events reach clients as they are written.
# http {} context
# Choose the Connection header per request:
# Upgrade sent -> "upgrade" (WebSocket handshake)
# no Upgrade sent -> "" (header omitted, so upstream keepalive works)
map $http_upgrade $connection_upgrade {
default upgrade;
'' '';
}
upstream app_backend {
server 10.0.0.12:8080;
keepalive 32; # idle pooled connections for plain HTTP
}
server {
listen 80;
server_name app.example.com;
# One location serves both normal API calls and WebSocket clients
location / {
proxy_pass
proxy_http_version 1.1; # HTTP/1.0 has no upgrade mechanism
proxy_set_header Host $host;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
# Close the tunnel only after 1h without data in that direction
proxy_read_timeout 3600s;
proxy_send_timeout 3600s;
}
# Server-sent events never upgrade, so turn off response buffering
location /events {
proxy_pass
proxy_http_version 1.1;
proxy_set_header Connection '';
proxy_buffering off;
proxy_read_timeout 3600s;
}
}
A map on $http_upgrade lets one location proxy both plain HTTP with upstream keepalive and WebSocket upgrades with long idle timeouts, while a separate SSE path disables response buffering.
How long should Nginx timeouts be for long-lived streams?
Nginx's 60-second defaults for proxy_read_timeout and grpc_read_timeout measure the gap between two successive reads from the upstream, not the total length of the stream. A WebSocket or server-streaming RPC that stays quiet for a minute gets closed, even though it is healthy. The usual fix of setting these values to hours is a mistake: dead peers then hold resources for hours. A better approach is a timeout of a few minutes plus traffic that resets the timer.
WebSocket ping frames work for this. After the upgrade, Nginx treats the connection as an opaque byte stream, so pings count as reads. gRPC is different. HTTP/2 PING frames apply only to a single hop, so client keepalives keep the client-to-Nginx leg alive but do not reset the upstream read timer. For gRPC, have the server send periodic heartbeat messages on long streams.
Every load balancer, NAT gateway and firewall in front of Nginx has its own idle timeout. The shortest timeout in the path decides when the connection drops, so set your ping interval below the smallest one.
| Directive | Applies to | Default | Recommended approach |
|---|---|---|---|
proxy_read_timeout |
WebSocket upstream reads | 60s | A few minutes, longer than the ping interval |
proxy_send_timeout |
WebSocket upstream writes | 60s | Usually leave as is |
grpc_read_timeout |
gRPC upstream reads | 60s | A few minutes, longer than the heartbeat interval |
grpc_send_timeout |
gRPC upstream writes | 60s | Usually leave as is |
Long-lived streams also hold connections open. Each proxied stream uses one client connection and one upstream connection, and both count against worker_connections and the worker's file-descriptor limit, which you can raise with worker_rlimit_nofile.
Illustrative estimate (assumed numbers, not a benchmark):
worker_connections ≥ (concurrent streams × 2 ÷ workers) × headroom
With 20,000 streams, 4 workers and 1.25× headroom: (20,000 × 2 ÷ 4) × 1.25 = 12,500 per worker. The file-descriptor limit should be at least that high.
# Two proxied legs per stream (client + upstream), so size both limits for that.
worker_rlimit_nofile 65536; # per-worker FD limit; keep above 2x worker_connections
events {
worker_connections 16384; # counts client AND upstream connections
}
http {
map $http_upgrade $connection_upgrade {
default upgrade;
'' close;
}
server {
listen 443 ssl;
http2 on; # needed for gRPC (nginx 1.25.1+ syntax)
ssl_certificate /etc/nginx/tls/example.crt;
ssl_certificate_key /etc/nginx/tls/example.key;
# WebSocket: app sends a ping every 30s, so 5 min of silence means the peer is dead.
location /ws/ {
proxy_pass
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $connection_upgrade;
proxy_read_timeout 300s; # longer than the ping interval, not hours
}
# gRPC: HTTP/2 PINGs are hop-by-hop and do not reset this timer,
# so the server sends a heartbeat message on long streams.
location /events.v1.EventService/ {
grpc_pass grpc://grpc_backend;
grpc_read_timeout 300s; # longer than the server heartbeat interval
}
}
upstream ws_backend { server 10.0.0.11:8080; }
upstream grpc_backend { server 10.0.0.12:50051; }
}
An Nginx config that uses five-minute read timeouts sized against application pings and heartbeats, and raises worker_connections and worker_rlimit_nofile, since each long-lived stream holds a client connection and an upstream connection.
How do you verify and debug the nginx grpc websocket proxy?
Treat verification as a fixed checklist you run after every config change. It is faster than editing snippets and reloading until something works, because each step isolates one layer.
The checklist
- Syntax and includes. Run
nginx -tbefore every reload. It catches typos, missing certificate files, and directives placed in the wrong context. - Protocol negotiation. Run
curl -v --http2 In the verbose output, look for the ALPN line confirmingh2and for an HTTP/2 status line. If you see HTTP/1.1, gRPC clients will fail before Nginx ever reachesgrpc_pass`. - Real gRPC calls. Use a gRPC CLI client or the service's own client. Make one unary call and one streaming call. A unary success with a failing stream usually means a timeout or buffering problem, not a routing one.
- WebSocket handshake. Connect with a WebSocket client and confirm a
101 Switching Protocolsresponse. A 200 or 400 means theUpgradeandConnectionheaders are not reaching the backend.
Reading the error log
Set error_log /var/log/nginx/error.log info; while debugging. At this level Nginx also logs client-side closes, which helps you see who dropped the connection. The common entries mean different things:
- "upstream timed out" means a
*_read_timeoutor*_send_timeoutexpired. Check whether the stream was simply idle. - "Connection reset by peer" or "upstream prematurely closed connection" means the backend closed the socket. Look at backend logs, restarts, or the backend's own idle limits.
- A protocol mismatch appears as a 502 with an invalid-header or premature-close message. A typical cause is sending HTTP/1.1 to an h2-only gRPC backend, for example by using
proxy_passinstead ofgrpc_pass, or plaintext wheregrpcs://is required.
When should you avoid Nginx for gRPC and WebSockets, and what are the alternatives?
Nginx works well as an edge for gRPC and WebSockets, but some limits only appear under sustained load or frequent deploys.
Reloads accumulate workers. A graceful reload starts new workers while old ones keep serving open streams. With hours-long connections, several generations of workers can coexist and hold memory. worker_shutdown_timeout caps this, but only by closing those streams forcibly, so clients must reconnect.
Streams pin to one backend. Load balancing happens per connection. After a scale-out, new backends receive only new connections, and existing streams stay where they were. Load stays uneven until clients reconnect.
Re-encryption costs CPU. Terminating TLS at Nginx and using plaintext upstream is simpler. grpcs:// adds a second handshake and a set of certificates to rotate.
Unbuffered proxying moves backpressure. With buffering off, a slow client stalls the upstream directly.
Nginx does not parse gRPC messages. It has no gRPC-Web translation and no per-method retry policy. Retrying a non-idempotent streaming call that is already in progress can duplicate side effects.
| Requirement | Better fit |
|---|---|
| Even load across replicas for long-lived streams | Client-side load balancing |
| mTLS, retries and traffic policy between services | Service mesh |
| gRPC-Web, per-method routing, retries or timeouts | gRPC-aware proxy layer |
| Simple TLS edge with few deploys | Nginx |
Each alternative adds its own cost. Client-side balancing needs changes in every client. A mesh adds sidecars and a control plane to operate. Choose based on which of these limits you are hitting today.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to nginx grpc websocket proxy, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.