Why do GraphQL subscriptions break when you run more than one instance?

A GraphQL subscription is a long-running operation: the client sends a subscription document once, and the server pushes a result each time a matching event occurs. Most servers implement this with a PubSub abstraction. Resolvers subscribe to a topic, and mutations publish to it. The default PubSub in many libraries is in-memory, so a publish only reaches subscribers held by the same process.

With one instance, that limitation is invisible. With three instances behind a load balancer, a mutation handled by node A never reaches a client whose socket lives on node B. Sticky sessions do not fix this. They pin a client's connection to one node, but the event can still originate anywhere: another user's mutation, a background job, or a webhook. Stickiness makes failures intermittent and harder to reproduce. The underlying problem is that event delivery has no shared bus.

The graphql-ws protocol makes the lifecycle explicit. The client opens a WebSocket and sends connection_init, often carrying auth. The server replies connection_ack. The client then sends subscribe with an operation ID, and the server streams next messages for that ID until either side sends complete. ping and pong keep the connection alive through proxies and detect dead peers.

That lifecycle drives most of the operational cost. Resolving a single event is usually cheap. The expensive parts are holding thousands of open sockets, tracking per-connection state, handling reconnect storms, and draining connections during deploys. Any team running horizontally scaled API instances with real-time features has to solve both cross-node delivery and connection management.

// Simulates two API instances in one process to show why an in-memory
// PubSub cannot deliver events across nodes.
import { PubSub } from 'graphql-subscriptions';

// Each instance creates its own PubSub, as each server process would.
const nodeA = new PubSub();
const nodeB = new PubSub();

const TOPIC = 'ORDER_UPDATED';

async function main() {
  // A client's WebSocket is held by node B, so its subscription
  // resolver registers a listener on node B's PubSub only.
  await nodeB.subscribe(TOPIC, (payload) => {
    console.log('client on node B received:', payload);
  });

  // The load balancer routes a mutation to node A, which publishes locally.
  await nodeA.publish(TOPIC, { orderId: 'ord_123', status: 'SHIPPED' });
  // Nothing is printed: node A's PubSub has no subscribers, and node B
  // never sees the event. Sticky sessions do not change this, because
  // the publisher (another user, a job, a webhook) can land on any node.

  // Only a publish on the same node reaches the client.
  await nodeB.publish(TOPIC, { orderId: 'ord_123', status: 'DELIVERED' });
  // Prints: client on node B received: { orderId: 'ord_123', status: 'DELIVERED' }
}

main();

Two separate in-memory PubSub instances stand in for two server nodes, showing that an event published on node A never reaches a subscriber whose socket lives on node B.

How does the Redis Pub/Sub pattern work for GraphQL subscriptions?

The most common fix for cross-node delivery is to keep the subscription logic in your API process and swap the in-memory event bus for Redis. When a client starts a subscription over a WebSocket (typically using the graphql-ws protocol), the instance holding that socket subscribes to one or more Redis channels. Mutations on any instance publish events to Redis, and Redis forwards each message to every instance subscribed to that channel. Each instance then runs the subscription resolver or filter for each local connection and writes the results to its own sockets.

Conceptually, the change is small. Your resolvers already call something like publish(channel, payload) and subscribe(channel) against an in-memory PubSub object. You replace that object with one that has the same two-method interface but passes calls through to a Redis client. Resolver code stays the same. Only the transport behind the abstraction changes.

Where filtering happens depends on how you design channels:

  • Per-topic channels (order_updated) keep the channel count low. Every instance receives every order event, and the per-connection filter discards events the client didn't ask for. Filtering happens on the API instance, after the network hop.
  • Per-entity channels (order_updated:8812) push filtering into Redis. An instance only receives events for entities its clients are watching. The cost is many more channels and more subscribe/unsubscribe calls as clients come and go.

This is the right default because it takes days of work, not months. Most teams already run Redis and know how to operate it. You also add no new service tier to deploy, monitor, or page someone about.

// The two-method interface resolvers already call. Only the implementation changes.
interface PubSub {
  publish(channel: string, payload: unknown): Promise<void>;
  subscribe(channel: string, onEvent: (payload: any) => void): () => void; // returns unsubscribe
}

// Thin adapter you write over your Redis client of choice (not a specific library's API).
interface RedisTransport {
  publish(channel: string, message: string): Promise<void>;
  onChannel(channel: string, handler: (message: string) => void): () => void;
}

class RedisPubSub implements PubSub {
  constructor(private redis: RedisTransport) {}

  async publish(channel: string, payload: unknown) {
    // Redis fans this out to every instance subscribed to the channel.
    await this.redis.publish(channel, JSON.stringify(payload));
  }

  subscribe(channel: string, onEvent: (payload: any) => void) {
    return this.redis.onChannel(channel, (msg) => onEvent(JSON.parse(msg)));
  }
}

// Per-topic: few channels; every instance gets every order event and filters locally.
function watchOrderByTopic(pubsub: PubSub, orderId: string, send: (o: any) => void) {
  return pubsub.subscribe("order_updated", (order) => {
    if (order.id === orderId) send(order); // filtering happens here, after the network hop
  });
}

// Per-entity: Redis routes only matching events; more channels and subscribe/unsubscribe churn.
function watchOrderByEntity(pubsub: PubSub, orderId: string, send: (o: any) => void) {
  return pubsub.subscribe(`order_updated:${orderId}`, send); // no local filter needed
}

// Mutation side, on any instance: publish once, and every instance holding a watcher receives it.
// await pubsub.publish(`order_updated:${order.id}`, order);

A conceptual Redis-backed PubSub that keeps the same publish/subscribe interface as the in-memory version, with per-topic channels that filter on the API instance and per-entity channels that let Redis do the routing.

What are the limits of Redis Pub/Sub for subscriptions?

Redis Pub/Sub is fire-and-forget. A message goes only to connections that are subscribed at the moment it is published, and Redis keeps no copy. If a client's WebSocket drops for ten seconds, or a node restarts during a deploy, every event published in that window is lost. Nothing reports the loss. The client reconnects, resubscribes, and continues with a gap it cannot see. For a typing indicator this is fine. For order status or a balance update it is a correctness bug.

Delivery is also coarse. Each instance that subscribes to a channel receives every message on it, even if that instance currently has no interested clients. The instance then filters the message out after paying for the network transfer and deserialization. On a hot channel with many instances, most of this work is discarded.

A single Redis instance is a shared failure point and a throughput ceiling for every subscription in the system. Clustering does not remove this cleanly. Classic Pub/Sub in Redis Cluster broadcasts each published message to all nodes in the cluster, so adding shards does not add publish capacity. Newer releases offer sharded Pub/Sub, which keeps each channel on a single shard. That spreads the load across the cluster, but you have to design channel names with sharding in mind.

When to move to a log

Switch to Redis Streams or another log-based broker when missed events are unacceptable. A log assigns each message an ID. A client sends its last-seen ID on reconnect, and the server replays everything after it. This adds work:

  • Clients must carry resume tokens, because the subscription protocol does not provide them.
  • You must set retention limits and handle tokens that point past the retained window.
  • Handlers must deduplicate replayed events.

Expect weeks of effort, not days.

How does a dedicated subscription gateway work?

A gateway moves the stateful half of the system into its own tier. Gateway nodes terminate WebSockets, speak the subscription protocol (for example graphql-ws), and track which connection holds which subscription with which arguments. Your GraphQL services become stateless again. When something changes, they send an event, or a payload already resolved against the subscription's selection set, to the gateway. The gateway then routes it to matching connections.

Redis Pub/Sub                       Dedicated gateway

clients                             clients
  |  WebSocket                        |  WebSocket
[API + subs] [API + subs]           [gateway] [gateway]   <- connection state
     \          /                        ^        ^
      [ Redis ]                          | events / payloads
                                    [API] [API] [API]     <- stateless

This split matters because the two halves have different load profiles. Connection count and idle memory drive gateway sizing. Request CPU drives API sizing. Each tier can now scale on its own metric. Deploying business logic no longer drops every open socket. Only gateway releases force reconnects, and those can be rarer and drained slowly.

The costs are real:

  • A new tier to operate. It needs its own monitoring, autoscaling, and security, and it becomes your public authentication boundary for long-lived connections.
  • An extra network hop between the service and the client, which adds latency and another failure point.
  • A versioned contract. The event or payload format between services and gateway becomes an API in its own right. Schema changes must stay compatible across independently deployed tiers.
REDIS PUB/SUB                           DEDICATED GATEWAY

  clients      clients                    clients      clients
     |            |                          |            |
     | WebSocket  |                          | WebSocket  |
     v            v                          v            v
+---------+  +---------+                +---------+  +---------+
| API +   |  | API +   |                | gateway |  | gateway |
| subs +  |  | subs +  |                | conn +  |  | conn +  |
| conns   |  | conns   |                | subs    |  | subs    |
+---------+  +---------+                +---------+  +---------+
   |   ^        |   ^                        ^            ^
pub|   |sub  pub|   |sub                     +-----+------+
   v   |        v   |                              |  events / payloads
+----------------------+                           |  (versioned contract)
|        Redis         |                     +-----+------+-------+
+----------------------+                     |            |       |
                                          +-----+      +-----+ +-----+
deploy API: all sockets drop              | API |      | API | | API |
scale on: CPU + connections               +-----+      +-----+ +-----+

                                        gateways: scale on connections
                                        APIs: stateless, scale on CPU

With Redis Pub/Sub, each API node holds sockets and subscription state and fans events through Redis. With a dedicated gateway, a separate tier holds connection state, and stateless API nodes push events or pre-resolved payloads to it over a versioned contract.

How much does subscription fan-out actually cost?

Illustrative estimate (assumed numbers, not a benchmark)

Assume N = 100,000 connected clients spread across I = 20 instances, E = 50 events per second, and f = 0.1 of clients subscribed to each event.

  • Deliveries per second = E × f × N = 50 × 0.1 × 100,000 = 500,000
  • Wasted broadcasts per second = E × (I − I_interested). If each event has subscribers on only 3 instances, that is 50 × 17 = 850 messages that are received, deserialized and discarded.

Client egress is roughly the same in both designs. Server-side work depends on where the payload is resolved:

Cost per second Resolve per subscriber Publish pre-resolved payload
Resolver executions E × f × N = 500,000 E = 50 (or one per distinct selection set)
Serializations 500,000 50
Downstream lookups (DB, services) Up to 500,000 ~50
Per-user field selection and authorization Natural Hard to express

Per-subscriber resolution scales with the audience size, not the event rate, so a popular topic can overload your data layer.

Authorization and filtering

Re-check permissions at delivery time, not only at subscribe time. Access can be revoked while a socket stays open for hours. Keep the check cheap by using cached claims instead of a fresh lookup for every event.

Filter before fan-out. If you broadcast every event and let each instance discard most of them, the waste grows with E × I. Narrow channels, such as per-tenant or per-resource topics, keep the waste close to zero.

Pre-resolved payloads have a cost. They fix one shape for every recipient, so field-level permissions and client-specific selection sets need multiple payload variants or a redaction step on the edge.

WebSocket scaling fundamentals that apply to both designs

Whichever design you pick, the edge layer that holds the sockets fails in the same ways.

Per-connection cost. Every idle WebSocket still holds kernel socket buffers, a file descriptor, and whatever your runtime allocates per connection: the protocol state, the subscription map, and the read and write buffers. Measure this under load instead of guessing. Raise the process file descriptor limit (ulimit -n or your service manager's equivalent) well above your target connection count. The default limits are sized for request/response traffic.

Timeouts and keepalive. Load balancers drop connections that stay silent past their idle timeout. A subscription with no events looks exactly like a dead one. Set your server's ping interval comfortably below the shortest idle timeout on the path. The graphql-ws protocol's ping/pong messages are enough for this, and they also let you detect half-open clients and reclaim their resources.

Draining. During a deploy, stop accepting new upgrades first. Then close existing sockets in batches over a window, using close code 1001 ("going away"), and exit only after that. If you close everything at once, every client reconnects in the same second.

Reconnect backoff. Clients should retry with exponential backoff and full jitter, picking a random delay between zero and the current cap. Without jitter, clients disconnected together reconnect together, and the retry waves stay synchronized.

Affinity skew. WebSocket connections never rebalance once they are open. When you scale out, the new instances receive only new connections while the old ones stay saturated. Sticky sessions make this worse. Plan for gradual, deliberate connection shedding so load can redistribute.

// --- server: refuse new upgrades, then close sockets in batches on SIGTERM ---
import { createServer } from 'node:http';
import { WebSocketServer, WebSocket } from 'ws';

const server = createServer();
const wss = new WebSocketServer({ noServer: true });
let draining = false;

server.on('upgrade', (req, socket, head) => {
  if (draining) { socket.destroy(); return; } // LB health check should already be failing
  wss.handleUpgrade(req, socket, head, (ws) => wss.emit('connection', ws, req));
});
server.listen(4000);

const DRAIN_WINDOW_MS = 60_000; // keep below the orchestrator's termination grace period
const TICK_MS = 1_000;

process.on('SIGTERM', () => {
  draining = true;
  server.close(); // stop accepting new TCP connections
  // Spread closes evenly across the window so clients don't all reconnect at once
  const perTick = Math.ceil(wss.clients.size / (DRAIN_WINDOW_MS / TICK_MS)) || 1;
  const timer = setInterval(() => {
    if (wss.clients.size === 0) { clearInterval(timer); process.exit(0); }
    [...wss.clients].filter((ws) => ws.readyState === WebSocket.OPEN)
      .slice(0, perTick).forEach((ws) => ws.close(1001, 'server draining'));
  }, TICK_MS);
  setTimeout(() => process.exit(0), DRAIN_WINDOW_MS + 5_000).unref(); // hard stop
});

// --- client: exponential backoff with full jitter (graphql-ws) ---
import { createClient } from 'graphql-ws';

export const client = createClient({
  url: 'wss://api.example.com/graphql',
  retryAttempts: Infinity,
  retryWait: async (retries) => {
    const cap = Math.min(30_000, 1_000 * 2 ** retries); // 1s, 2s, 4s ... capped at 30s
    await new Promise((r) => setTimeout(r, Math.random() * cap)); // full jitter: 0..cap
  },
});

A Node.js server that rejects new WebSocket upgrades and closes existing sockets with code 1001 in batches spread over a drain window, paired with a graphql-ws client that reconnects using capped exponential backoff with full jitter.

How does it work end to end? A side-by-side event walkthrough

Following one event through each design makes the difference concrete. Take a mutation that marks order 8812 as shipped while a client watches that order.

With Redis Pub/Sub:

  1. The client's WebSocket terminates on API instance B. Its subscribe message causes B to run the subscription resolver, which subscribes B to order_updated:8812 (or order_updated) in Redis.
  2. The mutation arrives at API instance A. A writes to the database, then publishes the event to the Redis channel.
  3. Redis forwards the message to every instance subscribed to that channel, including B.
  4. B runs the filter and resolver for each local connection watching that order, applies authorization, serializes the result, and sends a next message on the socket.
  5. If B is restarting or the socket is down at step 3, the event is gone.

With a dedicated gateway:

  1. The client's WebSocket terminates on gateway node G. G records the subscription, its arguments, and the authenticated identity in its connection registry.
  2. The mutation arrives at any stateless API instance. It writes to the database and emits an event, or a pre-resolved payload, to the gateway tier over the agreed contract (directly or through a broker).
  3. G looks up which local connections match, applies delivery-time authorization, and sends next messages.
  4. Whether a missed event can be replayed depends on what sits between the APIs and the gateway: a fire-and-forget bus gives at-most-once delivery, while a log gives replay.

In both designs the protocol the client sees is identical. What moves is where connection state lives and which process does the per-subscriber work.

Who should use Redis Pub/Sub, and who needs a gateway?

Redis Pub/Sub fits teams that:

  • Already run Redis and have someone who can operate it.
  • Run a handful to a few dozen API instances, with connection counts that fit comfortably alongside request traffic on those instances.
  • Push events where an occasional gap is acceptable or recoverable, such as presence, typing indicators, live counters, or notifications that the client also fetches on page load.
  • Deploy at a pace where dropping and re-establishing sockets on each rollout is tolerable once draining and jittered backoff are in place.

Redis Streams or another log-based broker fits teams that:

  • Have the same scale profile as above but cannot lose events, such as order status, financial balances, or workflow state.
  • Can afford the client-side work of carrying resume tokens and deduplicating replayed events.

A dedicated gateway fits teams that:

  • Hold enough long-lived connections that socket memory and connection count, rather than request CPU, dominate API sizing.
  • Deploy business logic frequently and cannot keep disconnecting users on every release.
  • Have several backend services that all need to push to clients, and want one place to own WebSocket termination, authentication of long-lived connections, and keepalive.
  • Have the capacity to run, monitor, and secure an additional stateful tier.

Teams that should not use subscriptions at all are those whose data changes every few seconds or less often, whose updates are one-way, or who have no real need for push. Polling or server-sent events will serve them with less machinery.

What are the alternatives to GraphQL subscriptions over WebSockets?

Subscriptions over WebSockets are one way to get fresh data to clients. The alternatives trade latency and flexibility for operational simplicity.

  • Short polling. The client re-runs a normal query on an interval. It reuses your existing HTTP stack, caching, rate limiting, and auth. The cost is latency equal to the interval and wasted requests when nothing changed. Conditional requests and a lightweight "has anything changed" field reduce that waste.
  • Long polling. The server holds the HTTP request open until there is new data or a timeout, then the client immediately re-requests. It gives lower latency than short polling without WebSockets, but each waiting client still holds a connection, so you inherit some of the same connection-count concerns.
  • Server-sent events (SSE). A one-way stream over a regular HTTP response. It works through most HTTP infrastructure, and the browser's EventSource reconnects automatically and sends a Last-Event-ID header, which gives you a natural hook for resumption if your server keeps a log. GraphQL subscriptions can also be carried over SSE instead of WebSockets. The cross-node delivery problem does not go away: you still need Redis or a broker behind it.
  • Log-based brokers behind your existing subscriptions. Not a client-facing alternative, but an alternative to Redis Pub/Sub as the bus. Redis Streams or another append-only log adds replay from an offset without changing the client protocol.
  • Webhooks. For server-to-server consumers rather than browsers or mobile apps, pushing HTTP callbacks to a registered endpoint avoids holding any long-lived connection on your side. Retries and signature verification become your concern instead.
  • Dedicated gateway tier. The option covered above: keep WebSocket subscriptions but move connection state out of the API.

The table in the final section compares delivery guarantees and overhead for the main options.

How can the right choice save engineering time?

Most of the engineering time spent on subscriptions goes into incidents and workarounds, not into writing resolvers. Picking the design that matches your actual requirements removes whole categories of that work.

  • Starting with Redis Pub/Sub avoids building a tier you may never need. Because it keeps the same publish/subscribe interface as the in-memory PubSub, resolver code does not change. The work is a transport adapter and channel naming, not a new service with its own deploy pipeline, dashboards, and on-call rotation.
  • Keeping the PubSub interface narrow keeps later migrations cheap. If resolvers only call publish and subscribe, moving from Redis Pub/Sub to Streams, or to a gateway that consumes the same events, changes the implementation behind the interface rather than every resolver.
  • Choosing a log only when loss matters avoids unnecessary resume-token work. Replay, retention, and deduplication take real effort. Spending it on typing indicators is waste; skipping it for order status means debugging "the client never saw the update" reports that leave no trace in Redis.
  • Draining and jittered backoff prevent recurring deploy incidents. Reconnect storms show up as CPU and auth spikes on every rollout. Fixing the edge behaviour once, as in the code above, is cheaper than investigating the same spike release after release.
  • A gateway decouples release schedules once socket churn is the bottleneck. When teams delay API deploys to avoid disconnecting users, the cost is coordination time across every team that ships to that service. Moving connection state to a separate tier lets business logic ship without touching sockets.
  • Polling or SSE for low-frequency data removes the connection registry entirely. No socket tier means no idle-timeout tuning, no file descriptor limits, and no reconnect storms to debug.

The common thread is to pay only for the guarantees and scale you need now, while keeping the abstraction that lets you move up a step without rewriting resolvers.

When should you move from Redis Pub/Sub to a gateway, or avoid subscriptions entirely?

Move to a gateway when the socket tier starts causing problems for the rest of the system, not when a diagram suggests it. The concrete signals:

  • Reconnect storms during deploys. Every API rollout drops every socket, and the reconnect wave spikes CPU and auth load.
  • Redis CPU or network saturation on hot channels. One popular topic fans out to every instance, whether or not that instance has subscribers for it.
  • Complaints about missed events. Pub/Sub is fire-and-forget. A client that reconnects mid-burst never sees what it missed.
  • Deploy coupling. You delay API releases because they would disconnect users.

If only the third signal applies, you may not need a gateway. Replacing Pub/Sub with Redis Streams or a log-based broker adds replay from an offset without adding a new tier.

Option Delivery guarantee Operational overhead Development cost Best fit
Redis Pub/Sub At-most-once Low Low Moderate fan-out, loss-tolerant events
Redis Streams / log broker At-least-once with replay Medium Medium Events clients must not miss
Dedicated gateway Depends on its backing broker High High Many long-lived sockets, independent deploys
Server-sent events At-most-once; resumable with event IDs Low Low One-way server-to-client updates
Polling Eventually consistent Lowest Lowest Updates every few seconds or slower

For low-frequency updates, polling or SSE is often simpler and cheaper than subscriptions. Both run over plain HTTP, so they work with your existing caching, load balancers, and auth. Neither needs a connection registry, and nothing breaks during a deploy.

Choose the simplest option that meets the delivery guarantee you actually need. Each step up the table adds on-call surface, extra failure modes, and migration work that someone has to maintain.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to graphql subscriptions scaling, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.