Redis Streams vs Pub/Sub for Background Events: Delivery and Replay

What Pub/Sub and Streams actually are

Redis has two primitives that both look like "send an event to workers," but they make opposite promises about the message's lifetime.

Pub/Sub is fire-and-forget fan-out with no storage. PUBLISH pushes a message to every client subscribed to that channel at that instant, and then the message is gone. A subscriber that is restarting, partitioned, or not yet connected never sees it. The integer PUBLISH returns is the number of clients the message was handed to. It does not confirm that any of them processed it.

Streams are an append-only log stored as a Redis key. XADD appends an entry and assigns it an ID such as 1700000000000-0: a millisecond timestamp plus a sequence number for entries within the same millisecond. IDs are strictly increasing, so they define a total order you can read from any point. Entries stay until you trim them, either explicitly or with a length or ID cap. Whether they survive a server restart depends on your RDB/AOF persistence settings.

The real question is whether a message must outlive the moment it was published. For background events like sending an email, charging a card, or rebuilding a cache, the answer is almost always yes. Workers deploy, crash, and scale to zero, and the work still has to happen. Pub/Sub cannot cover those gaps. A stream can, because the event is still there when a consumer returns.

How delivery works: at-most-once vs at-least-once

Pub/Sub is fire-and-forget. PUBLISH hands a message to whichever subscribers are connected at that moment, and Redis keeps no copy. A worker that is restarting, mid-deploy, or not yet subscribed never sees the message. Slow consumers are also at risk: if a subscriber's output buffer exceeds the client-output-buffer-limit set for the pubsub class, Redis disconnects it, and everything buffered for that client is discarded. This is at-most-once delivery.

Streams store entries, so delivery depends on how you read them. With plain XREAD, every reader sees every entry, and each reader tracks its own last-seen ID. That gives independent fan-out, but Redis keeps no record of what each reader has processed.

Consumer groups add that record. Within one group, each entry goes to only one consumer, which distributes the work. Across groups, every group receives every entry.

Producer → XADD → stream: orders

• group: billing → consumers b1, b2 → PEL (delivered, not yet acked)
• group: email → consumer e1 → PEL (delivered, not yet acked)

XREADGROUP with the ID > returns entries that have never been delivered to the group and records each one in the group's Pending Entries List (PEL) under the consumer that received it. XACK removes an entry from the PEL. If a consumer crashes before acking, it can restart and call XREADGROUP with ID 0, which returns its own pending entries so it can finish them.

An entry can therefore be processed more than once but is not silently lost. That is at-least-once delivery, so handlers must be idempotent.

Copy+----------+   XADD    +----------------------------------------+
| Producer |---------->| stream: orders                         |
+----------+           |  1-0  2-0  3-0  4-0  ...  (persisted)  |
                       +----------------------------------------+
                          |                               |
             XREADGROUP > |                  XREADGROUP > |
                          v                               v
+-------------------------------------+  +---------------------------------+
| group: billing                      |  | group: email                    |
| one consumer per entry              |  | one consumer per entry          |
|                                     |  |                                 |
|   +----+      +----+                |  |   +----+                        |
|   | b1 |      | b2 |                |  |   | e1 |                        |
|   +----+      +----+                |  |   +----+                        |
|     |           |                   |  |     |                           |
|     v           v                   |  |     v                           |
|  +-------------------------------+  |  |  +---------------------------+  |
|  | PEL: delivered, not acked     |  |  |  | PEL: delivered, not acked |  |
|  |  1-0 b1   2-0 b2   3-0 b1     |  |  |  |  1-0 e1   2-0 e1          |  |
|  +-------------------------------+  |  |  +---------------------------+  |
|   XACK: remove from PEL             |  |   XACK: remove from PEL         |
|   XREADGROUP 0: own pending         |  |   XREADGROUP 0: own pending     |
+-------------------------------------+  +---------------------------------+

Each consumer group receives every entry in the stream and assigns each entry to one of its consumers. Delivered entries stay in that group's PEL until XACK removes them, and a restarted consumer can call XREADGROUP with ID 0 to resume its own pending work, which gives at-least-once delivery.

Implementing a stream-based worker with consumer groups

The worker has to be built around that guarantee rather than around the hope of exactly-once.

Setup and the main loop

Create the group with XGROUP CREATE events workers $ MKSTREAM. MKSTREAM creates the stream if it does not exist yet, so deploy order doesn't matter. $ means the group starts at new entries only. Producers append with XADD events MAXLEN ~ 100000 * .... The ~ lets Redis trim at efficient node boundaries instead of to the exact count, which keeps XADD cheap.

On startup, each consumer calls XREADGROUP with ID 0 until it gets an empty result. This drains entries it was handed before a restart. It then switches to a loop of XREADGROUP GROUP workers <name> COUNT 10 BLOCK 5000 STREAMS events >. Here > requests entries never delivered to anyone. Call XACK only after the side effects commit. Acknowledging earlier turns a crash into silent loss.

Recovering from dead consumers

If a consumer dies for good, nobody reads its pending entries with 0. A periodic job runs XPENDING in its extended form, which reports idle time and delivery count per entry. It then calls XCLAIM with a minimum idle time to take over stale entries. On servers that support it, XAUTOCLAIM scans and claims in one call. Claiming increments the delivery count. Once the count passes a threshold such as 5, XADD the entry to events:dead and XACK the original. This stops a poison message from cycling forever.

Duplicates are normal

Claims and retries guarantee repeats. Make handlers idempotent. For non-idempotent effects, use a dedup key such as SET processed:<id> 1 NX EX 86400 and skip the work when it already exists.

Copy# One-time setup: '$' = only new entries; MKSTREAM creates the stream if missing
redis-cli XGROUP CREATE events workers '$' MKSTREAM

# Producer: '~' trims approximately at node boundaries, keeping XADD cheap
redis-cli XADD events MAXLEN '~' 100000 '*' type order.created order_id 4821

# Consumer startup: ID 0 re-reads this consumer's own pending entries.
# Repeat until the reply is empty, then switch to '>'.
redis-cli XREADGROUP GROUP workers worker-1 COUNT 10 STREAMS events 0

# Main loop: '>' = entries never delivered to any consumer; block up to 5s
redis-cli XREADGROUP GROUP workers worker-1 COUNT 10 BLOCK 5000 STREAMS events '>'

# Dedup guard before a non-idempotent side effect: OK = first time, nil = skip
redis-cli SET processed:1730275200000-0 1 NX EX 86400

# Acknowledge only after side effects have committed
redis-cli XACK events workers 1730275200000-0

# Recovery job: extended XPENDING lists id, owner, idle ms, delivery count
redis-cli XPENDING events workers - + 10

# Take over an entry idle for at least 60s (this increments its delivery count)
redis-cli XCLAIM events workers worker-2 60000 1730275200000-0

# Redis 6.2+: scan and claim stale entries in one call, starting from 0-0
redis-cli XAUTOCLAIM events workers worker-2 60000 0-0 COUNT 10

# Delivery count past threshold (e.g. 5): copy fields to the dead-letter stream,
# then ack the original so it stops cycling
redis-cli XRANGE events 1730275200000-0 1730275200000-0
redis-cli XADD events:dead '*' orig_id 1730275200000-0 type order.created order_id 4821
redis-cli XACK events workers 1730275200000-0

redis-cli walkthrough of a consumer-group worker lifecycle: setup, trimmed appends, pending drain, blocking reads, dedup, ack, claiming stale entries, and dead-lettering.

Replay, retention and durability limits

Three ways to replay

Stream entries stay in the stream after they are acknowledged, so you can re-read history without republishing anything. Choose the method based on who needs the history:

• Ad hoc reads. XRANGE or XREAD from an earlier ID read history without touching any group state. Use these for debugging or one-off scripts.
• Rewind an existing group. XGROUP SETID moves the group's last-delivered ID back, and its consumers receive those entries again. Your handlers must be idempotent for this.
• Backfill a new projection. Create a new group at ID 0. It processes the full retained history and leaves other groups alone.

Replay stops at retention

You can only replay what has not been trimmed. MAXLEN caps the stream by entry count. MINID drops entries older than a given ID, and because IDs start with a millisecond timestamp, this works as time-based retention. The ~ form trims only whole internal nodes, which makes it much cheaper but means the stream may hold more entries than the limit.

Trimming does not check consumer groups. If a group is lagging, the entries it has not read or acknowledged can be deleted before it gets to them. Size retention for the worst lag you expect, such as a consumer outage over a weekend, not for normal throughput.

Durability belongs to the server

A stream is no more durable than your persistence and replication settings. Replication is asynchronous, so a failover can lose an XADD that the client already saw succeed. A stream also cannot commit atomically with your database write. If you need both to happen together, use a transactional outbox and treat the stream as the delivery path.

Copy#!/usr/bin/env bash
# Replay and retention for one stream, using redis-cli.
set -euo pipefail
R="redis-cli -h redis.internal -p 6379"
STREAM=orders:events

# 1) Ad hoc read: history from an earlier ID. No group state changes.
$R XRANGE "$STREAM" 1761782400000-0 + COUNT 100
$R XREAD COUNT 100 STREAMS "$STREAM" 1761782400000-0

# 2) Rewind an existing group. Its consumers get these entries again,
#    so the handlers must be idempotent.
$R XGROUP SETID "$STREAM" billing 1761782400000-0

# 3) Backfill a new projection. A new group at ID 0 reads everything
#    still retained and does not affect other groups.
$R XGROUP CREATE "$STREAM" search-index 0

# Retention by count. The ~ form trims whole internal nodes only,
# so it is cheaper, but the stream may stay above the limit.
$R XADD "$STREAM" MAXLEN '~' 1000000 '*' type order.created id 42

# Retention by time. IDs begin with a millisecond timestamp, so this
# keeps roughly the last 7 days.
CUTOFF_MS=$(( ($(date +%s) - 7*24*3600) * 1000 ))

# Trimming ignores consumer groups. Before trimming, look at each group's
# last-delivered-id and its pending (unacked) entries. A group behind
# CUTOFF_MS would lose entries it has not processed.
$R XINFO GROUPS "$STREAM"
$R XPENDING "$STREAM" billing

$R XTRIM "$STREAM" MINID '~' "${CUTOFF_MS}-0"

redis-cli commands that replay a stream in three ways (XRANGE/XREAD, XGROUP SETID, a new group at ID 0) and trim it with approximate MAXLEN and MINID after checking for lagging groups.

When to use each, and when to avoid them

Pick Pub/Sub when a lost message costs nothing, or when the receiver can recover the state some other way. Cache invalidation hints, live UI notifications and presence all fit. A missed hint only means a cache entry lives until its TTL expires. A missed presence update is corrected by the next heartbeat. In these cases a dropped message is cheap, and the low latency is worth having. Do not use Pub/Sub for jobs or domain events. A subscriber that is restarting during a publish never receives that message, and nothing records that it was lost.

Pick Streams for background jobs, domain events, retries and audit trails. They are also the choice when a consumer must be able to rewind. Each of these needs a record that outlives the reader.

Streams still have limits:

• Long history. Redis holds the data in memory, so indefinite event-sourcing history and long replay windows quickly become a memory-sizing problem.
• Ordering. Strict per-entity ordering breaks once several consumers in a group process entries in parallel, unless you partition by entity.
• Hot keys in cluster mode. One stream key lives on one shard. A single hot stream cannot scale out across the cluster.

A common hybrid uses both. The stream holds the work, and a Pub/Sub message wakes idle workers so they do not have to poll. If a wake-up is lost, the work is only delayed, because the entry is still in the stream.

In cluster mode, check how Pub/Sub messages actually propagate in your deployment before you rely on them. Whether a channel is broadcast to every node or confined to one shard affects both fan-out cost and which clients receive the message.

Alternatives beyond Redis

Streams stop being the right tool when one of four pressures outgrows what an in-memory structure on a single primary can give you: how far back you need to replay, how strict ordering must be, how much data loss you can tolerate on failover, and how much traffic lands on a single key. Each pressure points to a different category.

A distributed log makes sense when consumers regularly rebuild state from history or new services need to backfill. It keeps ordering per partition and spreads load across partitions, whereas a stream is one key on one shard.

A message broker suits work queues where each message has its own lifecycle. You avoid writing and maintaining reclaim loops and retry counters yourself.

An outbox solves the dual-write problem. If you commit to the database and then add to a stream, a crash between the two steps loses the event. A relay process reads the outbox table and publishes the events, which gives you at-least-once delivery tied to durable commits. Because the relay publishes onward, you can keep Streams as the transport behind it.

Impact on cost and engineering time

Streams cost more to run. Someone has to watch pending-entries-list (PEL) growth and consumer lag, because a PEL that keeps growing means entries are being delivered but never acknowledged. Someone also has to maintain the claim and dead-letter path. The claim timeout needs deliberate tuning. If it is too short, healthy but slow consumers have their work reclaimed and run twice. If it is too long, work held by a crashed consumer sits idle until the timeout expires. Trimming policy needs an owner as well, because retention is a memory budget.

Size that budget before you set MAXLEN or MINID:

Illustrative estimate (assumed numbers, not a benchmark)

• Formula: memory ≈ event rate × retention seconds × average bytes per entry
• Assumed: 50 events/s, 24-hour retention, 500 bytes per entry including overhead
• 50 × 86,400 × 500 ≈ 2.16 GB

Per-entry overhead depends on field layout and encoding. Measure a sample stream with MEMORY USAGE and use that figure instead of guessing.

Pub/Sub costs almost nothing to build, so its cost shows up somewhere else. Messages are lost silently during subscriber restarts, deploys and network blips. Nothing alerts on this. You usually find out after an incident, when a customer reports a missing email or a reconciliation job finds gaps. Retrofitting catch-up at that point is rarely a patch. You need persistence, acknowledgements and offsets, and those are what Streams already provide. The result is usually a rewrite of producers, consumers and their tests.

For background work where losing a message matters, start on Streams. Building the monitoring and claim logic up front costs less than that later migration, and the rewrite often happens under incident pressure.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to redis streams vs pub sub, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• The dependency graph answers "what does this touch?" in seconds instead of a manual audit.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• You can size a change's blast radius before merging, avoiding surprise cross-team firefighting.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.