Redis Pub/Sub vs Streams for Service-to-Service Messaging

What Redis Pub/Sub and Redis Streams actually are

Redis ships two messaging primitives that look similar at the API level but are built on different models. One is a broadcast. The other is a log.

Pub/Sub: a broadcast with no memory

PUBLISH sends a message to every client subscribed to a channel at that moment, and then the message is gone. Channels are not keys: nothing is written to the keyspace, so nothing is persisted or replayable. A subscriber that is disconnected, restarting, or slow to connect never sees messages published in the meantime.

The integer PUBLISH returns is easy to misread. It counts how many subscribers the message was delivered to. It does not tell you whether any of them processed it. A return value of 3 means three connections received bytes, and all three might crash before acting on them.

Streams: an append-only log in the keyspace

A stream is a regular Redis key holding an ordered sequence of entries. XADD appends a field-value entry and returns its ID, which by default is a timestamp plus sequence number, so IDs increase monotonically. Because entries are stored, readers choose where to start. XRANGE scans a span of IDs, and XREAD returns entries after a given ID, optionally blocking for new ones. A consumer that was down for ten minutes resumes from the last ID it handled and catches up.

Copy# --- Pub/Sub: broadcast, nothing stored ---
# Terminal 1: a subscriber that is connected right now
redis-cli SUBSCRIBE orders

# Terminal 2: publish while that subscriber is listening
redis-cli PUBLISH orders "order 1001 created"
# (integer) 1   <- one connection received it; says nothing about processing

# Publish to a channel with no subscribers: the message is simply dropped
redis-cli PUBLISH invoices "invoice 77 issued"
# (integer) 0

# Channels are not keys, so there is nothing to read back later
redis-cli EXISTS orders
# (integer) 0

# --- Streams: append-only log stored in the keyspace ---
# Append entries; '*' lets Redis assign a monotonically increasing ID
redis-cli XADD orders:stream '*' order_id 1001 status created
# "1729600000000-0"   (example ID: milliseconds-sequence)
redis-cli XADD orders:stream '*' order_id 1002 status created

# The stream is a real key
redis-cli EXISTS orders:stream
# (integer) 1

# Scan the full range of IDs, oldest to newest
redis-cli XRANGE orders:stream - +

# A consumer coming back after downtime resumes after the last ID it handled
redis-cli XREAD COUNT 100 STREAMS orders:stream 1729600000000-0

# Block up to 5 seconds waiting only for entries added after this call
redis-cli XREAD BLOCK 5000 STREAMS orders:stream '$'

The same order events sent with PUBLISH and with XADD: Pub/Sub returns a receiver count and leaves nothing behind, while the stream stores entries under a key that XRANGE and XREAD can read from any ID.

How consumer groups distribute and track work

A consumer group is a named cursor stored inside the stream. XGROUP CREATE orders billing $ creates the group billing on the orders stream. The $ means the group starts at the current end of the stream. Use 0 instead to process the existing history. Each group tracks a last-delivered ID, so Redis, not your service, records how far the group has read.

Workers call XREADGROUP GROUP billing worker-1 COUNT 10 STREAMS orders >. The > asks for entries never delivered to anyone in this group, and Redis advances the group's cursor as it delivers them. Two workers in the same group therefore receive different entries. This is the competing-consumers pattern. Pub/Sub cannot do this, because every subscriber to a channel receives every message, so adding instances multiplies work instead of dividing it.

Delivery is not completion. Each delivered entry goes into the group's Pending Entries List (PEL), tagged with the consumer name, a delivery timestamp and a delivery count. It stays there until the worker calls XACK orders billing <id>. The PEL is what makes the next section's at-least-once guarantee possible.

For fan-out, give each downstream service its own group. The groups share the same entries but keep separate cursors and separate PELs. A slow analytics service builds its own backlog without delaying billing.

CopyPub/Sub                         Streams
                                          ┌─ group: billing   (cursor, PEL)
PUBLISH ─┬─> sub A (copy)       XADD ─> [stream] ─ group: shipping  (cursor, PEL)
         ├─> sub B (copy)                 └─ group: analytics (cursor, PEL)
         └─> sub C (copy)                        ├─> worker-1  (entries split
   every subscriber gets all                     └─> worker-2   within a group)

Copy# Start clean so the explicit IDs below are valid (demo only; never on a real stream).
redis-cli DEL orders

# One group per downstream service. '$' means each group starts at the current end.
# MKSTREAM creates the empty stream if it does not exist yet.
redis-cli XGROUP CREATE orders billing '$' MKSTREAM
redis-cli XGROUP CREATE orders analytics '$'

# Producer appends three orders. Explicit IDs keep the demo readable; normally use '*'.
redis-cli XADD orders 1700000000000-0 order_id 1001 total 49.90
redis-cli XADD orders 1700000000001-0 order_id 1002 total 12.50
redis-cli XADD orders 1700000000002-0 order_id 1003 total 80.00

# Competing consumers: two billing workers ask for never-delivered entries ('>').
# The group's cursor advances on each read, so they receive different entries.
redis-cli XREADGROUP GROUP billing worker-1 COUNT 2 STREAMS orders '>'   # gets 1001, 1002
redis-cli XREADGROUP GROUP billing worker-2 COUNT 2 STREAMS orders '>'   # gets 1003 only

# Delivered is not done: all three entries now sit in billing's PEL,
# recorded against the consumer that received them.
redis-cli XPENDING orders billing

# worker-1 finishes 1001 and acknowledges it, removing it from the PEL.
redis-cli XACK orders billing 1700000000000-0
redis-cli XPENDING orders billing   # 2 pending left: 1002 (worker-1), 1003 (worker-2)

# Fan-out: analytics has its own cursor and PEL, so it still sees all three entries,
# regardless of what billing has read or acknowledged.
redis-cli XREADGROUP GROUP analytics worker-1 COUNT 10 STREAMS orders '>'

# Each group's last-delivered ID and pending count, tracked by Redis.
redis-cli XINFO GROUPS orders

A redis-cli session showing two billing workers splitting entries within one consumer group, the PEL tracking unacknowledged entries until XACK, and a separate analytics group independently receiving every entry from the same stream.

Delivery semantics: at-most-once vs at-least-once

Pub/Sub delivers a message to whoever is subscribed at the moment of PUBLISH, then forgets it. A subscriber that is disconnected, restarting, or mid-deploy never sees messages published during that gap, and nothing reports the loss. A slow subscriber has a separate failure mode. Redis queues outbound messages in that client's output buffer. Once the buffer exceeds the client-output-buffer-limit configured for the pubsub class, Redis closes the connection, and every message still in the buffer is dropped. This is at-most-once delivery.

Streams with consumer groups take the opposite trade. XREADGROUP records each delivered entry in the pending entries list (PEL) until the consumer calls XACK. If a worker crashes after doing the work but before acknowledging, the entry stays pending. XCLAIM or XAUTOCLAIM later hands it to another worker, which runs the work a second time. This is at-least-once delivery. Neither model gives you exactly-once.

Because duplicates are expected, handlers must be idempotent. A non-idempotent handler simply repeats its side effect on each delivery. It charges the card twice, sends the email twice, or inserts a second row. There are two common ways to deduplicate:

• By entry ID. Record the stream ID with SET key NX and a TTL before acting, and skip the entry if the key already exists. This catches redelivery of the same entry. It does not catch a producer that retried and called XADD twice, because the second entry gets a new ID.
• By business key. Use an idempotency key such as a payment or order ID, enforced by a unique constraint in the system of record. This catches duplicates from both the producer and the consumer side.

Copyimport redis

r = redis.Redis(host="localhost", port=6379, decode_responses=True)
STREAM, GROUP, CONSUMER = "payments", "billing", "worker-1"
DEDUP_TTL = 7 * 24 * 3600  # must outlive the longest redelivery/retry window

def charge_card(order_id: str, amount_cents: int) -> None:
    # Pass order_id to the payment provider as its idempotency key, so two
    # workers racing on the same order still produce a single charge.
    ...

def handle(entry_id: str, fields: dict) -> None:
    order_id = fields["order_id"]
    entry_key = f"seen:entry:{entry_id}"   # catches redelivery/reclaim of this entry
    order_key = f"seen:order:{order_id}"   # catches a producer that XADDed twice
    if r.exists(entry_key) or r.exists(order_key):
        return  # already done; caller still XACKs so the entry leaves the PEL

    charge_card(order_id, int(fields["amount_cents"]))

    # Mark only after the side effect succeeds. Marking first would turn a
    # crash mid-charge into a silent loss on redelivery.
    pipe = r.pipeline()
    pipe.set(entry_key, "1", ex=DEDUP_TTL)
    pipe.set(order_key, entry_id, ex=DEDUP_TTL)
    pipe.execute()

while True:
    resp = r.xreadgroup(GROUP, CONSUMER, {STREAM: ">"}, count=10, block=5000)
    for _stream, entries in resp or []:
        for entry_id, fields in entries:
            handle(entry_id, fields)
            # XACK last: a crash before this line leaves the entry pending,
            # and the dedup markers make the eventual retry a no-op.
            r.xack(STREAM, GROUP, entry_id)

A redis-py consumer-group worker that deduplicates on both the stream entry ID and the order ID. It writes the markers only after the charge succeeds and acknowledges last. The payment provider's idempotency key covers the gap between the check and the mark.

Implementing crash recovery, poison messages and dead-lettering

Crash recovery starts with the pending entries list (PEL). Every entry delivered through XREADGROUP stays there until acknowledged. The extended form of XPENDING shows the owning consumer, idle time and delivery count for each entry, so a stuck message is easy to spot: its idle time keeps growing after its consumer has died.

A recovery loop runs XAUTOCLAIM with a minimum idle time. It moves stale entries to a live consumer, using a cursor to page through the PEL. XCLAIM does the same for specific IDs. Set the idle threshold above your worst-case processing time. If it is too low, two workers will process the same message at once.

Claiming increments the delivery count, which makes that count your poison-message signal. When it passes a limit, XADD the payload and its original ID to a dead-letter stream you create, then XACK the original. Order matters here: if the worker crashes between those two steps, you get a duplicate in the dead-letter stream, not a lost message.

Consumers are not removed automatically. Use XINFO CONSUMERS to find idle ones and remove them with XGROUP DELCONSUMER, but claim their pending entries first. Deleting a consumer discards its PEL entries.

Redis has no delayed delivery or per-message TTL for streams. The usual approach is a sorted set scored by due time, with a poller that moves due items into the stream. For TTL, store an expiry field and check it when reading.

Blocking reads (XREADGROUP BLOCK, SUBSCRIBE) each hold a dedicated connection. Size pools for concurrent blocking readers plus normal traffic. Multiplexing clients and proxies often cannot share those connections, and some proxies do not support them at all.

Copyimport redis

r = redis.Redis(host="localhost", port=6379, decode_responses=True)

STREAM, GROUP, ME = "orders", "billing", "worker-2"
DLQ = "orders:dead"   # dead-letter stream we create ourselves
MIN_IDLE_MS = 60_000   # above worst-case processing time
MAX_DELIVERIES = 5     # beyond this, treat the entry as poison

def recover_once(batch=100):
    # Extended XPENDING: owner, idle time and delivery count per stale entry
    stale = r.xpending_range(STREAM, GROUP, min="-", max="+",
                             count=batch, idle=MIN_IDLE_MS)
    to_claim = []
    for e in stale:
        if e["times_delivered"] < MAX_DELIVERIES:
            to_claim.append(e["message_id"])
            continue
        msg_id = e["message_id"]
        entries = r.xrange(STREAM, min=msg_id, max=msg_id)
        if entries:
            # Copy to the dead-letter stream FIRST, then ack the original.
            # A crash in between yields a duplicate in DLQ, never a loss.
            fields = dict(entries[0][1], original_id=msg_id)
            r.xadd(DLQ, fields)
        r.xack(STREAM, GROUP, msg_id)  # also clears entries trimmed away

    if to_claim:
        # XCLAIM re-checks idle time, so a peer that just claimed wins;
        # it also increments the delivery count for next time
        for msg_id, fields in r.xclaim(STREAM, GROUP, ME,
                                       MIN_IDLE_MS, to_claim):
            handle(msg_id, fields)             # your processing logic
            r.xack(STREAM, GROUP, msg_id)

def reap_dead_consumers(max_idle_ms=3_600_000):
    # Only delete consumers with an empty PEL; deleting discards pending entries
    for c in r.xinfo_consumers(STREAM, GROUP):
        if c["idle"] > max_idle_ms and c["pending"] == 0:
            r.xgroup_delconsumer(STREAM, GROUP, c["name"])

A redis-py recovery pass that uses extended XPENDING delivery counts to divert poison entries to a self-built dead-letter stream before acking, reclaims other stale entries with XCLAIM, and only removes idle consumers whose pending list is already empty.

Durability, retention and trimming limits

Retention is not durability

A stream retains entries while consumers are offline, so a consumer that restarts after an hour can still read what it missed. That is retention. Whether those entries survive a Redis restart or failover is a separate question, and the answer depends on persistence and replication:

• Persistence. With RDB snapshots alone, a crash loses everything written since the last snapshot. With AOF and appendfsync everysec, the exposure is roughly the last second of writes.
• Replication. Replication is asynchronous. A primary can acknowledge an XADD, fail before the replica receives the write, and the promoted replica will not have that entry. WAIT reduces this window but does not make Redis a consensus-backed log.

Trimming

Retention is bounded with MAXLEN (keep the newest N entries) or MINID (drop entries older than an ID, which works as a time cutoff because IDs are timestamp-based). Both are available on XADD and XTRIM.

The ~ modifier allows Redis to trim only whole internal nodes. This is much cheaper, but the stream may stay somewhat above the limit. Exact trimming costs more CPU on every write.

Trimming ignores consumer groups. If a group lags behind the trim point, its unread entries are deleted without any warning. Pending entries whose payloads were trimmed cannot be redelivered with their data. Set retention from worst-case consumer lag, and alert on that lag.

Memory, ordering and scale

All retained entries and every group's pending entries list (PEL) live in RAM. A group that never acknowledges messages grows its PEL without bound.

Ordering is guaranteed only within a single stream. In Redis Cluster, one stream key maps to one hash slot, so one shard handles all of its writes. To scale throughput, partition the data across multiple streams, which means giving up global ordering.

When to use Pub/Sub, when to use Streams, and when to avoid both

Default to Streams whenever a lost message costs something. Order and payment events, background jobs, and audit-style feeds all fit this. If a consumer crashes mid-task, the pending entries list lets another worker claim the message. If a downstream bug corrupts data, you can replay from a known ID instead of reconstructing state from logs. You pay for this with acknowledgement handling, pending-entry monitoring, and a trimming policy. That cost is worth it when the alternative is a silently missing order.

Keep Pub/Sub for ephemeral fan-out where a missed message is harmless or quickly superseded. Good examples are cache invalidation hints backed by a TTL, live UI or websocket notifications, and presence signals. In each case the next message, or a periodic refresh, corrects any gap. Persisting these would only add memory pressure and cleanup work.

Avoid Redis entirely in three situations:

• Strong durability. Replication is asynchronous, so a failover can drop writes the client already saw succeed.
• Global ordering at high throughput. A single stream is pinned to one shard, and partitioning gives up total order.
• Long retention. Every entry consumes RAM.

In these cases, use a replicated, disk-first log or a dedicated message broker.

Impact on development cost and engineering time

Pub/Sub costs little to build. A publisher calls PUBLISH, a subscriber loops on incoming messages, and the work is done in an afternoon. The cost shows up later. When a subscriber disconnects or falls behind, Redis drops the messages and reports nothing to the publisher, which only learns how many subscribers were connected at publish time. Engineers then spend hours comparing logs across services to prove that a message never arrived. You also end up writing the persistence layer that would have prevented the loss.

Streams cost more up front. You have to create groups, ack every message, write reclaim logic for consumers that die mid-message, route poison messages to a dead-letter stream and make every handler idempotent. Bugs in this code fail quietly. A missing ack makes the pending list grow without bound, and an aggressive reclaim loop delivers the same message over and over. In return, you get replay, catch-up after downtime and work distribution from Redis itself. For moderate workloads, that removes the need for custom persistence or a separate broker.

Metrics to monitor for Streams

• Pending list size per group (XPENDING). Steady growth means acks are missing.
• Idle time of the oldest pending entry. A high value means a consumer is stuck or dead.
• Consumer lag. This is the distance between the last-delivered ID and the end of the stream.
• Delivery count per message. Repeated delivery points to a poison message.
• Dead-letter stream length and growth rate.
• Stream length and memory usage, checked against your trimming policy.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to redis pubsub vs streams, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• The dependency graph answers "what does this touch?" in seconds instead of a manual audit.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• You can size a change's blast radius before merging, avoiding surprise cross-team firefighting.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.