What are Redis Streams consumer groups, and how do they differ from a list-based queue?

A Redis stream is an append-only log. XADD appends an entry and assigns it an ID made of a millisecond timestamp and a sequence number, so entries are ordered and individually addressable. Reading an entry does not remove it. The stream keeps every entry until you trim it, either by length (MAXLEN) or by minimum ID (MINID). Retention is a decision you have to make explicitly.

A list-based queue works differently. Producers LPUSH and workers BRPOP, and the pop deletes the job. If a worker crashes after popping, the job is lost. You can avoid that by atomically moving each job into a separate processing list, but then you have to build your own logic to track ownership, measure how long a job has been in flight, and requeue it.

Consumer groups provide that tracking in Redis itself. A group keeps a cursor into the stream and hands each new entry to one consumer in the group. Each group also maintains a Pending Entries List (PEL): entries that were delivered but not yet acknowledged with XACK. For each pending entry, the PEL records the owning consumer, the time since its last delivery, and its delivery count. Retries, crash recovery and dead-lettering are all built from these three fields, so the rest of this article treats the PEL as the core of the design.

Concern List queue Stream + consumer group
Read removes job Yes No; trim explicitly
In-flight tracking Manual processing list Built-in PEL
Delivery count Not tracked Tracked per entry
Multiple independent readers Not without duplicating jobs One group per reader

This approach suits teams that already run Redis and need moderate-throughput background jobs without operating a separate broker.

# Create the group first; MKSTREAM creates the stream if it does not exist.
# '0' means the group starts from the beginning of the stream.
redis-cli XGROUP CREATE jobs workers 0 MKSTREAM

# Producer: append a job. '*' lets Redis assign a <ms>-<seq> ID.
# MAXLEN ~ caps the stream at roughly 10000 entries (approximate trim is cheaper).
redis-cli XADD jobs MAXLEN '~' 10000 '*' type email to user@example.com

# Worker: read one never-delivered entry ('>') for this group, blocking up to 5s.
# The entry now sits in the group's PEL, owned by worker-1.
redis-cli XREADGROUP GROUP workers worker-1 COUNT 1 BLOCK 5000 STREAMS jobs '>'

# Inspect the PEL: entry ID, owning consumer, ms since last delivery, delivery count.
redis-cli XPENDING jobs workers - + 10

# After the job succeeds, acknowledge it. This removes it from the PEL only.
# Replace the ID with the one returned by XREADGROUP.
redis-cli XACK jobs workers 1731300000000-0

# The entry is still in the stream: reads and acks do not delete it.
redis-cli XRANGE jobs - + COUNT 10

# Retention is explicit: drop entries older than a given ID.
redis-cli XTRIM jobs MINID '~' 1731300000000-0

# For contrast, a list queue: BRPOP deletes the job on read,
# so a worker crash after this line loses the job.
redis-cli LPUSH jobs:list '{"type":"email","to":"user@example.com"}'
redis-cli BRPOP jobs:list 5

A redis-cli walkthrough of a job's lifecycle in a stream consumer group: it is added, delivered into the PEL, inspected, acknowledged and trimmed. A list queue is shown at the end for contrast, where BRPOP removes the job as soon as it is read.

How does a job move through the Pending Entries List?

Every job follows the same path, and the Pending Entries List (PEL) is where Redis tracks it between delivery and completion. Each consumer group keeps its own PEL. Every entry in it records the message ID, the consumer that currently owns it, the time since it was last delivered (idle time), and a delivery counter.

 producer
    │ XADD
    ▼
 ┌────────┐  XREADGROUP '>'   ┌──────────────────────────────┐
 │ stream │ ────────────────▶ │ PEL: id, owner=worker-A,     │
 └────────┘                   │      idle, deliveries=1      │
                              └──────────────┬───────────────┘
              ┌──────────────────────────────┼──────────────────────────────┐
              ▼                              ▼                              ▼
      XACK (success)          idle > threshold: XAUTOCLAIM     deliveries > limit:
      entry leaves PEL        owner=worker-B, deliveries+1     XADD to DLQ stream,
                              (back into the PEL)              then XACK original

XREADGROUP with the special ID > returns only entries that have never been delivered to any consumer in the group. It also advances the group's last-delivered ID and adds each returned entry to the PEL under the reading consumer's name. From there, an entry should leave the PEL in exactly one of three ways:

  • Acknowledged. The worker calls XACK after the job's side effects are durable.
  • Reclaimed. Another consumer takes ownership once the entry has been idle too long. Reclaiming increments the delivery count.
  • Dead-lettered. Redis has no built-in dead-letter queue. Your code checks the delivery count, copies the entry to a separate stream, and then acknowledges the original.

XREADGROUP on its own does not give you at-least-once processing. Suppose a worker crashes after reading an entry. The entry stays in the PEL, owned by a consumer that no longer exists. Because > never returns an entry twice, nothing will redeliver it. A worker restarting under the same name can see its own backlog by reading from ID 0, but a worker that never returns leaves its entries stranded. At-least-once delivery comes from two disciplines working together: reclaiming idle entries and acknowledging only after the work is done.

                                 producer
                                     │ XADD jobs * field value
                                     ▼
                              ┌──────────────┐
                              │ stream: jobs │
                              └──────┬───────┘
                                     │ XREADGROUP ... STREAMS jobs >
                                     │ (never-delivered entries only)
                                     ▼
      ┌──────────────────────────────────────────────────────────────┐
      │ PEL (one per consumer group)                                 │
      │ id | owner=worker-A | idle time | deliveries=1               │
      └──────────────────────────────┬───────────────────────────────┘
                                     │ worker-A runs the job
            ┌────────────────────────┼────────────────────────┐
            ▼                        ▼                        ▼
 ┌──────────────────────┐ ┌──────────────────────┐ ┌──────────────────────┐
 │ XACK (success)       │ │ idle > min-idle-time │ │ deliveries > limit   │
 │ after side effects   │ │ XAUTOCLAIM: worker-B │ │ XADD jobs:dlq ...    │
 │ are durable          │ │ deliveries += 1      │ │ then XACK original   │
 │ entry leaves PEL     │ │ stays in PEL, retry  │ │ entry leaves PEL     │
 └──────────────────────┘ └──────────────────────┘ └──────────────────────┘

 If worker-A crashes before XACK, the entry stays in the PEL and its idle
 time grows; only the reclaim path gets it processed again.

A job's path from XADD through XREADGROUP into the consumer group's PEL, and the three ways it leaves: XACK on success, reclaim by another consumer after an idle timeout, or dead-lettering once the delivery count exceeds a limit.

How do you implement a worker with XREADGROUP and XACK?

Create the group once, at deploy time or idempotently on worker startup: XGROUP CREATE jobs workers $ MKSTREAM. MKSTREAM creates the stream if it does not exist yet, so the first worker does not have to wait for a producer. If the group already exists, the command returns a BUSYGROUP error. Treat that as success. The start ID sets where the group's cursor begins. $ delivers only entries added after the group is created, which is the right choice for a fresh queue. 0 replays everything already in the stream, which is useful when you attach a new group to a stream that is already holding backlog.

Startup: drain your own pending entries first

Give each worker a stable consumer name so it can recover its own work after a restart. Before taking new jobs, call XREADGROUP GROUP workers <name> COUNT 10 STREAMS jobs 0. With an ID other than >, Redis returns the entries already in this consumer's PEL, meaning work it received but never acknowledged. Process each batch and pass the last returned ID as the next start ID. When a call returns no entries, the backlog is drained. Blocking is ignored for this history read, so the loop cannot hang.

Steady state

Switch to XREADGROUP GROUP workers <name> COUNT 10 BLOCK 5000 STREAMS jobs >. The > ID asks for entries never delivered to any consumer. BLOCK keeps the worker from busy-polling, and COUNT limits how much work one worker takes per call.

Call XACK jobs workers <id> only after the job's side effects have been committed. If you ack first and the process then crashes, the entry has already left the PEL and nothing will ever redeliver it. That changes the queue from at-least-once to at-most-once delivery.

import socket
import redis

r = redis.Redis(host="localhost", port=6379, decode_responses=True)
STREAM, GROUP = "jobs", "workers"
CONSUMER = f"worker-{socket.gethostname()}"  # stable name so restarts find their own PEL

def ensure_group():
    try:  # XGROUP CREATE jobs workers $ MKSTREAM
        r.xgroup_create(STREAM, GROUP, id="$", mkstream=True)
    except redis.ResponseError as e:
        if "BUSYGROUP" not in str(e):
            raise  # group already exists = success; anything else is a real error

def handle(entry_id, fields):
    try:
        if fields is not None:  # None = entry was trimmed while still pending
            process(fields)  # your job: must commit its side effects or raise
    except Exception:
        return  # no ack: entry stays in the PEL for retry or reclaim
    r.xack(STREAM, GROUP, entry_id)  # ack only AFTER side effects succeeded

def drain_own_pending():
    last_id = "0"  # non-'>' ID reads this consumer's PEL; BLOCK would be ignored
    while True:
        resp = r.xreadgroup(GROUP, CONSUMER, {STREAM: last_id}, count=10)
        entries = resp[0][1] if resp else []
        if not entries:
            return  # own backlog drained
        for entry_id, fields in entries:
            handle(entry_id, fields)
        last_id = entries[-1][0]  # continue after the last returned ID

def run():
    ensure_group()
    drain_own_pending()
    while True:  # '>' = never-delivered entries; BLOCK avoids busy-polling
        resp = r.xreadgroup(GROUP, CONSUMER, {STREAM: ">"}, count=10, block=5000)
        for _stream, entries in resp or []:
            for entry_id, fields in entries:
                handle(entry_id, fields)

A redis-py worker that creates the group idempotently, drains its own pending entries with ID '0', then switches to blocking reads with '>' and acks only after the job's side effects have committed.

How do you recover stuck jobs with XPENDING and XAUTOCLAIM?

at-most-once, which is rarely what a job queue wants.

Reading the PEL

XPENDING has two forms. The summary form, called with just the stream and group, returns the total pending count, the lowest and highest pending IDs, and a per-consumer breakdown. It is cheap enough to poll, and it suits dashboards. The extended form adds a range and a count, and can also filter by consumer. It returns one row per entry with the owning consumer, the milliseconds since the entry was last delivered, and its delivery count. Use the extended form when you need to decide what to do with a specific job.

These map directly to alerts. A pending count that keeps growing means workers are failing to ack. A high idle time on the oldest entry means a job is stuck. A rising delivery count points to a poison message, which section 5 covers. XINFO CONSUMERS shows how long each consumer has been idle, so you can spot dead workers that still own entries. XINFO GROUPS shows pending totals and the last-delivered ID for each group.

Reclaiming

A reclaim loop calls XAUTOCLAIM with a min-idle-time and a cursor. It transfers ownership of matching entries to the calling consumer, increments their delivery count, and returns the next cursor so you can page through a large PEL. If you need to inspect entries before claiming them, use XPENDING followed by XCLAIM instead.

Set min-idle-time well above the longest expected job duration. If you set it too low, a job that is slow but still alive gets claimed by another worker, and the two copies run concurrently. If you set it too high, a job held by a crashed worker sits unprocessed for that entire window before anything recovers it.

import time
import redis

r = redis.Redis(host="localhost", port=6379, decode_responses=True)
STREAM, GROUP, ME = "jobs", "workers", "reclaimer-1"
MIN_IDLE_MS = 5 * 60 * 1000  # longest expected job (~60s) plus a wide margin
MAX_DELIVERIES = 5           # beyond this, treat as poison (see DLQ section)

def check_pel():
    # Summary form: cheap enough to poll for dashboards
    s = r.xpending(STREAM, GROUP)
    print("pending:", s["pending"], "oldest id:", s["min"])
    # Extended form: one row per entry with owner, idle ms, delivery count
    for e in r.xpending_range(STREAM, GROUP, min="-", max="+", count=10):
        if e["time_since_delivered"] > MIN_IDLE_MS:
            print("stuck:", e["message_id"], "owner:", e["consumer"])
        if e["times_delivered"] >= MAX_DELIVERIES:
            print("possible poison:", e["message_id"])
    # Dead workers show a large idle time while still owning entries
    for c in r.xinfo_consumers(STREAM, GROUP):
        print(c["name"], "pending:", c["pending"], "idle ms:", c["idle"])

def process(msg_id, fields):
    ...  # job handler; must be idempotent because reclaim can re-run it

def reclaim_once():
    cursor = "0-0"
    while True:
        resp = r.xautoclaim(STREAM, GROUP, ME, min_idle_time=MIN_IDLE_MS,
                            start_id=cursor, count=50)
        cursor, entries = resp[0], resp[1]  # Redis 7+ adds deleted IDs as resp[2]
        for msg_id, fields in entries:
            if fields:  # skip entries trimmed from the stream
                process(msg_id, fields)
                r.xack(STREAM, GROUP, msg_id)
        if cursor == "0-0":  # full pass over the PEL complete
            break

while True:
    check_pel()
    reclaim_once()
    time.sleep(30)

A redis-py loop that reads the PEL with the XPENDING summary and extended forms and XINFO CONSUMERS to surface stuck and poison entries, then pages through XAUTOCLAIM with a min-idle-time set well above the longest expected job duration.

How should you handle poison messages and duplicate deliveries?

A delivery count protects you from the other failure mode: a job that crashes its handler every time. Each entry in the PEL carries a counter that increases whenever the entry is delivered or claimed, and the extended form of XPENDING and the XAUTOCLAIM path both expose it. Pick a limit, such as five attempts. When an entry exceeds that limit, XADD it to a dead-letter stream like jobs:dlq, along with its original ID, fields, and last error. Then XACK the original. Without the ack, the poison entry keeps cycling through reclaims and burns worker time on every pass. The two commands are not atomic, so wrap them in MULTI/EXEC or a Lua script, or accept that a crash between them can leave a duplicate in the DLQ.

Duplicates are guaranteed by design. If a worker finishes a job and dies before XACK, the entry stays pending and will run again. Handlers must therefore be idempotent:

  • Dedupe key: after a successful run, SET job:done:<id> 1 NX EX <ttl>, and check for that key before doing work. Set it only after success. Setting it before processing marks failed jobs as done. Keep the TTL longer than your maximum retry window.
  • Idempotent writes: make the side effect itself safe to repeat, for example an upsert keyed by job ID or a unique constraint that rejects the second insert.

Ordering also breaks down. Parallel consumers and reclaims let entry 2 finish before entry 1. If per-entity order matters, hash the entity ID across N streams so that all work for one entity lands in the same stream, and run one active consumer per stream.

What housekeeping does a Redis job queue need in production?

Consumers are never removed automatically. When a worker pod is replaced, its name stays in the group, and so do any entries it was holding. XGROUP DELCONSUMER deletes the consumer and its pending entries, so those jobs disappear from the PEL and no reclaim will ever find them. Run XAUTOCLAIM against the departed consumer's entries first, and delete it only once XPENDING shows it owns nothing.

Trimming without losing work

A stream keeps every entry until you trim it with MAXLEN or MINID, either on XADD or through XTRIM. Trimming does not check the PEL. If a pending entry is trimmed, the PEL still holds its ID, but the payload is gone, so a reclaim returns nothing to process. MINID is the safer choice. Set it below the smallest ID in the XPENDING summary for every group, so you only remove entries that all groups have acknowledged. MAXLEN caps memory, but under a backlog it will drop unprocessed jobs.

Memory and durability

The stream and the PEL both live in RAM. A slow consumer makes the stream grow, and missing XACK calls make the PEL grow. Alert on group lag and on PEL size.

How much survives a crash depends on your RDB and AOF fsync settings. Replication is asynchronous, so a failover can lose recent XADDs, which means lost jobs. It can also lose recent XACKs, which means completed jobs come back and run again.

A stream is a single key, so in Redis Cluster it lives on one shard. To scale throughput, partition the work across several streams.

When should you avoid Redis Streams, and what are the alternatives?

Redis Streams works well as a job queue when you already run Redis with persistence, replication and alerting you trust, and the job model is simple. That means one or a few job types, at-least-once delivery, idempotent handlers, and retries that can run as soon as a job is reclaimed. Reusing that infrastructure saves real engineering time, because there is no new system to deploy, secure, upgrade or learn to debug at 3 a.m.

The fit gets worse as requirements grow. Streams have no per-entry delay, so exponential backoff or scheduled jobs need a sorted set plus a process that moves due entries into the stream. Priorities require separate streams and read-ordering logic in the worker. Asynchronous replication limits durability guarantees. A single stream cannot scale past one shard.

Option Strengths Weaknesses Choose when
Redis Streams Reuses existing Redis, PEL tracking, replay No delays or priorities, one shard per stream Simple jobs, Redis already operated
Redis lists Minimal code, very simple No pending tracking, lost jobs on crash unless you build reliable-move patterns Loss is tolerable
Dedicated message broker Native acks, redelivery, dead-lettering, often delays and priorities Another cluster to run Complex routing, stronger delivery semantics
Log-based streaming platform High throughput, partitioned, long retention Poor fit for per-job retries, heavy to operate Event pipelines at scale
Job framework Retries, backoff, scheduling, dashboards built in Ties you to its model and language You want features over control

Count the code honestly. The reclaim loop, dead-letter handling, consumer cleanup and lag monitoring are all things you write, test and maintain. If your roadmap already includes backoff and scheduling, that code will keep growing, and a broker or job framework will usually cost less than extending a Redis-based queue.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to redis streams consumer groups, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Consumers learn about a breaking change at review time rather than from production errors.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.