What are Redis streams consumer groups and how do they work?

A Redis stream is an append-only log in which every entry has a unique, monotonically increasing ID (a millisecond timestamp plus a sequence number) and a set of field-value pairs. A consumer group is a named cursor over that log. Each group tracks a last-delivered ID, and every new entry past that point goes to exactly one consumer in the group. Separate groups on the same stream each receive every entry, so a "billing" group and an "audit" group can read the same jobs independently.

When a consumer calls XREADGROUP with the special > ID, Redis advances the group's last-delivered ID and records the entry in the group's pending entries list (PEL) under that consumer's name. From then on, that consumer owns the entry until it sends XACK. Each PEL record holds three things:

  • Owner: the consumer currently responsible for the entry.
  • Idle time: milliseconds since the entry was last delivered.
  • Delivery count: how many times it has been handed out.

This is why XREADGROUP on its own does not make a queue reliable. Delivery only moves an entry into the PEL. If a worker crashes after reading but before acknowledging, the entry stays pending under a consumer that will never return, and no new read with > will deliver it again. The PEL, not the stream, is the source of truth for in-flight work. Recovery, retries, and poison-message detection all depend on inspecting it.

Ordering is guaranteed per stream, not per consumer. Entries leave the group in ID order. Once several consumers are processing in parallel, completion order is not guaranteed. Redis also has no partition key that pins related jobs to one worker.

STREAM jobs (append-only log, entries in ID order)
+-----+-----+-----+-----+-----+-----+
| 1-0 | 2-0 | 3-0 | 4-0 | 5-0 | 6-0 | <-- XADD appends
+-----+-----+-----+-----+-----+-----+
                     |  last-delivered ID = 4-0
                     |  (5-0, 6-0 wait for the next '>' read)
                     v
+------------------------------------------------+
| GROUP 'billing'  (group 'audit' has its own)   |
| PEL (pending entries list) = in-flight work    |
| +-----+----------+---------+------------+      |
| | ID  | owner    | idle ms | deliveries |      |
| +-----+----------+---------+------------+      |
| | 2-0 | worker-a |     120 |          1 |      |
| | 3-0 | worker-b |   95000 |          1 |      |
| | 4-0 | worker-a |      40 |          1 |      |
| +-----+----------+---------+------------+      |
+------------------------------------------------+
          |                        |
     XREADGROUP >             XREADGROUP >
          v                        v
 +------------------+     +------------------+
 | worker-a         |     | worker-b         |
 | processes 2-0,4-0|     | crashed after    |
 | XACK -> PEL drop |     | reading 3-0;     |
 | (1-0 acked)      |     | '>' won't resend |
 +------------------+     +------------------+
Entries leave the group in ID order (per stream); completion order
across parallel workers is not guaranteed.

A consumer group advances its last-delivered ID over the stream, and each delivered entry sits in the PEL with its owner, idle time and delivery count until XACK, so a crashed worker's entry stays pending.

Who should use Redis streams for job queues, and when should you avoid them?

Where streams fit

Streams make sense when Redis is already in your stack and you understand how it behaves under failure. You get a durable-enough log, at-least-once delivery, and parallel workers without adding a broker or a new on-call rotation. Latency is low because entries never leave memory. A blocking XREADGROUP returns as soon as something is appended. This suits short tasks that are cheap to retry, such as sending notifications, invalidating caches, generating thumbnails, or fanning webhook events out to workers. Teams with a modest job volume and idempotent handlers get the most from it.

When to avoid them

  • You need exactly-once effects. Streams redeliver unacknowledged entries, so handlers will occasionally run twice. If you cannot make the side effect idempotent, the stream will not solve that for you.
  • Backlogs or payloads are large. Every entry lives in RAM until you trim it. A consumer outage that builds up millions of entries, or jobs that carry multi-megabyte bodies, consumes memory. Store payloads elsewhere and enqueue references.
  • Losing acknowledged writes is unacceptable. Replication is asynchronous. A primary can confirm an XADD and then fail before the replica receives it. Fsync policy determines how much is lost on a crash.
  • You need delays, priorities, or retry backoff. None of these is built in. You end up building them with sorted sets and extra code.
  • One stream outgrows one shard. A key maps to a single hash slot, so splitting the load means splitting streams yourself.

How do you create a consumer group and read jobs with XREADGROUP?

Start by creating the group with XGROUP CREATE jobs workers $ MKSTREAM. MKSTREAM creates an empty stream if the key doesn't exist yet, so you don't need a placeholder entry or a particular startup order between producers and workers. The last argument sets where the group's cursor begins. $ means only entries added after creation. Use it for a new queue where old data is irrelevant. 0 means everything already in the stream, which is useful when you attach a new group to existing history. Run this at deploy time or on worker startup. Treat the BUSYGROUP error as success, because it means the group already exists.

Producers call XADD jobs * type resize image_id 42. The * lets Redis assign a monotonically increasing ID.

Each worker runs XREADGROUP GROUP workers <consumer> COUNT 10 BLOCK 5000 STREAMS jobs >. The > ID asks for entries never delivered to any consumer in the group. Redis records each returned entry in the pending entries list (PEL) under this consumer's name. COUNT caps the batch size, which amortizes round trips. BLOCK long-polls, so idle workers wait on the server instead of spinning. After a job succeeds, call XACK jobs workers <id> to remove it from the PEL. If you never acknowledge, the PEL grows indefinitely.

Give each worker a stable consumer name, such as its hostname or pod name. On startup, read once with ID 0 instead of >. This returns entries already delivered to this consumer but not yet acknowledged, which is the work it held when it crashed. Process those entries, then switch to >. A random name on every restart would orphan that history, and recovering it would then depend entirely on another consumer claiming it.

import socket
import redis

r = redis.Redis(host="localhost", port=6379, decode_responses=True)
STREAM, GROUP = "jobs", "workers"
CONSUMER = socket.gethostname()  # stable name, so pending work survives restarts

# Idempotent setup: "$" = only new entries; MKSTREAM creates the key if missing
try:
    r.xgroup_create(STREAM, GROUP, id="$", mkstream=True)
except redis.exceptions.ResponseError as e:
    if "BUSYGROUP" not in str(e):  # group already exists -> treat as success
        raise

# Producer side: r.xadd(STREAM, {"type": "resize", "image_id": "42"})

def handle(job_id, fields):
    print("processing", job_id, fields)  # raise on failure -> no XACK, stays pending

def process(entries):
    for job_id, fields in entries:
        handle(job_id, fields)
        r.xack(STREAM, GROUP, job_id)  # remove from the PEL only after success

# 1) Startup: drain entries already delivered to THIS consumer but never acked
last_id = "0"
while True:
    resp = r.xreadgroup(GROUP, CONSUMER, {STREAM: last_id}, count=10)
    entries = resp[0][1] if resp else []
    if not entries:
        break
    process(entries)
    last_id = entries[-1][0]  # page forward through this consumer's history

# 2) Steady state: ">" = never-delivered entries; batch of 10, long-poll 5s
while True:
    resp = r.xreadgroup(GROUP, CONSUMER, {STREAM: ">"}, count=10, block=5000)
    if resp:
        process(resp[0][1])

A redis-py worker that idempotently creates the group with MKSTREAM, recovers its own unacknowledged history with ID 0, then switches to XREADGROUP with > using COUNT and BLOCK and acknowledges each job with XACK after it succeeds.

How do you recover stuck jobs with XAUTOCLAIM and XPENDING?

Before reclaiming anything, look at the pending entries list (PEL). The summary form, XPENDING jobs workers, returns the total pending count, the lowest and highest pending IDs, and a per-consumer count. That last part shows you which worker is hoarding entries, which usually points to a crash or a stall. The extended form, XPENDING jobs workers - + 10, lists individual entries with their owner, the milliseconds since last delivery, and the delivery count. You need the delivery count later to detect poison messages.

To recover work, have each worker periodically sweep the PEL with XAUTOCLAIM jobs workers <consumer> <min-idle-ms> 0-0 COUNT 50. It transfers ownership of entries idle longer than the threshold and returns a cursor. Pass that cursor back in to continue the scan, and stop when it returns 0-0. Each claim resets the entry's idle time and increments its delivery count. When you already know which IDs you want, such as ones picked out by an XPENDING scan, use XCLAIM with explicit IDs instead.

The hard part is choosing min-idle-time. Redis cannot distinguish a crashed worker from a slow one. Both just look idle. If you set the threshold too low, a healthy worker's long job gets stolen mid-run and executes twice. If you set it too high, a crashed worker's jobs sit stuck for that whole window.

Set the threshold well above your worst-case job runtime. If runtimes vary widely, don't raise the threshold to cover the slowest job. Instead, have long-running workers heartbeat by calling XCLAIM on their own entries with JUSTID. This resets the idle time without bumping the delivery count, so you can keep a short threshold for detecting real crashes.

import redis

r = redis.Redis(host="localhost", port=6379, decode_responses=True)
STREAM, GROUP, ME = "jobs", "workers", "worker-2"
MIN_IDLE_MS = 60_000   # must sit well above worst-case job runtime
MAX_DELIVERIES = 5     # beyond this, treat the entry as a poison message

# Summary form: total pending, lowest/highest ID, per-consumer counts
summary = r.xpending(STREAM, GROUP)
print(summary["pending"], summary["consumers"])

# Extended form: owner, ms since last delivery, delivery count per entry
for e in r.xpending_range(STREAM, GROUP, min="-", max="+", count=10):
    if e["times_delivered"] > MAX_DELIVERIES:
        print("poison candidate:", e["message_id"], e["consumer"])

# Sweep: claim entries idle longer than MIN_IDLE_MS, following the cursor
cursor = "0-0"
while True:
    resp = r.xautoclaim(STREAM, GROUP, ME, MIN_IDLE_MS, start_id=cursor, count=50)
    cursor, entries = resp[0], resp[1]
    for msg_id, fields in entries:
        # each claim reset idle time and bumped the delivery count
        print("reclaimed", msg_id, fields)  # process, then XACK
    if cursor == "0-0":  # full pass over the PEL complete
        break

# Targeted reclaim of a known ID (e.g. picked from the XPENDING scan)
r.xclaim(STREAM, GROUP, ME, MIN_IDLE_MS, ["1700000000000-0"])

# Heartbeat for a long job this worker already owns: min-idle 0 plus
# JUSTID resets idle time without incrementing the delivery count
def heartbeat(msg_id: str) -> None:
    r.xclaim(STREAM, GROUP, ME, 0, [msg_id], justid=True)

Using redis-py to inspect the PEL with XPENDING in summary and extended form, sweep stuck entries with cursor-driven XAUTOCLAIM, reclaim a specific ID with XCLAIM, and heartbeat a long-running job with XCLAIM JUSTID so a short min-idle-time doesn't steal it.

How do you handle poison messages and duplicate deliveries?

Even with careful claiming, some entries will never succeed. A malformed payload or a job that crashes the worker will be claimed, fail, go idle, and be claimed again indefinitely. The extended form of XPENDING reports a delivery count for each pending entry. XAUTOCLAIM and XCLAIM increment that count unless you pass JUSTID. Treat it as an attempt counter. When a claimed entry exceeds your limit, for example five deliveries, XADD its fields to a dead-letter stream along with the original entry ID and the last error, then XACK the original. Run both commands in a MULTI/EXEC block or a Lua script so a crash between them cannot lose the job or leave it in both places.

Idempotency is not optional

Consumer groups give you at-least-once delivery. If a worker finishes the side effect but dies before XACK, another worker will run the job again. Handlers must tolerate this. A common guard is SET job:done:<id> 1 NX EX 86400. If NX fails, the job already ran, so acknowledge it and skip it. Prefer a job ID carried in the payload over the stream entry ID. A producer that retries XADD after a timeout creates a second entry with a new ID but the same job. Set the marker only after the side effect succeeds. When the side effect writes to a database, store the marker in the same transaction.

Streams have no native delay or backoff. Scheduled retries need extra keys, typically a sorted set scored by due time, plus a process that moves due jobs back into the stream.

import redis

r = redis.Redis(host="localhost", port=6379, decode_responses=True)
STREAM, GROUP, CONSUMER = "jobs", "workers", "worker-1"
DLQ, ERRORS, MAX_DELIVERIES = "jobs:dead", "jobs:last_error", 5

def handle(entry_id, fields):
    # Key on the producer's job_id, which stays the same if XADD is retried
    marker = f"job:done:{fields['job_id']}"
    if r.exists(marker):
        r.xack(STREAM, GROUP, entry_id)  # duplicate delivery: already done
        return
    send_invoice(fields)  # the side effect; must succeed before marking
    r.set(marker, 1, nx=True, ex=86400)
    r.xack(STREAM, GROUP, entry_id)

def reclaim_once():
    # XAUTOCLAIM increments the delivery count of every entry it claims
    result = r.xautoclaim(STREAM, GROUP, CONSUMER, min_idle_time=60000,
                          start_id="0-0", count=10)
    for entry_id, fields in result[1]:
        info = r.xpending_range(STREAM, GROUP, min=entry_id, max=entry_id, count=1)
        if info and info[0]["times_delivered"] > MAX_DELIVERIES:
            # Poison message: copy to DLQ and ack atomically
            pipe = r.pipeline(transaction=True)  # MULTI/EXEC
            pipe.xadd(DLQ, {**fields, "orig_id": entry_id,
                            "last_error": r.hget(ERRORS, entry_id) or "unknown"})
            pipe.xack(STREAM, GROUP, entry_id)
            pipe.hdel(ERRORS, entry_id)
            pipe.execute()
            continue
        try:
            handle(entry_id, fields)
        except Exception as exc:
            # Leave it pending; it will be reclaimed after it goes idle again.
            # Real backoff would need a sorted set scored by due time.
            r.hset(ERRORS, entry_id, repr(exc))

A redis-py worker that uses the XPENDING delivery count to move poison entries to a dead-letter stream with XADD and XACK in one MULTI/EXEC, and guards the side effect with a job-ID marker written only after it succeeds.

How do you operate Redis streams in production?

Bound memory with approximate trimming

A stream grows until you trim it. Cap it at write time with XADD ... MAXLEN ~ <n>, or run XTRIM periodically. The ~ lets Redis trim only at internal node boundaries. This is much cheaper than exact trimming, and the stream may briefly hold somewhat more than n entries. Size n from your peak backlog, not your average one.

Trimming does not consult consumer groups. It deletes the oldest entries whether or not they have been acknowledged. If consumers fall far enough behind, unprocessed jobs disappear, and pending entries can point at messages that no longer exist. Alert on backlog well before it approaches the cap.

Watch lag and idle consumers

XINFO GROUPS reports each group's pending count and last-delivered ID, and newer Redis versions also report lag. XINFO CONSUMERS breaks pending counts down per consumer and shows idle time. A consumer with a high idle time and a nonzero pending count is holding work that nobody is processing.

Clean up autoscaled consumers

Autoscaling leaves dead consumer names behind. XGROUP DELCONSUMER discards a consumer's pending entries along with the consumer, so order matters:

  1. Reclaim the consumer's pending work with XAUTOCLAIM or XCLAIM into a live consumer.
  2. Confirm its pending count is zero.
  3. Delete the consumer.

Durability and scale

An acknowledgment is only as durable as Redis persistence. With AOF set to fsync every second, a crash can lose roughly the last second of writes. Replication is asynchronous, so a failover can also lose recently added jobs.

A stream is a single key, so it lives on one shard. To scale past that, partition jobs across several stream keys that hash to different slots, and run a consumer group on each one.

What are the alternatives to Redis streams, and what do they cost in engineering time?

Each option trades code you write against infrastructure you run.

Option Delivery Retries / dead-lettering Durability Ordering Ops overhead
Redis streams + groups At-least-once You build them; the PEL supplies delivery counts Redis persistence settings Per stream Existing Redis
Redis lists (BLMOVE to a processing list) At-least-once You build all of it, with no delivery counts Redis persistence settings FIFO per list Existing Redis
Redis pub/sub At-most-once. Offline subscribers miss messages None None Per channel Existing Redis
Database queue table At-least-once with row locking You build them, but SQL makes them straightforward Transactional Whatever your query enforces Existing DB, with extra write and vacuum load
Dedicated broker or log At-least-once is typical Usually built in for brokers. Log systems leave them to consumers Replicated Per queue or partition New cluster, monitoring and on-call

Choosing streams means you own five pieces: a reclaimer loop, a dead-letter stream, idempotency keys, trimming, and delayed jobs. Streams have no native scheduling, so delayed jobs usually need a sorted set plus a mover process. In return, you avoid deploying, securing, upgrading and paging on a new system.

Illustrative estimate (assumed numbers, not a benchmark)

Build cost = sum of component effort. Assume 3 days for the reclaimer, 2 for the DLQ, 2 for idempotency, 1 for cleanup and 3 for scheduling, which totals 11 engineer-days.

Broker cost = setup + (monthly ops × months). Assume 10 days of setup and 2 days per month of ops. Over 12 months that is 10 + 24 = 34 engineer-days.

On these inputs, streams come out cheaper. The comparison reverses when your team already runs a broker, because the broker's setup cost and most of its ongoing ops cost are then close to zero.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to redis streams consumer groups, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • The dependency graph answers "what does this touch?" in seconds instead of a manual audit.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • You can size a change's blast radius before merging, avoiding surprise cross-team firefighting.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.