How does a Redis List work as a job queue, and what happens when a worker crashes?
The simplest Redis job queue is a List. Producers LPUSH a serialized job onto one end, and workers block on BRPOP at the other end, which gives FIFO order without polling. The weakness is that BRPOP removes the job and returns it in one atomic step. After that, the job exists only in the worker's memory. If the worker crashes, is OOM-killed, or loses its connection mid-task, the job is gone. Redis has no record that it was ever handed out. This is at-most-once delivery: each job runs once or not at all.
The standard fix is the reliable-queue pattern. Instead of popping, a worker calls BLMOVE (or the older BRPOPLPUSH) to atomically move the job from the main queue into a processing list owned by that worker. When the job finishes, the worker deletes it from its processing list with LREM. A separate reaper process scans processing lists and pushes stale jobs back onto the main queue. A crash now leaves the job parked rather than lost, and delivery becomes at-least-once. Because of that, your handlers must be idempotent.
Redis gives you the atomic move, and you have to build the rest:
- Staleness detection. A list entry carries no timestamp. You must store claim times or worker heartbeats in separate keys, and the reaper needs them to tell a slow job from a dead worker.
- Completion cost.
LREMscans the list, so it is O(N). It gets slow if processing lists grow large. - Identity and retries. Jobs have no message ID and no delivery count. Duplicate payloads are indistinguishable, and you cannot detect a job that crashes every worker that takes it unless you embed and update that metadata yourself.
import json, socket
import redis
r = redis.Redis(host="localhost", port=6379, decode_responses=True)
QUEUE = "jobs:pending"
WORKER_ID = socket.gethostname()
PROCESSING = f"jobs:processing:{WORKER_ID}"
HEARTBEAT = f"worker:heartbeat:{WORKER_ID}"
def enqueue(job):
# Producers push on the left; workers take from the right (FIFO).
r.lpush(QUEUE, json.dumps(job))
def work_forever(handle):
while True:
# Liveness is our job: a key that expires if this worker stops refreshing it.
# Long jobs need a background thread to keep refreshing it during handle().
r.set(HEARTBEAT, "alive", ex=30)
# Atomically move the job into this worker's processing list instead of
# destroying it, as BRPOP would. Blocks up to 5 seconds.
raw = r.blmove(QUEUE, PROCESSING, 5, src="RIGHT", dest="LEFT")
if raw is None:
continue
handle(json.loads(raw)) # may run more than once: must be idempotent
# Ack by value. LREM scans the list (O(N)) and cannot tell identical payloads apart.
r.lrem(PROCESSING, 1, raw)
def reap():
# Run periodically in a separate process.
for key in r.scan_iter(match="jobs:processing:*"):
worker = key.rsplit(":", 1)[1]
if r.exists(f"worker:heartbeat:{worker}"):
continue # worker still alive (or just slow)
# Dead worker: push its jobs back to the consumer end so they run next.
# There is no delivery count, so a poison job will cycle forever.
while r.lmove(key, QUEUE, src="RIGHT", dest="RIGHT"):
pass
A reliable-list queue in redis-py. Workers claim jobs with BLMOVE into a per-worker processing list and ack with LREM, while a reaper uses hand-rolled heartbeat keys to requeue jobs left behind by dead workers.
What are Redis Streams and consumer groups, and how do they work for job queues?
A stream solves that by giving every job an identity. XADD jobs * type resize image_id 42 appends an entry and returns an ID such as 1700000000000-0: a millisecond timestamp plus a sequence number. Redis guarantees each new ID is greater than the last, so IDs give you a total order, a deduplication key and a cursor position all at once.
Reading a stream does not remove anything. Where RPOP destroys the job as it hands it over, a stream entry stays in place until you trim or delete it. Delivery state is tracked separately, per consumer group.
XGROUP CREATE jobs workers $ MKSTREAM creates a group named workers. MKSTREAM creates the stream if it does not exist yet, so deploy order between producers and consumers stops mattering. The start ID sets the group's cursor. $ means "only entries added from now on." 0 means "everything already in the stream," which is useful when you attach a new group to existing history.
Workers call XREADGROUP GROUP workers w1 ... STREAMS jobs >. The special ID > asks for entries this group has never delivered. Redis advances the group's cursor, so each entry goes to exactly one consumer in the group. It also records the delivery in the group's pending entries list (PEL).
Each group keeps its own cursor and PEL. Adding an audit group gives a second, independent pass over the same entries without copying any payloads:
producers ──XADD──▶ stream "jobs": [1-0][2-0][3-0][4-0] ...
│ │
group "workers" (cursor 4-0) group "audit" (cursor 2-0)
├─ w1 ├─ w2 └─ a1
PEL: 3-0→w1, 4-0→w2 PEL: 2-0→a1
redis-cli XGROUP CREATE jobs workers '$' MKSTREAM
redis-cli XADD jobs '*' type resize image_id 42
redis-cli XADD jobs '*' type resize image_id 43
redis-cli XGROUP CREATE jobs audit 0
redis-cli XREADGROUP GROUP workers w1 COUNT 1 STREAMS jobs '>'
redis-cli XREADGROUP GROUP workers w2 COUNT 1 STREAMS jobs '>'
redis-cli XREADGROUP GROUP audit a1 COUNT 10 STREAMS jobs '>'
redis-cli XLEN jobs
redis-cli XPENDING jobs workers
redis-cli XPENDING jobs audit
redis-cli XACK jobs workers 1700000000000-0
A redis-cli session that creates a stream with MKSTREAM, adds jobs, splits them across two workers in one group with XREADGROUP '>', reads the same entries again from an independent audit group, and inspects and clears each group's pending entries list.
How does it work end to end? The job lifecycle in each model
Both approaches follow the same lifecycle: enqueue, claim, process, acknowledge, and recover. They differ in how much of that lifecycle Redis tracks for you.
With a reliable List:
- Enqueue. A producer
LPUSHes a serialized job onto the main list. - Claim. A worker
BLMOVEs the job into its own processing list. The job now lives in exactly one place. - Process. The worker runs the handler. Redis knows nothing about the job's progress, so the worker refreshes a heartbeat key to show it is alive.
- Acknowledge. The worker
LREMs the job, matching by value, from its processing list. - Recover. A reaper finds processing lists whose heartbeat has expired and moves their contents back to the main list.
With a Stream and a consumer group:
- Enqueue. A producer
XADDs an entry. Redis assigns a unique, increasing ID. - Claim. A worker calls
XREADGROUP ... >. Redis hands the entry to that one consumer and writes a PEL record with the owner, the delivery time and a delivery count. - Process. The worker runs the handler. The PEL record's idle time grows until the worker acknowledges the entry or something claims it.
- Acknowledge. The worker
XACKs the entry by ID. The PEL record is removed, and the entry stays in the stream until it is trimmed. - Recover. A restarted worker re-reads its own pending entries with ID
0. A reaper usesXAUTOCLAIMorXCLAIMto take over entries idle past a threshold. It uses the delivery count to divert poison messages to a dead-letter stream.
In the List model, steps 3 and 5 depend on state you invent and maintain yourself, such as heartbeat keys, claim timestamps and retry counters. In the Stream model, Redis stores that state in the PEL. You still write the policy, meaning when to claim and when to dead-letter, but not the bookkeeping. The sections below cover each step in detail.
How do XACK and the Pending Entries List track in-flight jobs?
When XREADGROUP delivers an entry with the special ID >, Redis records it in the group's Pending Entries List (PEL). Each PEL record holds the entry ID, the name of the consumer it went to, the time of the last delivery, and a delivery counter. The entry stays there, assigned to that consumer, until the consumer calls XACK with the group name and entry ID. A List has no equivalent. Once LPOP returns, the server has no record that the job exists.
XACK only removes the record from the PEL. The entry itself stays in the stream, and other groups can still read it. It is removed only by XDEL or by trimming. This is why acknowledging work and controlling memory are separate concerns.
Restarting a worker
On startup, a worker should first call XREADGROUP with ID 0 instead of >. That call returns entries already pending against its own consumer name, which are jobs it received before crashing but never acknowledged. The worker processes them until the call returns nothing, then switches to > for new work. This only works if the consumer name is stable, such as a pod ordinal or hostname, and not a random UUID per process. With a new name each time, the old entries stay assigned to a consumer that will never return. They then need claiming from outside, and the group collects dead consumer records.
At-least-once delivery
A worker can finish a job and crash before XACK reaches Redis. On restart it receives that job again. Handlers must be idempotent, or they must deduplicate. You can record the entry ID or a business key, such as an order ID, in the same transaction as the job's side effect, and skip any entry already recorded.
import os
import redis
r = redis.Redis(host="localhost", port=6379, decode_responses=True)
STREAM, GROUP = "jobs", "workers"
CONSUMER = os.environ["HOSTNAME"]
def handle(entry_id, fields):
# Placeholder: in ONE database transaction, apply the side effect and insert
# entry_id (or fields["order_id"]) into a processed-keys table with a unique
# constraint. If the key already exists, skip: this is a redelivery.
...
last_id = "0"
while True:
resp = r.xreadgroup(GROUP, CONSUMER, {STREAM: last_id}, count=100)
entries = resp[0][1] if resp else []
if not entries:
break # PEL for this consumer is drained; move on to new work
for entry_id, fields in entries:
last_id = entry_id # advance so a failing entry can't loop forever
if fields: # empty if the entry was XDEL'd or trimmed while pending
handle(entry_id, fields)
# Removes the PEL record only; the entry itself stays in the stream.
r.xack(STREAM, GROUP, entry_id)
while True:
resp = r.xreadgroup(GROUP, CONSUMER, {STREAM: ">"}, count=10, block=5000)
for _stream, entries in resp or []:
for entry_id, fields in entries:
handle(entry_id, fields)
# A crash between handle() and xack() means redelivery on restart,
# which is why handle() must be idempotent.
r.xack(STREAM, GROUP, entry_id)
A redis-py worker that uses a stable consumer name, first replays its own pending entries with XREADGROUP ID 0, then reads new entries with ">", and acknowledges each one only after an idempotent handler has run.
How do you recover jobs from dead consumers and handle poison messages?
A consumer that crashes leaves its entries in the Pending Entries List indefinitely. Nothing returns them to the group, so another worker has to take ownership. The extended form of XPENDING lists each pending entry with its owner, its idle time in milliseconds, and its delivery count. XCLAIM reassigns specific IDs, but only if they have been idle for at least the min-idle-time you pass. XAUTOCLAIM scans the PEL from a cursor and claims every entry past that threshold in one call. That makes it the usual choice for a periodic reaper loop.
Setting the idle threshold is a trade-off. If it is shorter than your slowest legitimate job, a healthy worker's entry gets claimed while that worker is still processing it, and the job runs twice at the same time. If it is too long, jobs owned by a dead consumer sit untouched for that whole window. Base it on your p99 job duration plus a margin. If some jobs run long, have the worker call XCLAIM on its own entry periodically. That resets the idle time and works like a heartbeat.
Claiming increments the delivery count, which makes the count a good signal for poison messages. When a claimed entry's count passes a limit such as five, XADD its fields to a dead-letter stream, then XACK the original so it stops cycling through the group.
XGROUP DELCONSUMER removes a consumer, and any entries it still owns are discarded from the PEL. Claim them first. Autoscaled workers that generate random names make this worse. Every new pod registers a new consumer, and departed consumers stay in the group until you delete them. XINFO CONSUMERS grows without bound, and orphaned entries can only be recovered by claiming them. Stable worker identities avoid this, because a restarted worker comes back under the same name and picks up its own pending entries.
import redis
r = redis.Redis(host="localhost", port=6379, decode_responses=True)
STREAM, GROUP, ME = "jobs", "workers", "reaper-1"
DLQ = "jobs:dead"
IDLE_MS = 300_000 # p99 job duration plus margin (assumed 5 min)
MAX_DELIVERIES = 5 # past this, treat the entry as poison
GONE_MS = 3_600_000 # consumer idle 1h with nothing pending = departed
def reap():
# Extended XPENDING: only entries idle past the threshold
stuck = r.xpending_range(STREAM, GROUP, min="-", max="+", count=100, idle=IDLE_MS)
for p in stuck:
msg_id = p["message_id"]
if p["times_delivered"] >= MAX_DELIVERIES:
# Copy fields to the dead-letter stream, then ack the original
for _, fields in r.xrange(STREAM, min=msg_id, max=msg_id):
r.xadd(DLQ, {**fields, "orig_id": msg_id, "owner": p["consumer"]})
r.xack(STREAM, GROUP, msg_id)
continue
# min_idle_time re-checks idleness, so a heartbeating worker keeps its entry
for _, fields in r.xclaim(STREAM, GROUP, ME, IDLE_MS, [msg_id]):
process(fields) # placeholder for the job handler
r.xack(STREAM, GROUP, msg_id)
def prune_consumers():
for c in r.xinfo_consumers(STREAM, GROUP):
# Delete only consumers owning nothing; otherwise their PEL entries are lost
if c["pending"] == 0 and c["idle"] > GONE_MS:
r.xgroup_delconsumer(STREAM, GROUP, c["name"])
def process(fields):
print("processing", fields)
if __name__ == "__main__":
reap()
prune_consumers()
A redis-py reaper that finds stale entries with extended XPENDING, dead-letters those past five deliveries before XACKing them, reclaims the rest with XCLAIM, and deletes idle consumers that own no pending entries.
How should you trim and monitor a Redis Stream in production?
Acknowledging an entry removes it from the group's Pending Entries List, not from the stream. Every entry stays in memory until something trims it. A List shrinks on every pop. A stream grows until you bound it.
The usual approach is to cap the stream on write with XADD ... MAXLEN (keep the newest N entries) or MINID (drop entries with IDs older than a threshold, which maps to a time window because IDs start with a millisecond timestamp). You can also run XTRIM periodically with the same options. Add ~ to either command for approximate trimming. Redis then removes only whole internal nodes, which is much cheaper than exact trimming, and the stream may keep somewhat more entries than the limit. XDEL removes individual entries, but memory is reclaimed only when an entire internal node is empty, so it is not a substitute for trimming.
Trimming ignores consumer groups. If consumers fall behind the cap, Redis drops entries that were never delivered, and also entries that are still pending. A later claim on those IDs returns no payload. The job is gone, and nothing reports an error. Size the cap for your worst expected backlog, not your average one.
A List gives you one number, LLEN. Streams expose more:
XINFO STREAMreports length and the first and last IDs.XINFO GROUPSreports each group's pending count and last-delivered ID, and lag where your version supports it.XINFO CONSUMERSreports each consumer's pending count and idle time.
Alert on:
- Growing group lag.
- Rising pending count.
- The idle time of the oldest pending entry, which you get from
XPENDING. - Consumers that have been idle long enough to suggest they are dead.
What limits apply to both Streams and Lists?
Both data structures inherit Redis's durability model, so neither is safer than the persistence and replication settings underneath it. Replication is asynchronous by default. If a primary fails before it has shipped recent writes to a replica, the promoted replica will not have them. A lost LPUSH or XADD means a lost job. A lost XACK or LMOVE means the job runs again. With AOF set to fsync once per second, a crash can drop roughly the last second of writes for the same reason. WAIT can reduce this window, but it does not make Redis strongly consistent. Assume at-least-once delivery and make handlers idempotent.
Ordering is guaranteed only within a single list or stream. Once a consumer group spreads entries across several workers, two jobs for the same entity can run concurrently or finish out of order. If you need strict per-entity ordering, partition by entity into separate streams, for example by hashing an order ID into N keys, and give each partition a single active consumer.
In Redis Cluster, each key maps to one hash slot on one shard. A single queue key cannot grow past one node's memory or throughput. To scale, you split the queue across multiple keys yourself and route producers and consumers to the right one.
Neither structure supports delayed jobs or priorities natively:
- Delayed jobs: use a sorted set scored by the due timestamp. A poller moves entries that are due into the work queue.
- Priorities: use one key per priority level.
BLPOPchecks the keys in the order you list them, so higher-priority keys are served first.
If you need durable multi-node replication, very long retention, or complex routing, a dedicated message broker is a better fit than Redis.
Who should use Redis Streams or Lists as a job queue?
Redis-backed queues fit teams that already run Redis and want background job processing without adding and operating another piece of infrastructure. The typical users are backend engineers who build worker pools for tasks like image processing, sending email, webhooks, report generation and provisioning. Their jobs are small enough to serialize into a single entry, and their handlers can be made idempotent.
Lists suit:
- Teams whose jobs are cheap to lose or are re-enqueued automatically, such as cache warm-ups, periodic syncs and metrics flushes.
- Single pools of identical workers, where nobody needs to ask which worker holds which job.
- Teams that want the smallest possible operational surface: one key and one length metric.
Streams suit:
- Teams for whom a dropped job is a correctness bug and who need at-least-once delivery with a record of what is in flight.
- Systems where more than one independent consumer needs the same events, such as a worker pool plus an audit or analytics consumer.
- Teams that want to inspect a stuck job, see who owns it and how often it has been retried, and replay past entries after fixing a bug.
- Operators who can run a reaper loop, set trimming limits and alert on pending counts.
Neither is a good fit for workloads that need strong durability across node failures, retention measured in long periods, strict global ordering across many workers, or routing rules beyond "one queue key per purpose." The limits in the previous section apply there, and a dedicated broker is usually the better choice.
How can Redis Streams and Lists save engineering time?
The main saving is in code and infrastructure you do not have to build or run.
Compared with adopting a separate broker, either structure reuses the Redis deployment, client libraries, monitoring and on-call knowledge you already have. You do not have to provision a new cluster, design a new failover setup or learn a new set of client semantics. For a team with modest queueing needs, that removes a whole category of setup and maintenance work.
Compared with a hand-built reliable List, Streams move bookkeeping into the server. The list example above needed:
- Heartbeat keys and a background refresher to tell slow workers from dead ones.
- A reaper that scans processing lists by key pattern.
- Embedded metadata for retry counts, because the list cannot count deliveries.
- Value-based acknowledgement with
LREM, which is O(N) and cannot tell duplicate payloads apart.
With Streams, the PEL records the owner, idle time and delivery count. XAUTOCLAIM or XCLAIM replaces the scan-and-requeue logic, and XACK acknowledges by unique ID. You still write the reaper loop and the dead-letter policy, but that code is smaller and leaves fewer edge cases to test.
Debugging is faster. When a job goes missing in a plain List, there is often nothing left to inspect. With a Stream, XPENDING shows which consumer holds the entry, how long it has been idle and how many times it has been delivered. XRANGE shows the payload. That turns an investigation into a few commands.
Reprocessing is cheaper. After you fix a handler bug, you can create a new consumer group starting at an earlier ID, or read a range with XRANGE, and rerun the affected entries. You do not need to reconstruct the jobs from logs or ask producers to resend them. Adding a second consumer, such as an audit pass, also needs no change to producers, because a new group reads the same entries.
Lists save time in the opposite direction. When jobs are disposable, a plain LPUSH/BRPOP queue is the least work to write, deploy and operate. Choosing Streams there buys guarantees nobody needs and adds permanent operational work. The time saving comes from matching the structure to how much a lost job actually costs.
Redis Streams vs Lists: when should you use each?
Pick a List when losing a job is cheap: cache warm-ups, metrics flushes, or work a scheduler will re-enqueue on its next tick anyway. A List is one key, popped entries free their memory immediately, and there is no group state to create or inspect. If you have a single pool of identical workers and throughput matters more than per-job guarantees, that simplicity is worth more than anything Streams add.
Pick a Stream when a lost job is a bug: payments, emails, provisioning. Acks, the Pending Entries List and delivery counts give you at-least-once delivery and a basis for dead-lettering. Retained entries let you replay history and support several independent consumer groups.
Avoid Streams for fire-and-forget queues. You would still have to create groups, run a reaper loop, route poison messages, trim the stream and alert on pending counts, and none of it pays off if nobody cares about a dropped job.
Development cost cuts both ways. Streams save you from hand-building the reliable-list pattern (processing lists, timeouts, retry counters), but you take on that operational work permanently.
| Concern | List | Stream |
|---|---|---|
| Delivery semantics | At-most-once (plain pop) | At-least-once |
| Ack | None built in | XACK |
| Crash recovery | Hand-built with LMOVE |
PEL plus claiming |
| Delivery count | Track yourself | Tracked per entry |
| Fan-out | No, each job goes to one consumer | Multiple groups |
| Replay | No, popped entries are gone | Yes, until trimmed |
| Observability | Length only | Pending, idle time, lag |
| Memory growth | Shrinks as consumed | Grows until trimmed |
| Operational effort | Low | Moderate to high |
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to redis streams vs lists, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Consumers learn about a breaking change at review time rather than from production errors.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.