Dead Letter Queue Redrive: Replaying Failed Messages Safely

What dead letter queue redrive is and how messages get there

A dead letter queue (DLQ) holds messages that a consumer repeatedly failed to process. Redrive moves those messages back to a source queue for another attempt. The DLQ exists to isolate failures. Without one, a poison message that always throws keeps getting redelivered. It burns consumer capacity and, on ordered queues, blocks every message behind it.

In SQS, a redrive policy on the source queue names a target DLQ and a maxReceiveCount, for example {"deadLetterTargetArn": "<dlq-arn>", "maxReceiveCount": "5"}. Each receive increments the message's ApproximateReceiveCount attribute. When a receive would exceed the limit, SQS moves the message to the DLQ. The body and message attributes are preserved.

RabbitMQ uses a dead-letter exchange (DLX), set with the queue arguments x-dead-letter-exchange and, optionally, x-dead-letter-routing-key. A message is dead-lettered in three cases:

• it is rejected or nacked with requeue=false
• its TTL expires (x-message-ttl)
• the queue overflows its length limit (x-max-length)

RabbitMQ adds an x-death header recording the source queue, the reason (rejected, expired, maxlen) and a count. Redrive tooling should read this header before replaying.

Choosing maxReceiveCount, or any retry budget, is a trade-off:

• A low value sends healthy messages to the DLQ during a brief downstream outage.
• A high value leaves poison messages cycling longer before they are isolated.

Copyimport json
import boto3
import pika

# --- SQS: attach a redrive policy to the source queue ---
sqs = boto3.client("sqs", region_name="us-east-1")
sqs.set_queue_attributes(
    QueueUrl="",
    Attributes={
        # After 5 receives without a delete, SQS moves the message to the DLQ
        "RedrivePolicy": json.dumps({
            "deadLetterTargetArn": "arn:aws:sqs:us-east-1:123456789012:orders-dlq",
            "maxReceiveCount": "5",
        })
    },
)

# --- RabbitMQ: declare a DLX, a DLQ, and a source queue that dead-letters into it ---
conn = pika.BlockingConnection(pika.ConnectionParameters("rabbitmq.internal"))
ch = conn.channel()
ch.exchange_declare(exchange="orders.dlx", exchange_type="direct", durable=True)
ch.queue_declare(queue="orders.dlq", durable=True)
ch.queue_bind(queue="orders.dlq", exchange="orders.dlx", routing_key="orders.dead")

ch.queue_declare(
    queue="orders",
    durable=True,
    arguments={
        "x-dead-letter-exchange": "orders.dlx",       # where rejected/expired/overflowed messages go
        "x-dead-letter-routing-key": "orders.dead",   # optional; defaults to the original key
        "x-message-ttl": 600000,                      # 10 min TTL triggers reason 'expired'
        "x-max-length": 100000,                       # overflow triggers reason 'maxlen'
    },
)

# Redrive tooling should inspect x-death before replaying a dead-lettered message
method, props, body = ch.basic_get(queue="orders.dlq")
if method:
    for death in (props.headers or {}).get("x-death", []):
        print(death.get("queue"), death.get("reason"), death.get("count"))
conn.close()

Configuring an SQS redrive policy with boto3 and RabbitMQ dead-letter queue arguments with pika, then reading the x-death header from a dead-lettered message.

Who needs a redrive process and when to replay versus discard

This section is for backend and platform engineers who own queue consumers or write the on-call runbook for them. If a DLQ alarm pages you, you need a triage process before you need a redrive button. Replaying everything at once tends to turn one incident into two.

Triage before you replay

Sample the dead-lettered messages and sort them by cause. The failure metadata and consumer logs usually point to one of three categories:

• Poison messages. The payload is malformed or doesn't match the schema the consumer expects. Retrying can't fix these because the input itself is wrong.
• Transient failures. A downstream outage or throttling ran through the retry budget. The message is valid, and it failed because of timing.
• Bug-induced failures. The consumer code mishandled valid input. These look like poison messages until you see that many differently shaped payloads fail the same way.

When not to replay

Some messages are valid but out of date. Replaying an old price update can overwrite a newer price. Replaying a fulfilment event for an order that was later cancelled can ship goods nobody wants. Before replaying, compare each message's timestamp or version against current state.

Discarding is a business decision, not an engineering one. Get explicit sign-off from the owner of the affected data, and record which messages were dropped and why.

Prerequisites: fix the cause, then make consumers idempotent

A redrive is a production change, so treat it like one. Messages that reached the DLQ will fail again if you replay them before the cause is gone, and they will consume the retry budget they have left. Deploy the consumer fix first, or confirm that the downstream dependency has recovered and is stable. Then move a small sample of DLQ messages, five to twenty for example, and check the results against the source of truth before you start any bulk redrive. Choose the sample on purpose. It should include the oldest messages and every distinct error type you found during triage.

Idempotency is a hard requirement for redrive. Many messages in a DLQ were partly processed before they failed. The charge went through but the acknowledgement did not, or the email was sent but the status update timed out. If you replay these without a guard, customers get charged twice, emails go out twice, and counters drift.

You can add that guard in three common ways:

• Dedupe key. Derive a stable key from the business operation, such as order_id plus the action. Do not use the broker's message ID, because it can change on republish.
• Idempotency table. Insert the key and the side effect in the same transaction. A primary-key violation tells you the work is already done.
• Conditional write. Update the row only if version = expected. A replay that arrives after the state has moved on becomes a no-op.

Each option has costs. The idempotency table grows over time and needs a retention policy. Each message now costs an extra write and sometimes an extra read, which adds latency on the hot path. The retention window is a real risk. If the table expires keys after seven days and you redrive messages that are three weeks old, the guard no longer protects you. Size the window to cover your longest realistic DLQ retention, or fall back to conditional writes on version.

Copy# Idempotency table + conditional write in one transaction (PostgreSQL, psycopg2).
# Schema assumed:
#   processed_messages(idempotency_key text PRIMARY KEY, processed_at timestamptz)
#   orders(order_id text PRIMARY KEY, status text, version int)

def handle_charge(conn, msg, payment_client):
    # Key comes from the business operation, not the broker message ID,
    # so it stays the same when the message is republished from the DLQ.
    key = f"{msg['order_id']}:charge"

    with conn:  # psycopg2: commit on success, roll back on exception
        with conn.cursor() as cur:
            # Claim the key. If it already exists, this work was done before.
            cur.execute(
                "INSERT INTO processed_messages (idempotency_key, processed_at) "
                "VALUES (%s, now()) ON CONFLICT (idempotency_key) DO NOTHING",
                (key,),
            )
            if cur.rowcount == 0:
                return "duplicate"  # replay of completed work: ack and move on

            # Conditional write: apply only if the row is still in the state
            # this message expects. A late replay becomes a no-op.
            cur.execute(
                "UPDATE orders SET status = 'charged', version = version + 1 "
                "WHERE order_id = %s AND version = %s AND status = 'pending'",
                (msg["order_id"], msg["expected_version"]),
            )
            if cur.rowcount == 0:
                return "stale"  # state moved on; nothing to do

            # The external call is not covered by this transaction. Pass the same
            # key to the provider so that it can also dedupe a retried request.
            # If this call raises, the transaction rolls back and the key is released.
            payment_client.charge(msg["order_id"], msg["amount"], idempotency_key=key)
            return "charged"

A consumer claims a business-derived idempotency key and applies a version-checked update in the same transaction, so a replayed DLQ message results in a no-op instead of a second charge.

How to implement redrive in SQS and RabbitMQ

SQS: native redrive

SQS attaches a redrive policy to the source queue: after maxReceiveCount failed receives, the message moves to the DLQ. To replay, start a message move task from the DLQ. It sends messages back to their original source queue by default, or to a custom destination such as a dedicated replay queue. Only visible messages move, so stop any process that is reading the DLQ first. Messages it holds in flight are skipped.

Retention matters because, on standard queues, a message keeps its original enqueue timestamp when it moves to the DLQ. If the DLQ's retention is equal to or shorter than the source queue's, messages can expire before anyone looks at them. Set the DLQ's retention to the maximum. After a redrive, the source queue's visibility timeout and receive count apply again. If the fix isn't deployed, the messages loop back to the DLQ.

RabbitMQ: build your own

RabbitMQ has no redrive command. You write a shovel-style consumer instead:

1. Read from the DLQ with manual acks and a small prefetch.
2. Republish each message to the original exchange. Take its routing key from the first x-death entry, which records the queue and routing key where the message died.
3. Strip or rename x-death before republishing. If you keep it, death counts accumulate, and retry logic that reads them will misjudge fresh failures.
4. Enable publisher confirms. Ack the DLQ message only after the broker confirms the republish. If you ack first and the publish is lost, the message is gone for good.

Copy producer ──► [source queue] ──► consumer
                   │ reject/expire
                   ▼
                [ DLQ ] ──► replay script ──► source exchange
                                  │
                                  └──(optional)──► [replay queue] ──► consumer

Copyimport pika

DLQ = "orders.dlq"
params = pika.URLParameters("amqp://replay:changeme@rabbitmq.internal:5672/%2F")
conn = pika.BlockingConnection(params)
ch = conn.channel()
ch.confirm_delivery()  # basic_publish now waits for the broker's ack or nack

# Only process what is in the DLQ right now, so requeued failures don't loop forever.
pending = ch.queue_declare(queue=DLQ, passive=True).method.message_count

for _ in range(pending):
    # One message at a time with manual acks: at most one message is ever unconfirmed.
    method, props, body = ch.basic_get(queue=DLQ, auto_ack=False)
    if method is None:
        break  # DLQ drained
    headers = dict(props.headers or {})
    deaths = headers.pop("x-death", None)  # strip so death counts start fresh
    if not deaths:
        ch.basic_nack(method.delivery_tag, requeue=True)  # no origin info; leave it in the DLQ
        continue
    last = deaths[0]  # first entry = most recent death: original exchange and routing key
    headers["x-replayed-from"] = DLQ
    headers["x-prior-death-count"] = last.get("count", 1)  # keep history for humans, not retry logic
    props.headers = headers
    try:
        ch.basic_publish(exchange=last["exchange"],
                         routing_key=last["routing-keys"][0],
                         body=body, properties=props,
                         mandatory=True)  # unroutable messages raise instead of vanishing
    except (pika.exceptions.UnroutableError, pika.exceptions.NackError):
        ch.basic_nack(method.delivery_tag, requeue=True)  # publish not confirmed: keep original
        continue
    ch.basic_ack(method.delivery_tag)  # ack only after the broker confirmed the republish

conn.close()

A pika-based replay script that drains a RabbitMQ DLQ, republishes each message to the exchange and routing key recorded in its most recent x-death entry with x-death stripped, and acks the DLQ copy only after a publisher confirm.

Controlling blast radius during a replay

A replay is a burst of traffic you choose to send, so you can decide how big it is. Move messages in small batches, such as 50 at a time, and pause between batches. Size the rate against the slowest dependency the consumer calls, not against the queue's throughput. The queue can absorb almost any rate, but the database or third-party API behind the consumer usually cannot.

While the replay runs, track three signals:

• Consumer error rate. Compare it to the baseline for live traffic, not to zero.
• DLQ re-entry. Count replayed messages that land back in the DLQ. Tag each one with a replay attempt header so you can tell re-entries apart from fresh failures.
• Downstream load. Watch latency, connection pool saturation, and throttling responses from dependencies.

Set stop conditions before you start, and have the loop check them after every batch. For example, halt if re-entry exceeds 5% of a batch, if the error rate doubles, or if downstream p99 latency crosses an agreed threshold. These numbers are illustrative; choose your own. If a message fails a second time, park it instead of retrying it again. This keeps a broken message from cycling between the queues.

Replayed messages arrive late and out of order. Consumers should compare an entity version or event timestamp against current state, discard updates that are already superseded, and send anything ambiguous to a reconciliation path.

You can redrive to the original queue or to a separate replay queue. The original queue needs no new infrastructure, but replayed messages compete with live traffic and are hard to tell apart in metrics. A dedicated replay queue with its own consumer keeps the two isolated and lets you throttle the replay independently. The cost is one more queue and one more deployment to maintain.

The loop shown below reads a batch, checks the stop conditions, publishes the messages, and then sleeps before the next batch.

Copyimport time

BATCH_SIZE = 50          # small, fixed batches
PAUSE_SECONDS = 10       # give consumers time to process and metrics time to settle
MAX_REENTRY_RATIO = 0.05 # illustrative thresholds; agree on your own before starting
MAX_ERROR_MULTIPLIER = 2.0
MAX_P99_MS = 800

def redrive(dlq, target, parking, metrics):
    # Baseline is live-traffic error rate measured before the replay, not zero.
    baseline = metrics.consumer_error_rate()
    while True:
        batch = dlq.receive(max_messages=BATCH_SIZE)
        if not batch:
            return "done: DLQ drained"
        reentries_before = metrics.dlq_count(header="replay-attempt")
        for msg in batch:
            attempt = int(msg.headers.get("replay-attempt", "0"))
            if attempt >= 1:
                parking.send(msg)  # already failed after a replay: park it, never loop
            else:
                msg.headers["replay-attempt"] = str(attempt + 1)
                target.send(msg)
            dlq.delete(msg)        # delete only after the send above succeeded

        time.sleep(PAUSE_SECONDS)

        # Stop conditions are checked after every batch, before sending more.
        reentered = metrics.dlq_count(header="replay-attempt") - reentries_before
        if reentered / len(batch) > MAX_REENTRY_RATIO:
            return f"stopped: {reentered} of {len(batch)} re-entered the DLQ"
        if metrics.consumer_error_rate() > baseline * MAX_ERROR_MULTIPLIER:
            return "stopped: consumer error rate doubled vs baseline"
        if metrics.downstream_p99_ms() > MAX_P99_MS:
            return "stopped: downstream p99 latency over threshold"

A broker-agnostic redrive loop that moves fixed-size batches, pauses between them, tags each message with a replay-attempt header, parks messages that fail a second time, and halts on DLQ re-entry, error-rate, or downstream-latency thresholds checked after every batch.

Alternatives to manual DLQ redrive

Manual redrive is not the only way to recover failed work. Each alternative moves the retry decision somewhere else, and that affects how quickly failures get noticed.

Scheduled automatic redrive is convenient because it handles outages with no one involved. That same convenience is the problem. A malformed payload fails, gets redriven, and fails again, and the DLQ never empties long enough for anyone to look. Put a replay-count attribute on each message and stop redriving once it passes a threshold.

In-consumer retries and delayed retry queues catch transient errors before a message ever reaches the DLQ. With both in place, whatever does land in the DLQ is more likely to be a real bug.

Exporting payloads to object storage gets around retention limits. It also creates a second copy of the data, which must follow the same encryption, access control, and deletion requirements as the primary store. Personal data needs particular care here.

Reconciliation removes the need to replay at all, because it recomputes state directly. It only works if a source of truth exists. Manual redrive is still the slowest of these options, but it is the safest, because a person checks the cause before anything is reprocessed.

Operational hygiene and impact on cost and engineering time

Alert on two signals, not one. DLQ depth tells you something is failing. The age of the oldest message tells you nobody has looked yet. In SQS, ApproximateAgeOfOldestMessage on the DLQ covers the second. In RabbitMQ, watch the dead letter queue's length, and record the arrival time in a header so the age can be checked. Page on age as well as depth, because a slow trickle of failures never trips a depth threshold.

Set DLQ retention longer than your realistic triage time, and check how your broker measures it for dead-lettered messages. Then decide in advance what happens at expiry: archive to durable storage or discard deliberately. Silent expiry should never be the outcome.

Keep a runbook, and log every replay: which message IDs, who triggered it, and which fix or commit it followed. That log is how you answer "did this order get charged twice?" weeks later.

The upfront cost is real. It includes idempotency work in each consumer, replay tooling with filters and rate limits, and sometimes a separate replay queue or consumer pool.

Illustrative estimate (assumed numbers, not a benchmark)

Hours saved per year = incidents × (ad hoc hours − runbook hours)

Assume:

• 6 DLQ incidents a year
• 10 engineer-hours each when handled ad hoc (scripting plus cleaning up duplicate side effects)
• 1 hour each with the runbook

Saved: 6 × (10 − 1) = 54 hours a year. Against an assumed 60-hour build, the tooling pays back in just over a year. It also shortens every recovery after that.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to dead letter queue redrive, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Consumers learn about a breaking change at review time rather than from production errors.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.