Dead Letter Queue Guide: Handling Poison Messages Safely

What a dead letter queue is and what counts as a poison message

A dead letter queue (DLQ) is a separate destination where a consumer parks messages it has decided not to process further, so the main stream can keep moving. A poison message is a message that fails deterministically: every attempt produces the same error, because the cause is in the message or its context, not in the environment. Typical causes are a malformed payload, a schema mismatch, a violated business invariant such as a negative quantity, or a reference to data that does not exist.

Transient failures look similar in logs but behave differently. They depend on time and conditions, so a later attempt can succeed.

Most DLQ pain comes from mixing up these two categories. If a consumer treats an outage as poison, the DLQ fills with valid messages that need bulk replay later. If it treats poison as transient, retries consume resources and never succeed.

Without a DLQ, one poison message is redelivered indefinitely. Each attempt uses compute and produces another error log. In ordered or partitioned consumers, the message also blocks everything behind it on that partition. Consumer lag then grows for healthy messages, and alerts fire repeatedly for one stuck record until on-call engineers start ignoring them.

Any team running queue-based or log-based consumers on business-critical events needs a deliberate plan for this situation.

Who should use a dead letter queue

A DLQ fits teams whose consumers process asynchronous messages where losing a message is unacceptable but stalling the whole stream is also unacceptable. The most common cases:

• Teams that own business-critical event consumers, such as order, billing, fulfilment, or account-update pipelines. Every message matters, so dropping failures is not an option, and a single bad record must not hold up thousands of good ones.
• Teams consuming data produced by other teams or external systems. When you don't control the producer, schema drift and malformed payloads will reach you eventually. A DLQ gives you a place to hold that input while you coordinate a fix.
• Teams running partitioned or log-based consumers at meaningful volume. Here a stuck message blocks a whole partition, and the lag grows quickly. Dead-lettering is how you unblock it without losing the record.
• Platform teams building shared consumer libraries or frameworks. Standardising classification, headers, and replay tooling once saves every service team from reinventing it with different gaps.

A DLQ only pays off if someone will act on it. The team that adopts it must also be prepared to own alerts, triage, and replay, as described later in this guide.

When to avoid a dead letter queue

A DLQ is not always the right tool, and in some situations it makes things worse:

• Strict ordering is required and handlers cannot tolerate gaps. If skipping one event corrupts state, such as a balance or a state machine, a plain DLQ is unsafe. Block the consumer or park the affected key instead. The ordering section below covers this in detail.
• Nearly all failures are transient. If failures come from a dependency being down, a circuit breaker or paused consumer is the better response. A DLQ in that situation fills with healthy messages that all need replay.
• Nobody will own it. A DLQ without alerts, retention planning, and a named owner becomes a place where messages quietly expire. That is data loss with extra steps. If you cannot staff ownership, failing loudly is often safer.
• The work is synchronous request/response. When a caller is waiting for a result, return an error to the caller so it can decide what to do. Parking the request for later processing usually leaves the caller with no answer.
• The messages are genuinely disposable. For low-value telemetry, best-effort metrics, or events that a later message always supersedes, drop-and-log may be enough. The DLQ's operational cost may not be worth it.
• Copying the data is restricted. If the payload carries regulated data that cannot be duplicated into another store with different retention, either apply the same controls to the DLQ or redact before storing. If neither is possible, don't add the DLQ until that is resolved.

How a dead letter queue works: message flow, classification, retry, and dead-lettering

At its core, a dead letter queue works by separating failed messages from the main stream according to why they failed. Retryable failures go back for another attempt. Permanent or exhausted failures go to a holding destination where people can inspect them and replay them later.

A message starts on the primary topic or queue and reaches a consumer, which tries to process it. The consumer classifies each failure. A transient failure, such as a timeout or a throttled downstream call, goes to a retry stage. That stage might be a delayed redelivery or a separate retry topic with a backoff interval, and the backoff gives the dependency time to recover. A permanent failure, such as a schema violation or a reference to an entity that doesn't exist, skips retries and goes straight to the dead letter queue, because retrying it only burns capacity. A message that runs out of retries ends up in the same place. From the DLQ, a replay path sends corrected or re-validated messages back to the original destination, so they pass through normal processing again instead of a side channel.

The DLQ write can live in one of two places:

• Broker-native dead-lettering. The broker counts deliveries and moves a message to the DLQ once that count passes a configured limit. It takes little effort to set up and keeps working even if the consumer crashes in the middle of processing. The drawback is how little it records: usually the payload, the delivery count and perhaps a timestamp. You don't get the exception, the stack trace or the consumer version that failed.
• Application-level dead-lettering. The consumer keeps its own attempt count in a header and publishes failures to the DLQ itself. This lets it attach whatever context triage will need. It also creates new failure paths. The worst one is a failed DLQ write. If the consumer acknowledges the original anyway, the message is lost. If it holds back the acknowledgement, the poison message keeps blocking the partition or redelivering. Make that choice on purpose ahead of time.

Copy+---------------------+     replay: corrected / re-validated messages
| Primary topic/queue |<-----------------------------------------------+
+---------------------+                                                |
           |  ^                                                        |
           v  | broker redelivers (delivery count++)                   |
+---------------------+  transient    +------------------------+       |
|      Consumer       |-------------->|      Retry stage       |       |
|  classifies failure |<--------------| delayed redelivery or  |       |
+---------------------+ after backoff | retry topic + backoff  |       |
    |            |                    +------------------------+       |
    | native:    | app-level: permanent           |                    |
    | delivery   | failure, or header attempt     | retries            |
    | count >    | count exhausted; consumer      | exhausted          |
    | limit      | publishes with context         |                    |
    v            v                                v                    |
+------------------------------------------------------------+         |
|                  Dead letter queue (DLQ)                   |---------+
| native:    payload, delivery count, maybe timestamp        |
| app-level: + exception, stack trace, consumer version      |
+------------------------------------------------------------+
         |
         | if app-level DLQ write fails
         v
+------------------------------------------------------------+
| ack original   -> message lost                             |
| withhold ack   -> poison blocks partition / redelivers     |
| choose one on purpose, ahead of time                       |
+------------------------------------------------------------+

Message flow from the primary queue through consumer classification, backoff retries, and broker-native or application-level dead-lettering, with the replay path back to the original destination and the ack decision when a DLQ write fails.

Implementing failure classification and bounded message retry

Classify every failure explicitly instead of sending all exceptions down one path. The decision is short: if the payload fails deserialization or schema validation, dead-letter it immediately. If the error is transient, such as a timeout, connection reset, throttling response, or 5xx from a dependency, retry it. If the error is unknown, treat it as retryable but count the attempt. A malformed message fails the same way on every attempt, so retrying it only burns time and holds up the partition.

Bound retries with an attempt limit and exponential backoff with jitter. Compute min(cap, base × 2^attempt), then pick a random delay up to that value. The randomness stops consumers that hit the same blip from retrying in lockstep. Store the attempt count with the message, in a header, not in consumer memory. If it lives in memory, a rebalance or restart resets it, and a poison message can retry indefinitely.

Log-based systems cannot delay a single message in place without stalling its partition. The usual pattern is a set of tiered retry topics with increasing delays, for example 1 minute, 10 minutes, then 1 hour. Each tier's consumer waits until the message is due, reprocesses it, and forwards failures to the next tier and finally to the DLQ.

Tune the limits against the dependency's realistic recovery time. Too few attempts dead-letter messages that would have succeeded after a failover. Too many delay detection of real poison messages.

When a dependency is fully down, stop consuming. A circuit breaker or paused consumer leaves messages where they are. Retrying to exhaustion instead fills the DLQ with healthy messages that all need replay.

Copyimport random

MAX_ATTEMPTS = 5          # size this to the dependency's realistic recovery time
BASE_DELAY_S = 1.0
CAP_DELAY_S = 60.0

class PoisonError(Exception): pass      # deserialization / schema validation failures
class TransientError(Exception): pass   # timeouts, resets, throttling, 5xx

def backoff_with_jitter(attempt):
    # "Full jitter": random delay up to min(cap, base * 2^attempt)
    ceiling = min(CAP_DELAY_S, BASE_DELAY_S * (2 ** attempt))
    return random.uniform(0, ceiling)

def handle(message, process, send_to_dlq, schedule_retry):
    # Attempt count travels with the message so restarts and rebalances cannot reset it
    attempt = int(message.headers.get("x-attempt", "0"))
    try:
        process(message)
    except PoisonError as err:
        # Fails identically on every attempt: dead-letter now, never retry
        send_to_dlq(message, reason=f"non-retryable: {err}")
    except Exception as err:
        # TransientError and unknown errors are retried, but every attempt is counted
        attempt += 1
        if attempt >= MAX_ATTEMPTS:
            send_to_dlq(message, reason=f"exhausted {attempt} attempts: {err}")
            return
        message.headers["x-attempt"] = str(attempt)
        # In log-based systems, this republishes to a delayed retry tier
        # instead of sleeping and stalling the partition
        schedule_retry(message, delay_s=backoff_with_jitter(attempt))

Consumer-side failure classification that dead-letters poison messages immediately, retries transient and unknown errors with a header-stored attempt count, and applies capped exponential backoff with full jitter.

What to capture when dead-lettering, and how to keep the DLQ observable and owned

A dead-lettered message is only useful if someone can diagnose it later without reproducing the failure. Store the unmodified original payload and attach the context as headers or an envelope. An example header set:

• dlq.source: orders.v1, partition 7, offset 48211 (or the broker's message ID)
• dlq.error.class: SchemaValidationError, plus a truncated stack trace or error summary
• dlq.attempts: 5, first and last failure timestamps
• dlq.consumer: billing-projector, version 2.3.1
• trace-id: the correlation ID carried from the producer

Never re-serialize or "fix" the payload before storing it. Replay and root-cause analysis both depend on the exact bytes that failed.

A DLQ copies whatever the source carried, including personal or payment data. Give it the same access controls and retention limits as the source, or redact deliberately and record which fields were removed.

For operations, alert on both DLQ depth and arrival rate, because a sudden spike usually means a deploy or schema change rather than one bad record. Set retention longer than realistic triage time, including weekends. Build dashboards grouped by error class. Assign a named owning team. A DLQ without an owner fills up quietly, ages out under retention, and becomes a data-loss bucket that nobody notices.

Copyimport time, traceback
from confluent_kafka import Producer

dlq = Producer({"bootstrap.servers": "kafka-1.internal:9092"})

CONSUMER_NAME = "billing-projector"
CONSUMER_VERSION = "2.3.1"

def dead_letter(msg, exc, attempts, first_failure_ts):
    """Forward a failed Kafka message to this consumer's own DLQ topic."""
    # Carry over the original headers so the producer's trace-id survives.
    headers = list(msg.headers() or [])
    summary = "".join(traceback.format_exception_only(type(exc), exc)).strip()
    stack = "".join(traceback.format_exception(type(exc), exc, exc.__traceback__))
    headers += [
        ("dlq.source.topic", msg.topic()),
        ("dlq.source.partition", str(msg.partition())),
        ("dlq.source.offset", str(msg.offset())),
        ("dlq.error.class", type(exc).__name__),
        ("dlq.error.summary", summary[:500]),
        # Keep the tail of the trace: that is where the failing frame is.
        ("dlq.error.stack", stack[-2000:]),
        ("dlq.attempts", str(attempts)),
        ("dlq.first.failure.ms", str(first_failure_ts)),
        ("dlq.last.failure.ms", str(int(time.time() * 1000))),
        ("dlq.consumer", CONSUMER_NAME),
        ("dlq.consumer.version", CONSUMER_VERSION),
    ]
    # Pass the original key and value bytes through untouched.
    # Do not decode, re-serialize or "fix" them. Replay needs the exact bytes.
    dlq.produce(
        "orders.v1.billing-projector.dlq",  # per-consumer DLQ, so ownership is clear
        key=msg.key(),
        value=msg.value(),
        headers=headers,
    )
    # Block until the broker acknowledges the write. Commit the source offset
    # only after this returns, or a crash here would lose the message silently.
    dlq.flush(10)

A Kafka consumer helper, written with confluent-kafka-python, that sends a failed message to a per-consumer DLQ topic with its original key and value bytes unchanged and attaches source, error, attempt, timing and consumer-version headers.

DLQ replay as a controlled, idempotent operation

Replay is only safe if the consumer is idempotent. A message that partially succeeded before failing (charged the card, then timed out writing the order) will run that side effect again on replay. Before replaying anything, confirm that the handler derives a stable deduplication key from the message, such as a producer-assigned event ID or an idempotency token passed to the downstream API. The handler should record that key in the same transaction as its state change and use upserts instead of blind inserts. When the handler sees a key it has already processed, it acknowledges the message and does nothing else.

Idempotency has costs. Each message needs an extra lookup, which adds latency. The dedup store grows with throughput. Most teams expire keys after a fixed window, so a replay older than that window can slip through and be applied twice. Check the age of the oldest dead-lettered message against your retention period before you start.

The replay procedure

1. Fix the root cause and deploy the fix. Replaying into the same bug only moves messages back into the DLQ.
2. Replay in small, rate-limited batches. A dependency that just recovered can fall over again under a sudden backlog.
3. Send messages either to the original destination or to a dedicated replay topic that the same consumer reads at lower priority.
4. Add a replay header and a replay count to each message so that logs and metrics can tell replayed traffic from live traffic.
5. Decide in advance what happens when a replayed message fails again. Re-dead-letter it with an incremented count. Once the count passes a threshold, stop replaying it automatically and send it to manual review, so it cannot loop forever.

CopyMAX_REPLAYS = 3
BATCH_SIZE = 50
BATCH_PAUSE_S = 5  # gives a recovering dependency room to breathe

def handle(msg, db, payments):
    key = msg.headers["event_id"]  # stable, producer-assigned dedup key
    with db.transaction() as tx:
        # Atomic claim. If the key already exists, this is a duplicate.
        if not tx.insert_if_absent("processed_events", key=key):
            return  # already applied: ack and do nothing else
        # Downstream dedupes on the same token, so a retried charge is a no-op.
        payments.charge(msg.body["amount"], idempotency_key=key)
        tx.upsert("orders", id=msg.body["order_id"], status="paid")
    # An exception rolls back the claim, so a later retry can still run.

def replay(dlq, target, manual_review):
    while batch := dlq.receive(max_messages=BATCH_SIZE):
        for msg in batch:
            count = int(msg.headers.get("x-replay-count", 0))
            if count >= MAX_REPLAYS:
                manual_review.send(msg)  # stop automatic replay
            else:
                msg.headers["x-replayed"] = "true"  # separates replayed from live traffic
                msg.headers["x-replay-count"] = str(count + 1)
                target.send(msg)  # original destination or replay topic
            dlq.ack(msg)  # ack only after the message is forwarded
        sleep(BATCH_PAUSE_S)  # rate-limit between batches

Pseudocode for an idempotent handler that claims a dedup key in the same transaction as its state change, plus a rate-limited replay loop that tags messages with a replay header and sends them to manual review once they pass a replay-count threshold.

Ordering semantics: when to skip a message and when to block the key

A dead letter queue breaks ordering. When a consumer of a partitioned or keyed stream dead-letters message N for key account-42, message N+1 for that key is processed next as if N never existed. For many workloads this is fine. For some it silently corrupts state.

Skipping is a correctness bug when each event depends on the one before it. Examples are account balances, inventory counts, and state machines such as order lifecycles, where a "shipped" event applied without its "paid" event produces an invalid state. In these domains, park the key. You can also make the handler detect the gap from a sequence number and refuse to advance.

Skipping is acceptable when events are independent or commutative. Examples are notifications, analytics, audit logs, and last-write-wins updates that carry a timestamp.

A simple DLQ pattern is safe only when two things are true: the topic has no ordering contract, or the handlers tolerate gaps. If neither holds, extend the pattern with per-key parking. Treat a parked key as an incident. A parked key holds back every later event for that entity, so its lag keeps growing until someone resolves the poison message.

Triage runbook, alternatives, and impact on cost and engineering time

That someone needs a runbook, not improvisation.

Triage runbook

1. Group by failure class, not by message. Sort by count and first-seen time, because one schema change can account for thousands of entries.
2. Decide per group: fix and replay (consumer bug, now deployed), transform and replay (correctable payload, rewritten by a reviewed script), or discard with a written justification such as "test traffic" or "superseded by later event."
3. Record the outcome with class, count, decision, and owner. When the same class appears in a second incident, turn it into a code fix or producer-side schema validation so it stops reaching the DLQ.

Alternatives and complements

Cost versus time saved

The build cost covers dead-lettering code paths, idempotency storage, dashboards, and on-call ownership. The payoff comes from collapsing many failures into a few diagnosable patterns and making recovery a repeatable procedure.

Illustrative estimate (assumed numbers, not a benchmark): Hours saved per year = incidents × (ad-hoc triage hours − runbook triage hours). Assume 12 incidents, 6 h ad hoc, and 1.5 h with the runbook: 12 × (6 − 1.5) = 54 h/year. Compare that figure against your build and maintenance estimate.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to dead letter queue, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• The dependency graph answers "what does this touch?" in seconds instead of a manual audit.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• You can size a change's blast radius before merging, avoiding surprise cross-team firefighting.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.