Poison Message Handling: Detect, Retry, and Quarantine Safely

What a poison message is and why retries alone don't fix it

A poison message fails deterministically: every delivery hits the same bug, so every retry fails the same way. Typical causes are a payload that violates the expected schema, an edge case the code never handled, a body too large to process within memory limits, or a reference to an entity that has since been deleted. A transient failure, by contrast, depends on conditions outside the message, and a later attempt can succeed.

Effective poison message handling therefore starts with classification. Retrying without first deciding which kind of failure occurred just repeats the same work. Any team running queue consumers with side effects needs this, because the failures compound:

• Hot loops. Nack-with-requeue returns the message immediately, and it is redelivered again.
• Head-of-line blocking. On ordered or single-consumer queues, valid messages wait behind the poison one.
• Crash loops. An unhandled exception or OOM kills the consumer, which restarts and fetches the same message.
• Noise. Repeated stack traces bury real incidents.

Illustrative estimate (assumed inputs, not measured data): suppose a single consumer with prefetch 1, a failing attempt that takes 200 ms, and immediate redelivery. One poison message then produces about 5 attempts per second, or 18,000 per hour. It occupies 100% of that consumer's capacity. At an assumed 2 KB of logs per failure, it also generates roughly 36 MB of log output per hour.

Who should use this approach

The full pattern described here (classification, capped retries with backoff, a context-rich DLQ, and replay tooling) is aimed at teams in these situations:

• Consumers with side effects. If processing a message writes to a database, charges a payment, sends a notification, or calls another service, uncontrolled retries cause duplicate effects and confused state.
• Queues carrying data you cannot lose. Orders, account changes, and billing events need a place to land when they fail, not a silent discard.
• Consumers that run unattended. If nobody is watching the logs in real time, a hot loop or a stalled queue can run for hours before anyone notices.
• Shared brokers. When several teams publish to and consume from the same broker, one team's poison message can consume capacity that others depend on, and a common quarantine convention makes ownership clear.
• Ordered or single-consumer queues. Head-of-line blocking hurts most here, so detecting and moving the poison message quickly matters more.

Teams that can likely use less of it: workloads where messages are cheap to lose and easy to regenerate, such as low-value telemetry or caches that are rebuilt on a schedule. For those, a retry limit plus a counted, logged drop is usually enough. The section on when to avoid quarantining covers these cases in more detail.

How poison message handling works, end to end

The pattern is a pipeline with four stages. Each stage handles a failure the previous one cannot.

1. Classify in the consumer. When processing throws, the consumer decides whether the same message could succeed unchanged. Non-retryable failures (malformed payloads, violated invariants) go straight to quarantine. Retryable failures (timeouts, connection resets, contention) go to the retry path. Unknown errors are treated as retryable but capped.
2. Retry with delay and a limit. Retryable messages are not requeued immediately. They are parked in a delay mechanism (in RabbitMQ, a retry queue with a TTL that dead-letters back to the work queue) so the dependency has time to recover. A counter, ideally maintained by the broker, tracks attempts. Once it exceeds the limit, the message is quarantined.
3. Backstop at the broker. Failures that kill the process (OOM, segfault) never reach the consumer's error handler. A broker-side delivery limit counts redeliveries regardless of cause and dead-letters the message after too many, which breaks crash loops.
4. Quarantine with context, then operate the DLQ. The quarantined message carries its original bytes plus diagnostic headers: reason, exception class, attempt count, timestamps, consumer version, and correlation ID. Operators alert on DLQ depth and inflow, group failures by cause, ship fixes, and replay selected messages. Idempotency keys on the consumer side make those replays safe.

The rest of this article covers each stage in that order.

How to classify failures in consumer code

The consumer is the only place where a failure's cause is visible, so classification belongs there, at the moment the exception is caught. The question to ask is whether the same message could succeed if processed again unchanged. If it could not, retrying only burns capacity.

Validate the schema and business invariants before any side effect: no database writes, no outbound calls, no emitted events. This keeps a rejected message from leaving partial state behind, and it lets you send the message to quarantine on the first attempt rather than the fifth.

Misclassification costs you in both directions. If you label a transient error as permanent, valid work goes to the dead letter queue during a short outage, and someone has to replay it. If you label a permanent error as transient, you get back the retry storm described above. The classifier is also code, and it can have bugs of its own. A broad catch clause or a stale exception mapping can quietly send whole categories of messages down the wrong path. Test it the way you would test business logic.

Some failures never reach a catch block. An out-of-memory kill or a segfault ends the process before your handler runs. Only a broker-side delivery limit or crash-loop detection will catch these, and they record a delivery count but not the cause.

Copyimport json
import logging

log = logging.getLogger("orders.consumer")
MAX_ATTEMPTS = 5
TRANSIENT = (ConnectionError, TimeoutError)

class NonRetryable(Exception):
    """The same message can never succeed if reprocessed unchanged."""

def validate(raw: bytes) -> dict:
    # Pure checks only: nothing here writes, calls out, or emits events.
    try:
        msg = json.loads(raw)
    except (json.JSONDecodeError, UnicodeDecodeError) as e:
        raise NonRetryable(f"malformed payload: {e}") from e
    if not isinstance(msg, dict) or "order_id" not in msg:
        raise NonRetryable("missing required field order_id")
    qty = msg.get("quantity")
    if type(qty) is not int or qty <= 0:  # rejects bools and negatives
        raise NonRetryable(f"invalid quantity: {qty!r}")
    return msg

def handle(raw: bytes, attempt: int, apply_order) -> str:
    """Return 'ack', 'retry', or 'quarantine' for the broker adapter to act on."""
    try:
        msg = validate(raw)   # fail fast before any side effect
        apply_order(msg)      # DB writes and outbound calls happen only here
        return "ack"
    except NonRetryable as e:
        log.warning("quarantining on attempt %d: %s", attempt, e)
        return "quarantine"   # retrying cannot help, so don't burn attempts
    except TRANSIENT as e:
        log.info("transient failure on attempt %d: %r", attempt, e)
        return "retry" if attempt < MAX_ATTEMPTS else "quarantine"
    except Exception:
        # Unmapped errors default to retryable, but capped so they can't storm.
        log.exception("unclassified failure on attempt %d", attempt)
        return "retry" if attempt < MAX_ATTEMPTS else "quarantine"
    # An OOM kill or segfault never reaches this function; only a broker-side
    # delivery limit or crash-loop detection can catch those.

A broker-agnostic Python consumer that validates schema and invariants before side effects, quarantines non-retryable failures on the first attempt, and caps retries for transient and unclassified errors.

Enforcing a message retry limit

A retry limit needs a counter, and the AMQP redelivered flag is not one. It is a boolean: it tells you a message has been delivered before, not whether this is the second attempt or the two-hundredth. You can keep a real count in one of three places.

The broker count is the only one that sees crashes your code never observed. That makes it the backstop for the out-of-process failures described above. Header counts are portable but drift under crashes. An external store gives you a precise count, but you must decide what consumers do when it is unreachable. If they block, a cache outage halts processing. If they proceed, retries become unbounded until the store recovers.

Choosing the limit is a trade-off between outage tolerance and detection speed. A low limit, such as two or three attempts, sends healthy messages to the DLQ when a downstream dependency is briefly unavailable. Someone then has to replay them. A high limit keeps a genuine poison message blocking its queue for longer and delays the signal that something is wrong. Set the limit with your backoff schedule in mind. The product of attempts and delay should cover a typical transient outage and not much more.

Copyimport pika

MAX_ATTEMPTS = 5  # pick with backoff in mind: attempts x delay ~ a typical transient outage

def attempts_so_far(props, method):
    headers = props.headers or {}
    # Quorum queues add x-delivery-count on redelivery. This is the only count that
    # also sees deliveries lost to consumer crashes or disconnects.
    if "x-delivery-count" in headers:
        return int(headers["x-delivery-count"]) + 1
    # Fallback: our own header, carried on copies we republish (see below).
    if "x-retry-count" in headers:
        return int(headers["x-retry-count"]) + 1
    # Do NOT use method.redelivered as a count. It is a boolean, so it only
    # means "seen before", never "how many times".
    return 1

def on_message(ch, method, props, body):
    attempt = attempts_so_far(props, method)
    try:
        handle(body)  # your business logic
        ch.basic_ack(method.delivery_tag)
    except Exception:
        if attempt >= MAX_ATTEMPTS:
            # requeue=False routes to the queue's dead-letter exchange, if one is configured
            ch.basic_nack(method.delivery_tag, requeue=False)
            return
        headers = dict(props.headers or {})
        headers["x-retry-count"] = attempt
        # Republish-then-ack is NOT atomic. A crash between these two calls leaves
        # the original unacked: it is redelivered with its old, stale count and the
        # copy also exists, so handlers must tolerate duplicates.
        ch.basic_publish(exchange="", routing_key=method.routing_key, body=body,
                         properties=pika.BasicProperties(headers=headers,
                                                         message_id=props.message_id))
        ch.basic_ack(method.delivery_tag)

conn = pika.BlockingConnection(pika.ConnectionParameters("rabbitmq.internal"))
ch = conn.channel()
ch.basic_consume(queue="orders", on_message_callback=on_message)
ch.start_consuming()

A pika consumer that prefers the broker's x-delivery-count, falls back to a header it sets when republishing, ignores the boolean redelivered flag, and dead-letters the message once the attempt limit is reached.

How RabbitMQ poison message handling works: dead-letter exchanges, retry queues, and delivery limits

RabbitMQ's dead-letter exchange (DLX) is configured per queue, either with the x-dead-letter-exchange and optional x-dead-letter-routing-key arguments at declaration or with a policy, for example {"dead-letter-exchange":"orders.dlx"} applied to ^orders\.work$. Policies are usually the better choice because you can change them without redeclaring queues. A message is dead-lettered when a consumer rejects or nacks it with requeue=false, when its TTL expires, or when a queue length limit drops it. The broker adds an x-death header recording the queue, reason, count, and time, so consumers and operators can see why and how often a message was dead-lettered.

Quorum queues also support a delivery-limit that dead-letters a message after repeated redeliveries. This guards against consumers that crash before acking, which application-side counters cannot catch. Check your broker version's documentation for the exact behavior and default.

Immediate requeue is rarely right. The message comes straight back to a consumer that just failed on it, which creates a hot loop. A common topology instead routes failures to a retry queue that has a TTL and dead-letters back to the work queue:

work ──nack──▶ retry.30s (TTL, DLX→work) ──expire──▶ work ⋯ count > limit ──▶ dlq

For tiered backoff, use one retry queue per delay, such as retry.5s, retry.1m, and retry.10m. Classic queues expire TTL messages only from the head, so a long-TTL message can hold back shorter ones behind it if delays are mixed in one queue. Separate queues per tier avoid this, at the cost of more topology to manage.

Copy#!/usr/bin/env bash
set -euo pipefail

rabbitmqctl set_policy orders-work '^orders\.work$' \
  '{"dead-letter-exchange":"orders.retry","dead-letter-routing-key":"retry.30s","delivery-limit":10}' \
  --apply-to queues --priority 10

rabbitmqctl set_policy orders-retry-30s '^orders\.retry\.30s$' \
  '{"message-ttl":30000,"dead-letter-exchange":"orders","dead-letter-routing-key":"work"}' \
  --apply-to queues --priority 10

rabbitmqctl list_policies

rabbitmqctl policies that wire a quorum work queue to a 30-second TTL retry queue that dead-letters back to it, with a delivery-limit guard and an x-death-driven path to a DLQ.

What a quarantined message must carry, and running the DLQ as an operational surface

A dead-lettered message is only useful if someone can diagnose it without reproducing the failure. At minimum it should carry:

RabbitMQ's x-death header records the queue, reason, and count, but it knows nothing about your exceptions or build. The consumer has to add those as headers before it rejects the message.

Treat the DLQ as a queue you operate. Alert on depth and on arrival rate. Depth tells you how much work is waiting, and a sudden spike in arrivals usually points to a bad deploy. Set retention long enough to cover a weekend, and document it. A DLQ that nobody watches is silent data loss with extra steps.

Build tooling to filter by exception class or version, inspect payloads, edit when necessary, and replay selected messages. Avoid blind bulk replay. Messages that still trigger the bug will poison consumers again, and messages whose side effects partly ran will apply them twice.

During an incident, this tooling changes the question an engineer asks. Instead of "what happened to these orders?", which means digging through logs, the question becomes "which 40 messages failed with this exception, and has the fix shipped?" That is a query you can answer in minutes.

Copyimport time
import pika

CONSUMER_VERSION = "orders-consumer 2.14.3"  # placeholder: inject from your build
DLX = "orders.dlx"  # exchange bound to the orders DLQ

def quarantine(ch, method, props, body, exc, attempts):
    """Republish a poison message with diagnostic headers, then ack the original.
    basic_reject cannot add headers, so the consumer publishes the copy itself."""
    headers = dict(props.headers or {})  # keep upstream headers
    now = int(time.time())
    headers.update({
        "dlq-original-exchange": method.exchange,
        "dlq-original-routing-key": method.routing_key,  # replay target
        "dlq-exception-class": type(exc).__name__,  # group failures by cause
        "dlq-failure-reason": str(exc)[:500],  # bounded so huge errors don't bloat headers
        "dlq-attempts": attempts,
        "dlq-first-failure-at": headers.get("dlq-first-failure-at", now),
        "dlq-last-failure-at": now,
        "dlq-consumer-version": CONSUMER_VERSION,  # has the fix shipped?
    })
    ch.basic_publish(
        exchange=DLX,
        routing_key=method.routing_key,
        body=body,  # the exact bytes that failed, never re-serialized
        properties=pika.BasicProperties(
            headers=headers,
            correlation_id=props.correlation_id,  # link to traces and logs
            message_id=props.message_id,
            content_type=props.content_type,
            delivery_mode=2,  # persistent
        ),
    )
    # Ack only after the quarantined copy is published. With confirm_delivery()
    # enabled on a BlockingChannel, a failed publish raises an exception instead
    # of silently dropping the message.
    ch.basic_ack(delivery_tag=method.delivery_tag)

A pika consumer helper that copies a poison message to the dead letter exchange with its original bytes, routing key, exception class, attempt count, timestamps, consumer version, and correlation ID, and acks the original only after the copy is published.

Idempotency, ordering, and when to avoid quarantining

Every retry and every replay from the DLQ is another delivery of the same message. If a consumer charged a card or sent an email before it crashed, it will do that again unless it can recognise the message. The producer should assign each message a stable idempotency key. The consumer should record processed keys in the same transaction as the side effect, so that "done" and "recorded as done" cannot diverge. Broker-level message IDs are not enough on their own, because a redelivery after a consumer crash looks exactly like the first delivery.

Replays also reorder. If a quarantined update is replayed an hour later, it lands after newer updates for the same entity. For ordered streams, you have three policies for a poisoned message:

Some workloads should not use DLQ quarantine at all:

• Strictly ordered ledgers: Skipping an entry silently corrupts every balance after it. Block or park the key, and page a human.
• Low-value telemetry: A DLQ nobody reviews is just storage. Drop the message, increment a metric labelled by error class, and log a sample.
• Broker defaults alone: Redelivery followed by discard, with no tooling around it, is acceptable only where loss is tolerable and volume is low enough that someone would notice a spike.

Alternatives to DLQ-based quarantine

A dead-letter queue is the most common quarantine mechanism, but it is not the only one. Each alternative trades off loss, ordering, and operational effort differently.

These options can be combined. A common arrangement is a short in-process retry for blips, broker-level retry queues for longer outages, a DLQ for most quarantined messages, and drop-and-count for telemetry streams on the same broker. The deciding questions are the same each time: can you afford to lose the message, does order matter for it, and who will look at it once it fails?

Impact on development cost and engineering time

The upfront work is modest but spread across teams. Classification code has to be written and maintained in every consumer. Retry topology adds queues, bindings, and TTL policies, all of which need provisioning and review. Enriching DLQ messages with context means agreeing on a header schema. Replay tooling needs authorization, rate limiting, and an audit trail. Most of this is built once, as a shared library or platform module, and then reused.

Without these pieces, the cost recurs and shows up during incidents. A hot loop burns consumer CPU and pushes healthy messages behind a single bad one. A stalled queue causes SLA breaches downstream. Messages that were discarded silently can only be reconstructed from producer logs, if those logs still exist. Debugging a DLQ entry with no error class or stack trace starts with someone trying to reproduce the failure.

Illustrative estimate (assumed numbers, not a benchmark): annual incident cost ≈ incidents per year × engineers involved × hours per incident × loaded hourly rate. With 6 incidents, 2 engineers, 5 hours, and a rate of 100, the result is 6,000 per year. Compare that with your own estimate for the build. Then add the cost of lost data, which this formula leaves out.

The investment pays off once queues carry data you cannot afford to lose, consumers run unattended, or more than one team shares the broker.

Rollout order

1. Retry limit. This stops hot loops immediately and costs the least.
2. DLQ with context. Quarantined messages become diagnosable instead of opaque.
3. Alerting on DLQ depth and inflow rate, so failures surface before customers report them.
4. Replay tooling. Build this once the volume of manual reprocessing justifies it.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to poison message handling, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.