What a poison message is and why a dead letter queue exists

A poison message is one that fails every time it is processed, no matter how often you retry it. That separates it from a transient failure, such as a timeout or a briefly unavailable dependency, which a retry usually fixes. It also differs from an ordinary permanent failure in its effect. A malformed payload, a schema the consumer cannot deserialize, or a reference to a record that will never exist all fail deterministically. Each retry spends work and produces nothing.

A dead letter queue (DLQ) holds these messages once retries run out. The main stream keeps moving, and the failed message is kept rather than dropped. How much the DLQ matters depends on the broker model:

Broker model What a poison message does What the DLQ provides
Queue-based (competing consumers) Repeated redelivery uses consumer capacity and retry budget, which can starve healthy messages Removal from rotation once a delivery or retry limit is reached
Partitioned log Blocks its partition, because consumers commit offsets in order and cannot advance past it A skip-and-park step: copy the record elsewhere, commit past it, and keep going

On a log, there is often no built-in dead-lettering. The consumer must decide to park a record and move on, or the partition stalls.

A DLQ only isolates poison messages. Diagnosing them, fixing the cause, and replaying them safely is the other half of the job, and most incidents are decided there. The team that owns the consuming service should own its DLQ, because only that team can say whether a parked message is a bug, bad data, or something safe to discard. Platform teams should provide the shared tooling for parking and replay.

How the retry, park and replay flow works

A message moves through four tiers, and each tier handles a different class of failure.

Immediate retries run in-process, a few attempts spaced milliseconds apart. They absorb blips such as a dropped connection or a lock conflict. Keep them few, because the consumer holds the message, and often a lease on it, for the whole time.

Delayed retries republish the message to a retry queue or topic with increasing delays. This frees the consumer to keep processing healthy traffic while a dependency recovers, and it avoids hammering a struggling downstream service.

The DLQ receives the message once attempts are exhausted. The message is parked with its failure context, and nothing consumes it automatically.

Replay is a separate, deliberate path. A tool republishes selected messages to the consumer, ideally through a dedicated replay queue so that replayed traffic can be throttled and tagged.

When choosing thresholds, make the total delayed-retry window longer than the dependency outage you want to ride out without a human. Illustrative estimate: delays of 1, 5 and 30 minutes cover roughly a 36-minute outage. Immediate retries must also finish well inside the visibility timeout or lock duration. If they don't, the broker redelivers the message to another consumer mid-retry, which causes duplicate processing and inflates the delivery count.

If the threshold is set too low, a routine outage floods the DLQ with valid messages, replay becomes a daily chore, and DLQ alerts stop meaning anything. If it is set too high, true poison messages tie up consumers, stall ordered partitions, and go undetected for hours.

+--------------+       +----------------------------------------+
| Main queue / |------>| Consumer                               |
| topic        |       |  Tier 1: immediate retries (few, ms    |
+--------------+       |  apart) - must end well inside the     |
                       |  visibility timeout / lock duration    |
                       +----------------------------------------+
                          |            ^             ^         |
              still fails |            | due again   | replay  | ok
                          v            |             |         v
+-----------------------------------+  |             |        ack
| Tier 2: delayed retry queue/topic |  |             |
|  retry-1m -> retry-5m -> retry-30m|--+             |
|  (illustrative: ~36 min window)   |                |
+-----------------------------------+                |
                          |                          |
       attempts exhausted |                          |
                          v                          |
+-----------------------------------+   +------------------------+
| Tier 3: DLQ (parked)              |   | Tier 4: replay queue   |
|  payload + failure context        |   |  tagged, throttled     |
|  nothing consumes automatically   |   +------------------------+
+-----------------------------------+                ^
                          |                          |
                          v                          |
                 +------------------------------------------+
                 | Replay tool (deliberate, human-driven):  |
                 |  inspect, fix cause, select messages     |
                 +------------------------------------------+

A failing message moves from in-process immediate retries to delayed retry queues, then to the DLQ once attempts are exhausted, and returns to the consumer only through a deliberate, throttled replay path.

Implementing dead-lettering with classification metadata

Copy the original payload bytes and headers to the dead letter topic exactly as received. Do not deserialize and re-serialize them, because replay needs the precise input that failed and re-encoding can hide the schema or encoding defect that caused the failure. Put diagnostics in separate, namespaced headers:

  • Origin: source topic, partition and offset (or message ID on a queue broker)
  • Consumer: consumer name and deployed version, so you can tell whether a later fix covers this message
  • History: attempt count, first and last failure timestamps
  • Error: exception class, error message truncated to a fixed length (stack traces can exceed header size limits), and a failure category such as deserialization, validation, dependency or bug

The category drives triage and replay decisions later, so set it in code at the point where the exception is caught. Do not try to infer it afterwards from log text.

For a log-based consumer, skip-and-park runs in this order:

  1. Classify the failure.
  2. Produce to the dead letter topic.
  3. Wait for the broker to acknowledge the write.
  4. Commit the source offset.

Committing before the acknowledgement can lose the message. If the DLQ write fails, stop the partition rather than skip, because skipping would drop data silently.

Every DLQ needs a named owning team, the same team that owns the consumer. Alert on three signals: depth, inflow rate and oldest-message age. Inflow rate catches a bad deploy within minutes. Oldest-message age catches a backlog that nobody is working.

Set DLQ retention shorter than the source topic's retention. A parked message whose original offset has expired from the source topic is harder to replay and audit in context. A shorter limit also forces a decision on each message instead of letting the DLQ become an archive.

# Broker-neutral sketch: consumer/producer methods are placeholders for your client library.
import time
CONSUMER, VERSION, DLQ_TOPIC, MAX_ERR = "orders-projector", "2.4.1", "orders.dlq", 512

class Classified(Exception):
    """Raised where the error is caught, so the category is set in code, not inferred later."""
    def __init__(self, category, cause):
        super().__init__(str(cause))
        self.category, self.cause = category, cause  # deserialization|validation|dependency|bug

def park(msg, err, first_failure_ms, attempts, dlq_producer, consumer):
    diag = {
        "dlq.source.topic": msg.topic,
        "dlq.source.partition": str(msg.partition),
        "dlq.source.offset": str(msg.offset),
        "dlq.consumer.name": CONSUMER,
        "dlq.consumer.version": VERSION,
        "dlq.attempts": str(attempts),
        "dlq.first_failure_ms": str(first_failure_ms),
        "dlq.last_failure_ms": str(int(time.time() * 1000)),
        "dlq.error.class": type(err.cause).__name__,
        "dlq.error.message": str(err.cause)[:MAX_ERR],  # stack traces can exceed header limits
        "dlq.error.category": err.category,
    }
    # Original key, value bytes and headers are copied as received; never re-serialized.
    headers = list(msg.headers) + [(k, v.encode("utf-8")) for k, v in diag.items()]
    try:
        # Block until the broker acknowledges the DLQ write.
        dlq_producer.send(DLQ_TOPIC, key=msg.key, value=msg.value,
                          headers=headers).wait(timeout_s=10)
    except Exception:
        # DLQ write failed: halt this partition instead of skipping and losing data.
        consumer.pause(msg.topic, msg.partition)
        raise
    # Commit the source offset only after the DLQ write is durable.
    consumer.commit(msg.topic, msg.partition, msg.offset + 1)

A broker-neutral skip-and-park sketch that copies the original bytes and headers to a dead letter topic, adds namespaced diagnostic headers, waits for the broker ack, and only then commits the source offset.

Triage and designing DLQ replay tooling

Triage starts by grouping parked messages by failure category and by the consumer version that rejected them. A thousand messages that failed schema validation under one build usually share a single cause. If you look at them one by one, you will fix the same bug a thousand times or miss that it is the same bug.

Each group gets one disposition:

  • Fix code: the consumer was wrong, and the message replays unchanged.
  • Fix data: an upstream record or reference table was wrong, so correct it at the source first.
  • Transform: the payload must be rewritten before it can succeed.
  • Discard: drop the messages and record the reason, so the loss is a decision someone made and can find later.

Do not replay any group until its root-cause fix is deployed. A replay against the old consumer only refills the DLQ, and it burns retry budget along the way.

Redrive tool requirements

A usable message redrive tool needs:

  • Selective filters by category, consumer version, time range and key, so you can replay one group and leave the others parked.
  • Rate limiting, because a backlog arrives faster than live traffic ever did.
  • Dry runs that report what would be sent without sending it.
  • An explicit target, with no default that guesses the destination.
  • An audit trail recording who replayed what, when, and why.

Prefer a dedicated replay path over republishing to the original topic. Other subscribers on that topic would otherwise reprocess the messages.

Rate limits should come from downstream capacity, not from the size of the backlog. Illustrative estimate: say 200,000 messages are parked and the downstream service has 100 req/s of spare capacity. Applying a 0.5 headroom factor gives 50 msg/s, so the redrive takes 4,000 seconds, or about 67 minutes. That is long enough that someone should watch it.

import argparse, datetime, getpass, json, time

def parse_args():
    p = argparse.ArgumentParser(description="Replay one triaged DLQ group")
    p.add_argument("--category", required=True)          # e.g. schema_validation
    p.add_argument("--consumer-version", required=True)   # build that rejected the group
    p.add_argument("--target", required=True)             # no default destination
    p.add_argument("--reason", required=True)             # recorded in the audit trail
    p.add_argument("--spare-rps", type=float, required=True)  # downstream spare capacity
    p.add_argument("--headroom", type=float, default=0.5)
    p.add_argument("--dry-run", action="store_true")
    return p.parse_args()

def redrive(store, publish, audit_log, args):
    # store/publish/audit_log are adapters for your broker and storage.
    rate = args.spare_rps * args.headroom  # illustrative: 100 * 0.5 = 50 msg/s
    batch = [m for m in store.parked()
             if m["category"] == args.category
             and m["consumer_version"] == args.consumer_version]
    eta_min = len(batch) / rate / 60       # illustrative: 200,000 / 50 = 4,000 s, ~67 min
    print(f"{len(batch)} msgs -> {args.target} at {rate:.0f}/s, ~{eta_min:.0f} min")
    if args.dry_run:
        return  # report what would be sent, send nothing
    audit_log.append(json.dumps({
        "who": getpass.getuser(),
        "when": datetime.datetime.now(datetime.timezone.utc).isoformat(),
        "filter": {"category": args.category, "consumer_version": args.consumer_version},
        "target": args.target, "reason": args.reason, "count": len(batch),
    }))
    for msg in batch:
        publish(args.target, msg)  # dedicated replay path, not the original topic
        time.sleep(1 / rate)       # pace to downstream capacity, not backlog size

A broker-agnostic Python sketch of a redrive tool that filters one triaged group, requires an explicit target and reason, supports dry runs, writes an audit record, and paces replay from downstream spare capacity.

Making replay safe: idempotency, ordering and transformation

Live traffic keeps changing the same records for that entire hour, so replayed messages never land on a frozen world.

Expect duplicates

Replay delivers duplicates. A message may have partially succeeded before it failed, or a broker may redeliver after a timeout. Handlers must therefore be idempotent on a stable ID assigned by the producer, such as an event ID or a business key plus operation. Do not use a broker delivery ID, because it changes on redrive. A dedupe table keyed on that ID, written in the same transaction as the state change, turns a second delivery into a no-op.

Watch the retention mismatch. If dedupe entries expire after 7 days but the DLQ keeps messages for 14, a replay on day 10 passes the dedupe check and applies twice. The dedupe window must be at least as long as DLQ retention plus the longest expected time to triage.

Reject stale updates

Idempotency stops a message from applying twice. It does not stop an old message from overwriting newer state. Carry an entity version or source timestamp on each message and apply it only if it is newer than the stored version. Otherwise, acknowledge and drop it. The example below combines both checks.

Transform with an audit trail

When a payload needs fixing, leave the original untouched in the DLQ. Write a transformed copy that references the original's ID. Keep transformation scripts in version control, put them through code review, and run them first on a small sample routed to a test consumer.

# Version-checked idempotent handler (PostgreSQL via psycopg2).
# Assumes tables:
#   processed_events(event_id text PRIMARY KEY, processed_at timestamptz)
#   accounts(account_id text PRIMARY KEY, status text, version bigint)

def handle_account_updated(conn, msg):
    # Producer-assigned ID, stable across retries and DLQ redrive.
    # Never use the broker's delivery/receipt ID here.
    event_id = msg["event_id"]

    with conn:  # one transaction: dedupe row and state change commit or roll back together
        with conn.cursor() as cur:
            # 1. Idempotency: record the event ID; a conflict means we already applied it.
            cur.execute(
                "INSERT INTO processed_events (event_id, processed_at) "
                "VALUES (%s, now()) ON CONFLICT (event_id) DO NOTHING",
                (event_id,),
            )
            if cur.rowcount == 0:
                return "duplicate"  # ack and drop

            # 2. Staleness: write only if the message's entity version is newer.
            cur.execute(
                "INSERT INTO accounts (account_id, status, version) "
                "VALUES (%s, %s, %s) "
                "ON CONFLICT (account_id) DO UPDATE "
                "SET status = EXCLUDED.status, version = EXCLUDED.version "
                "WHERE accounts.version < EXCLUDED.version",
                (msg["account_id"], msg["status"], msg["version"]),
            )
            if cur.rowcount == 0:
                # Newer state already stored. Keep the dedupe row so a later
                # replay is also skipped, then ack and drop.
                return "stale"

    return "applied"

A replay-safe handler that records the producer's event ID and applies a version-guarded upsert in a single PostgreSQL transaction, so duplicates become no-ops and stale replays cannot overwrite newer state.

When to avoid it, trade-offs and alternatives

A DLQ adds little when nearly all failures are transient. If a downstream dependency is down, every message fails, and dead-lettering them turns a short outage into a large backlog that someone has to replay by hand. Dead-letter only deterministic failures, such as schema or validation errors, and pause consumption on dependency errors. When you do drain a large backlog, rate-limit it. Replaying everything at once recreates the load spike that caused the failures.

Ordering is the sharpest trade-off. Parking one message and moving on keeps the partition available, but later events for the same key can overtake it. If consumers rely on per-entity order, park every subsequent message for that key until the poison one is resolved.

Replaying into the original queue is simple, but replayed traffic then mixes with live traffic and can't be throttled or traced separately. A dedicated replay path lets you tag, throttle and monitor it. DLQ payloads often contain PII, so they need the same retention limits and access controls as your primary data store. Self-service replay buttons suit idempotent, low-risk handlers. Anything that moves money or contacts customers should go through a reviewed replay.

Approach Fits when Cost
Retry in place with backoff Failures are transient and order matters Blocks the partition while retrying
Park all messages per key Strict per-entity ordering Extra state and release logic
Discard with recorded decision Message is obsolete or superseded Deliberate data loss, needs audit trail
Generic tooling Many services share one envelope format Cannot do domain-specific repairs
Per-service tooling Repairs need domain logic Duplicated effort across teams

Impact on development cost and engineering time

The costs arrive up front and keep recurring. A dedupe store needs capacity planning, a retention window longer than your longest plausible replay delay, and an owner when it degrades. Replay infrastructure is another service to run: a parked-message store, a redrive worker and rate limiting. Classification metadata requires every consumer to populate error class, attempt count and original source consistently. That touches shared libraries and every team that depends on them. Access controls need role design and audit logging, because a replay can rewrite production state. The tooling also decays. Envelope formats evolve, and a replay tool that cannot parse last quarter's messages tends to fail exactly when you need it.

The return shows up during incidents. Classification metadata turns "read thousands of stack traces" into a grouped query. A reviewed redrive path replaces the untested script someone writes at 2 a.m. Idempotent replay avoids the long cleanups that follow double-applied writes. Parking messages instead of letting them expire means nothing is lost silently.

Illustrative estimate (assumed numbers, not a benchmark): hours saved per year = incidents × (manual hours − tooled hours). Assume 12 incidents a year, 10 hours each without tooling and 2 hours with it: 12 × 8 = 96 hours saved. Set that against an assumed 120 hours to build and 30 hours a year to maintain. Under these assumptions, break-even falls in year two. Substitute your own figures.

Treat every replay as a production deployment:

  • Record the selection criteria and the target consumer version in a reviewed change.
  • Run a dry run that reports what would be applied.
  • Release a small batch first.
  • Watch consumer error rates and downstream lag.
  • Then continue or halt.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to dead letter queue microservices, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • The dependency graph answers "what does this touch?" in seconds instead of a manual audit.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • You can size a change's blast radius before merging, avoiding surprise cross-team firefighting.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.