What is a Kafka dead letter queue and why does a poison message block a partition?
A poison message is a record that your consumer cannot process no matter how many times it tries. Typical causes are a payload that fails deserialization, a schema the code doesn't recognize, or a value that violates a business invariant. Retrying will never fix it. A dead letter topic is a separate Kafka topic where the consumer writes such records, usually with headers describing the failure, so the main stream can keep moving.
The plain Kafka consumer client has no built-in dead letter queue. The broker stores records and tracks committed offsets, and nothing else. Deciding when a record is "dead," where it goes, and what metadata travels with it is your responsibility.
The blocking comes from how consumption works. Within a partition, records are read in order, and the committed offset is a single position: committing offset N means everything before N is done. If record 1,042 throws and you don't commit past it, the consumer will read it again on the next restart, rebalance, or explicit seek back to the failed offset, and it will fail again. Every record behind it in that partition waits. Consumer lag grows, and a crash-restart loop is common.
The tempting fix is to catch the exception, log it, and commit anyway. That unblocks the partition but silently drops data. Usually nobody notices until a downstream total stops matching.
A DLQ solves both problems only if you treat it as an operational workflow. Failed records need owners, alerts, diagnosis, and a replay path. A topic that nobody reads is just a slower way to lose data.
How should you classify consumer errors before choosing a path?
Before a record goes anywhere, the consumer has to decide what kind of failure it is. Treating every exception the same way is the most common reason a dead letter queue becomes either useless or overwhelming.
Most consumer failures fall into three classes:
| Error class | Typical causes | Will retrying help? | Handling path |
|---|---|---|---|
| Deserialization | Malformed bytes, schema mismatch, wrong format on the topic | No. The bytes never change | Send straight to the DLQ with raw key and value preserved |
| Permanent business or validation | Missing required field, unknown entity ID, rule violation | No, unless the code or reference data changes | Send to the DLQ after one attempt |
| Transient | Timeouts, connection resets, throttling, downstream unavailable | Usually, after a delay | Retry topics with backoff, then the DLQ if attempts run out |
Getting the class wrong in either direction is costly. If you mark transient errors as permanent, a database outage pushes every in-flight record into the DLQ within minutes. The DLQ fills with valid messages, real poison messages get buried, and someone has to bulk-replay everything once the dependency recovers. If you mark permanent errors as transient, each bad record uses up every retry tier before it fails. That consumes retry capacity and broker throughput, and it can hide a bug for hours that should have alerted on the first attempt.
Build the classification as an explicit mapping from exception type to category, kept in one place and covered by unit tests. Default unknown exceptions to transient with a bounded retry count. A safe default delays detection slightly but does not lose data. Classify HTTP and database errors by status or error code, not by exception class alone. A 429 or 503 is transient, while a 400 or 422 is permanent. Review the mapping whenever records with unexpected exception types show up in the DLQ.
import json
from enum import Enum
class Category(Enum):
DESERIALIZATION = "deserialization" # straight to DLQ, keep raw key/value
PERMANENT = "permanent" # DLQ after one attempt
TRANSIENT = "transient" # retry topics with backoff, then DLQ
class ValidationError(Exception): ...
class HttpError(Exception):
def __init__(self, status: int):
super().__init__(f"HTTP {status}")
self.status = status
# Single source of truth. Order matters: isinstance also matches subclasses.
EXCEPTION_MAP = [
(json.JSONDecodeError, Category.DESERIALIZATION),
(UnicodeDecodeError, Category.DESERIALIZATION),
(ValidationError, Category.PERMANENT),
(TimeoutError, Category.TRANSIENT),
(ConnectionError, Category.TRANSIENT),
]
TRANSIENT_STATUS = {408, 429, 502, 503, 504}
def classify(exc: Exception) -> Category:
# HTTP errors are classified by status code, not by exception class.
if isinstance(exc, HttpError):
return Category.TRANSIENT if exc.status in TRANSIENT_STATUS else Category.PERMANENT
for exc_type, category in EXCEPTION_MAP:
if isinstance(exc, exc_type):
return category
# Unknown exceptions are treated as transient. The bounded retry count limits the cost.
return Category.TRANSIENT
# Unit tests pin the mapping so a refactor cannot silently change routing.
assert classify(HttpError(429)) is Category.TRANSIENT
assert classify(HttpError(422)) is Category.PERMANENT
assert classify(json.JSONDecodeError("bad", "{", 0)) is Category.DESERIALIZATION
assert classify(RuntimeError("surprise")) is Category.TRANSIENT
An explicit exception-to-category mapping that classifies HTTP errors by status code, defaults unknown exceptions to transient, and has its routing pinned by tests.
How do Kafka retry topics and the three-tier pattern work?
The pattern has three tiers, and each record moves down a tier only when the one above has given up.
Tier 1: bounded in-process retry. The consumer retries the same record a few times with short exponential backoff, typically totaling under a few seconds. This absorbs brief network failures and leader elections cheaply. It is a blocking retry. Order is preserved, but every other key on that partition waits too. Keep the total retry time well below max.poll.interval.ms. If the gap between polls exceeds it, the broker evicts the consumer, triggers a rebalance, and redelivers the same record to another instance, which then stalls the same way.
Tier 2: non-blocking retry topics. If tier 1 is exhausted, the consumer publishes the record to a retry topic, such as orders.retry.1m or orders.retry.10m, commits the original offset, and moves on. Kafka has no native per-message delay, so the producer writes a due timestamp into a header. The retry consumer reads that header. If the record is not yet due, it pauses the partition with pause() and keeps calling poll() until the time arrives, then resumes. Pausing keeps the consumer in the group. Sleeping inside the poll loop does not, and it triggers the eviction described above. Because each retry topic holds records with the same delay, they become due in roughly arrival order, and a paused partition blocks little. Retry topics usually need fewer partitions than the main topic, since they carry only failures. However, too few partitions concentrate load during a downstream outage.
Tier 3: the dead letter topic. After the last retry tier fails, the record goes to the terminal topic with its error metadata attached. No automatic consumer processes it.
+--------------------------------------------+
| orders (main topic) |
+--------------------------------------------+
v
+--------------------------------------------+
| TIER 1 main consumer: in-process retry |
| short exp. backoff, blocking | ok
| order kept; all keys on partition wait |------> commit
| total retry time << max.poll.interval.ms |
+--------------------------------------------+
| exhausted: produce + due-at header,
v commit original offset
+--------------------------------------------+
+---->| orders.retry.1m, orders.retry.10m, ... |
| | one delay per topic; fewer partitions |
| +--------------------------------------------+
| v
| +--------------------------------------------+
| | TIER 2 retry consumer (non-blocking) |
| | read due-at header | ok
| | not due: pause(), keep calling poll() |------> commit
| | due: resume() and process; never sleep() |
| +--------------------------------------------+
| | |
+-------+ fails: next retry topic | last tier fails
v
+--------------------------------------------+
| TIER 3 orders.dlt: record + error metadata|
| terminal; no automatic consumer |
+--------------------------------------------+
A record moves from blocking in-process retry through delay-specific retry topics. Their consumers pause partitions until the due-at header time arrives. After the last retry tier fails, the record lands in a terminal dead letter topic that no automatic consumer reads.
How do you implement a dead letter topic producer in a Kafka consumer?
The most reliable way to catch poison messages is to configure the consumer with a byte-array deserializer for both key and value, then deserialize inside your processing loop. If deserialization happens inside the client, a malformed payload can fail before your handler ever receives a record. With raw bytes in hand, a schema mismatch becomes an ordinary exception, and you still hold the exact bytes to forward.
When a record fails, produce a new record to the retry or dead letter topic. Copy the original key, value bytes and headers unchanged. Reusing the key keeps related records on the same partition in the retry tier. Leaving the value untouched means a replay sends what the producer actually wrote, not your partial interpretation of it. Then add metadata headers:
| Header | Purpose |
|---|---|
dlq.source.topic, dlq.source.partition, dlq.source.offset |
Locate the original record for audit and replay |
dlq.exception.class, dlq.exception.message |
Triage without reading consumer logs |
dlq.attempt |
Decide which tier the record goes to next |
dlq.first.failure.ts |
Measure how long the record has been failing across tiers |
Commit the source offset only after the forwarding produce is acknowledged. Use acks=all and wait on the send result. If the consumer crashes between the acknowledgement and the commit, the record is forwarded twice on restart. That duplicate is recoverable, while a committed offset for a record that never reached the DLQ is a silent loss.
If the DLQ produce itself fails, do not commit and do not skip the record. Handle the failure by type:
- Transient failures such as a broker outage or timeouts: pause the assigned partitions and retry with backoff.
- Permanent failures such as a record that exceeds the size limit: fail hard and alert, because retrying will never succeed and a human has to decide what happens to the record.
// Consumer: key/value.deserializer=ByteArrayDeserializer, enable.auto.commit=false
// dlqProducer: acks=all, enable.idempotence=true
while (running) {
ConsumerRecords<byte[], byte[]> batch = consumer.poll(Duration.ofMillis(500));
for (TopicPartition tp : batch.partitions()) {
for (ConsumerRecord<byte[], byte[]> rec : batch.records(tp)) {
try {
// Deserialize here so a schema mismatch is an ordinary exception
handler.process(valueDeserializer.deserialize(rec.topic(), rec.value()));
} catch (Exception e) {
// Original key, value bytes and headers, copied unchanged
ProducerRecord<byte[], byte[]> out = new ProducerRecord<>(
"orders.dlq", null, rec.key(), rec.value(), rec.headers());
Headers h = out.headers();
h.add("dlq.source.topic", utf8(rec.topic()));
h.add("dlq.source.partition", utf8(String.valueOf(rec.partition())));
h.add("dlq.source.offset", utf8(String.valueOf(rec.offset())));
h.add("dlq.exception.class", utf8(e.getClass().getName()));
h.add("dlq.exception.message", utf8(String.valueOf(e.getMessage())));
h.add("dlq.attempt", utf8("1")); // on retry tiers, read and increment
h.add("dlq.first.failure.ts", utf8(String.valueOf(System.currentTimeMillis())));
try {
dlqProducer.send(out).get(); // block until acks=all acknowledgement
} catch (ExecutionException ex) {
if (ex.getCause() instanceof RecordTooLargeException) {
throw new IllegalStateException("DLQ cannot accept record; stopping", ex); // permanent: fail hard
}
consumer.pause(Set.of(tp)); // transient: stop fetching this partition
consumer.seek(tp, rec.offset()); // re-deliver the same record after resume
scheduleResumeWithBackoff(tp); // app helper that calls consumer.resume(tp) later
break; // nothing committed for this record
}
}
// Commit only after processing or an acknowledged DLQ write
consumer.commitSync(Map.of(tp, new OffsetAndMetadata(rec.offset() + 1)));
}
}
}
A byte-array consumer loop that forwards failed records to a DLQ with original bytes, headers and metadata, commits only after the acks=all send succeeds, and pauses or fails hard when the DLQ produce fails.
What happens to ordering and idempotency when messages are retried?
Kafka guarantees order only within a partition. When a consumer moves a failed record to a retry topic and commits past it, later records with the same key are processed first. For many workloads that is fine. Examples include sending notifications, indexing documents for search, and enriching analytics events. Each record stands alone, or the latest write wins anyway.
It is not fine when records describe transitions. Suppose OrderCancelled is applied before OrderCreated, or a ledger debit lands before the credit that funds it. The result is invalid state or a rejected operation, which may itself end up in the dead letter topic. For these streams you have two options:
- Block the partition. Retry in place with backoff until the record succeeds. This keeps ordering but delays every key on that partition.
- Park the key. Record the failed key in a store and divert every later record for that key to the retry path until the head record succeeds. Other keys keep flowing, but you now maintain parking state that must survive restarts and rebalances.
| Approach | Per-key order | Impact on other keys | Complexity |
|---|---|---|---|
| Non-blocking retry topic | Broken | None | Low |
| Blocking retry | Preserved | Whole partition stalls | Low |
| Per-key parking | Preserved | Only parked key lags | High |
Whichever path you choose, expect duplicates. A crash between processing and committing the offset causes redelivery, and DLQ replays reprocess records that may have partially succeeded. Handlers must therefore be idempotent. Common techniques:
- Deduplication table. Store a unique message or business ID in the same transaction as the side effect, and skip IDs you have already seen.
- Conditional writes. Use upserts keyed by entity, or apply an update only if the stored version is lower than the record's version.
- Natural idempotency. Prefer "set balance to X" or "status = shipped" over increments, so applying a record twice gives the same result.
- Idempotency keys downstream. Pass a stable key to external APIs that support one, so they can discard repeated requests.
How do you operate, alert on and replay a dead letter queue?
A DLQ only works if someone is looking at it. An unwatched dead letter topic defers data loss: records sit until retention deletes them, and nobody ever fixed the bug that put them there.
Ownership, retention and alerting
Assign a named owning team, usually the team that owns the consumer, not a shared platform group. Set retention to cover your realistic investigation window, including weekends and holidays. If triage can take two weeks, a seven-day retention guarantees loss.
Alert on DLQ arrival rate, not only consumer lag. Lag on the DLQ consumer tells you little, because nothing may consume it. A steady trickle can go to a ticket queue. A sudden spike is an incident signal. It usually means a bad deploy, a schema change or a downstream outage. In that case, trip a circuit breaker or pause the consumer rather than dead-lettering healthy traffic. Otherwise you move thousands of good records into a side channel and turn an outage into a replay project.
Replay
After a fix ships, a replay tool republishes selected records to the source topic or to a retry topic, filtered by time range, error type or key. Replay carries three risks:
- Out-of-order arrival. Replayed records land behind newer events for the same key.
- Changed consumer code. The fixed consumer may interpret old payloads differently than the original would have.
- Duplicates. Records that partially succeeded before failing will be applied again, so the idempotency measures above are a prerequisite.
Security
DLQ records copy payloads verbatim, including PII and secrets. Apply the same access control, encryption and retention policy as the source topic.
from confluent_kafka import Consumer, Producer, TopicPartition
DLQ, TARGET = "orders.dlq", "orders.retry" # replay into the retry topic, not straight into source
START_MS, END_MS = 1793000000000, 1793086400000 # window covering the bad deploy
ERROR_TYPE = b"SchemaValidationError" # replay only the error class the fix addresses
consumer = Consumer({"bootstrap.servers": "kafka:9092", "group.id": "dlq-replay-2026-11-08",
"enable.auto.commit": False, "auto.offset.reset": "earliest"})
producer = Producer({"bootstrap.servers": "kafka:9092", "enable.idempotence": True})
# Start each partition at the first record at or after START_MS.
parts = list(consumer.list_topics(DLQ).topics[DLQ].partitions)
start = consumer.offsets_for_times([TopicPartition(DLQ, p, START_MS) for p in parts], timeout=10)
consumer.assign(start)
replayed, done = 0, set()
while len(done) < len(parts):
msg = consumer.poll(5.0)
if msg is None:
break # caught up on all partitions
if msg.error():
continue
if msg.timestamp()[1] > END_MS: # past the window: stop reading this partition
done.add(msg.partition())
consumer.pause([TopicPartition(DLQ, msg.partition())])
continue
headers = msg.headers() or []
if dict(headers).get("error.type") != ERROR_TYPE: # header written by the DLQ producer
continue
# Keep the original key so the record lands on the same partition as newer events.
# It will still arrive after them, so consumers must tolerate out-of-order and duplicates.
producer.produce(TARGET, key=msg.key(), value=msg.value(),
headers=headers + [("dlq.replayed", b"true")])
replayed += 1
producer.poll(0)
producer.flush()
consumer.close()
print(f"replayed {replayed} records to {TARGET}")
A minimal replay tool using the confluent-kafka Python client that republishes DLQ records from a time window and a single error type to a retry topic, preserving keys and tagging each record as replayed.
When should you avoid a DLQ, and what are the alternatives?
A DLQ pays off when losing a message has a real cost and the data feeds several downstream services: payments, order state, inventory, audit trails. In those streams, a stuck partition stalls every consumer behind it, and a dropped record becomes a reconciliation job weeks later. For a low-volume or loss-tolerant stream, such as clickstream sampling, cache warming or best-effort notifications, the extra topics, consumer groups, dashboards and on-call runbooks often cost more than the occasional lost record. Structured logging and skipping is simpler and easier to reason about.
| Approach | Loss risk | Partition blocking | Build and run cost | Fits |
|---|---|---|---|---|
| Log and skip | High | None | Minimal | Telemetry, derived or rebuildable data |
| Blocking retry only | None | Yes, until fixed | Low | Strict ordering, rare transient faults |
| Per-key parking | Low | Only the affected key | Moderate, needs a state store | Per-entity ordering with mixed failures |
| Tiered retry plus DLQ | Low | None | Highest | High-value, multi-consumer streams |
The tiered approach costs real engineering time. You build header propagation, retry consumers and a replay tool, then keep them tested as schemas change. That investment comes back during incidents. Without a DLQ, a poison message usually means manually resetting offsets on a consumer group under pressure, which is error-prone and can skip valid records. With original topic, partition, offset and exception class in headers, triage starts at the failing record rather than in log searches. Once the bug is fixed, replaying the DLQ through the patched consumer replaces ad hoc scripts and one-off data repairs.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to kafka dead letter queue, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Changing a shared variable is one edit and one review instead of editing every service's
.envby hand.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.