What is a Kafka dead letter queue, and why should it be treated as a work queue?

A Kafka dead letter queue is an ordinary topic that holds messages a consumer could not process. Kafka has no special DLQ type. It is a convention: when a record fails in a way the consumer cannot resolve on its own, the consumer writes it to a separate topic, commits the offset on the source partition, and moves on.

The reason to do this is isolation. Kafka consumers read partitions in order. A single poison message, such as a payload that fails deserialization or violates a schema constraint, can block every record behind it if the consumer keeps retrying. Moving that record aside lets healthy traffic keep flowing while the bad one waits for a human.

The pattern suits backend and platform teams whose consumer failures need investigation rather than silent drops: payments, order state, entitlements, anything where a lost event becomes a data correctness bug.

A DLQ only has value if it is treated as a work queue. It needs an owning team, a triage process, and a safe replay path. Without those, it becomes write-only storage. Records land, nobody reads them, and the topic's retention policy deletes them before anyone looks. The failure is still lost. It just takes a few days longer.

The rest of this guide compares the DLQ with the alternatives: blocking retries, partition pausing, logging and skipping, and version checks on write.

When should a failed message go to a retry topic and when straight to the DLQ?

Routing decisions belong in one place, before any retry logic runs. The consumer should ask a single question of every failure: will the same message succeed if processed again later, unchanged? If yes, the failure is transient and earns bounded retries. If no, the message is poison and retrying it only burns time.

Failure category Route Rationale
Network timeout calling a downstream service Retry topic The dependency is likely slow or restarting, and a later attempt can succeed
Dependency unavailable or rate-limited Retry topic The cause sits outside the message, so waiting fixes it
Deserialization failure DLQ immediately The bytes will never parse, no matter how often you try
Schema or validation violation DLQ immediately The payload breaks the contract, and a code or producer fix is needed first
Business-rule rejection (for example, an unknown account) DLQ immediately The data is wrong for this system, and retrying does not change it
Unrecognised exception Retry topic, then DLQ Assume it is transient but cap the attempts

Misclassification costs you in both directions. If you retry a permanent error, you spend every attempt and every backoff delay before the message reaches the DLQ. That delays triage and occupies retry capacity that real transient failures need. If you treat a transient error as permanent, the problem is worse. A five-minute database outage sends every message processed during that window to the DLQ, which buries any genuine poison messages under thousands of records that would have succeeded on their own.

The classifier should map known exception types to transient or permanent and return unknown for everything else. Count how often unknown appears. A high rate means the mapping needs new entries.

import java.net.ConnectException;
import java.net.SocketTimeoutException;
import java.util.concurrent.atomic.AtomicLong;
import org.apache.kafka.common.errors.SerializationException;

public final class FailureClassifier {

    public enum FailureClass { TRANSIENT, PERMANENT, UNKNOWN }

    // Application-defined exceptions for contract and business-rule failures.
    public static class ValidationException extends RuntimeException {}
    public static class BusinessRuleException extends RuntimeException {}

    // Track UNKNOWN results so unclassified exception types become visible.
    private final AtomicLong unknownCount = new AtomicLong();

    public FailureClass classify(Throwable error) {
        // Walk the cause chain: frameworks often wrap the exception that matters.
        for (Throwable t = error; t != null; t = t.getCause()) {
            // The dependency is slow or down; the same message can succeed later.
            if (t instanceof SocketTimeoutException || t instanceof ConnectException) {
                return FailureClass.TRANSIENT;
            }
            // The message itself is bad; reprocessing it unchanged can never succeed.
            if (t instanceof SerializationException
                    || t instanceof ValidationException
                    || t instanceof BusinessRuleException) {
                return FailureClass.PERMANENT;
            }
        }
        // Not recognised: the caller retries with a capped attempt count, then routes to the DLQ.
        unknownCount.incrementAndGet();
        return FailureClass.UNKNOWN;
    }

    public long unknownCount() {
        return unknownCount.get();
    }
}

A Java classifier walks the exception cause chain and maps network failures to TRANSIENT and serialization, validation and business-rule failures to PERMANENT. It returns UNKNOWN for anything else and counts those results so you can see which exception types still need a classification.

How do retry topics and the DLQ fit together in a consumer pipeline?

Once a failure is classified, it needs a route. The common shape is a chain of retry topics with increasing delays and a fixed attempt cap, ending in the DLQ:

orders ──fail──▶ orders.retry.1 (30s) ──fail──▶ orders.retry.2 (5m) ──fail──▶ orders.retry.3 (30m) ──fail──▶ orders.dlq
  ▲                                                                                                              │
  └─────────────────────────────── replay (manual, after fix) ───────────────────────────────────────────────────┘

Permanent failures skip the tiers and go straight to orders.dlq. Each retry consumer reads a record, waits until its timestamp plus the tier delay has passed, and reprocesses it. A failure there forwards the record to the next tier with an incremented attempt header.

The alternative is retrying in place. In-place retries preserve order, but one stuck record halts every key behind it on that partition. Retry topics keep the main partition moving. The cost is per-key ordering: a later update for the same key can be applied before an earlier one comes back from retry.2.

The operational cost is real. Every source topic gains several topics and consumer groups to provision and monitor. The delay mechanism also needs care. A consumer that sleeps inside its poll loop past max.poll.interval.ms is evicted from the group, which triggers a rebalance and redelivery. Pausing partitions avoids that because the consumer keeps polling, but paused state does not survive reassignment. Your rebalance listener must re-pause the partitions.

Approach Per-key ordering Partition availability Complexity
Blocking in-place retry Preserved Stalls until success or give-up Low
Pause partition, retry later Preserved Stalls that partition only, group stays healthy Medium
Non-blocking retry topics Lost across tiers Main flow continues High
    ┌───────────────────────────┐
┌──▶│ orders (main topic)       │
│   └─────────────┬─────────────┘
│                 ▼
│   ┌───────────────────────────┐
│   │ main consumer             │── ok ──▶ commit offset
│   └─────────────┬───────┬─────┘
│        transient│       └── permanent: skip tiers ──────────────┐
│                 ▼                                               │
│   ┌───────────────────────────┐                                 │
│   │ orders.retry.1  delay 30s │                                 │
│   └─────────────┬─────────────┘                                 │
│  fail, attempt+1▼                                               │
│   ┌───────────────────────────┐                                 │
│   │ orders.retry.2  delay 5m  │                                 │
│   └─────────────┬─────────────┘                                 │
│  fail, attempt+1▼                                               │
│   ┌───────────────────────────┐                                 │
│   │ orders.retry.3  delay 30m │                                 │
│   └─────────────┬─────────────┘                                 │
│     max attempts▼                                               │
│   ┌───────────────────────────┐                                 │
│   │ orders.dlq                │◀────────────────────────────────┘
│   └─────────────┬─────────────┘
│                 │ triage, fix root cause, then replay (manual)
└─────────────────┘

Transient failures move through retry tiers with increasing delays and an attempt header that increments at each tier. Permanent failures skip the tiers, and both paths end in orders.dlq, which is replayed manually into the main topic after a fix.

How should you structure a DLQ record so it can be triaged and replayed?

The envelope rule is simple: the DLQ record carries the original key, value bytes and headers unchanged, and everything you add goes into new headers. Replay then means "send these exact bytes back", with no reconstruction step that could drift from what the producer originally wrote.

Write raw bytes, never deserialized objects

The DLQ handler must read and write byte[]. Configure its consumer with ByteArrayDeserializer and its producer with ByteArraySerializer. A malformed payload, an unknown schema ID, or a truncated message are among the most common reasons a record lands here. If the handler deserializes before writing, it throws on exactly the messages it exists to capture, and those messages either stall the partition or vanish.

In the companion code example, the producer builds a ProducerRecord with the original key and value, copies every entry from the source record's Headers, then calls headers().add(...) for each metadata field. Values are encoded as UTF-8 strings. Reusing the original key keeps related failures on the same DLQ partition.

Header Purpose
dlq.source.topic Where to replay to
dlq.source.partition Locate the original record
dlq.source.offset Deduplicate; check whether replay already happened
dlq.consumer.group Which service failed
dlq.exception.class Group failures during triage
dlq.exception.message Human-readable cause
dlq.attempt.count Distinguish transient from persistent failures
dlq.first.failure.ts / dlq.last.failure.ts Age and retry span

There is a cost to this. Full payloads, and especially stack traces stored in headers, increase storage. They can also copy personal data or secrets into a topic whose access controls and retention are looser than the source topic's. Truncate messages, omit stack traces or store them elsewhere, and give the DLQ the same access policy as the source topic.

import java.nio.charset.StandardCharsets;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.common.header.Header;

public class DlqWriter {
  // Producer configured with ByteArraySerializer for both key and value
  private final KafkaProducer<byte[], byte[]> producer;
  private final String dlqTopic;
  private final String consumerGroup;

  public DlqWriter(KafkaProducer<byte[], byte[]> producer, String dlqTopic, String consumerGroup) {
    this.producer = producer; this.dlqTopic = dlqTopic; this.consumerGroup = consumerGroup;
  }

  public void send(ConsumerRecord<byte[], byte[]> rec, Throwable err, int attempts, long firstFailureTs) {
    // Same key and raw value bytes: no deserialization, and related failures share a DLQ partition
    ProducerRecord<byte[], byte[]> out = new ProducerRecord<>(dlqTopic, rec.key(), rec.value());
    // Copy original headers unchanged so replay sends back exactly what was produced
    for (Header h : rec.headers()) out.headers().add(h);
    // Append metadata as UTF-8 strings
    add(out, "dlq.source.topic", rec.topic());
    add(out, "dlq.source.partition", String.valueOf(rec.partition()));
    add(out, "dlq.source.offset", String.valueOf(rec.offset()));
    add(out, "dlq.consumer.group", consumerGroup);
    add(out, "dlq.exception.class", err.getClass().getName());
    // Message only, no stack trace: limits header size and exposure of sensitive data
    add(out, "dlq.exception.message", String.valueOf(err.getMessage()));
    add(out, "dlq.attempt.count", String.valueOf(attempts));
    add(out, "dlq.first.failure.ts", String.valueOf(firstFailureTs));
    add(out, "dlq.last.failure.ts", String.valueOf(System.currentTimeMillis()));
    producer.send(out);
  }

  private static void add(ProducerRecord<byte[], byte[]> r, String k, String v) {
    r.headers().add(k, v.getBytes(StandardCharsets.UTF_8));
  }
}

A DLQ producer using the standard Java Kafka client that forwards the original key, value bytes and headers untouched and appends triage metadata as UTF-8 headers.

How do you triage a Kafka dead letter queue in practice?

Give every DLQ exactly one owning team, usually the team that owns the consumer. A DLQ shared by several teams has no clear owner, so messages pile up until someone audits it.

Alert on flow and age, not size

Total message count is a poor signal. A large backlog from last month's resolved incident does not need a page, while a sudden burst of new failures does. Alert on two things instead:

  • Ingress rate, which shows that something just started failing.
  • Age of the oldest unhandled message, which shows that triage is falling behind.

Group, then decide

Group DLQ records by exception type, plus the failing service or schema version if it is in the headers. For each group, choose one of three actions:

  • Fix the code and replay.
  • Fix the data upstream or in a transform, then replay.
  • Discard, and record the reason.

Grouping saves the most time. A thousand scattered failures often come from three root causes. Ownership plus grouping turns repeated investigations into a short list of fixes, each of which is resolved once.

Retention and cleanup policy

Set retention.ms so it comfortably outlasts your triage SLA. If triage can take a week, a three-day retention deletes evidence before anyone reads it.

Use cleanup.policy=delete, not compaction. Compaction keeps only the latest record per key. Two distinct failures for the same order ID, such as a deserialization error followed by a validation error, would collapse into one, and you would never see the first.

Apply the same access controls, encryption and data-retention obligations to the DLQ as to its source topics.

#!/usr/bin/env bash
# Create a DLQ topic for the orders consumer with explicit retention and delete cleanup.
# Placeholder values: adjust broker address, partitions and replication to your cluster.
set -euo pipefail

BOOTSTRAP="broker-1.internal.example:9092"
DLQ_TOPIC="orders-consumer.dlq"

# 14 days (14 * 24 * 60 * 60 * 1000 ms), chosen to outlast a 7-day triage SLA
# with margin, so evidence is still there when someone gets to it.
RETENTION_MS=1209600000

# cleanup.policy=delete: never compact a DLQ. Compaction keeps only the latest
# record per key, so two different failures for the same order ID would collapse
# into one and the earlier failure would be lost.
kafka-topics.sh --bootstrap-server "$BOOTSTRAP" \
  --create \
  --topic "$DLQ_TOPIC" \
  --partitions 3 \
  --replication-factor 3 \
  --config cleanup.policy=delete \
  --config retention.ms="$RETENTION_MS"

# Verify the topic-level overrides were applied as intended.
kafka-configs.sh --bootstrap-server "$BOOTSTRAP" \
  --describe \
  --entity-type topics \
  --entity-name "$DLQ_TOPIC"

Creates a DLQ topic with cleanup.policy=delete and a 14-day retention.ms that outlasts a one-week triage SLA, then confirms the topic-level config overrides.

How can you replay failed Kafka messages without causing new incidents?

Replay should go through a dedicated tool or consumer, never an ad hoc script. The tool reads from the DLQ and selects records by error type, time range or key. It can apply a transform, such as a schema fix. It republishes at a fixed rate and supports a dry-run mode that reports what would be sent without producing anything. Dry-run catches a selector that is too broad before it matters.

Target Isolation Effect on other consumers Subscription changes
Original topic None. Replayed records mix with live traffic Every subscriber sees them again, including services that succeeded the first time None
Dedicated replay topic Full. Can be paused, throttled or discarded on its own Only the failing consumer group sees them The consumer must also subscribe to the replay topic

Replay is at-least-once, so make consumers idempotent first. Use deduplication keys or idempotent downstream writes such as upserts keyed on a business ID.

Track attempts in a header such as replay-count. Check it before republishing, and once it exceeds the cap, tag the record as terminally failed instead of sending it back. Without this, a message that still fails will cycle between the DLQ and the main topic.

Old messages can overwrite newer state for the same key. Carry a version or event timestamp, and have writers reject anything older than what is stored.

After an outage, throttle bulk replay. Otherwise the backlog hits the recovering dependency all at once and can take it down again. The sketch below combines the selector, dry-run flag, rate limit and replay-count check.

When should you avoid a DLQ, and what does it cost to run one?

A DLQ deliberately breaks ordering. When a message is parked, later messages for the same key keep flowing. In domains where order is the invariant, such as account ledgers, inventory counts, or state machines driven by event sequence, a later event can then be applied on top of a missing earlier one. Two approaches fit these cases better:

  • Blocking retry. Accept the stall on that partition.
  • Version-checked writes. The consumer rejects any event whose expected version does not match the stored version, so an out-of-order apply cannot happen silently.

At the other extreme, low-value streams such as debug telemetry or cache-warming hints do not justify the machinery. Log the failure, emit a metric, and skip.

Strategy Ordering Availability Operational burden Data-loss risk
Blocking retry Preserved Partition stalls until fixed Low Low
Partition pause Preserved Partition stalls; others continue Medium (resume logic, lag alerts) Low
Log and skip Broken for skipped keys High Very low High (only logs remain)
DLQ with replay Broken for parked keys High High Low, if retention outlives triage

What a DLQ costs to run

A DLQ is a product with its own operating costs:

  • Infrastructure. You run extra topics and their retention, plus retry consumers.
  • Tooling. You need a replay tool with selectors and rate limits.
  • Idempotency work. Every downstream write must be made safe to repeat.
  • Monitoring. Alerts must cover DLQ depth and the age of the oldest message.
  • Ownership. A named on-call owner is needed. Without one, the topic becomes a place where failures pile up unexamined.

The cost is justified by what happens during incidents. Structured DLQ records turn root-cause analysis into a query instead of a search through logs. A tested replay path replaces scripts written under pressure, which are run once against production and rarely reviewed. If your team has written that kind of script more than once for the same stream, the DLQ is probably worth building.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kafka dead letter queue, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.