Spring Kafka Error Handling: DefaultErrorHandler vs Retry Topics

What happens when a @KafkaListener throws

Spring Kafka error handling is the set of decisions the listener container makes when your @KafkaListener method throws. Those decisions matter because Kafka has no per-message acknowledgement. The consumer tracks one committed offset per partition, so a single failing record affects everything behind it in that partition.

When a listener throws, the container passes the exception to its error handler, which in current Spring Kafka is DefaultErrorHandler. The handler does not skip the record. It seeks the consumer back to the failed offset, discarding the rest of that batch, so the next poll() redelivers the same record. By default it does this a small, fixed number of times, then logs the failure and moves on.

Problems start when engineers configure this without understanding it. A back-off with unlimited attempts, or long delays across many attempts, keeps the consumer on one offset. Every later record in that partition waits, and consumer lag grows even though the rest of the topic is healthy.

Deserialization failures are worse. If the value deserializer throws inside poll(), the exception happens before your listener runs. Without an ErrorHandlingDeserializer wrapping the real deserializer, the consumer can fetch the same unreadable bytes again and again in a tight loop.

Spring Kafka offers two ways out:

• Blocking in-place retry: DefaultErrorHandler with a bounded back-off.
• Non-blocking retry: @RetryableTopic, which moves the failed record to separate retry topics.

Both should end in a dead letter topic when retries run out. Both give at-least-once delivery, so a record can be processed more than once.

Blocking retries with DefaultErrorHandler and a dead letter topic

DefaultErrorHandler retries by re-seeking the consumer to the failed offset. The partition makes no progress while it waits, so retries must be bounded. A FixedBackOff takes an interval and a retry count. The total number of deliveries is that count plus one. An ExponentialBackOff grows the delay and stops once the maximum elapsed time is reached. Pass the backoff and a recoverer to the handler's constructor, then register it with setCommonErrorHandler on the ConcurrentKafkaListenerContainerFactory. Keep the total backoff well below max.poll.interval.ms, or the broker will rebalance the consumer out of the group.

Retrying is pointless when the failure is deterministic. Call addNotRetryableExceptions for validation errors so those records go straight to the recoverer. Several conversion and deserialization exceptions are already non-retryable by default. You can adjust the list with removeClassification or setClassifications.

DeadLetterPublishingRecoverer publishes the failed record through a KafkaTemplate. By default it sends to the source topic name with a .DLT suffix, on the same partition number. The DLT therefore needs at least as many partitions as the source topic, or you must supply a custom destination resolver. The recoverer also adds headers prefixed kafka_dlt- that hold the exception class, message, stack trace, original topic, partition and offset. These headers let you diagnose a failure without searching the logs.

If the payload can't be deserialized, the failure happens inside poll(), before the listener is called, and the consumer can crash-loop on that record. Wrap your deserializer in ErrorHandlingDeserializer so the failure reaches the handler and the raw bytes go to the DLT.

Batch listeners need explicit handling. Throw BatchListenerFailedException with the index of the failing record. The handler then commits the offsets before that record, retries or recovers the failed record, and redelivers the rest of the batch.

Copy@Configuration
public class KafkaErrorHandlingConfig {

  @Bean
  public ConsumerFactory<String, Order> consumerFactory() {
    Map<String, Object> props = new HashMap<>();
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    props.put(ConsumerConfig.GROUP_ID_CONFIG, "orders-service");
    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
    // Wrap the real deserializer so bad payloads reach the error handler instead of failing poll()
    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ErrorHandlingDeserializer.class);
    props.put(ErrorHandlingDeserializer.VALUE_DESERIALIZER_CLASS, JsonDeserializer.class);
    props.put(JsonDeserializer.TRUSTED_PACKAGES, "com.example.orders");
    return new DefaultKafkaConsumerFactory<>(props);
  }

  @Bean
  public DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> dltTemplate) {
    // Sends to <topic>.DLT on the same partition and adds kafka_dlt-* diagnostic headers.
    // The template's value serializer must also handle raw byte[] from deserialization failures.
    DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(dltTemplate);
    // 1s interval, 2 retries => 3 deliveries; the partition is blocked ~2s per failing record
    DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 2L));
    // Deterministic failures go straight to the recoverer without retrying
    handler.addNotRetryableExceptions(InvalidOrderException.class);
    return handler;
  }

  @Bean
  public ConcurrentKafkaListenerContainerFactory<String, Order> kafkaListenerContainerFactory(
      ConsumerFactory<String, Order> consumerFactory, DefaultErrorHandler errorHandler) {
    ConcurrentKafkaListenerContainerFactory<String, Order> factory =
        new ConcurrentKafkaListenerContainerFactory<>();
    factory.setConsumerFactory(consumerFactory);
    factory.setCommonErrorHandler(errorHandler);
    return factory;
  }
}

Spring configuration that wraps the JSON deserializer in ErrorHandlingDeserializer, bounds blocking retries with a FixedBackOff, marks validation failures as non-retryable, and routes exhausted records to a .DLT topic via DeadLetterPublishingRecoverer.

Non-blocking retries with @RetryableTopic

@RetryableTopic moves the failed record out of the way. It does not hold the partition while retrying. You annotate the listener and set attempts, which counts the first delivery, so attempts = 4 means three retries. Set backoff with @Backoff(delay, multiplier, maxDelay). Use include or exclude to choose which exceptions are retried. Certain exceptions go straight to the DLT without being retried, such as deserialization and message conversion failures, because retrying them cannot succeed. If you want the same policy across many listeners, declare a RetryTopicConfiguration bean instead of repeating the annotation.

Spring creates the retry topics and the dead letter topic from the main topic's name. With index-based suffixing, orders gets orders-retry-0, orders-retry-1 and orders-dlt. A method annotated @DltHandler in the same class receives records that run out of attempts. This is your terminal handling, typically an alert or a write to a failure store.

Copyorders ──fail──> orders-retry-0 ──fail──> orders-retry-1 ──fail──> orders-dlt
  │               (wait 1s)                (wait 2s)                  │
  └─ continues with next record                                 @DltHandler

When a delivery fails, the framework publishes the record to the next retry topic. The record carries a header with its due timestamp. The consumer on that retry topic checks the header. If the record is not due yet, the consumer pauses that partition, seeks back, and resumes once the time has passed. Meanwhile the main topic keeps flowing.

This approach has two costs:

• Per-key ordering is lost. A later record for the same key can be processed before an earlier one that is still waiting in a retry topic.
• Delays are approximate. A record is reprocessed no earlier than its due time, but it can run later. The actual delay depends on poll timing and how long the partition stays paused.

Copyimport org.apache.kafka.clients.consumer.ConsumerRecord;
import org.springframework.kafka.annotation.DltHandler;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.annotation.RetryableTopic;
import org.springframework.kafka.retrytopic.TopicSuffixingStrategy;
import org.springframework.retry.annotation.Backoff;
import org.springframework.stereotype.Component;

@Component
public class OrderListener {

    // attempts counts the first delivery: 4 = 1 original + 3 retries.
    // Exponential backoff 1s, 2s, 4s (capped at 10s); each delay gets its own retry topic.
    // Index suffixing yields orders-retry-0, orders-retry-1, orders-retry-2 and orders-dlt.
    @RetryableTopic(
        attempts = "4",
        backoff = @Backoff(delay = 1000, multiplier = 2.0, maxDelay = 10000),
        topicSuffixingStrategy = TopicSuffixingStrategy.SUFFIX_WITH_INDEX_VALUE,
        // Bad input will never succeed on retry: send it straight to the DLT.
        exclude = { IllegalArgumentException.class })
    @KafkaListener(topics = "orders", groupId = "order-service")
    public void onOrder(ConsumerRecord<String, String> record) {
        // Throwing here forwards the record to the next retry topic with a due-time header;
        // the main partition moves on to the next record immediately.
        processOrder(record.value());
    }

    // Called for records that exhausted all attempts or hit an excluded exception.
    @DltHandler
    public void onDlt(ConsumerRecord<String, String> record) {
        // Terminal handling: alert and persist for manual inspection or replay.
        System.err.printf("Order dead-lettered: topic=%s key=%s offset=%d%n",
            record.topic(), record.key(), record.offset());
    }

    private void processOrder(String payload) {
        // business logic that may throw
    }
}

A @RetryableTopic listener with four attempts, exponential backoff, index-suffixed retry topics, an exception exclude list, and a @DltHandler for terminal handling.

Choosing a strategy: when to use each and when to avoid it

Use blocking retries when order matters and failures clear within seconds. Avoid them for anything that may last minutes, because one bad dependency freezes the partition and can trigger a rebalance. Use retry topics when throughput matters more than order. Avoid them for entity-keyed streams where a later event must not overtake an earlier one. Use DLT-only for errors that retrying cannot fix. Most setups combine it with one of the other two by classifying those exceptions as not retryable.

The service team owns the decision because it knows which failures are transient. Platform engineers should review it, since retry topics add topics, partitions and ACLs.

Alternatives:

• Spring Retry, or a retry loop inside the listener. This is simple, but it hides retries from the container.
• Pausing the container while a dependency is down.
• An application-managed outbox or delay store, for precise or very long delays.
• Validating upstream so bad records are never produced.

Idempotent listeners and failure modes of the recoverer

Both strategies give at-least-once delivery, so any listener can see the same record more than once. A consumer can finish its side effect (a database write, an HTTP call, an email) and then crash, or lose its partitions in a rebalance, before the offset is committed. The next owner of the partition reprocesses the record. Retries repeat side effects in the same way: if an attempt wrote one row and then failed on the second call, the next attempt writes that row again.

Making the listener idempotent

• Idempotency keys. Derive a stable key from the producer's business identifier, or from topic, partition and offset if nothing better exists. Carry the key through to downstream APIs that support deduplication.
• Upserts. Write with insert-or-update semantics keyed on that identifier, so a replay converges to the same state.
• Processed-message table. Record the key in the same database transaction as the side effect, and skip the record if the key is already there. This only works when the side effect and the marker share one transaction.

When the recoverer itself fails

The DLT publish can fail for several reasons:

• The broker is unavailable.
• The producer is not authorized to write to the DLT.
• The DLT does not exist, or it has fewer partitions than the source topic and the default resolver targets the same partition number.
• The producer's serializer cannot handle the payload, which is common when a deserialization failure leaves raw bytes.

In each case recovery fails, the offset is not committed, and the record is redelivered. That is safe, but on a blocking setup it can stall the partition indefinitely. Alert on recoverer failures, and provision DLTs explicitly.

Misclassification costs you in both directions. If you retry permanent failures, you burn attempts and add latency, and under blocking retries you hold up every record behind the bad one. If you mark transient errors as non-retryable, recoverable records go to the DLT and someone has to replay them by hand.

Copy@Bean
DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> dltTemplate) {
    // Partition -1 leaves the partition unset, so the producer chooses one.
    // A DLT with fewer partitions than the source topic still accepts the record.
    var recoverer = new DeadLetterPublishingRecoverer(dltTemplate,
        (rec, ex) -> new TopicPartition(rec.topic() + ".DLT", -1));

    // 3 retries, 1s apart. If the DLT send fails, recovery fails and the
    // record is redelivered rather than having its offset committed.
    var handler = new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 3L));

    // Permanent failures go to the DLT immediately instead of wasting retries
    handler.addNotRetryableExceptions(InvalidOrderException.class);
    return handler;
}

@KafkaListener(topics = "orders", groupId = "order-service")
@Transactional // marker and side effect commit or roll back together
public void onOrder(Order order) {
    // Processed-message table: a redelivered record inserts 0 rows and is skipped
    int inserted = jdbc.update(
        "INSERT INTO processed_messages (message_id) VALUES (?) ON CONFLICT DO NOTHING",
        order.id());
    if (inserted == 0) {
        return;
    }
    // Upsert keyed on the business id, so a replay converges to the same state
    jdbc.update(
        "INSERT INTO orders (id, amount) VALUES (?, ?) "
            + "ON CONFLICT (id) DO UPDATE SET amount = EXCLUDED.amount",
        order.id(), order.amount());
}

A DefaultErrorHandler that sends permanent failures straight to the DLT and lets the producer choose the DLT partition, paired with a listener that uses a processed-message table and an upsert in one database transaction so redeliveries do not duplicate side effects.

Operating the dead letter topic and replaying records

Treat the DLT as a queue with a named owner and a response time, not as an archive. A record there is a business event that did not happen. Alert on the DLT's message rate rather than its total size, because any new arrival deserves a look. Also alert on consumer lag for the main and retry consumer groups using kafka-consumer-groups.sh --describe. Growing lag on a retry topic usually means a dependency is down, before anything reaches the DLT.

To triage, read the DLT with kafka-console-consumer.sh and set print.key=true and print.headers=true. The DeadLetterPublishingRecoverer adds headers such as kafka_dlt-exception-fqcn, kafka_dlt-exception-message, kafka_dlt-original-topic and kafka_dlt-original-offset. Group failures by exception class to separate one bug from many unrelated problems.

Replay only after the fix is deployed. Use a small dedicated tool, not an ad hoc console producer, and have it do the following:

• Select records by exception type or offset range.
• Republish each record to the original topic with its original key, so partitioning and per-key ordering are preserved.
• Add a replay-count header and refuse to replay a record more than once or twice. Without that limit, a fix that did not work sends records around the same loop indefinitely.

Replays can deliver duplicates, so the idempotent listeners described earlier are what make this safe.

Finally, provision retry topics and DLTs the same way you provision production topics, through your infrastructure tooling or NewTopic beans, and turn off auto-creation. The default recoverer publishes to the same partition number as the source record, so the DLT needs at least as many partitions as the source topic. Set its replication factor and ACLs to match production. Set retention to cover your realistic time-to-fix.

Copy#!/usr/bin/env bash
# Operational checks for a DLT, plus explicit provisioning of retry/DLT topics.
set -euo pipefail
BOOTSTRAP=kafka-1.internal:9092
SRC=orders
DLT=orders-dlt

# 1. Lag for the main and retry consumer groups. Lag that grows on the retry
#    group usually means a dependency is down, before records reach the DLT.
kafka-consumer-groups.sh --bootstrap-server "$BOOTSTRAP" --describe --group orders-service
kafka-consumer-groups.sh --bootstrap-server "$BOOTSTRAP" --describe --group orders-service-retry

# 2. Triage: print keys and headers (kafka_dlt-exception-fqcn,
#    kafka_dlt-original-topic, kafka_dlt-original-offset, ...) without
#    committing offsets for any real consumer group.
kafka-console-consumer.sh --bootstrap-server "$BOOTSTRAP" \
  --topic "$DLT" --from-beginning --max-messages 20 \
  --property print.key=true --property print.headers=true

# 3. Provision the DLT explicitly. The broker should run with
#    auto.create.topics.enable=false so a typo fails loudly.
#    The default recoverer writes to the same partition number as the source
#    record, so the DLT needs at least as many partitions as the source topic.
kafka-topics.sh --bootstrap-server "$BOOTSTRAP" --describe --topic "$SRC"

kafka-topics.sh --bootstrap-server "$BOOTSTRAP" --create --if-not-exists \
  --topic "$DLT" \
  --partitions 12 \
  --replication-factor 3 \
  --config min.insync.replicas=2 \
  --config retention.ms=1209600000  # 14 days: longer than your triage SLA

# Grant the service principal write access to the DLT. Grant read access only
# to the replay tool, not to everyone.
kafka-acls.sh --bootstrap-server "$BOOTSTRAP" --add \
  --allow-principal User:orders-service --operation Write --topic "$DLT"

Standard Kafka CLI commands that check consumer lag on the main and retry groups, inspect DLT records with their keys and Spring's dead-letter headers, and provision the DLT with a partition count, replication factor, retention and ACLs that match production.

Testing failure paths, and the impact on cost and engineering time

Run failure-path tests against a real broker. @EmbeddedKafka from spring-kafka-test starts quickly and is enough for most cases. A Testcontainers Kafka container is better when you need production-like broker settings. In both cases, use the same container factory and error handler beans as production, but shrink backoff intervals to milliseconds through test properties.

Make the listener throw on demand and count invocations with an AtomicInteger, so you can assert the exact number of attempts. Record a timestamp per attempt and assert that the gaps grow as your backoff policy says. Then consume from the DLT and check the payload, key and partition. Also check the headers your replay runbook depends on: KafkaHeaders.DLT_ORIGINAL_TOPIC, DLT_ORIGINAL_OFFSET and DLT_EXCEPTION_FQCN. Add one test that throws a non-retryable exception and asserts it reaches the DLT after a single attempt. That test catches classification mistakes before they reach production.

On cost, retry topics add topics, partitions, consumers and lag alerts. Blocking retries add none of these.

Illustrative estimate (assumed numbers, not a benchmark):

• Extra topics = source topics × (retry topics + 1 DLT). With 10 source topics and 3 retry topics, that is 10 × 4 = 40 extra topics to provision, secure and monitor.
• Hours saved per month = incidents × (ad-hoc repair hours − runbook hours). With 4 incidents, 6 hours of ad-hoc repair and 1.5 hours with a runbook, that is 4 × (6 − 1.5) = 18 hours/month.

The savings come from failures behaving predictably. Classified exceptions, standard recoverers and tested headers make failed records easy to find and replay, so engineers spend less time on forensics and manual data repair.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to spring kafka error handling, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.