Spring Kafka Error Handling: Retries, DLTs and Offset Commits

What happens when a listener throws: offsets, seeks and the DefaultErrorHandler

Spring Kafka error handling is an offset-commit problem. Kafka tracks progress per partition as a single committed offset, so the container must never commit past a record the listener failed to process. This is why the listener container only commits after your listener returns normally: a thrown exception means "this offset is not done."

When a record listener throws, the DefaultErrorHandler takes over. It seeks the failed partition back to the failed record's offset and discards the rest of the in-memory batch, so the next poll() redelivers the failed record and everything after it. Between attempts it waits according to a configurable BackOff. When the BackOff is exhausted, it calls a ConsumerRecordRecoverer, which by default logs the record. After recovery, the handler can commit the recovered offset so the partition moves on.

Copypoll() --> listener(record) --throws--> DefaultErrorHandler
  ^                                        |
  |            seek(partition, offset) <---+ (BackOff not exhausted)
  +----------- next poll redelivers -------+
                                           |
                     BackOff exhausted --> recoverer --> commit offset

The container AckMode controls when successful offsets are committed:

Because a seek rewinds the consumer's position before any commit for the failed offset, a crash during retries replays the failed record, not data after it.

Copy+-----------+       +------------------+
|  poll()   |------>| listener(record) |
+-----------+       +------------------+
   ^    ^             |              |
   |    |          returns         throws
   |    |             v              v
   |    |   +----------------+  +---------------------+
   |    +---| commit offsets |  | DefaultErrorHandler |
   |        | per AckMode    |  +---------------------+
   |        +----------------+     |               |
   |                          BackOff left   BackOff exhausted
   |                               v               v
   |                 +------------------+  +-------------------+
   +-----------------| seek(partition,  |  | recoverer(record) |
     next poll       |  failed offset)  |  +-------------------+
     redelivers      | wait per BackOff |            |
     failed record   +------------------+            v
     and the rest                        +----------------------+
                                         | commit recovered     |
                                         | offset; partition    |
                                         | moves on             |
                                         +----------------------+

When a listener throws, DefaultErrorHandler seeks back to the failed offset for redelivery until the BackOff is exhausted, then calls the recoverer and commits past the record.

Configuring blocking retries, non-retryable exceptions and a dead-letter topic

Construct a DefaultErrorHandler with a recoverer and a BackOff, then register it on the listener container factory. FixedBackOff(interval, maxAttempts) counts retries, not deliveries, so a value of 2 means three attempts in total. ExponentialBackOff fits transient faults such as a briefly unavailable downstream service. Cap it with a maximum interval and a maximum elapsed time so it cannot run indefinitely. Use addNotRetryableExceptions for failures that will never succeed on a retry, like validation errors or a missing required field. Those records go straight to the recoverer, and the partition is not held up.

DeadLetterPublishingRecoverer sends exhausted records to <original-topic>.DLT by default, on the same partition number. The DLT therefore needs at least as many partitions as the source topic. If you want different routing, pass a destination resolver that maps the record and the exception to a TopicPartition. A negative partition lets the producer choose one. Each published record carries headers that describe the exception (class, message, stack trace) and the original topic, partition, offset and timestamp. With these headers, a replay tool can work out where a record came from and why it failed.

Wrap your key and value deserializers in ErrorHandlingDeserializer. Without it, a malformed record fails inside poll() before your listener runs, and the consumer can loop on that offset forever. With the wrapper, the failure reaches the error handler as a DeserializationException, which is non-retryable by default, and the record goes to the DLT.

Blocking retries keep per-partition ordering, but they stall the partition while they run. Illustrative estimate: with exponential backoff starting at 1 s, a multiplier of 2 and five retries, the waits are 1+2+4+8+16 = 31 s. Add processing time and the rest of the polled batch on top of that. The total must stay comfortably below max.poll.interval.ms. If it does not, the broker considers the consumer dead and starts a rebalance.

Copy@Configuration
public class KafkaErrorHandlingConfig {

  @Bean
  public DefaultErrorHandler errorHandler(KafkaTemplate<Object, Object> template) {
    // Route failed records to <topic>.DLT; partition -1 lets the producer pick one,
    // so the DLT does not need to match the source topic's partition count
    var recoverer = new DeadLetterPublishingRecoverer(template,
        (record, ex) -> new TopicPartition(record.topic() + ".DLT", -1));

    // Transient faults: 1s, 2s, 4s, ... capped per wait and in total
    var backOff = new ExponentialBackOff(1_000L, 2.0);
    backOff.setMaxInterval(10_000L);
    backOff.setMaxElapsedTime(30_000L); // keep well below max.poll.interval.ms

    var handler = new DefaultErrorHandler(recoverer, backOff);
    // Retrying will never fix these, so send them straight to the DLT
    // (InvalidOrderException is an application-defined validation exception)
    handler.addNotRetryableExceptions(InvalidOrderException.class);
    return handler;
  }

  @Bean
  public ConsumerFactory<String, Object> consumerFactory() {
    Map<String, Object> props = new HashMap<>();
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
    props.put(ConsumerConfig.GROUP_ID_CONFIG, "orders-service");
    // Wrap deserializers so poison pills reach the error handler
    // instead of failing inside poll() on every attempt
    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, ErrorHandlingDeserializer.class);
    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, ErrorHandlingDeserializer.class);
    props.put(ErrorHandlingDeserializer.KEY_DESERIALIZER_CLASS, StringDeserializer.class);
    props.put(ErrorHandlingDeserializer.VALUE_DESERIALIZER_CLASS, JsonDeserializer.class);
    props.put(JsonDeserializer.TRUSTED_PACKAGES, "com.example.orders");
    return new DefaultKafkaConsumerFactory<>(props);
  }

  @Bean
  public ConcurrentKafkaListenerContainerFactory<String, Object> kafkaListenerContainerFactory(
      ConsumerFactory<String, Object> cf, DefaultErrorHandler errorHandler) {
    var factory = new ConcurrentKafkaListenerContainerFactory<String, Object>();
    factory.setConsumerFactory(cf);
    factory.setCommonErrorHandler(errorHandler);
    return factory;
  }
}

Configures a DefaultErrorHandler that uses a capped ExponentialBackOff, skips retries for a validation exception, and sends failed records to a .DLT through a custom destination resolver; ErrorHandlingDeserializer passes poison pills to that handler.

Non-blocking retries with retry topics

Retry topics avoid that constraint because they never hold the main partition. With @RetryableTopic on a @KafkaListener method, a failed record is republished to a separate retry topic, and the original offset is committed. The main partition moves on to the next record right away. You set the attempt count, the backoff (for example, a delay with a multiplier), and which exceptions to include or exclude on the annotation. If you want one place to apply the same policy to several listeners, declare a RetryTopicConfiguration bean, usually built with RetryTopicConfigurationBuilder.

The framework creates and manages these pieces:

• Retry topics. These are the main topic name plus a suffix. By default each backoff step gets its own topic. A fixed delay can share a single topic.
• A dead-letter topic. It takes the -dlt suffix and receives the record after the final attempt.
• Listener containers. Each retry topic and the DLT get one, and they all run in the same consumer group.

Each republished record carries a header with the time it becomes due. If the retry consumer polls a record before that time, it pauses the partition, seeks back to the record, and resumes once the delay has passed. Delays therefore depend on pause/resume timing and the poll cycle. They are approximate, with no precise timer behind them.

Avoid retry topics when per-key order matters. A retried record can land after later records with the same key that were already processed. Order-dependent work such as state transitions, event sourcing, and balance updates can apply events out of order. Retry topics also add topics, consumers, lag metrics, and partition-count decisions that your team has to provision and monitor.

Copy// Option 1: per-listener policy on the annotation.
// 4 attempts total = 1 main delivery + 3 retries (1s, 2s, 4s), then the DLT.
@RetryableTopic(
    attempts = "4",
    backoff = @Backoff(delay = 1000, multiplier = 2.0),
    exclude = { IllegalArgumentException.class }) // bad data: send straight to DLT
@KafkaListener(topics = "orders", groupId = "order-service")
public void onOrder(OrderEvent event) {
    orderService.apply(event); // a throw republishes to a retry topic; main offset is committed
}

// Receives the record after the final attempt (from the orders DLT).
@DltHandler
public void onOrderDlt(OrderEvent event,
                       @Header(KafkaHeaders.RECEIVED_TOPIC) String topic) {
    log.error("Giving up on order {} from {}", event.id(), topic);
}

// Option 2: one shared policy applied to listeners on the listed topics.
@Bean
public RetryTopicConfiguration paymentsRetryConfig(KafkaTemplate<String, PaymentEvent> template) {
    return RetryTopicConfigurationBuilder
        .newInstance()
        .exponentialBackoff(1000, 2.0, 10000) // initial 1s, x2 each step, capped at 10s
        .maxAttempts(4)
        .retryOn(TransientDataAccessException.class) // retry only transient failures
        .includeTopic("payments")
        .create(template); // template publishes to the retry topics and the DLT
}

A @RetryableTopic listener with a @DltHandler, and a shared RetryTopicConfiguration bean. Both send failed records through delayed retry topics to a dead-letter topic, so the main partition does not wait on a failing record.

Manual acknowledgment and batch listeners

Set the container's ack mode to MANUAL or MANUAL_IMMEDIATE and add an Acknowledgment parameter to the listener. Call acknowledge() only after the record's side effects are durable: the database transaction has committed, the downstream call has returned success, the file is written. If you acknowledge first and the process dies before the work finishes, the offset says the record is done when it is not, and nothing will ever redeliver it.

To back off without throwing, call nack with a sleep duration. The container seeks back to that record and pauses before redelivering it. This is useful when a dependency is briefly unavailable and you want to wait rather than run the error handler.

A Kafka commit is a single position per partition, so it cannot express "everything except offset 1042." If a record is never acknowledged, there are two possible outcomes. The committed offset stays at that record while the consumer keeps working ahead of it, and a restart or rebalance replays everything processed since. Or a later acknowledgment moves the committed offset past the unacknowledged record, and that record is lost without any error. Track unacknowledged records explicitly.

Batch listeners make this harder because a single offset covers many records. If record 37 of 100 fails, records 0–36 are done and should not run again. Throw BatchListenerFailedException with the failing record's index. DefaultErrorHandler then commits the offsets before that index and seeks so retries start at record 37. If you throw a plain exception instead, the handler cannot tell which record failed. It retries the whole batch, so records 0–36 are reprocessed on every attempt.

Copy@Component
public class OrderListeners {
    private final OrderService orders; // writes to DB in its own transaction
    OrderListeners(OrderService orders) { this.orders = orders; }

    // Container factory must use AckMode.MANUAL or MANUAL_IMMEDIATE:
    // factory.getContainerProperties().setAckMode(ContainerProperties.AckMode.MANUAL);
    @KafkaListener(topics = "orders", groupId = "order-writer")
    public void onOrder(ConsumerRecord<String, String> rec, Acknowledgment ack) {
        try {
            orders.save(rec.value());   // returns only after the DB commit
            ack.acknowledge();          // commit the offset after the side effect is durable
        } catch (DependencyUnavailableException e) {
            // Seek back to this record, pause, then redeliver without invoking the error handler
            ack.nack(Duration.ofSeconds(5));
        }
    }

    // Batch factory: factory.setBatchListener(true) plus a DefaultErrorHandler
    @KafkaListener(topics = "orders-bulk", groupId = "order-bulk-writer",
                   containerFactory = "batchFactory")
    public void onBatch(List<ConsumerRecord<String, String>> batch, Acknowledgment ack) {
        for (int i = 0; i < batch.size(); i++) {
            try {
                orders.save(batch.get(i).value());
            } catch (RuntimeException e) {
                // The index tells DefaultErrorHandler to commit records 0..i-1
                // and seek so retries start at record i.
                // A plain exception would retry the whole batch instead.
                throw new BatchListenerFailedException("save failed", e, i);
            }
        }
        ack.acknowledge(); // every record in the batch succeeded
    }
}

A record listener acknowledges only after a durable write and uses nack(Duration) to pause and redeliver. A batch listener throws BatchListenerFailedException with the failing index so DefaultErrorHandler commits the earlier records and retries from the failed one.

Choosing a strategy: who should use which approach and when to avoid it

Each approach trades ordering and partition availability against throughput and the number of moving parts you have to run.

Blocking retries are a poor choice when backoffs run into minutes, because every record behind the failure waits. Retry topics are wrong for state-machine or account-balance streams, where processing event N+1 before N corrupts state. Use manual ack only when the commit point really must differ from "listener returned". Without that need, it adds code that can lose or duplicate commits.

The hardest call is classifying exceptions. If you retry a deserialization error or a null-pointer bug, the outcome is the same failure every time, and the partition stays blocked through the whole backoff. If you mark a connection timeout as non-retryable, a database outage sends every in-flight record to the DLT, and you must replay them all after recovery. Classify by cause rather than exception type. Wrapped exceptions often hide the real cause, so check what the handler actually traverses.

Idempotency, duplicates and the limits of exactly-once

Kafka consumers in Spring are at-least-once by default, and no combination of error handler settings changes that. A record is processed first and its offset is committed afterward, so anything that happens between those two steps produces a redelivery. A pod can be killed after the database write but before the commit. A rebalance can revoke a partition while a batch is mid-flight. The DefaultErrorHandler can seek back and re-fetch records whose side effects already ran. A retry topic can deliver a copy after the original attempt partially succeeded. Your listener will see some records twice, so design for it.

An idempotent listener pattern

Make the side effect safe to repeat. There are two common ways to do this:

• Deduplication key. Have producers stamp each event with a stable ID, usually in a header. Insert that ID into a processed_events table that has a unique constraint, in the same database transaction as the business write. If the insert hits a duplicate-key violation, skip the record and let the offset commit. Don't use the topic-partition-offset as the key, because a producer retry can write the same event at a new offset.
• Upsert. Key the write on the business identifier and include a version or event timestamp. The write only applies if it is newer than what is stored, so replays converge on the same final state.

What Kafka transactions do not cover

Kafka transactions make consume-transform-produce atomic: the output records and the consumer offsets commit together. That guarantee ends at the Kafka boundary. A database insert, an HTTP call or an email cannot take part in a Kafka transaction, so those writes still need one of the patterns above.

Testing failure paths, operating DLTs and the impact on engineering time

Test the failure paths, not just the happy path

Run integration tests against a real broker, either the embedded broker from spring-kafka-test or a Kafka container via Testcontainers. Mocks cannot show how seeks, back-off and commits interact. For each failure scenario, assert three things:

• Retry count. Have the listener increment a counter. A poison record should be attempted exactly as many times as your back-off allows. A non-retryable exception should be attempted once.
• DLT contents. Consume from the dead-letter topic. Check the original key, the payload, and the exception headers the recoverer adds.
• Committed offsets. Read the group's offsets with the Kafka Admin client. Confirm the consumer moved past the failed record and did not stall on it or skip unrelated records.

A DLT needs an owner

A dead-letter topic that nobody watches still loses data. The loss is just slower and less visible. Every DLT needs three things:

• An owning team.
• An alert on its growth rate. Growth that keeps climbing usually means a systemic bug, not a few bad records.
• A documented replay process. That includes a tool that republishes selected records to the source topic once the fix is deployed. Replay only works safely if consumers are idempotent.

The cost, and what it buys

This work adds retry and DLT topics, consumers or tooling for them, dashboards, alerts and a replay utility. All of it has to be built and maintained. In return, an incident becomes easier to handle. When commit semantics are explicit and the failure paths are tested, responders can tell which records were processed, which were parked and which need replay. They can then replay instead of reconstructing state by hand from logs and database queries.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to spring kafka error handling, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.