What does exactly-once semantics actually mean in Kafka?

Kafka delivery guarantees are defined by what happens when something fails partway through. Three failure points matter:

  • Producer retries. The broker writes a record, but the acknowledgement is lost, so the producer sends it again.
  • Consumer crashes between processing a record and committing its offset.
  • Broker failover. A partition leader dies with requests in flight, and clients retry against the new leader.
Semantics Typical configuration Producer retry after lost ack Consumer crash before offset commit
At-most-once Commit offsets before processing, no retries Record may be lost Record skipped
At-least-once Process, then commit, with retries enabled Duplicate write Record reprocessed, output duplicated
Exactly-once Idempotent producer, transactions, read_committed consumers Broker deduplicates Output and offsets roll back together

The exactly-once guarantee is narrower than the name suggests. The effect on Kafka state is applied once. Your processing code may still run more than once. After a crash, a consumer re-reads records from the last committed offset and runs your logic again. Kafka discards the aborted output and the uncommitted offsets, so downstream read_committed consumers see each result once. Anything your code did outside Kafka, such as an HTTP call, a database write or an email, has already happened and is not rolled back.

This matters mainly to teams running consume-transform-produce pipelines, where input and output are both Kafka topics. Duplicate output there inflates aggregates, double-counts payments, or forces every downstream consumer to deduplicate on its own. Engineers reach for exactly-once to handle that problem once, at the pipeline level, instead of in every consumer.

Who should use Kafka exactly-once?

Exactly-once is a good fit when both ends of a processing step are Kafka topics and a duplicate or missing output record has a real cost. Typical users:

  • Stream processing teams on Kafka Streams. Enabling the exactly-once processing guarantee covers output topics, consumed offsets and state-store changelogs in one transaction. For stateful operations such as counts, sums, joins and windowed aggregations, this keeps state and output consistent after a crash without extra application code.
  • Teams running hand-written consume-transform-produce services with the plain producer and consumer clients, such as enrichment, filtering, routing or format conversion between topics, where downstream consumers would otherwise each need their own deduplication.
  • Pipelines feeding financial, billing or audit topics, where a duplicated event becomes a double charge or a wrong balance further down the chain.
  • Platform teams that own many downstream consumers and want one correctness guarantee at the producing pipeline instead of asking every consuming team to build deduplication.

It is a poor fit, or adds cost without benefit, in these cases:

  • The main side effect is outside Kafka: a database write, an external API call, an email. Transactions do not cover these. An idempotent consumer is simpler and correct (see the alternatives section below).
  • Latency-sensitive fan-out, where waiting for transaction commits before consumers can read is unacceptable.
  • Pipelines whose consumers you do not control and cannot guarantee will run with read_committed. Those consumers stay at-least-once regardless of what the producer does.
  • Workloads that already tolerate duplicates, such as idempotent upserts by key into a compacted topic, or metrics where occasional double counting does not matter.

How does exactly-once work, end to end?

Exactly-once in Kafka is three mechanisms layered on top of each other. Each one closes a gap the previous one leaves open:

  1. Idempotent producer. The broker assigns each producer session a producer ID, and every batch carries a per-partition sequence number. The partition leader drops retried batches it has already written. This removes duplicates caused by network retries within one producer session and one partition.
  2. Transactions. A stable transactional.id lets the broker-side transaction coordinator recognize a producer across restarts and fence older instances using an epoch. A transaction groups writes to many partitions, plus the consumer's offset commit, into one atomic unit. The coordinator finishes it by writing commit or abort markers to every partition involved.
  3. read_committed consumers. Downstream readers only receive records from committed transactions. The broker serves data up to the last stable offset, and the client filters out aborted records and the markers themselves.

Put together, a consume-transform-produce step reads input, writes output and records its input position as one transaction. If the step crashes, the output and the offset advance are discarded together, the input is re-read, and downstream readers only ever see the committed result. The sections below cover each layer in detail and then describe where the guarantee ends.

How does the idempotent producer prevent duplicate writes?

The first layer of protection sits in the producer. When idempotence is enabled, the broker assigns the producer a producer ID (PID) at initialization. The producer then stamps every batch with that PID and a sequence number that increases monotonically per partition. The partition leader stores the most recent sequence numbers it has accepted for each PID.

Consider a common failure. The producer sends a batch, the broker writes it, and the acknowledgment is lost on the network. The producer retries. The broker sees a PID and sequence number it has already accepted, so it returns success without appending the batch again. If a sequence number arrives out of order, which means a gap where a batch went missing, the broker rejects it with an error instead of silently reordering the log.

acks=all is required because the sequence state must survive a leader change. If the leader acknowledged before followers had replicated the batch and then failed, the new leader would have neither the data nor the record of that sequence number. The guarantee would be lost.

The minimal settings are enable.idempotence=true, acks=all, retries left enabled, and max.in.flight.requests.per.connection no higher than 5. Defaults differ between clients, so check yours.

The scope of this protection is narrow. It covers one producer session writing to one partition. If the application catches an exception and calls send() again with the same record, the producer assigns a new sequence number, and the broker stores a duplicate. If the process restarts, it receives a new PID, and the broker has no way to link its writes to those of the previous session. Closing that gap across restarts requires transactions.

import java.util.Properties;
import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.StringSerializer;

Properties props = new Properties();
props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "broker1:9092,broker2:9092");
props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class.getName());

// Broker assigns a PID; each batch carries PID plus a per-partition sequence number
props.put(ProducerConfig.ENABLE_IDEMPOTENCE_CONFIG, "true");
// Sequence state must be replicated so it survives a leader change
props.put(ProducerConfig.ACKS_CONFIG, "all");
// Internal retries are what idempotence deduplicates, so keep them enabled
props.put(ProducerConfig.RETRIES_CONFIG, Integer.toString(Integer.MAX_VALUE));
// Must be 5 or fewer for the ordering and dedup guarantee to hold
props.put(ProducerConfig.MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION_CONFIG, "5");

Producer<String, String> producer = new KafkaProducer<>(props);
ProducerRecord<String, String> record =
    new ProducerRecord<>("orders", "order-1042", "{\"status\":\"paid\"}");

try {
    producer.send(record).get();
} catch (Exception e) {
    // NOT protected: this application-level resend gets a NEW sequence number,
    // so if the first send actually landed, the broker stores a duplicate.
    // producer.send(record);
    throw new RuntimeException("send failed; handle explicitly", e);
}

// NOT protected either: after close() or a process restart, a new producer
// receives a new PID and the broker cannot link it to this session's writes.
producer.close();

A minimal Java idempotent producer configuration, with comments marking the two cases it does not cover: an application-level resend and a restart that receives a new producer ID.

How do Kafka transactions make read-process-write atomic?

Linking a producer's writes across restarts is the job of the transactional.id. Unlike the PID, it is a stable name you configure, and the broker-side transaction coordinator maps it to a PID and an epoch stored in the internal transaction log (__transaction_state). When a producer calls initTransactions(), the coordinator bumps the epoch and resolves any transaction the previous incarnation left open. Any producer still holding the old epoch is now a fenced zombie, and its writes are rejected.

The processing loop is beginTransaction(), one or more send() calls, then sendOffsetsToTransaction() with the consumer's group metadata, and finally commitTransaction(). The offsets are just more writes, to __consumer_offsets, and they belong to the same transaction. On commit, the coordinator records a prepare-commit state in its log and then writes a commit marker to every partition the transaction touched, including the offsets partition. Output records and consumed offsets therefore become visible together or not at all. On abort, abort markers are written, and the records stay in the log but are skipped by transactional readers.

Fencing works only if one logical task maps to one transactional.id. If two instances share an id, they fence each other repeatedly. If ids are reassigned carelessly during scaling, a zombie and its replacement can both commit. A ProducerFencedException means another instance now owns the identity, so the correct response is to close the producer, not retry.

Kafka Streams wraps this loop when you enable its exactly-once processing guarantee. It manages ids, offsets, and state-store changelogs for you.

// Producer: one stable transactional.id per logical task (not per process or host)
producerProps.put(ProducerConfig.TRANSACTIONAL_ID_CONFIG, "orders-enricher-task-0");
// Consumer: offsets are committed through the transaction; read only committed data
consumerProps.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");
consumerProps.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");

KafkaProducer<String, String> producer = new KafkaProducer<>(producerProps);
KafkaConsumer<String, String> consumer = new KafkaConsumer<>(consumerProps);
consumer.subscribe(List.of("orders"));
producer.initTransactions(); // coordinator bumps the epoch and fences older holders of this id

while (true) {
  ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
  if (records.isEmpty()) continue;
  producer.beginTransaction();
  try {
    Map<TopicPartition, OffsetAndMetadata> offsets = new HashMap<>();
    for (ConsumerRecord<String, String> r : records) {
      producer.send(new ProducerRecord<>("orders-enriched", r.key(), enrich(r.value())));
      // Commit the position of the next record to read, not the one just processed
      offsets.put(new TopicPartition(r.topic(), r.partition()), new OffsetAndMetadata(r.offset() + 1));
    }
    // Offsets join the same transaction, so they commit or abort together with the outputs
    producer.sendOffsetsToTransaction(offsets, consumer.groupMetadata());
    producer.commitTransaction(); // coordinator writes commit markers to every touched partition
  } catch (ProducerFencedException | OutOfOrderSequenceException | AuthorizationException e) {
    // Fatal: another instance owns this identity, or the producer can't continue. Do not retry.
    producer.close();
    throw e;
  } catch (KafkaException e) {
    producer.abortTransaction(); // abort markers: transactional readers skip these records
    // Rewind so the batch is reprocessed in a new transaction
    for (TopicPartition tp : records.partitions()) {
      consumer.seek(tp, records.records(tp).get(0).offset());
    }
  }
}

A minimal Java read-process-write loop that commits output records and consumer offsets in one transaction, aborts and rewinds on recoverable errors, and closes the producer on fencing or other fatal errors.

Why does read_committed matter on the consumer side?

Transactions only change what producers write. Whether a consumer honors them is decided by its own isolation.level setting, and the default is read_uncommitted. A consumer on the default reads every record in the log as soon as it is replicated, including records from transactions that are still open or that later abort. Kafka Streams sets read_committed for you when exactly-once is enabled. Plain consumers do not.

With read_committed, the broker serves data only up to the last stable offset (LSO). The LSO is the offset of the earliest transaction on that partition that is still open, or the high watermark if no transaction is open. Records past the LSO stay invisible until their transaction resolves. Along with each fetch, the broker returns the list of aborted transactions in that range. The client uses that list to drop the aborted records and skips the commit and abort markers. Those markers still occupy offsets, which is why consumed offsets show gaps. One side effect is that a single slow or hung producer holds back the LSO, so every read_committed consumer on that partition stalls until the transaction commits, aborts, or hits its timeout.

The serious failure is a configuration that looks harmless. If one downstream service stays on read_uncommitted, it processes aborted records and the duplicates left behind by retried transactions. That service is now at-least-once while the rest of the pipeline is exactly-once, and nothing reports an error. It also makes debugging harder. When a record shows up in one service's database but not another's, the cause is often isolation level rather than data loss. Check isolation.level on every consumer group before you start investigating anything else.

import java.time.Duration;
import java.util.List;
import java.util.Properties;
import org.apache.kafka.clients.consumer.*;
import org.apache.kafka.common.serialization.StringDeserializer;

public class LedgerConsumer {
  public static void main(String[] args) {
    Properties props = new Properties();
    props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "kafka-1.internal:9092");
    props.put(ConsumerConfig.GROUP_ID_CONFIG, "billing-ledger");
    props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
    props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class.getName());
    // The default is read_uncommitted. Without this line, the consumer sees
    // records from open and aborted transactions, so this service is at-least-once.
    props.put(ConsumerConfig.ISOLATION_LEVEL_CONFIG, "read_committed");
    // Commit offsets only after the records have been processed.
    props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "false");

    try (KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props)) {
      consumer.subscribe(List.of("payments"));
      while (true) {
        // The broker returns data only up to the last stable offset (LSO).
        // A hung producer transaction makes this poll return nothing for that partition.
        ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(500));
        for (ConsumerRecord<String, String> r : records) {
          // Aborted records and commit/abort markers have already been dropped
          // client-side, so offsets can jump (e.g. 41 -> 43). That gap is expected.
          System.out.printf("partition=%d offset=%d key=%s%n", r.partition(), r.offset(), r.key());
        }
        consumer.commitSync();
      }
    }
  }
}

A plain Java consumer that sets isolation.level=read_committed explicitly, because only Kafka Streams with exactly-once enabled does this for you. Comments mark where the LSO stall and offset gaps from markers and aborted records show up.

Where does Kafka exactly-once stop working?

A Kafka transaction covers two things: the records a producer writes and the consumer offsets it commits through the same transaction. Nothing else is included. The transaction coordinator tracks partitions and offsets. It has no knowledge of your Postgres connection, your Redis client, or the HTTP call you made to a payment provider. Exactly-once is a guarantee about Kafka-to-Kafka flows only.

A worked failure

A service consumes OrderPlaced for order 1042 and processes it in four steps:

  1. It inserts a row into orders in Postgres and commits that database transaction.
  2. It produces OrderConfirmed inside its Kafka transaction.
  3. It adds the input offset to the transaction.
  4. It crashes before commitTransaction.

The Kafka transaction never commits. When a new instance starts with the same transactional.id, it fences the old producer and the coordinator aborts the open transaction. The OrderConfirmed record stays in the log but is marked aborted, so read_committed consumers skip it. The offset was never committed, so the new instance reads OrderPlaced for order 1042 again.

The Postgres row is a different story. It was committed in a separate system before the crash, so the Kafka abort cannot undo it. On replay, the service inserts again. Downstream consumers see exactly one confirmation, but the database now has two rows for order 1042. If one of those steps had been a card charge, the customer would have been billed twice.

Any side effect that is not idempotent therefore needs its own deduplication, keyed on something stable such as the order ID or event ID. Examples include incrementing external counters, sending email, and calling payment APIs. The next section covers how to build that deduplication.

What are the alternatives for side effects outside Kafka?

Three patterns cover most cases. Each one moves the atomicity boundary into a system that can enforce it, usually your database.

Idempotent consumer with a deduplication key

Store the order ID or event ID in a table with a unique constraint, and insert it in the same database transaction as the business write. Rerun the order walkthrough: the consumer writes the order row, crashes before committing its offset, and receives OrderPlaced again. The second insert violates the unique key, so the consumer skips the event and commits the offset. Delivery is still at-least-once, but the effect happens only once. For external APIs, pass the same key as the provider's idempotency key, if the provider supports one.

Transactional outbox (DB to Kafka)

The service writes the order row and an outbox row in one local transaction. A separate relay publishes outbox rows to Kafka. If the service crashes after commit, the relay still publishes the event. If it crashes before commit, neither the order nor the event exists. The relay can publish the same event twice, so downstream consumers still need deduplication.

Offsets stored with the side effect (Kafka to DB)

The consumer saves its partition offset in the same transaction as the order write. On restart, it seeks to the stored offset instead of the committed group offset. After a crash, the order and the offset are either both present or both absent.

Pattern Guarantees Requires Effort
Dedup key Effect applied once Stable event ID, unique constraint Low
Outbox DB write and event are atomic Relay process, outbox cleanup Medium
Offsets in DB Offset and write are atomic Manual seek, rebalance handling Medium–High
-- PostgreSQL. Dedup key and consumer offset live next to the business data.
CREATE TABLE orders (order_id text PRIMARY KEY, customer_id text NOT NULL, total_cents bigint NOT NULL);
CREATE TABLE processed_events (event_id text PRIMARY KEY, processed_at timestamptz NOT NULL DEFAULT now());
CREATE TABLE consumer_offsets (
  consumer_group text NOT NULL,
  topic          text NOT NULL,
  partition      int  NOT NULL,
  next_offset    bigint NOT NULL,  -- offset of the next record to read (last processed + 1)
  PRIMARY KEY (consumer_group, topic, partition)
);

-- Handling one OrderPlaced record (partition 3, offset 18420) in ONE transaction.
BEGIN;

-- Dedup: claim the event ID. On a redelivery the insert does nothing, returns
-- no row, and the order insert below inserts zero rows.
WITH claimed AS (
  INSERT INTO processed_events (event_id) VALUES ('evt-7f3a9c')
  ON CONFLICT (event_id) DO NOTHING
  RETURNING event_id
)
INSERT INTO orders (order_id, customer_id, total_cents)
SELECT 'ord-1042', 'cust-88', 4999 FROM claimed;

-- Offset: advance it whether the event was new or a duplicate, so the
-- consumer never re-reads past work after a crash.
INSERT INTO consumer_offsets (consumer_group, topic, partition, next_offset)
VALUES ('order-service', 'orders', 3, 18421)
ON CONFLICT (consumer_group, topic, partition)
DO UPDATE SET next_offset = EXCLUDED.next_offset;

COMMIT;  -- order row, dedup key and offset become visible together or not at all

-- On startup or partition assignment: read this value and seek the consumer
-- to it instead of relying on the group's committed offset in Kafka.
SELECT next_offset FROM consumer_offsets
WHERE consumer_group = 'order-service' AND topic = 'orders' AND partition = 3;

A single PostgreSQL transaction that records the dedup key, the order row and the consumer's next offset together, so a crash leaves all three committed or none of them.

How can exactly-once save engineering time?

Used in the right place, exactly-once moves duplicate handling out of application code and into the platform. The time savings come from work you no longer have to write, review and debug:

  • Deduplication is solved once, not per consumer. Without transactions, every service reading a pipeline's output topic needs its own dedup logic: an event ID, a store of seen IDs, and a retention policy for that store. With exactly-once on the producing pipeline and read_committed on readers, Kafka-internal consumers can treat each committed record as unique. One configuration change in the pipeline replaces many separate implementations in the consumers.
  • Stateful stream processing needs no hand-built recovery. In Kafka Streams, state-store updates, output records and offsets commit together. Recovering correctly after a crash or rebalance is handled by the library, so you don't have to write custom checkpointing or reconcile aggregates that drifted after a failure.
  • Crash recovery becomes routine. Because output and offsets advance atomically, restarting a failed instance or letting a rebalance move partitions does not leave partial output. Incidents of the form "the service restarted and some events were doubled" largely disappear for Kafka-to-Kafka steps. Those incidents tend to be expensive because they are intermittent and hard to reproduce.
  • Fewer reconciliation jobs. Teams without transactional guarantees often add periodic jobs that compare counts or totals between topics and repair discrepancies. When the pipeline is exactly-once end to end, those jobs can shrink to verification checks instead of repair tooling.
  • Simpler reasoning during code review. Reviewers do not need to examine every send path for "what if this runs twice" when the effect stays inside Kafka. They can focus that scrutiny on the steps that leave Kafka.

The savings stop at the Kafka boundary. Transactions do not remove the need for idempotent handling of database writes or external calls, and they add their own operational work: stable transactional.id management, timeout tuning, and monitoring of aborts and the last stable offset. Exactly-once saves time when many consumers or stateful operators would otherwise each carry their own dedup logic. For a single service whose main output is a database write, the idempotent-consumer patterns above usually take less effort.

When should you avoid Kafka transactions, and what do they cost?

Every transaction adds coordinator round trips (registering partitions, adding consumer offsets, committing) and writes a commit or abort marker to each partition it touched. That fixed overhead is why you batch, but batching moves the cost elsewhere.

Cost Why it happens When it hurts
Consumer latency read_committed readers see records only after commit Low-latency fan-out
Redone work on abort The whole batch is discarded and reprocessed Large transactions
LSO stall An open transaction holds back the last stable offset for all read_committed readers A hung producer blocks them until the transaction timeout aborts it
Repeated aborts A slow external call outlives the transaction timeout HTTP or DB calls inside the loop
transactional.id upkeep IDs must stay stable per instance so the broker can fence zombies Autoscaling, ID collisions

Illustrative estimate (assumed numbers, not a benchmark). Assume a 100 ms commit interval, a 10 ms commit round trip, and 5,000 records/s per producer.

  • Average added latency ≈ interval / 2 + commit time = 50 + 10 = ~60 ms. The worst case is about 110 ms.
  • Records redone per abort ≈ throughput × interval = 5,000 × 0.1 = 500.

A shorter interval lowers both numbers but raises coordinator load per second.

The rule: if a service's main side effect is a database write, use at-least-once delivery with an idempotent consumer. Two options work: an upsert keyed by event ID, or offsets stored in the same DB transaction as the write. The database's own atomicity already gives you correctness, and Kafka transactions cannot extend it. You avoid fencing errors, LSO investigations and timeout tuning. Debugging also gets simpler: a duplicate shows up as a rejected insert you can see, not as an aborted batch you have to work out.

Operational checklist for running exactly-once pipelines

Align the configuration end to end

  • Producer: enable.idempotence=true and acks=all. Anything weaker lets a leader failover drop or duplicate writes, and the transactional guarantees rest on that foundation.
  • transactional.id: It must be stable across restarts and unique per logical producer. If you generate it randomly at startup, a restarted instance cannot fence its predecessor, and the zombie's open transaction stays open until it times out.
  • Consumer: isolation.level=read_committed on every downstream reader, including ad hoc tools and connectors. Disable auto-commit. Offsets must go through the transaction, not through a separate commit.
  • Timeouts: Set transaction.timeout.ms above your worst-case batch processing time, including retries and GC pauses. It must also stay below the broker's transaction.max.timeout.ms. If processing outlives the timeout, the coordinator aborts the transaction and your commit fails.

Monitor what breaks silently

Track the transaction abort rate per producer. A steady trickle usually means timeouts are too tight or there is a fencing loop. Watch the gap between the last stable offset and the high watermark on output partitions. A growing gap means an open transaction is stalling every read_committed consumer behind it. Alert on that gap, not just on consumer lag.

Treat fencing as terminal

A fenced-producer error means another instance now owns the transactional.id. Close the producer and stop processing that work. Do not retry. For other transactional errors, abort the transaction and reprocess from the last committed offsets.

Prove it with crashes

Configuration review is not verification. Kill instances with SIGKILL between the send and the commit, partition brokers from the network, and force rebalances under load. Afterward, compare output against input by key and count, and check for both duplicates and gaps.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to kafka exactly once semantics, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.