Why do consumers see duplicate messages under at-least-once delivery?

Under at-least-once delivery, the broker guarantees a message won't be lost. It does not guarantee the message arrives only once. Any failure between "work done" and "work acknowledged" leads to redelivery, because the broker cannot tell a consumer that crashed before processing from one that crashed after it.

The classic case: your consumer writes an order to its database, commits the transaction, then dies before committing the Kafka offset or sending the ack. On restart, it resumes from the last committed offset and processes the same message again. Rebalances cause the same effect at scale. When a partition moves to another group member, everything since the last offset commit is redelivered to the new owner.

Source of duplicate Where it happens Covered by broker idempotence/transactions?
Crash after DB commit, before offset commit/ack Consumer No
Consumer group rebalance Consumer group No, for external side effects
Producer retry after a lost ack Producer → broker Yes, within an idempotent producer session
Deliberate replay (offset reset, reprocessing) Operator action No, by design

Kafka's idempotent producer suppresses retry duplicates when writing to a partition. Transactions make consume-transform-produce atomic across topics and offsets. Both stop at the broker boundary. A row inserted into Postgres or a call to a payment API is outside that transaction. Exactly-once processing is therefore never provided end to end. The consumer itself has to be idempotent.

What is the inbox pattern and how does it work?

The consumer has to make its own work idempotent, and the inbox pattern is the most common way to do that when the consumer owns a relational database.

An inbox is a table that records which messages a consumer has already handled. Each row is keyed by a stable message identifier. If several consumers share one database, the key is scoped by consumer name. A minimal schema has three columns:

Column Purpose
consumer_name Which handler processed the message
message_id Stable identifier taken from the message
processed_at Timestamp, used later for retention

The primary key is (consumer_name, message_id). Duplicate detection depends entirely on this unique constraint.

The flow

  1. The broker delivers a message.
  2. The consumer opens a local database transaction.
  3. It inserts the inbox row first, then applies the business change, such as updating an order or writing a ledger entry.
  4. It commits the transaction.
  5. Only after the commit succeeds does it acknowledge the message or commit the offset.

If the insert hits a unique-constraint violation, another delivery already committed this message. The consumer rolls back, skips the work, and acknowledges. If the process crashes between the commit and the acknowledgement, the broker redelivers, and the constraint catches the duplicate.

The inbox row and the business change commit or roll back together, so neither can exist without the other. That gives you effectively-once side effects inside one service's database. The guarantee stops there. An HTTP call or a message published from the handler sits outside the transaction and can still run more than once.

-- Minimal inbox table (PostgreSQL syntax).
-- The composite primary key is the deduplication mechanism.
CREATE TABLE inbox (
    consumer_name text        NOT NULL,  -- which handler processed the message
    message_id    text        NOT NULL,  -- stable id carried in the message
    processed_at  timestamptz NOT NULL DEFAULT now(),  -- used for retention
    PRIMARY KEY (consumer_name, message_id)
);

-- Per-message handling, run by the consumer for each delivery.
BEGIN;

-- 1. Claim the message first. On a duplicate this fails with
--    unique_violation (SQLSTATE 23505). The application then issues
--    ROLLBACK, skips the business change, and acknowledges the message.
INSERT INTO inbox (consumer_name, message_id)
VALUES ('order-service.payment-captured', 'msg-7f3c9a2e-0001');

-- 2. Apply the business change in the same transaction.
UPDATE orders
SET    status = 'paid'
WHERE  order_id = 'ord-10442';

INSERT INTO ledger_entries (order_id, amount_cents, entry_type)
VALUES ('ord-10442', 4999, 'payment');

-- 3. The inbox row and the business change become visible together.
COMMIT;

-- 4. Only after COMMIT succeeds does the application acknowledge the
--    message or commit the offset. A crash before that point causes a
--    redelivery, and step 1 rejects the duplicate.

A minimal PostgreSQL inbox table and the single local transaction that records the message ID and applies the business change, with acknowledgement deferred until after commit.

How do you choose a deduplication key that survives retries and replays?

The inbox only deduplicates as well as its key does. The key must identify the event, not the delivery, so it has to be assigned once by the producer and carried unchanged through every retry, republish and replay. A business identifier (an order ID plus an event type) or a dedicated event ID in a message header both work, provided the producer generates it when the event is created, not when it is sent.

Topic, partition and offset look tempting because the broker provides them for free. But they describe a position in a log, not an event. Replay the same events into a new topic, mirror them to another cluster, or recreate a topic, and every coordinate changes. The consumer then treats old events as new.

Payload hashes fail in the opposite direction. Two genuine "add one item to cart" events from the same user can be byte-for-byte identical, and a hash collapses them into one. You lose real work silently.

Key option Survives producer retry Survives replay or mirroring Distinguishes identical events
Topic + partition + offset Not if the producer resends No Yes
Payload hash Yes Yes No
Producer-assigned event ID Yes, if reused Yes Yes

The limit is outside the consumer's control. A producer that mints a fresh UUID on each send attempt produces distinct keys for the same event, and no consumer logic can tell them apart. Treat a stable event ID as part of the message contract and enforce it in producer code review and tests.

How do you implement an idempotent consumer with an inbox table?

Turn off auto-commit in the consumer client and commit offsets yourself. Auto-commit advances the offset on a timer, independent of whether your transaction succeeded, so a crash can skip a message that was never applied.

For each record, the loop does this:

  1. Open a database transaction.
  2. Insert the event ID into the inbox table, which has a unique constraint on that column.
  3. If the insert added a row, run the business logic in the same transaction. If it hit a conflict, skip the logic because this event was already handled.
  4. Commit the transaction.
  5. Commit the offset.

The order of steps 4 and 5 is what makes this safe. If the process crashes between them, the broker redelivers the message. The redelivered copy then hits the conflict and is skipped. If you reverse the order, the same crash loses the message silently.

Do not check for the ID with SELECT and then INSERT if it is absent. During a rebalance, two consumers can briefly hold the same partition. Both SELECTs can return no row, and both consumers then process the event. Use an insert that fails or does nothing on conflict, such as INSERT ... ON CONFLICT DO NOTHING or your database's equivalent. Then check the affected-row count. The unique index serializes the two consumers: the second one blocks until the first commits, then sees the conflict.

That blocking has a cost. A slow transaction holds the inbox row lock, and it can also keep the consumer from polling within the poll or session timeout. When that happens, the broker assumes the consumer is dead and rebalances the group. Rebalances are the main source of the duplicates you are trying to absorb, and they stall throughput while partitions are reassigned. Keep transactions short, and move slow side effects out of the transaction.

-- Schema: the unique constraint on event_id is what deduplicates
CREATE TABLE inbox (
  event_id     VARCHAR(64) PRIMARY KEY,
  processed_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);

-- Consumer loop (pseudocode). Auto-commit of offsets is disabled.
consumer = Consumer(group = "orders-service", auto_commit = false)

loop:
  records = consumer.poll(timeout = 500ms)
  for record in records:
    tx = db.begin()
    try:
      -- Atomic claim. No SELECT first: two consumers racing during a
      -- rebalance would both see "no row" and both process the event.
      rows = tx.execute(
        "INSERT INTO inbox (event_id) VALUES (?)
         ON CONFLICT (event_id) DO NOTHING",   -- or your DB's equivalent
        record.event_id)
      if rows == 1:
        apply_business_logic(tx, record)       -- same transaction, keep it fast
      -- rows == 0: already handled, skip the logic
      tx.commit()                              -- step 4: durable state first
    except error:
      tx.rollback()
      raise                                    -- offset not committed; message is redelivered

    -- Step 5: commit the offset only after the DB commit succeeds.
    -- A crash before this line causes a redelivery, which the inbox absorbs.
    consumer.commit(record.partition, record.offset + 1)

-- Keep each transaction well under the poll/session timeout. Slow work
-- (HTTP calls, emails) holds the inbox row lock and delays the next poll,
-- which triggers a rebalance and produces more duplicates.

An inbox table with a unique event ID and a consumer loop that claims each event with a single conflict-tolerant insert, commits the database transaction, and only then commits the offset.

Should you use synchronous dedupe-and-process or store-then-process?

The synchronous variant keeps those side effects inside the handler. One database transaction inserts the message ID into the inbox and runs the business logic, and the offset is committed only after that transaction commits. A redelivered message fails the unique constraint and is skipped. Partition order is preserved, but every millisecond of handler time is spent inside the consumer's poll loop.

The store-then-process variant splits the work. The consumer inserts the raw payload and commits the offset right away. A separate worker claims pending rows, for example with SELECT … FOR UPDATE SKIP LOCKED, and runs the logic. Because a row now outlives the consumer's transaction, it needs explicit state:

  • status: received, processing, done, failed or dead_letter
  • attempts: an integer counter
  • next_attempt_at: when the row is next eligible for a retry
  • last_error
  • processed_at

After a fixed number of attempts, set the row to dead_letter and alert. A poison message must never be marked done just to clear the backlog. The broker has already forgotten that message, so a falsely completed row is silent data loss.

You also take on an added cost: latency of up to one polling interval, plus a poller or scheduler that you must run and monitor. Parallel workers also break per-key ordering unless you route rows by a hash of the key or claim only the oldest pending row for each key.

Aspect Synchronous Store-then-process
Transactions per message One Two (ingest, process)
Consumer throughput Bound by handler time Bound by insert time
Ordering Preserved per partition Needs key-based routing
Failure handling Retry or block the partition Retries, dead-letter state
Extra components None Poller or scheduler
Main risk Rebalances from slow handlers Hidden loss from false done
-- Inbox table for the store-then-process variant (PostgreSQL)
CREATE TABLE inbox (
  message_id      text PRIMARY KEY,          -- dedupe key from the producer
  message_key     text NOT NULL,             -- business key, for ordering/routing
  payload         jsonb NOT NULL,
  status          text NOT NULL DEFAULT 'received'
                  CHECK (status IN ('received','processing','done','failed','dead_letter')),
  attempts        integer NOT NULL DEFAULT 0,
  next_attempt_at timestamptz NOT NULL DEFAULT now(),
  last_error      text,
  received_at     timestamptz NOT NULL DEFAULT now(),
  processed_at    timestamptz
);
-- Keep the poller's scan cheap: index only rows that still need work
CREATE INDEX inbox_pending ON inbox (next_attempt_at)
  WHERE status IN ('received','failed');

-- Consumer: fast ingest; a redelivery is a no-op, then commit the offset
INSERT INTO inbox (message_id, message_key, payload)
VALUES ($1, $2, $3) ON CONFLICT (message_id) DO NOTHING;

-- Worker: claim a batch without blocking other workers
UPDATE inbox SET status = 'processing', attempts = attempts + 1
WHERE message_id IN (
  SELECT message_id FROM inbox
  WHERE status IN ('received','failed') AND next_attempt_at <= now()
  ORDER BY received_at LIMIT 10
  FOR UPDATE SKIP LOCKED)
RETURNING message_id, message_key, payload;

-- On failure: back off, and park poison messages instead of marking them done
UPDATE inbox
SET status = CASE WHEN attempts >= 5 THEN 'dead_letter' ELSE 'failed' END,
    next_attempt_at = now() + interval '30 seconds' * attempts,
    last_error = $2
WHERE message_id = $1;

A PostgreSQL inbox schema with status, attempts and next_attempt_at columns, plus the idempotent ingest, SKIP LOCKED claim and retry-or-dead-letter updates used by the store-then-process variant; this claim does not preserve per-key order across parallel workers, which still needs routing by key.

How do you operate an inbox table in production: retention, cost, testing and monitoring?

Every consumed message now costs an extra insert and a unique-index lookup inside the transaction. On a busy topic that index becomes the hottest structure in the database: it grows without bound and concentrates write contention on the same pages. Partitioning the table by time (daily or hourly buckets) keeps the active index small. It also turns cleanup into dropping old partitions instead of running large DELETE statements.

Retention

The retention window must exceed the longest realistic redelivery or replay horizon. That horizon includes broker retention, consumer lag during an outage, and any manual offset reset you might run. If the window is too short, a replay reprocesses messages whose keys were already purged, and duplicates reach downstream systems silently. If it is too long, you pay for storage and slower index maintenance with no added safety.

Illustrative estimate (assumed numbers, not a benchmark): rows = rate × 86,400 × days; storage ≈ rows × bytes per row (key + timestamp + index overhead). With an assumed 2,000 msg/s, 7 days and 100 bytes per row, that is 1.21 billion rows, or about 121 GB.

Testing

Write three tests early:

  • Inject a crash after the database commit but before the offset commit, and confirm the message is skipped on redelivery.
  • Force consumer-group rebalances while messages are in flight.
  • Replay a whole partition from an earlier offset and assert that no side effects repeat.

Monitoring

Emit a duplicates-skipped counter per topic and partition. Its normal baseline is low and steady. A sudden spike points to an upstream retry storm or a misconfigured offset reset, and you see it before it becomes a data incident.

When should you avoid the inbox pattern, and what are the alternatives?

The inbox pays off when a consumed message drives a non-idempotent change, such as incrementing a balance, appending a ledger row or decrementing stock, and every write lands in one transactional store. In that case the dedupe insert and the business write commit or roll back together.

It is unnecessary when the operation is already idempotent. Upserting a target state ("order 42 is SHIPPED") or applying a version-checked update (WHERE version = 7) gives the same result however many times it runs. To tell which case applies, ask: if this handler ran twice back to back, would any row, counter or side effect differ from running it once? If not, skip the table.

It is insufficient once processing touches a second database, a cache, a search index or third-party HTTP, email or payment calls. The inbox transaction cannot cover those systems. A crash after the external call but before the commit replays the call. You have three options:

Approach How it works Guarantee for the external effect
Downstream idempotency key Send the dedupe key; the receiver rejects repeats Effectively-once, if the receiver honours keys
Outbox in the same transaction Record the intended call; a relay performs it later At-least-once delivery to the relay target, so it still needs a key
Accept and document Tolerate duplicates such as a repeated cache refresh At-least-once

The outbox is the publishing-side counterpart to the inbox. The inbox dedupes what arrives, and the outbox makes what leaves consistent with your commit.

Do not describe the inbox as "exactly-once." Teams that hear that phrase drop idempotency in downstream services, and duplicates then reappear at the first boundary the inbox cannot reach.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to inbox pattern message deduplication, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • The dependency graph answers "what does this touch?" in seconds instead of a manual audit.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • You can size a change's blast radius before merging, avoiding surprise cross-team firefighting.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.