Transactional Outbox Pattern: Outbox Polling vs Debezium CDC

What is the dual write problem and why the transactional outbox pattern fixes it

A service that updates its database and then publishes to a message broker performs a dual write: two systems, no shared transaction. If it commits first and then publishes, a crash, a broker timeout or a network partition between the two steps loses the event, and downstream services never learn about the change. If it publishes first and then commits, a failed commit leaves a phantom event describing state that does not exist.

Two-phase commit across the database and the broker would solve this in theory. In practice, many brokers don't support it, it holds locks across network round trips, and a stalled coordinator blocks both sides.

The transactional outbox pattern avoids the problem by never writing to two systems at once. The business row and an outbox row are inserted in the same local transaction, so either both exist or neither does. A separate relay reads committed outbox rows and publishes them to the broker.

Service -> DB transaction { orders row + outbox row } -> Relay -> Broker -> Consumers

A minimal outbox table needs these columns:

• id: a unique event ID that consumers use to discard duplicates.
• aggregate_type: selects the destination topic.
• aggregate_id: used as the message key so events for one entity stay ordered.
• event_type: lets consumers route or ignore events.
• payload: the serialized event body.
• created_at: used for ordering and cleanup.
• published_at (optional): marks rows the relay has already sent.

This pattern fits any service that changes state and must reliably notify other services about it.

Copy-- Minimal outbox table (PostgreSQL)
CREATE TABLE outbox (
  id             UUID PRIMARY KEY,           -- unique event ID; consumers dedupe on it
  aggregate_type TEXT        NOT NULL,       -- selects the destination topic, e.g. 'order'
  aggregate_id   TEXT        NOT NULL,       -- message key; keeps one entity's events ordered
  event_type     TEXT        NOT NULL,       -- e.g. 'OrderCreated'; consumers route or ignore
  payload        JSONB       NOT NULL,       -- serialized event body
  created_at     TIMESTAMPTZ NOT NULL DEFAULT now(),  -- ordering and cleanup
  published_at   TIMESTAMPTZ                 -- optional: set by a polling relay after send
);

-- Business change and event in ONE local transaction:
-- either both rows commit or neither does, so there is no dual write.
BEGIN;

INSERT INTO orders (id, customer_id, total_cents, status)
VALUES ('7f3c2a9e-4b1d-4c8e-9a2f-1d6e5b7c8a90', 'cust-1042', 4999, 'PLACED');

INSERT INTO outbox (id, aggregate_type, aggregate_id, event_type, payload)
VALUES (
  'b2e8d4f1-6a3c-4e7b-8d9f-0c1a2b3e4f56',
  'order',
  '7f3c2a9e-4b1d-4c8e-9a2f-1d6e5b7c8a90',
  'OrderCreated',
  '{"orderId": "7f3c2a9e-4b1d-4c8e-9a2f-1d6e5b7c8a90", "customerId": "cust-1042", "totalCents": 4999}'
);

COMMIT;
-- The service never talks to the broker here; a separate relay
-- reads committed outbox rows and publishes them.

A minimal PostgreSQL outbox table and a single local transaction that writes the order row and its OrderCreated event together, leaving publishing to a separate relay.

How outbox polling works

A polling relay is a loop that reads unpublished rows in order, publishes them in a batch and then records that they were sent. A typical cycle runs inside one transaction: SELECT ... FROM outbox WHERE published_at IS NULL ORDER BY id LIMIT 100 FOR UPDATE SKIP LOCKED, publish each row to the broker, wait for broker acknowledgements, then UPDATE the rows to set published_at or DELETE them and commit. SKIP LOCKED lets several relay instances run at once without double-sending, because each one skips rows another instance has already claimed. PostgreSQL, MySQL and Oracle support it. If your database does not, use leader election so that only one relay polls at a time.

The polling interval sets the trade-off. A short interval lowers latency, but it adds query load even when the table is empty. A long interval saves database work, but events reach consumers later. Many teams poll quickly while batches come back full and back off when they come back empty.

Sequence and identity columns carry a subtle risk. Transaction A takes id 10, transaction B takes id 11, and B commits first. A relay that tracks "last id seen = 11" will never read row 10. You can avoid this in three ways:

• Lock rows and mark each one individually instead of keeping a high-water mark.
• Read only rows older than a small lag window.
• Track a commit-time watermark.

Add a partial index on WHERE published_at IS NULL where the database supports partial indexes. This keeps the polling query small and fast as published rows pile up.

Copy-- PostgreSQL. Outbox table written in the same transaction as the business change.
CREATE TABLE outbox (
  id             bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  aggregate_type text        NOT NULL,
  aggregate_id   text        NOT NULL,
  event_type     text        NOT NULL,
  payload        jsonb       NOT NULL,
  created_at     timestamptz NOT NULL DEFAULT now(),
  published_at   timestamptz
);

-- Partial index covers only unpublished rows, so it stays small
-- no matter how many published rows accumulate.
CREATE INDEX outbox_unpublished_idx ON outbox (id) WHERE published_at IS NULL;

-- One relay cycle. Each relay instance runs this in its own transaction.
BEGIN;

-- Claim a batch. SKIP LOCKED makes concurrent relays skip rows another
-- relay has already locked, so no row is claimed twice.
-- No high-water mark is kept: every unpublished row stays eligible,
-- so a lower id that commits late is picked up on a later cycle.
SELECT id, aggregate_type, aggregate_id, event_type, payload
FROM outbox
WHERE published_at IS NULL
ORDER BY id
LIMIT 100
FOR UPDATE SKIP LOCKED;

-- Application code publishes each row to the broker and waits for acks.
-- Then it marks only the rows that were acknowledged.
-- $1 is a bigint[] bind parameter supplied by the database driver.
UPDATE outbox
SET published_at = now()
WHERE id = ANY($1::bigint[]);
-- Alternative: DELETE FROM outbox WHERE id = ANY($1::bigint[]);

COMMIT;

A PostgreSQL outbox table with a partial index on unpublished rows, and one relay cycle that claims a batch with FOR UPDATE SKIP LOCKED and marks each row after the broker acknowledges it, so the relay never depends on a high-water mark that could skip late-committing ids.

How Debezium outbox CDC works

Debezium does not query the outbox table. It runs as a source connector inside Kafka Connect and reads the database's change log: the PostgreSQL write-ahead log through a logical replication slot, or the MySQL binlog in row format. A committed transaction that inserts an outbox row appears in the log in commit order. Rolled-back transactions never reach the log, so you don't need a watermark or a published_at column.

The outbox event router is a single message transformation that turns each outbox insert into a domain event. It routes the event to a topic based on the aggregate type column, uses the aggregate id as the Kafka message key so events for one aggregate stay in order within a partition, and uses the payload column as the message value. An illustrative fragment, using property names you should check against your Debezium version:

• connector.class: the PostgreSQL connector, with plugin.name set to pgoutput
• table.include.list: only the outbox table
• transforms: outbox
• transforms.outbox.type: the EventRouter class
• transforms.outbox.route.by.field: aggregatetype
• transforms.outbox.table.field.event.key: aggregateid

Because the log is the source, the outbox table can be insert-only. You can delete rows right after insert, even in the same transaction, because the insert is already in the log.

On PostgreSQL, the connector needs wal_level = logical and a role with replication privileges. Some managed or restricted environments limit or gate logical replication, so confirm it is available before you commit to this design. An inactive slot also retains WAL, which grows disk usage until the connector catches up.

Copy{
  "name": "orders-outbox-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "plugin.name": "pgoutput",
    "database.hostname": "orders-db.internal",
    "database.port": "5432",
    "database.user": "debezium_repl",
    "database.password": "change-me",
    "database.dbname": "orders",
    "topic.prefix": "orders",
    "slot.name": "orders_outbox_slot",
    "table.include.list": "public.outbox",

    "transforms": "outbox",
    "transforms.outbox.type": "io.debezium.transforms.outbox.EventRouter",
    "transforms.outbox.route.by.field": "aggregatetype",
    "transforms.outbox.route.topic.replacement": "outbox.event.${routedByValue}",
    "transforms.outbox.table.field.event.key": "aggregateid",
    "transforms.outbox.table.field.event.payload": "payload"
  }
}

Illustrative Kafka Connect registration for a Debezium PostgreSQL connector that reads only the outbox table via pgoutput and uses the EventRouter to route events by aggregatetype, keyed by aggregateid; verify property names against your Debezium version.

Delivery guarantees, duplicates and ordering

Neither approach gives exactly-once delivery end to end. Both are at-least-once, so every consumer must expect to see the same event more than once.

Where duplicates come from

A polling relay publishes a row, then marks it sent. If it crashes between those two steps, the row is still unmarked and gets published again on the next poll. Debezium has the same gap in a different place. It records its position as connector offsets, and those are committed periodically, not after every message. After a restart it resumes from the last committed offset, so anything emitted after that point is emitted again. A re-snapshot or a failover of the connector or database can replay a much larger range.

Idempotent consumers

Every outbox event should carry a unique event id. The consumer inserts that id into a processed-messages table with a unique constraint, in the same database transaction as its business update. If the insert conflicts, the event was already applied, so the consumer skips it and acknowledges the message. Because the dedup record and the state change commit together, neither can exist without the other.

Some updates are idempotent on their own and need no table. Setting status = 'SHIPPED' and upserting a row guarded by a version check both fall into this category.

Ordering

Key messages by aggregate id so that all events for one aggregate land on the same partition and keep their relative order. Polling needs extra care. Sequence values are assigned at insert time, not at commit time, so a lower id can commit after a higher one. A relay that tracks only "last id seen" will silently skip that row. Select unpublished rows by status instead, or delay reading recent ids until in-flight transactions have finished. CDC avoids this problem because it reads the log in commit order.

Copy-- PostgreSQL. One row per event the consumer has already applied.
CREATE TABLE processed_messages (
  event_id     uuid PRIMARY KEY,               -- unique constraint does the dedup
  processed_at timestamptz NOT NULL DEFAULT now()
);

-- Handling a PaymentReceived event. The increment is NOT idempotent,
-- so a redelivered event must not apply it twice.
BEGIN;

WITH dedup AS (
  -- Record the event id. On a duplicate this inserts nothing and RETURNING
  -- yields no rows. A concurrent insert of the same id waits on the unique
  -- index until the first transaction commits, then also yields nothing.
  INSERT INTO processed_messages (event_id)
  VALUES ('3f6c2a9e-8b1d-4c7e-9a52-1d0e4b7f6a21')
  ON CONFLICT (event_id) DO NOTHING
  RETURNING event_id
)
UPDATE accounts
SET    balance = balance + 25.00
WHERE  account_id = 'acc-1042'
  AND  EXISTS (SELECT 1 FROM dedup);  -- apply only if the id was new

-- The dedup row and the balance change commit or roll back together.
-- 0 rows updated means duplicate: skip the work and acknowledge the message.
COMMIT;

-- By contrast, this update is naturally idempotent and needs no table:
-- running it twice leaves the same final state.
UPDATE orders
SET    status = 'SHIPPED'
WHERE  order_id = 'ord-7781';

An idempotent PostgreSQL consumer that records the event id in a processed-messages table in the same transaction as a non-idempotent balance update, so a redelivered event changes nothing, contrasted with an update that is idempotent on its own.

When to use polling, when to use Debezium, and when to avoid an outbox

Default to polling. It runs on infrastructure you already operate, and its failure modes are visible: a stuck relay leaves a growing backlog of unpublished rows that you can query directly. Move to Debezium when latency targets, write volume or polling load on the primary clearly justify it.

Illustrative estimate. These inputs are assumed, not measured: a 500 ms poll interval, 2 relay instances and 50 events/s. That gives 4 polls/s, or about 345,600 queries per day, whether or not any events exist. Average added latency is about 250 ms. Worst case is about 500 ms plus publish time. Halving the interval doubles the query count.

Skip the outbox, or reconsider it, in three cases:

• The database is already the bottleneck. The outbox adds a write to every transaction.
• Events are fire-and-forget. If losing an event is acceptable, you don't need a durable outbox.
• Payloads are large or expose internal models. Publish IDs, or design a public event contract, instead.

Alternatives to consider:

• Two-phase commit. It works only if both resources support it, and you have to accept coordinator blocking.
• Event sourcing. The event store is the source of truth, so there is no dual write.
• Listen-to-yourself. Publish first, then update state by consuming your own event.
• Best-effort publish with reconciliation jobs. Publish without guarantees and run jobs that detect and repair drift.

Operating the outbox: housekeeping and monitoring

Housekeeping

A polling relay runs its "find unpublished rows" query constantly, so that predicate needs an index. A partial index on created_at where published_at IS NULL works well because it stays small no matter how much history accumulates. Published rows must still be deleted or moved to an archive table on a schedule. Otherwise the table grows without limit, and every scan, vacuum and backup gets slower.

With Debezium, the connector reads the log, not the table. That means rows can be deleted in the same transaction that inserts them, or pruned periodically. On PostgreSQL, constant insert-then-delete churn still produces dead tuples, so autovacuum must keep up with the outbox table.

What to monitor

• Polling: backlog size and the age of the oldest unpublished row. A query like SELECT count(*), now() - min(created_at) FROM outbox WHERE published_at IS NULL covers both. Age is the better alert signal, because a small backlog that never drains is still an outage.
• CDC: connector and task status, plus how much log the database is retaining for the connector. On PostgreSQL, query pg_replication_slots and compute pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) per slot. Also track end-to-end event age, from created_at to the time the consumer processes the event. Connectors can report healthy while events go stale.

The replication slot risk

A replication slot keeps WAL until its consumer confirms it has processed it. If the connector stalls, or someone removes it without dropping the slot, WAL keeps piling up until the primary's disk fills. Alert on retained WAL size, not only on connector status.

Write runbooks for primary failover (where slots may need recreating), major version upgrades, and log retention limits such as MySQL binlog expiry. Rehearse each one before you need it.

Impact on cost and engineering time, and a migration path from polling to CDC

Each outbox event costs at least two extra writes in the producer's database: the insert inside the business transaction, and a later delete or update by housekeeping. Both are also written to the transaction log, so log volume, replication traffic and vacuum or purge work all increase.

Illustrative estimate (assumed numbers, not a benchmark): extra row writes per day = events/day × (1 insert + 1 delete). At an assumed 2 million events/day, that is 4 million additional row writes. Polling also adds query load of roughly relay instances × polls per second, even when the table is empty.

The database also becomes part of the publishing path. If it is slow or unavailable, events stop. The producer now owns the event schema, the envelope and payload versioning, because consumers depend on what it writes. Debezium adds Kafka Connect, connector configuration, schema handling and a new on-call surface. For a team that does not already run Connect, this operational cost is often larger than the code change.

The pattern pays this back by removing the ad hoc reconciliation scripts teams write after dual writes go wrong. It also removes the work of debugging events that were lost, or "phantom" events published for transactions that rolled back.

A low-risk migration path:

1. Start with polling, behind a stable outbox table schema and event envelope with a unique event ID.
2. While the relay is still running, create the replication slot or record the binlog position. This gives you a known starting point that is at or before the relay's progress.
3. Stop the relay.
4. Start the connector from that recorded point. Configure it to emit the same envelope to the same topics, so producers and consumers do not change.

Some events will be published twice: once by the relay and once by the connector. This overlap is bounded by the gap between the recorded position and the moment the relay stopped. Idempotent consumers that deduplicate on event ID absorb these duplicates. Because the connector starts at or before the relay's last position, no event falls into a gap.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to transactional outbox pattern, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.