Outbox Pattern with Debezium: Reliable Events from PostgreSQL

Why dual writes between PostgreSQL and a broker fail

A service that updates PostgreSQL and then publishes to a message broker is performing a dual write: two separate systems, two separate commits, and no shared transaction between them. Every ordering of those two steps has a window where one succeeds and the other does not.

Commit succeeds, publish fails. The order row commits. Then the broker is unreachable, the network times out, or the process is killed by a deploy before the publish call returns. The database says the order exists, but downstream services never hear about it. Nothing records that an event is missing, so the gap stays silent until a customer or an audit notices.

Publish succeeds, commit rolls back. To avoid the first case, you publish inside the transaction, before COMMIT. Now consumers can act on an event for data that never persisted, because a constraint violation, serialization failure or crash aborted the transaction afterward. The event is already out and cannot be recalled.

Application-level retries only cover the cases where the process survives to retry. A crash between the commit and the publish loses the retry state too. Retries also introduce duplicates whenever a publish succeeded but its acknowledgment was lost. Two-phase commit would close the gap in theory, but many brokers do not take part in XA transactions. Where they do, holding locks across a coordinator makes every write depend on the availability and latency of both systems.

This pattern is for teams whose services must emit an event whenever their PostgreSQL state changes, with no silent losses and no phantom events.

How the transactional outbox and PostgreSQL CDC work together

The fix is to stop writing to two systems. The service writes to PostgreSQL only. In one local transaction it updates the business tables and inserts a row into an outbox table that describes the event: aggregate type, aggregate ID, event type, and payload. Atomicity does the rest. If the transaction commits, the event exists. If it rolls back, the event never existed. No code path can produce one without the other.

Getting those rows to the broker without polling is the job of change data capture. PostgreSQL already records every committed change in its write-ahead log (WAL). With wal_level = logical, the built-in pgoutput plugin decodes WAL records into row-level changes. A publication limits decoding to the tables you choose, ideally just outbox. A replication slot tracks how far the consumer has read, and PostgreSQL keeps WAL from that position until the consumer confirms it. Debezium, running inside Kafka Connect, streams from the slot and turns each outbox insert into a broker message. Its outbox event router routes the message to a topic, typically named after the aggregate type.

Copy Service ──one tx──► PostgreSQL
                     ├─ orders (business rows)
                     └─ outbox (event rows)
                            │ commit
                            ▼
                           WAL ──► replication slot (publication: outbox)
                                          │ pgoutput
                                          ▼
                              Kafka Connect + Debezium
                                          │ outbox event router
                                          ▼
                              topics: order.events, ...
                                          │
                                          ▼
                                      consumers

Delivery is at-least-once. Debezium saves its read position periodically, not after every message. A crash after publishing but before that save replays events on restart, so every consumer has to handle duplicates.

Copy [ Service ]
       | one local tx: UPDATE orders + INSERT INTO outbox
       v
 +-----------------------------------------------------------+
 | PostgreSQL  (wal_level = logical)                         |
 |   +----------+   +----------+                             |
 |   |  orders  |   |  outbox  |                             |
 |   +----------+   +----------+                             |
 |        | commit       |                                   |
 |        v              v                                   |
 |   +---------------------------+                           |
 |   |  WAL (write-ahead log)    |                           |
 |   +---------------------------+                           |
 |        | logical decoding                                 |
 |        v                                                  |
 |   +---------------------------+                           |
 |   |  replication slot         |  WAL kept until           |
 |   |  plugin: pgoutput         |  Debezium confirms        |
 |   |  publication: outbox      |                           |
 |   +---------------------------+                           |
 +-----------------------------------------------------------+
       | logical replication stream
       v
 +-----------------------------------------------------------+
 | Kafka Connect + Debezium (outbox event router)            |
 +-----------------------------------------------------------+
       | at-least-once: one message per outbox row
       v
 [ topics: order.events, ... ] ---> consumers (idempotent)

The service commits business rows and an outbox row in one transaction; Debezium reads outbox inserts from the WAL through a pgoutput replication slot and routes them to topics with at-least-once delivery.

Implementing it: outbox table, PostgreSQL settings, and connector configuration

Step 1: Create the outbox table

The table needs six columns: id (UUID primary key), aggregatetype, aggregateid, type, payload (jsonb) and created_at (timestamptz). These column names match the outbox event router's defaults, so the connector needs less configuration. The router uses aggregateid as the Kafka message key. Kafka assigns partitions by key, so all events for one order land on the same partition and keep their commit order.

Step 2: Write the business row and the event together

In the same transaction as the business write, for example INSERT INTO orders, insert the outbox row and then COMMIT. Both rows become visible together or not at all.

Step 3: Configure PostgreSQL

1. Set wal_level = logical. This change requires a server restart.
2. Set max_replication_slots and max_wal_senders high enough to cover this connector plus any existing replicas.
3. Create a connector role with the REPLICATION attribute and SELECT on the outbox table.
4. As the table owner, run CREATE PUBLICATION ... FOR TABLE outbox. Creating it yourself means the connector role does not need ownership rights.

Step 4: Configure the connector

Use the pgoutput plugin and set the publication name. Limit table.include.list to the outbox table. Add the io.debezium.transforms.outbox.EventRouter transform. It routes each event to a topic based on aggregatetype, sets the key from aggregateid, and emits payload as the message value.

Step 5: Keep the table small

You have two options:

• Delete in the same transaction. Delete the outbox row right after inserting it. The insert is still written to the WAL, so Debezium captures it, and the router skips the delete. A polling publisher would never see the row, so this only works with log-based CDC.
• Run a periodic cleanup job. Delete rows older than a retention window. This keeps recent events queryable for debugging.

Copy-- Server settings (run as superuser; wal_level takes effect only after a restart)
ALTER SYSTEM SET wal_level = 'logical';
ALTER SYSTEM SET max_replication_slots = 10;  -- this connector plus existing slots
ALTER SYSTEM SET max_wal_senders = 10;        -- this connector plus existing replicas

-- Step 1: outbox table; column names match the EventRouter defaults
CREATE TABLE outbox (
  id            uuid         PRIMARY KEY,
  aggregatetype varchar(255) NOT NULL,  -- picks the topic, e.g. 'order'
  aggregateid   varchar(255) NOT NULL,  -- becomes the Kafka message key
  type          varchar(255) NOT NULL,  -- event type, e.g. 'OrderCreated'
  payload       jsonb        NOT NULL,  -- becomes the Kafka message value
  created_at    timestamptz  NOT NULL DEFAULT now()
);

-- Step 2: business row and event commit together or not at all
BEGIN;
INSERT INTO orders (id, customer_id, total)
  VALUES ('7f3c2a10-5b1e-4d2a-9c6e-1a2b3c4d5e6f', 42, 99.90);
INSERT INTO outbox (id, aggregatetype, aggregateid, type, payload)
  VALUES (gen_random_uuid(), 'order', '7f3c2a10-5b1e-4d2a-9c6e-1a2b3c4d5e6f',
          'OrderCreated', '{"orderId": "7f3c2a10-5b1e-4d2a-9c6e-1a2b3c4d5e6f", "total": 99.90}');
-- Optional (Step 5): delete in the same transaction. The INSERT is still in the WAL,
-- so Debezium captures it, and the router ignores the DELETE.
-- DELETE FROM outbox WHERE aggregateid = '7f3c2a10-5b1e-4d2a-9c6e-1a2b3c4d5e6f';
COMMIT;

-- Step 3: connector role with replication rights and read access to the outbox only
CREATE ROLE debezium WITH LOGIN REPLICATION PASSWORD 'change-me-placeholder';
GRANT SELECT ON outbox TO debezium;

-- Run as the table owner; set the connector's publication.name to 'dbz_outbox'
CREATE PUBLICATION dbz_outbox FOR TABLE outbox;

SQL for the PostgreSQL side of the outbox setup: logical replication settings, the outbox table DDL, a transactional business-plus-event insert with an optional same-transaction delete, the connector role, and a publication limited to the outbox table.

Building idempotent consumers and a stable event contract

Debezium and Kafka give you at-least-once delivery, so consumers will see duplicates. They come from a few places. A connector that restarts resumes from its last committed offset and re-emits everything after it. Offset commits happen periodically, not per event, so a crash between publishing and committing replays a batch. Consumer group rebalances hand partitions to new members before the old member committed its progress. Producer and broker retries can also write a message twice.

Deduplicate on the event id

Keep a processed_events table in the consumer's own database, keyed on the outbox event id. In a single local transaction, insert the id and apply the side effects. If the insert hits a unique-constraint violation, the event was already handled, so roll back and commit the offset. Because the marker and the side effects commit or fail together, a crash at any point leaves them consistent. A separate cache or an in-memory set gives you none of that.

Know the limits of ordering

Kafka orders messages only within a partition, so per-aggregate ordering holds only if every event for an aggregate uses the same key. If you change the partition count or the keying scheme, keys map to different partitions. Events produced before and after the change can then be consumed out of order.

Treat payloads as a public contract

Consumers you don't control will depend on your event shape. Design each payload as a domain event, such as OrderPlaced with the fields consumers need. Don't mirror your internal table columns. Carry an explicit type and version in each event, make changes additive, and publish a new version when you have to break compatibility.

Copyimport json
import psycopg2
from psycopg2 import errors
from confluent_kafka import Consumer

# Schema in the consumer's own database:
# CREATE TABLE processed_events (event_id uuid PRIMARY KEY, processed_at timestamptz NOT NULL DEFAULT now());

consumer = Consumer({
    "bootstrap.servers": "kafka:9092",
    "group.id": "billing-service",
    "enable.auto.commit": False,  # commit offsets only after the DB transaction finishes
    "auto.offset.reset": "earliest",
})
consumer.subscribe(["outbox.event.Order"])
db = psycopg2.connect("host=billing-db dbname=billing user=billing password=change-me")

while True:
    msg = consumer.poll(1.0)
    if msg is None or msg.error():
        continue
    # The Debezium outbox router puts the outbox row id in the "id" header by default
    event_id = dict(msg.headers() or [])["id"].decode()
    event = json.loads(msg.value())  # assumes a plain JSON payload with type and version fields
    try:
        with db:  # one local transaction: marker and side effects commit or roll back together
            with db.cursor() as cur:
                cur.execute("INSERT INTO processed_events (event_id) VALUES (%s)", (event_id,))
                if event["type"] == "OrderPlaced" and event["version"] == 1:
                    cur.execute(
                        "INSERT INTO invoices (order_id, amount_cents) VALUES (%s, %s)",
                        (event["orderId"], event["totalCents"]),
                    )
    except errors.UniqueViolation:
        pass  # already processed; the context manager rolled the transaction back
    # Duplicate or not, the event is handled, so advance the offset
    consumer.commit(message=msg, asynchronous=False)

A Kafka consumer that inserts the outbox event id into processed_events in the same PostgreSQL transaction as its side effect, treats a unique-constraint violation as a duplicate, and commits the offset only after the transaction is resolved.

Operating it safely in production

Replication slots hold WAL until they are consumed

Debezium reads from a logical replication slot. PostgreSQL keeps every WAL segment the slot has not confirmed. If the connector stops, crashes, or falls behind, WAL keeps growing until the disk fills and the primary stops accepting writes. A slot left behind after a decommissioned connector is worse, because nothing will ever consume it.

Query pg_replication_slots for active, restart_lsn, and confirmed_flush_lsn. To measure retained WAL in bytes, compare restart_lsn with pg_current_wal_lsn() using pg_wal_lsn_diff(). Alert on that byte count, and alert separately when the slot is inactive or the connector task is not running. If your PostgreSQL version supports max_slot_wal_keep_size, set it. It trades disk exhaustion for an invalidated slot, which is a recoverable failure.

Snapshots, failover, and lost offsets

On first start, or if its stored offsets are lost, the connector may snapshot the captured tables according to its snapshot mode. For an outbox, that means rows that still exist get republished. This is one reason consumers must be idempotent.

Logical slots may not carry over to a promoted replica, depending on your PostgreSQL version and HA tooling. After failover, the connector may find no slot. It can then re-snapshot, miss events committed before the new slot was created, or replay duplicates. Rehearse failover with the connector attached before you rely on it.

Long transactions and table churn

Decoding emits a transaction only after it commits. Long-running transactions therefore hold back restart_lsn and delay every event behind them.

The outbox table sees constant inserts and deletes, which produces dead tuples quickly. Tune autovacuum per table so it runs often enough on the outbox to prevent bloat.

Copy-- 1. Retained WAL per logical slot. Alert on retained_bytes, and separately on active = false.
SELECT slot_name,
       active,
       wal_status,  -- PostgreSQL 13+: 'lost' means the slot was invalidated
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal,
       pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS retained_bytes,
       pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn) AS unconfirmed_bytes
FROM pg_replication_slots
WHERE slot_type = 'logical';

-- 2. Cap retained WAL (PostgreSQL 13+). A slot that exceeds this is invalidated
--    and the connector must re-snapshot, but the primary keeps running.
ALTER SYSTEM SET max_slot_wal_keep_size = '50GB';
SELECT pg_reload_conf();

-- 3. Long-running transactions hold back restart_lsn and delay later events.
SELECT pid, usename, state,
       now() - xact_start AS xact_age,
       left(query, 80) AS query
FROM pg_stat_activity
WHERE xact_start IS NOT NULL
  AND now() - xact_start > interval '5 minutes'
ORDER BY xact_start;

-- 4. The outbox table gets constant inserts and deletes. Vacuum on a fixed
--    dead-tuple count instead of a fraction of a table that stays small.
ALTER TABLE public.outbox SET (
  autovacuum_vacuum_scale_factor = 0.0,
  autovacuum_vacuum_threshold    = 1000
);

-- Verify that autovacuum keeps up.
SELECT relname, n_dead_tup, last_autovacuum
FROM pg_stat_user_tables
WHERE relname = 'outbox';

SQL to measure WAL retained by logical slots, cap it with max_slot_wal_keep_size, find long-running transactions that hold back decoding, and tune autovacuum for the high-churn outbox table.

When to use it, when to avoid it, and the alternatives

Debezium-based outbox earns its keep when several services consume the same domain events, write volume is high enough that polling would either lag or hammer the primary, and consumers care about seconds rather than minutes. It also assumes Kafka (or Kafka Connect) already runs somewhere, because standing up a broker just for this multiplies the cost.

Skip it when two or three services owned by one small team exchange a few hundred events an hour. The failure modes covered above (stuck replication slots, WAL retention, connector restarts) need someone who can read pg_replication_slots at 2 a.m. If nobody can, a polling publisher on the same outbox table keeps the transactional guarantee and drops the CDC layer entirely.

Two-phase commit looks attractive on paper, but most modern brokers do not participate in XA, and a crashed coordinator leaves prepared transactions holding locks. Dual writes with retries only shrink the inconsistency window. They never close it.

Impact on development cost and engineering time

The outbox does not remove that cost. It moves it from application code into shared infrastructure. Whether that trade pays off depends mostly on how many services publish events.

The last row matters most. A stalled connector produces a metric you can alert on. A missed dual write produces a support ticket.

Illustrative estimate (assumed numbers, not a benchmark)

Assume the following inputs:

• S = 8 services
• R = 3 engineer-days per service per quarter spent maintaining retry and reconciliation code
• I = 2 inconsistency incidents per quarter, each taking D = 2 engineer-days to investigate
• P = 10 engineer-days per quarter for the platform team to run, upgrade, and monitor Connect and Debezium

Quarterly savings = (S × R) + (I × D) − P = (8 × 3) + (2 × 2) − 10 = 18 engineer-days.

With S = 2, the same formula gives (2 × 3) + 4 − 10 = 0. A small system may never recover the fixed platform cost.

The extra write load on the primary does not appear in the formula. Account for it separately in capacity planning before you adopt the pattern.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to outbox pattern debezium, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.