Outbox Relay Lag: Measuring and Tuning Each Stage of Event Delay

What outbox relay lag is and why one number misleads

The transactional outbox pattern solves a dual-write problem. A service needs to change its own state and tell other systems about it, but writing to the database and publishing to a broker are two separate operations that can't share a transaction. If the event is instead inserted into an outbox table in the same database transaction as the state change, both either commit or roll back together. A separate relay then reads the outbox and publishes to the broker. That relay is where outbox relay lag accumulates.

Teams usually track lag as one figure, such as "time since the oldest unpublished row" or "commit-to-consume latency." That figure combines four distinct stages:

1. Commit to row visible. The outbox row exists only once its transaction commits.
2. Row visible to relay read. This stage depends on the polling interval or on how far behind log capture is running.
3. Read to broker acknowledgement. This covers batching, network time, and the broker's replication and acknowledgement settings.
4. Acknowledgement to downstream consumption. This depends on consumer group lag, which belongs to whoever owns the consumers.

A single number of 30 seconds can't tell you whether the relay is starved, the broker is throttling, or a consumer is stuck. Each cause calls for a different fix and is often owned by a different team.

The first stage deserves particular attention. A long-running or very large transaction keeps its outbox rows invisible until it commits. No change to relay configuration can publish data the database hasn't exposed yet.

This matters most to backend and platform engineers who own event-driven integrations with explicit latency expectations, such as fulfilment triggers, cache invalidation, or cross-service read models.

Copy +-------------------------------------------+
 | Service DB txn: UPDATE state + INSERT row |
 +---------------------+---------------------+
                       |
                       |  [1] commit -> visible: long/large txns hide rows
                       v
 +-------------------------------------------+
 | Outbox table (row readable after COMMIT)  |
 +---------------------+---------------------+
                       |
                       |  [2] visible -> read: poll interval / capture lag
                       v
 +-------------------------------------------+
 | Outbox relay (polling or log capture)     |
 +---------------------+---------------------+
                       |
                       |  [3] read -> ack: batching, network, broker acks
                       v
 +-------------------------------------------+
 | Message broker                            |
 +---------------------+---------------------+
                       |
                       |  [4] ack -> consumed: consumer group lag
                       v
 +-------------------------------------------+
 | Downstream consumers                      |
 +-------------------------------------------+

 |<--- one lag metric = [1]+[2]+[3]+[4] ---->|

Outbox relay lag splits into four stages, from transaction commit to downstream consumption, and a single end-to-end lag metric adds all four together without showing which one is slow.

The outbox relay in brief: what, who, when and how

What it is

An outbox relay is the process that moves events from an outbox table in a service's own database to a message broker. The service never publishes directly. It writes each event as a row in the same transaction as the business change, and the relay publishes those rows afterwards. Outbox relay lag is the time between an event's transaction committing and that event being available to, or consumed by, downstream systems.

Who should use it

The pattern and the per-segment lag measurement in this article are aimed at:

• Service owners who publish domain events and cannot tolerate lost or phantom events, such as payment, order or inventory state changes.
• Platform teams that run a shared relay or CDC connector for many services and need to show which stage is slow when someone reports a delay.
• On-call engineers who have to decide quickly whether a delay belongs to the database, the relay, the broker or a consumer.

If you only run a single service with no latency expectations on its events, end-to-end lag plus a stalled-pipeline alert may be enough. Per-segment measurement becomes worth the effort once more than one team owns parts of the path.

When to use it

Use the outbox pattern when all of these hold:

• The service already writes to a transactional database, and the event describes a change made in that database.
• Losing an event, or emitting one for a change that rolled back, would cause incorrect behaviour downstream.
• Consumers can tolerate at-least-once delivery and handle duplicates idempotently.
• A delay of a poll interval or a few hundred milliseconds of capture lag fits within what consumers need.

Measure lag per segment as soon as consumers have a stated latency expectation, so that an SLO breach points to a specific stage.

When to avoid it

The pattern is a poor fit when:

• There is no transactional database to make the outbox write atomic with the business write.
• Delivery must be strictly sub-millisecond. The extra write, capture step and publish hop set a latency floor.
• Events can be lost or rebuilt cheaply, for example best-effort metrics or notifications that a periodic reconciliation job already corrects. A direct publish is simpler.
• The broker and database already share a transaction, through transactional messaging that spans both. The outbox would duplicate that guarantee.
• The primary database cannot absorb the extra load. Every event becomes an additional row write, and a polling relay adds recurring queries. Log-based capture adds log retention risk on the source.

How it works

1. The service opens a transaction, applies the business change, and inserts an outbox row containing the serialized event, a routing key and a database-stamped creation time.
2. The transaction commits. The state change and the event become visible together, or neither does.
3. The relay finds new rows, either by polling the table for unpublished rows or by reading the database's change log through a CDC connector.
4. The relay publishes each event to the broker, keyed so that events for one aggregate stay in order, and waits for the broker's acknowledgement.
5. The relay records progress. A polling relay marks rows as published or deletes them. A CDC connector stores its log offset.
6. Consumers read from the broker and process events idempotently, because a crash between publish and progress recording causes the relay to send some events again.

Each of these steps is a place where time accumulates, and the rest of this article measures and tunes them separately.

How to measure event publishing delay per segment

Measuring this starts with timestamps you trust, captured at fixed points.

Stamp creation time with the database clock

Set the creation time inside the same transaction that writes the business change, using the database's clock rather than the application's. A minimal outbox table might look like this:

The published_at column only applies to polling relays. CDC-based capture never writes back to the row.

Record publish and consume times

When the relay publishes, it adds created_at and its own send time as message headers. Consumers record the time they receive each message. That gives three segments:

• Commit to publish: relay lag.
• Publish to consume: broker and consumer lag.
• Commit to consume: end-to-end delay.

Emit each segment as a histogram metric, not a single average, so tail latency stays visible.

For a polling relay, the most useful single query takes the current database time minus the minimum created_at among rows where published_at is null. This gives the age of the oldest unpublished row, and it keeps rising when the relay stalls, even if no new events are being written.

Clock skew

Each segment compares timestamps from different machines. A consumer clock that runs behind the database produces negative delays. One that runs ahead inflates every value. Using the database clock as the single reference for created_at avoids fleet-wide drift between application hosts. For cross-host segments, treat values below roughly your expected skew as noise.

Copy-- PostgreSQL: outbox table where the database clock stamps creation time.
CREATE TABLE outbox (
  id             BIGSERIAL PRIMARY KEY,           -- monotonically assigned
  aggregate_type TEXT        NOT NULL,            -- routing key
  aggregate_id   TEXT        NOT NULL,            -- ordering key
  payload        JSONB       NOT NULL,            -- serialized event
  -- now() is the transaction start time on the DB server, not the app host
  created_at     TIMESTAMPTZ NOT NULL DEFAULT now(),
  published_at   TIMESTAMPTZ NULL                 -- set by a polling relay after send
);

-- Partial index keeps the 'unpublished' lookup cheap as the table grows.
CREATE INDEX outbox_unpublished_idx
  ON outbox (created_at)
  WHERE published_at IS NULL;

-- The business change and the event commit together; created_at is not supplied.
BEGIN;
UPDATE orders SET status = 'paid' WHERE id = 'ord-1042';
INSERT INTO outbox (aggregate_type, aggregate_id, payload)
VALUES ('order', 'ord-1042', '{"type":"OrderPaid","orderId":"ord-1042"}');
COMMIT;

-- Polling relay: mark a row sent using the same DB clock.
UPDATE outbox SET published_at = clock_timestamp() WHERE id = 1042;

-- Age of the oldest unpublished row. Both sides come from the DB clock, so
-- there is no cross-host skew. This keeps growing while the relay is stalled.
-- It returns 0 when nothing is pending (an empty set yields NULL, so COALESCE it).
SELECT COALESCE(
         EXTRACT(EPOCH FROM (clock_timestamp() - MIN(created_at))),
         0
       ) AS oldest_unpublished_age_seconds
FROM outbox
WHERE published_at IS NULL;

A PostgreSQL outbox schema that stamps created_at with the database clock inside the business transaction, plus a query that reports the age of the oldest unpublished row in seconds for monitoring a polling relay.

How polling relays and Debezium outbox capture differ in where lag comes from

Polling relay

A polling relay queries the outbox for unpublished rows every poll interval, publishes them in batches, and marks them sent. Worst-case lag for a single event is about one poll interval plus batch processing time. Order by a monotonic sequence column rather than created_at, since timestamps can tie or commit out of order. A partial index on the unpublished predicate (for example, WHERE published_at IS NULL) keeps each poll cheap because the index holds only the backlog, not the full table history. The unpublished-row count is your direct backlog signal. Shortening the interval cuts lag, but every poll costs a query even when the table is empty.

Debezium outbox

Debezium reads the database change log, not the table, so lag comes from how far the connector trails the log. Row counts tell you nothing here. Watch three things:

• Time behind source: the connector's milliseconds-behind-source metric.
• Task state: whether the connector tasks are running or failed.
• Log position: replication slot position on PostgreSQL, or binlog position on MySQL.

A stalled PostgreSQL slot also retains WAL on the primary, so this lag can turn into a disk problem on the database.

Alternatives

Dual writes, where the application commits to the database and then publishes separately, can leave the two out of sync. A crash between the commit and the publish loses the event, and a publish followed by a rollback emits an event for a change that never happened. LISTEN/NOTIFY-style triggers reduce polling latency, but notifications are not durable, so you still need a poll to catch missed ones. Where the broker supports transactional messaging across the database and the queue, it can replace the outbox, at the cost of coupling the two systems.

Copy-- Outbox table: a monotonic sequence gives a stable publish order
CREATE TABLE outbox (
  id            bigserial PRIMARY KEY,
  aggregate_id  text        NOT NULL,
  event_type    text        NOT NULL,
  payload       jsonb       NOT NULL,
  created_at    timestamptz NOT NULL DEFAULT now(),
  published_at  timestamptz
);

-- Partial index holds only the backlog, so each poll stays cheap
CREATE INDEX outbox_unpublished_idx ON outbox (id) WHERE published_at IS NULL;

-- Polling relay: claim one batch in sequence order.
-- SKIP LOCKED lets multiple relays poll without blocking each other,
-- but it gives up strict global ordering across those relays.
SELECT id, aggregate_id, event_type, payload
FROM outbox
WHERE published_at IS NULL
ORDER BY id
LIMIT 500
FOR UPDATE SKIP LOCKED;

-- After the broker acknowledges the batch, mark it sent in the same transaction
UPDATE outbox SET published_at = now() WHERE id = ANY(:published_ids);

-- Polling backlog signal: unpublished count and age of the oldest event
SELECT count(*) AS backlog,
       now() - min(created_at) AS oldest_unpublished_age
FROM outbox
WHERE published_at IS NULL;

-- Debezium signal on PostgreSQL: how much WAL each slot is retaining
SELECT slot_name,
       active,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS consumer_lag,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn))         AS retained_wal
FROM pg_replication_slots
WHERE slot_type = 'logical';

PostgreSQL sketch contrasting the two lag signals: a polling relay ordered by a sequence column with a partial index, measured by unpublished row count and oldest-event age, versus a Debezium setup measured by how far the logical replication slot trails the current WAL position.

CDC monitoring and alerting that distinguishes quiet from stalled

Raw counts are a poor alert signal. A backlog of 5,000 rows can be normal after a batch import and alarming at 3 a.m. Alert on two things instead. The first is the age of the oldest unpublished event, which maps directly to how late consumers are. The second is the backlog growth rate sustained over several minutes, which shows the relay falling behind before the age threshold trips.

Telling quiet from stalled

A pipeline with no traffic and a pipeline that has stopped both report zero new events. To tell them apart, write a synthetic canary row to the outbox on a fixed schedule and alert if it doesn't reach the broker within your budget. Give canaries a distinct aggregate type so the router sends them to a dedicated topic that business consumers never subscribe to. For log-based capture, the connector's own heartbeat setting keeps offsets advancing on low-traffic databases. It does not prove that the outbox table path works end to end.

Also export connector restarts, task failures and error counts. A connector that restarts every few minutes can look healthy on lag graphs while repeatedly reprocessing the same window.

Database-side risk

A stalled log-based connector stops acknowledging its position, so the database retains WAL or binlog it would otherwise discard. That can fill the source disk and take down the primary. Monitor retained log size and free disk alongside lag. In PostgreSQL, query pg_replication_slots and compute pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) per slot. Alert on growth, and on inactive slots.

Copy-- Retained WAL per replication slot. Run on the primary:
-- pg_current_wal_lsn() raises an error on a standby.
-- wal_status and safe_wal_size require PostgreSQL 13 or later.
SELECT
  slot_name,
  plugin,                 -- e.g. pgoutput for a log-based CDC connector
  slot_type,
  active,                 -- false means no connector is attached
  active_pid,
  wal_status,             -- 'reserved' is normal; 'extended', 'unreserved' or 'lost' need attention
  -- Bytes of WAL the server must keep because this slot has not advanced.
  -- Alert on this value growing, not only on a fixed threshold.
  pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS retained_wal_bytes,
  pg_size_pretty(
    pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)
  ) AS retained_wal,
  -- How far the consumer's acknowledged position trails the current WAL position.
  pg_size_pretty(
    pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)
  ) AS unconfirmed_wal,
  -- WAL that can still be written before this slot is at risk of being invalidated.
  -- NULL when max_slot_wal_keep_size is not set, which means WAL retention is unbounded.
  pg_size_pretty(safe_wal_size) AS safe_wal_remaining
FROM pg_replication_slots
ORDER BY retained_wal_bytes DESC NULLS LAST;

-- Separate alert: an inactive slot still pins WAL even though nothing is consuming it.
SELECT slot_name,
       pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal
FROM pg_replication_slots
WHERE NOT active;

PostgreSQL queries that report how much WAL each replication slot is forcing the primary to retain and flag inactive slots, so you can alert on slot growth alongside source disk usage.

Tuning levers for each lag segment

Each lever shortens one segment and usually costs something in another. Tune the segment your measurements blame, not the end-to-end number.

Key by aggregate ID, not event type. Events for one order then land on one partition in commit order, while different orders proceed in parallel.

Delivery semantics

Relays are at-least-once. Duplicates happen when a send succeeds but the ack is lost, or when a connector restarts from its last committed offset and replays events. Consumers must be idempotent, for example by recording processed event IDs. Short timeouts and aggressive retries make this worse because they resend events the broker already accepted.

Table hygiene

A polling relay must delete or archive published rows. Without that, scans slow down and dead tuples bloat the table and its indexes. Log-based capture reads from the transaction log, so the application can delete each row in the same transaction that inserts it. Retaining rows gives you an audit trail and a replay source, but you pay for it in storage and vacuum work.

Setting a latency budget and SLO for commit-to-publish delay

Start from what consumers need. If a downstream cache must reflect a write within one second for 99% of events, set the SLO on p99 commit-to-publish delay at 1 s, measured from the transaction commit timestamp to the broker acknowledgment. Then split that budget across segments so each one has a ceiling you can alert on independently.

Illustrative estimate (assumed numbers, not a benchmark), for a polling relay:

• Worst case ≈ poll interval + batch processing (worst) + publish (worst)
• Typical case ≈ poll interval ÷ 2 + batch processing (typical) + publish (typical)

Against a 1 s target, the worst case leaves 270 ms of headroom for GC pauses, broker leader elections, and transaction-log backlog. Replace each input with your own measured p99 values.

When the SLO is breached, do not start by shortening the poll interval. Compare the per-segment p99 histograms first. If publish time is the segment that grew, a faster poll loop only builds a larger queue in front of the broker.

The outbox pattern fits poorly in two cases:

• Strict sub-millisecond delivery. The pattern adds a write, a capture step, and a publish hop, and that structure has a latency floor no tuning removes.
• No transactional database. Without a transaction there is nothing to make the outbox write atomic with the business write, and atomicity is the reason to use the pattern in the first place.

Lag incident runbook and impact on cost and engineering time

Runbook: walk the pipeline in order

Check each stage from source to consumer and stop at the first unhealthy one. Downstream symptoms usually have upstream causes, so starting at the consumer wastes time.

1. Oldest unpublished event age. If it is low and flat, the relay is keeping up and the delay is downstream. If it climbs, the fault is at or before the relay.
2. Relay or connector status. Confirm the process is running, the task is not in a failed state, and the logs do not show repeating errors. A restart loop looks like slow progress on a dashboard.
3. Source log position and retained log size. For log-based capture, a replication slot or binlog position that stops advancing while retained log grows means the connector is stuck. It also means database disk is now at risk.
4. Broker health. Produce errors, under-replicated partitions or throttling force the relay to retry, and those retries appear as relay lag.
5. Consumer lag. Treat the delay as a consumer problem only after publishing is confirmed healthy.

Cost and engineering time

A polling relay is cheaper to build because it needs only a loop, a query and an index. Its cost is recurring database load, which grows with polling frequency and outbox table size. Log-based CDC costs more to operate. You run a connector platform, watch log retention and coordinate schema changes. In return, it reads the log instead of querying tables.

Segment-level metrics and a fixed runbook shorten diagnosis. A single end-to-end lag number invites engineers to tune whatever they know best, such as adding consumers while the connector is stalled. Ordered checks identify the failing stage quickly, so tuning effort goes to the stage that is actually slow.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to outbox relay lag, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a shared variable is one edit and one review instead of editing every service's .env by hand.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• A wrong or stale value is caught at review time by the services it affects, not by a failed deploy.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.