What is a partial failure in a multi-service write?

A multi-service write is any request whose outcome depends on changing state in two or more places: an order service that records the order, a payment service that captures funds, an inventory service that reserves stock. Each of those writes commits independently. There is no shared transaction spanning them, so the request can end with any subset of the changes applied. The order exists but the payment never happened. The payment was captured but the reservation failed. All three succeeded but the caller was never told.

The last case is the one that catches teams off guard. When a call times out, the caller learns nothing about what happened on the other side. The remote service may have rejected the request, may still be processing it, or may have committed it a millisecond before the connection dropped. A timeout is the absence of an answer, not a negative answer. Treating it as "failed" and moving on is how you get duplicate charges or orphaned records.

This is not an edge case. Network blips, deploys, pod evictions, and slow dependencies happen during normal operation, and at enough request volume, some fraction of multi-service writes will land mid-failure every day. The goal is not to prevent intermediate states, which is impossible without a distributed transaction you probably don't want. The goal is to make every intermediate state detectable, safe to retry, and eventually resolved.

If any request path in your system writes to more than one service or data store, this applies to you, including a database plus a message broker.

Who should use it, and who can skip it?

These patterns are for teams that own request paths where a lost, duplicated, or half-applied write costs money, inventory, or user trust. Typical cases:

  • Commerce and payments flows, where a duplicate charge or an unpaid shipped order becomes a refund, a chargeback, or a support ticket.
  • Event-driven systems where a service updates its database and publishes an event that other teams' services depend on.
  • Workflows spanning several team-owned services, where no single database can hold the whole operation and someone needs to know what state each request is in.
  • Platform teams that build shared libraries or service templates. Building the outbox relay, dedup table, and retry policy once and reusing them is cheaper than having each team discover the dual write problem on its own.

You can skip most of this, or adopt only part of it, in these cases:

  • All the state lives in one database. Use a local transaction. It is simpler and stronger than anything in this article.
  • The write path touches one store and the rest is reads. Aggregating data from several services without changing it does not create partial writes.
  • Loss is acceptable. Best-effort analytics or cache warm-up events may not justify an outbox and dedup table.
  • The service boundary is wrong. If two services constantly need to change together, consider merging that data under one owner before building sagas around the split.

Adoption can be incremental. The outbox and idempotent consumers solve the most common failures on their own. Sagas are only needed once a workflow has several steps that may need to be reversed. Reconciliation is added where the cost of an undetected mismatch is high.

Why can't you just write to the database and then publish an event?

The obvious implementation is two calls in sequence: commit the row, then publish to the broker. The database and the broker each guarantee atomicity only for their own writes, so the pair has no shared commit point. This is the dual write problem. A process crash, network timeout, or deploy can interrupt the sequence at several points:

Crash point Database Broker Resulting state
Before commit Nothing written Nothing published Consistent; caller sees an error and can retry
After commit, before publish Row exists No event Downstream services never learn of the change
After publish, before ack Row exists Event may or may not be stored Producer cannot tell; a retry may publish a duplicate
Publish first, then commit fails No row Event delivered Consumers act on a change that never happened

Reordering the calls only changes which failure you get. Whichever write comes second can fail after the first has already taken effect, and neither system can undo the other's work.

Two-phase commit addresses this in theory by having a coordinator ask every participant to prepare and then commit. In practice it is rarely used across services. If the coordinator fails after participants have prepared, they must hold locks and wait, because they cannot safely decide the outcome on their own. Every participant must be reachable for any write to succeed, so overall availability is lower than that of any single participant. Many message brokers and managed databases also either do not expose a prepare phase to external coordinators or support it poorly.

How does it work end to end?

The approach in this article does not replace a distributed transaction with a single mechanism. It uses several layers, and each one handles a specific way the request can break. Here is one request, such as cancelling an order that other services must react to:

  1. Local atomic write. The owning service updates its business row and inserts an outbox row in the same database transaction. Either both commit or neither does, so the service's state and its intent to notify others cannot diverge.
  2. Honest response. If the rest of the workflow is asynchronous, the service returns a pending status to the caller rather than claiming completion.
  3. Reliable publish. A relay reads committed outbox rows and publishes them to the broker, keyed by aggregate ID so per-aggregate order is preserved. If the relay crashes, it re-sends on restart, so delivery is at-least-once.
  4. Idempotent consumption. Each consumer records the message ID in the same transaction as its own state change. Redelivered messages are detected and skipped. Calls that time out are retried with the same idempotency key instead of being treated as failures.
  5. Workflow coordination. For multi-step flows, a saga persists each step's outcome before starting the next. When a step fails permanently, it runs compensations in reverse order and escalates to a human if a compensation keeps failing.
  6. Bounded retries and dead letters. Transient failures are retried with backoff and jitter up to a limit. Anything left over goes to a dead-letter queue instead of looping forever.
  7. Reconciliation. A periodic job compares the sources of truth across services, auto-repairs the safe classes of mismatch, and routes the rest to people.
  8. Visibility. Correlation IDs, outbox lag, saga age, compensation counts, and mismatch rates let you see where a request is stuck.

Each layer assumes the one before it can fail. The outbox assumes the relay will crash. Consumers assume the relay will send duplicates. Reconciliation assumes all of the above can still have bugs. The sections below cover each layer in detail.

How does the transactional outbox pattern work?

The outbox pattern avoids the need for a distributed coordinator by keeping the atomic step inside one database. In the same local transaction that updates the business row, the service inserts a row into an outbox table. That row holds the event payload, the aggregate ID, and a sequence number. Either both rows commit or neither does, so an event can never describe a state change that did not happen.

A separate relay reads committed outbox rows and publishes them to the broker. The relay can crash after publishing but before recording that it did, so it will sometimes send the same event twice. Delivery is therefore at-least-once by design, and consumers must deduplicate. Ordering matters per aggregate, not globally. Use the aggregate ID as the partition or message key, and have the relay publish each aggregate's rows in sequence order, so that OrderCancelled never arrives before OrderCreated. Delete or archive published rows on a schedule. Otherwise the table grows without bound and relay queries slow down.

Polling relay vs. change data capture

A polling relay queries for unpublished rows and marks them sent. It is simple and portable, but it adds write amplification: every event costs an insert, an update, and eventually a delete. Publish latency is bounded by the poll interval, and it is one more process to deploy and monitor.

Change data capture tails the database's replication log instead. This removes the marking writes and lowers latency. The cost is coupling to engine-specific log formats and retention settings, plus extra infrastructure that someone has to operate and upgrade.

-- PostgreSQL. One row per event; (aggregate_id, aggregate_seq) gives per-aggregate order.
CREATE TABLE outbox (
  id            bigserial PRIMARY KEY,
  aggregate_id  uuid        NOT NULL,
  aggregate_seq bigint      NOT NULL,
  event_type    text        NOT NULL,
  payload       jsonb       NOT NULL,
  published_at  timestamptz,
  UNIQUE (aggregate_id, aggregate_seq)
);
CREATE INDEX outbox_unpublished ON outbox (id) WHERE published_at IS NULL;

-- Write path: business row and outbox row commit or roll back together.
-- The row lock on orders serializes writers per aggregate, so seq and id stay in order.
BEGIN;
WITH o AS (
  UPDATE orders SET status = 'CANCELLED', version = version + 1
  WHERE id = '7f3c2a10-0000-4000-8000-000000000001' AND status = 'CREATED'
  RETURNING id, version
)
INSERT INTO outbox (aggregate_id, aggregate_seq, event_type, payload)
SELECT id, version, 'OrderCancelled', jsonb_build_object('orderId', id, 'version', version)
FROM o;
COMMIT;

-- Polling relay (run a single active relay so per-aggregate order is preserved).
BEGIN;
SELECT id, aggregate_id, aggregate_seq, event_type, payload
FROM outbox
WHERE published_at IS NULL
ORDER BY id
LIMIT 100
FOR UPDATE SKIP LOCKED;
-- App code: publish each row in order with message key = aggregate_id, wait for broker ack.
-- A crash here means rows are re-sent next poll: at-least-once, consumers dedupe on (aggregate_id, aggregate_seq).
UPDATE outbox SET published_at = now() WHERE id = ANY($1::bigint[]);  -- ids that were acked
COMMIT;

-- Scheduled cleanup keeps the table and relay scans small.
DELETE FROM outbox WHERE published_at < now() - interval '7 days';

A PostgreSQL outbox: the order update and its event row commit in one transaction, a polling relay publishes unpublished rows in order and marks them sent, and a scheduled job deletes old published rows.

How do you make consumers and retries idempotent?

Whether events come from an outbox relay or log tailing, delivery is at-least-once. A relay can crash after publishing but before recording the send, and a broker can redeliver after a lost acknowledgment. Exactly-once delivery across a network isn't available, so the consumer has to make duplicates harmless.

Record the key with the side effect

Store each processed message ID or client-supplied idempotency key in the same database transaction as the state change, using a unique constraint. A duplicate then fails the insert, and you skip it. If the key lives in a separate store, a crash between the two writes recreates the dual write problem inside your consumer.

Keys can't be kept forever, so pick a retention window longer than your worst realistic redelivery delay. That includes broker retention, replay from backups, and a stalled consumer catching up. When a key expires, a very late duplicate will be processed again. Where that would cause real harm, make the operation naturally idempotent, for example by setting a status rather than incrementing a counter.

Treat timeouts as unknown

A timeout means you don't know the outcome. It does not mean the call failed. You have three options:

  • Query the downstream service for the result before acting.
  • Retry with the same idempotency key so the server can deduplicate.
  • Park the operation in a pending state for later resolution.

Never mark it failed and move on.

Bound your retries

Use exponential backoff with jitter so clients don't retry in synchronized waves. Cap the number of attempts, then route the operation to a dead-letter queue. Unbounded retries turn a brief outage into sustained load that keeps the dependency down.

-- PostgreSQL. The primary key on message_id is the dedup guard.
CREATE TABLE processed_messages (
  message_id   text        PRIMARY KEY,
  processed_at timestamptz NOT NULL DEFAULT now()
);

-- Consumer handling one message. The key and the side effect commit or roll back together.
BEGIN;

WITH claimed AS (
  -- A duplicate hits the primary key and inserts nothing, so RETURNING yields no row.
  -- A concurrent duplicate waits on the key until the first transaction commits or rolls back.
  INSERT INTO processed_messages (message_id)
  VALUES ('msg-2f9c41e0-payment-captured')
  ON CONFLICT (message_id) DO NOTHING
  RETURNING message_id
)
-- The side effect runs only if this transaction claimed the key.
-- Setting a status instead of incrementing a counter keeps this safe
-- even if a very late duplicate arrives after the key has expired.
UPDATE orders
SET    status = 'PAID', updated_at = now()
WHERE  order_id = 'ord-10427'
  AND  EXISTS (SELECT 1 FROM claimed);

COMMIT;

-- Periodic cleanup job. The retention window must exceed the worst realistic
-- redelivery delay: broker retention, replays and a stalled consumer catching up.
-- Duplicates older than this window will be processed again.
DELETE FROM processed_messages
WHERE  processed_at < now() - interval '14 days';

A PostgreSQL consumer records the message ID in the same transaction as an idempotent status update, so a redelivered message changes nothing, and a cleanup job deletes keys once they fall outside a 14-day retention window.

When should you use a saga, and orchestration or choreography?

Retries and the outbox get a single step delivered reliably. A workflow that spans several services, such as reserving stock, charging a card, and booking a shipment, needs more than that. A saga models the workflow as an explicit state machine. Each step's outcome is persisted before the next step starts. If the coordinator crashes mid-flow, it reloads the saved state and resumes from the last recorded step instead of guessing.

Orchestration Choreography
Where the flow lives One coordinator service Spread across event subscribers
Visibility One place to read and debug Must be reconstructed from events
Changing the flow Edit one component Coordinate changes across teams
Cost Another service to run and scale Hidden coupling through event contracts

Orchestration is usually the safer default once a flow has more than two or three steps. Choreography works for short, stable flows where no single team owns the end-to-end outcome.

When a later step fails, the saga runs compensations. These are semantic reversals, not rollbacks: a refund instead of an erased charge, a released reservation instead of a deleted row. Some effects cannot be reversed. A sent email stays sent, and data already shared with a partner stays shared. Compensations can also fail, so they need the same idempotency, retry limits, and alerting as forward steps.

Avoid sagas when an invariant must never be violated, even briefly, such as "never oversell the last seat." Use a reservation pattern instead, or give one service sole ownership of that data.

STEPS = [  # (name, forward step, compensation as a semantic reversal)
    ("reserve_stock", inventory.reserve, inventory.release),
    ("charge_card", payments.charge, payments.refund),
    ("book_shipment", shipping.book, shipping.cancel),
]

def run_saga(store, saga_id, order):
    # A restarted coordinator reloads saved state and resumes from it.
    state = store.load(saga_id) or {"done": [], "status": "RUNNING"}
    if state["status"] == "RUNNING":
        for name, forward, _ in STEPS:
            if name in state["done"]:
                continue  # recorded before a crash, so skip it on resume
            try:
                # The key makes a replay safe if we crashed after the call succeeded.
                forward(order, idempotency_key=f"{saga_id}:{name}")
            except PermanentFailure:
                state["status"] = "COMPENSATING"
                store.save(saga_id, state)
                break
            state["done"].append(name)
            store.save(saga_id, state)  # persist each outcome before the next step
        else:
            state["status"] = "COMPLETED"
            store.save(saga_id, state)
    if state["status"] == "COMPENSATING":
        for name, _, compensate in reversed(STEPS):
            if name not in state["done"]:
                continue
            # Compensations get their own idempotency key, bounded retries and alerting.
            undo = lambda: compensate(order, idempotency_key=f"{saga_id}:{name}:undo")
            if not retry(undo, attempts=5):
                state["status"] = "NEEDS_OPERATOR"
                store.save(saga_id, state)
                alert(f"saga {saga_id}: compensation {name} failed")
                return state
            state["done"].remove(name)
            store.save(saga_id, state)
        state["status"] = "COMPENSATED"
        store.save(saga_id, state)
    return state

A minimal orchestrator sketch that persists each step's outcome so it can resume after a crash, runs compensations such as refunds in reverse order when a step fails, and escalates to an operator when a compensation keeps failing.

How do reconciliation jobs catch what everything else misses?

Outboxes, idempotent consumers, and sagas each prevent a known class of failure. Reconciliation catches the rest: a bug in a compensation step, a message dropped by a misconfigured retention policy, a manual database fix that skipped the event stream. A reconciliation job periodically reads the source of truth in each service and compares them, for example orders marked paid against ledger entries.

Scope each run to a time window, such as records updated between two and twenty-six hours ago. The lower bound skips in-flight work that is still converging and would otherwise show up as false mismatches. Scan with a cursor on a stable, indexed key, such as (updated_at, id), and persist the cursor after each batch. A crashed run then resumes where it stopped, and no single query locks or scans a whole table.

Classify every discrepancy before acting on it:

Discrepancy class Example Action
Missing downstream effect Payment captured, no ledger entry Auto-repair by re-emitting the event
Stale value Shipping address differs Auto-repair from the owning service
Conflicting facts Refunded in one service, fulfilled in another Route to a human queue
Unknown shape Record exists in only one place, no rule applies Route to a human queue

The costs are real. Reconciliation runs after the fact, so customers may already have seen the inconsistency. Scanning large datasets is expensive. A wrong auto-repair rule spreads damage faster than the original bug did. Silent repairs also hide upstream defects. Record the mismatch rate per class as a metric and alert when it rises, because a rising rate means something upstream broke.

from datetime import datetime, timedelta, timezone

BATCH_SIZE = 500
AUTO_REPAIRABLE = {"missing_downstream"}  # every other class goes to a human

def classify(order, entry):
    if order.status == "paid" and entry is None:
        return "missing_downstream"
    if entry is not None and order.status != "paid":
        return "unknown_shape"  # no rule applies; never guess
    if entry is not None and entry.amount_cents != order.amount_cents:
        return "conflicting_facts"
    return None  # consistent

def reconcile(db, ledger, cursor_store, metrics, republish, human_queue):
    now = datetime.now(timezone.utc)
    window_start = now - timedelta(hours=26)
    window_end = now - timedelta(hours=2)  # skip in-flight work still converging
    cursor = cursor_store.load() or (window_start, 0)  # resume after a crash
    while True:
        # Keyset scan on indexed (updated_at, id): bounded batches, no OFFSET
        rows = db.query(
            "SELECT id, updated_at, status, amount_cents FROM orders"
            " WHERE (updated_at, id) > (%s, %s) AND updated_at < %s"
            " ORDER BY updated_at, id LIMIT %s",
            (cursor[0], cursor[1], window_end, BATCH_SIZE))
        if not rows:
            break
        entries = ledger.entries_for([r.id for r in rows])  # one lookup per batch
        for order in rows:
            kind = classify(order, entries.get(order.id))
            if kind is None:
                continue
            # Count every mismatch by class so a rising rate triggers an alert
            metrics.increment("reconciliation.mismatch", tags={"class": kind})
            if kind in AUTO_REPAIRABLE:
                republish("OrderPaid", order.id)  # idempotent consumers absorb repeats
            else:
                human_queue.put(kind, order.id)
        cursor = (rows[-1].updated_at, rows[-1].id)
        cursor_store.save(cursor)  # persist progress after each batch

A windowed, cursor-paginated reconciliation pass that compares paid orders against ledger entries, auto-repairs only missing downstream effects, routes everything else to a human queue, and emits a per-class mismatch metric for alerting.

What does eventual consistency cost, and how do you keep it visible?

Eventual consistency is a contract with clients, so make it explicit in the API. Return 202 Accepted with a resource whose status is pending, and move it to confirmed or failed only when the saga reaches a terminal state. For read-your-writes, route a user's reads to the primary for a short window after their write, or return a version token that the client sends back so the read waits until a replica has caught up. In the UI, show "Processing" instead of a success screen. A premature "Order placed" turns a later compensation into a support ticket.

Signals to instrument

  • Correlation ID on every command, event, outbox row and log line in a saga.
  • Outbox lag: age of the oldest unpublished row, not the row count.
  • Saga age: sagas still in a non-terminal state past their expected duration.
  • Compensation counts per step. A spike points to a failing downstream service.
  • Dead-lettered messages: alert on any growth.
  • Reconciliation mismatches per class.
Approach Failure behaviour Main cost
Outbox + idempotency + saga + reconciliation Converges, and stuck state is visible Extra tables, relay process, client handling of pending states
Two-phase commit Atomic, but blocks if the coordinator fails Tight coupling, and many brokers and datastores do not support it
Single owning service No cross-service write Requires redrawing service boundaries
Synchronous calls, no safety net Silent divergence Manual data fixes

The layered approach has real costs: more schema, another process to run, and clients that must handle intermediate states. It pays back in operations. Engineers spend less time on hand-written repair scripts. During incidents, a correlation ID plus a stuck-saga query identifies the affected records in minutes, without anyone having to reconstruct events from logs across several services.

How can it save engineering time?

Without these patterns, partial failures still happen. They just surface later and cost more to handle. The time usually goes to the same few activities:

  • Finding the damage. Someone reports a missing order or a double charge. An engineer then searches logs across several services to work out which writes landed.
  • Writing one-off repair scripts. Each incident gets its own SQL or script, written under pressure, reviewed in a hurry, and thrown away afterwards.
  • Unsafe retries. When retries are not idempotent, on-call engineers cannot simply replay failed work. Each replay needs a manual check that it will not duplicate a charge or a shipment.
  • Chasing silent drift. Divergence that no one detects becomes a data-quality project months later, when the records are harder to trace.

The layered approach changes each of these:

  • Retries become routine. Because consumers deduplicate and calls carry idempotency keys, replaying a dead-letter queue or re-running a relay is a standard operation, not a risk assessment.
  • Repairs are written once. Auto-repair rules in reconciliation replace recurring hand-written fixes for known discrepancy classes. Engineers only handle the classes routed to the human queue.
  • Diagnosis starts from state, not logs. A saga record shows which steps completed and which compensation failed. Correlation IDs connect the remaining pieces.
  • Problems show up as metrics. Outbox lag, saga age, and per-class mismatch rates point to the failing component before customers report it.
  • The work is reusable. The outbox table, relay, dedup table, and retry policy can be packaged as a shared library or service template. Later services adopt them without re-solving the problem.

Illustrative estimate, with all inputs assumed rather than measured: suppose a team handles an assumed 4 partial-failure data-fix tickets per week, at an assumed 3 engineer-hours each for investigation, scripting, and review. That is 12 hours per week. If an assumed half of those tickets are "missing downstream effect" cases that reconciliation can auto-repair, and correlation IDs plus saga state cut investigation time on the rest by an assumed third, weekly effort falls to about 4 hours: 6 tickets' worth removed, and the remaining 6 hours reduced by a third. Weigh that against the one-time cost of building the outbox, dedup, and reconciliation pieces and the ongoing cost of running the relay. Substitute your own ticket counts and handling times. If partial-failure incidents are rare in your system, the payback period will be long. In that case adopt the outbox and idempotent consumers first, and add sagas and reconciliation only where incidents actually occur.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to microservice partial failure handling, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • Consumers learn about a breaking change at review time rather than from production errors.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.