Saga Compensating Transactions: Designing Undo as Forward Actions

What saga compensating transactions are and why they are not rollbacks

A saga is a sequence of local transactions, each committed independently in the database of the service that owns it. If step n fails, the saga does not abort the earlier steps, because they have already committed and other processes may have read their results. Instead, it runs saga compensating transactions: new local transactions that counteract the business effect of steps 1 through n–1.

This is why compensation is semantic, not physical. A database rollback discards uncommitted writes as if they never happened. A compensation adds new writes on top of committed ones. Refunding a captured payment does not delete the charge. It creates a refund record, and the customer sees both lines on their statement. Releasing an inventory hold leaves an audit trail of the reservation and its release. The system ends up in a state that is equivalent for business purposes, not identical to the state before.

Some actions cannot be counteracted at all, only mitigated. A sent email cannot be unsent. You can send a correction. A shipped parcel cannot be un-shipped. You can arrange a return. Classifying each step as compensatable or mitigation-only is the first design decision, because it determines where that step can safely sit in the sequence.

Teams adopt the saga pattern for distributed transactions when two-phase commit is impractical. Services own separate databases, often on different engines, and a coordinator holding locks across them couples their availability. Sagas suit any team whose business process crosses several service-owned databases and can tolerate brief, visible inconsistency.

How to structure a saga: compensatable, pivot, and retryable steps

A saga is a sequence of local transactions, each committed in its own service. The useful design question is not "how do I undo each step?" but "which steps must be undoable at all?" Classifying every step into one of three categories answers that.

• Compensatable steps can be semantically reversed by a later local transaction. They come first, because a failure anywhere in this region triggers compensation of everything already done.
• The pivot step is the go/no-go point. If it commits, the saga is committed to finishing. It is neither compensated nor retried indefinitely. If it fails, the saga compensates backward.
• Retryable steps follow the pivot and must eventually succeed. They need idempotent handlers and retry with backoff. They never need compensation because the saga will not roll back past the pivot.

The payoff comes from ordering. If you push hard-to-undo actions, such as handing a parcel to a carrier or emailing a customer, behind the pivot, you never have to design an undo for them. You only need to make them reliably repeatable, which is a much easier problem.

Orchestration versus choreography for compensation

With orchestration, one component owns the saga definition. It invokes each step, records the outcome in its own durable log, and on failure walks that log backwards, issuing compensations for completed steps only. Reverse order is natural here because the orchestrator already holds the list of what succeeded. Nobody has to reconstruct it from scattered events.

CopyOrchestrator (durable saga log)
  forward:  [1] reserve stock --> [2] charge payment --> [3] create shipment  X fails
  reverse:  [2'] refund payment <-- triggered first
            [1'] release stock  <-- then this
  log:      1=done, 2=done, 3=failed -> compensate 2, then 1

With choreography, there is no single owner. Each service listens for events and publishes its own. Compensation is spread across services: inventory releases stock when it sees PaymentFailed, and payment refunds when it sees ShipmentFailed. The saga's "completed steps" exist only implicitly, as each service's local state plus the event history. That is why ordering mistakes are easy to introduce and hard to spot.

Choreography works well for two or three steps. Beyond that, the orchestrator's single readable flow is usually worth its operational cost.

CopyORCHESTRATION: one owner, explicit saga log, reverse order enforced

+---------------------------------------------------------------+
|  Orchestrator           durable saga log                      |
|                         1=done  2=done  3=failed              |
|                         on failure: compensate 2', then 1'    |
+---------------------------------------------------------------+
     |     |                 |     |                 |
    [1]  [1']               [2]  [2']               [3]
     v     v                 v     v                 v
+---------------+       +---------------+       +---------------+
|   Inventory   |       |    Payment    |       |   Shipping    |
+---------------+       +---------------+       +---------------+
                                                     X fails
  [n]  forward step       [n'] compensation, issued 2' then 1'

CHOREOGRAPHY: no owner, completed steps inferred from events

+-----------+  StockReserved  +-----------+  PaymentCharged +------------+
| Inventory |---------------->|  Payment  |---------------->|  Shipping  |
|           |<----------------|           |<----------------|            |
+-----------+  PaymentFailed  +-----------+  ShipmentFailed +------------+
  release stock                 refund payment

An orchestrator records each step in its durable saga log and issues compensations in reverse order, while in choreography each service compensates on its own when it sees a failure event, so the overall order depends on how the events are wired.

Implementing compensations safely: idempotency, ordering, and durable state

Every message in a saga will eventually be delivered more than once, so every step and every compensation must tolerate replays. The simplest mechanism is to key each command on the saga instance ID plus the step name, for example (saga_id, 'reserve_stock'). The receiving service stores this key in a processed_commands table with a unique constraint, inside the same local transaction as the business change. A duplicate then fails the insert, and the service returns the stored result. It does not act twice. Checking first and writing later is not enough, because two concurrent deliveries can both pass the check.

Compensation that arrives first

Under retries and reordering, cancel_reservation can reach inventory before reserve_stock does. If the compensation does nothing, the late reservation succeeds and the stock stays locked with no owner. The fix is to have the compensation write a tombstone row for that key, with status CANCELLED. When the original command arrives later, it finds the tombstone and is rejected. The rule is that a compensation for an action never seen still records that it happened.

State and messages that cannot diverge

The orchestrator persists each saga as a row holding saga_id, current_state, and a version used for optimistic concurrency. Transitions happen only through conditional updates such as WHERE version = ?. Each transition writes its outgoing command to an outbox table in the same transaction. A separate relay process publishes outbox rows and marks them sent. A crash can then cause a resend, which deduplication already absorbs, but it can never cause a lost message.

Copy-- Receiving service: one row per (saga_id, step). The primary key makes duplicates fail.
CREATE TABLE processed_commands (
  saga_id VARCHAR(64) NOT NULL,
  step    VARCHAR(64) NOT NULL,
  status  VARCHAR(16) NOT NULL,   -- 'DONE', or 'CANCELLED' (tombstone)
  result  VARCHAR(255),
  PRIMARY KEY (saga_id, step)
);

-- reserve_stock: the key and the business change share one transaction.
-- If the INSERT fails on the key, roll back and read the existing row:
-- DONE means return the stored result, CANCELLED means reject the late command.
BEGIN;
INSERT INTO processed_commands (saga_id, step, status, result)
VALUES ('saga-7f3a', 'reserve_stock', 'DONE', 'reservation r-1001');
UPDATE stock SET reserved = reserved + 2
 WHERE sku = 'SKU-123' AND available - reserved >= 2;  -- 0 rows: roll back, reply failure
COMMIT;

-- cancel_reservation: claim the forward step's key as a tombstone.
-- If the key already exists as DONE, release the stock instead and set
-- status = 'CANCELLED'. If it is already CANCELLED, this is a replay, so do nothing.
BEGIN;
INSERT INTO processed_commands (saga_id, step, status)
VALUES ('saga-7f3a', 'reserve_stock', 'CANCELLED');
COMMIT;

-- Orchestrator: the guarded state transition and the outgoing command commit together.
BEGIN;
UPDATE sagas SET current_state = 'RESERVING_STOCK', version = version + 1
 WHERE saga_id = 'saga-7f3a' AND version = 4;  -- 0 rows: lost the race, roll back
INSERT INTO outbox (id, saga_id, command, payload, sent)
VALUES ('msg-0001', 'saga-7f3a', 'reserve_stock',
        '{"sku":"SKU-123","qty":2}', FALSE);
COMMIT;

-- Relay loop: SELECT unsent rows, publish each one, then
-- UPDATE outbox SET sent = TRUE WHERE id = ?. A crash in between causes a resend, not a loss.

Generic SQL for deduplication keyed on (saga_id, step), a CANCELLED tombstone that rejects a late forward command, and an optimistic-concurrency saga transition that writes its outgoing command to an outbox in the same transaction.

Living without isolation: semantic locks, pending states, and failed compensations

A saga commits each local transaction as it goes, so it has no isolation. Between reserving stock and capturing payment, another saga can read inventory that a compensation will release a moment later. The results are the classic anomalies: lost updates, and decisions made on data that is about to be undone.

Several countermeasures help:

• Semantic locks. Write an explicit status such as PENDING_RESERVATION. Other sagas then know the row is tentative and can wait, reject, or proceed while accepting the risk.
• Commutative updates. Apply relative changes, such as decrementing available stock, instead of setting absolute values. Steps and compensations from different sagas then produce the same result in any order.
• Re-read and version. Re-read state immediately before acting, and make writes conditional on a version column. A stale write fails loudly instead of overwriting a compensation.

Pending states need timeouts. Without them, a crashed orchestrator leaves rows locked forever. A reaper that expires them can race with a saga that is slow but still alive. Guard both paths with the same conditional transition: move out of PENDING only if the row is still PENDING, so exactly one writer wins.

Compensations can fail too. Retry them with bounded exponential backoff and jitter, because most failures are transient. Once the retry budget is exhausted, move the saga to FAILED_NEEDS_INTERVENTION and route it to a dead-letter queue or a human exception workflow, with full context attached. Treat a stuck saga as a real state with an owner and an alert, not something to drop silently.

For observability, tag every log line, span, and message with the saga ID. Expose the statuses running, compensating, compensated, and failed-needs-intervention. Record when each state was entered so you can alert on time spent in it.

Copyimport logging, random, time

log = logging.getLogger("saga")

class TransientError(Exception):
    pass

def transition(conn, saga_id, expected, new):
    # Conditional write: succeeds only if the row is still in `expected`.
    # The saga and the reaper both call this, so exactly one of them wins.
    cur = conn.execute(
        "UPDATE sagas SET status = ?, version = version + 1, "
        "status_since = CURRENT_TIMESTAMP "  # supports time-in-state metrics
        "WHERE saga_id = ? AND status = ?",
        (new, saga_id, expected),
    )
    conn.commit()
    return cur.rowcount == 1

def compensate(conn, saga_id, undo_step, dead_letter, max_attempts=5, base=0.5, cap=30.0):
    if not transition(conn, saga_id, "RUNNING", "COMPENSATING"):
        log.info("saga=%s already moved by another writer", saga_id)
        return
    for attempt in range(1, max_attempts + 1):
        try:
            undo_step(saga_id)  # must be idempotent: it may run more than once
            transition(conn, saga_id, "COMPENSATING", "COMPENSATED")
            log.info("saga=%s compensated attempt=%d", saga_id, attempt)
            return
        except TransientError as exc:
            # Bounded exponential backoff with full jitter.
            delay = random.uniform(0, min(cap, base * 2 ** attempt))
            log.warning("saga=%s attempt=%d failed: %s; retry in %.1fs",
                        saga_id, attempt, exc, delay)
            time.sleep(delay)
    # Budget exhausted: make the stuck saga a visible, owned state.
    transition(conn, saga_id, "COMPENSATING", "FAILED_NEEDS_INTERVENTION")
    dead_letter({"saga_id": saga_id, "attempts": max_attempts})
    log.error("saga=%s escalated to dead-letter queue", saga_id)

A guarded status transition, implemented as a conditional UPDATE, lets one writer win against the reaper; compensation is retried with bounded, jittered backoff and escalated to FAILED_NEEDS_INTERVENTION, with every log line tagged by saga ID.

Worked example: order placement across payment, inventory, and shipping

Take a four-step order saga: reserve inventory, capture payment, send the order-confirmation email, then book the shipment with the carrier. Booking the shipment is the pivot. Once the carrier accepts the parcel, the saga can only move forward. Compensations run in reverse order of completion, so the most recent side effect is undone first.

Two compensations are only mitigations. A sent email cannot be recalled, so its compensation is a correction notice. A refund is a new transaction that follows the payment provider's own rules. It may take days to settle, and the provider may not return its processing fee, so the customer sees a charge and a separate credit rather than a vanished charge.

A failed email does not unwind the order. The step is retryable, and cancelling a paid order because an SMTP relay timed out costs more than a delayed message.

The last row shows a design lesson. Moving the confirmation email after the pivot removes the only step that needs a mitigation. Order your steps so that irreversible side effects happen as late as possible.

When to avoid sagas, alternatives, and the impact on cost and engineering time

Sagas fit when every step before the pivot can be undone by a business action, and the business accepts that other readers may briefly see intermediate state. Avoid them when you need strict all-or-nothing semantics with isolation, such as ledger transfers that must never be observed half-applied. Also avoid them when a customer-visible intermediate state is unacceptable. A charge followed by a refund still appears on a card statement and still generates support contacts.

The development cost is real. Each forward step needs a counterpart with its own validation, failure modes and schema dependencies, which roughly doubles the business-logic surface. Compensation paths run rarely, so they rot quietly. A field renamed on the forward path breaks the undo, and nobody notices until an incident. Testing needs fault injection: kill the process between steps, deliver duplicate messages, reorder them, and make the compensation itself fail.

Designing compensations up front still costs less than adding them later. Without them, partial failures turn into manual data repair and incident investigations that rebuild state from logs. Retrofitting is also hard when the original records never stored what an undo needs, such as the captured amount or the reservation ID. Retry budgets and escalation thresholds remain a judgment call. Weigh the cost of a stuck order against operator time, then tune both from your own incident history.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to saga compensating transactions, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• The dependency graph answers "what does this touch?" in seconds instead of a manual audit.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• You can size a change's blast radius before merging, avoiding surprise cross-team firefighting.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.