Saga Compensating Transactions: Designing Undo Steps That Work

What saga compensating transactions are and why 2PC is avoided

A saga splits one business workflow into a sequence of local transactions. Each local transaction commits in its own service's database and then triggers the next step. If a later step fails, the saga runs saga compensating transactions for the steps that already committed, in reverse order.

Two-phase commit (2PC) would make the whole workflow atomic, but teams usually avoid it across microservices for three reasons:

• The coordinator can block participants. After a participant votes "prepared", it holds its locks until it hears the decision. If the coordinator crashes, the participant can be stuck waiting.
• Availability is coupled. A commit succeeds only when every participant is reachable at that moment.
• Support is missing. Many datastores and message brokers cannot take part in a distributed transaction at all.

Sagas give up isolation in exchange. Other requests can see intermediate states, such as a reserved seat whose payment later fails.

A compensation is not a physical rollback. It is a new forward business operation: a refund, a cancellation, or a reversal entry. The original record stays in place, so you get an audit trail of what happened.

Sagas fit teams whose business workflows span several services, each owning its own database.

Avoid sagas when:

• one database can hold the whole transaction,
• law or finance requires strict isolation, or
• a step cannot be meaningfully offset, such as a sent email or a shipped parcel.

Classifying steps: compensatable, pivot and retriable

Each saga step falls into one of three types, and the type comes from what the step does in the business domain.

• Compensatable steps can be semantically undone by a later action. A reservation can be released and a pending order can be cancelled. The original record stays, and a new action reverses its effect.
• The pivot is the go/no-go point. When it succeeds, the saga is committed to finishing. When it fails, every earlier step is compensated. Choose the first step whose effect is costly or impossible to reverse cleanly.
• Retriable steps come after the pivot. They must eventually succeed, so they need to be idempotent and safe to repeat until they complete.

The ordering rule follows from these types. Put irreversible actions, such as charging a card or sending an email, at or after the pivot. If a charge runs before a step that can still fail, a failure later in the flow forces a refund. A refund is a separate financial event that the customer sees, not an undo.

Some effects have no true undo, such as a shipped package or a sent notification. Offsets for these are business decisions. A return label, a correction email, or simply writing off the cost each involve product, finance and support trade-offs, so engineering should not pick one on its own.

How a saga runs and compensates: orchestration vs choreography

Under orchestration, a single coordinator issues each command, records the outcome, and on failure walks back through the completed steps in reverse. Reverse order matters because later steps often depend on earlier ones: you release the stock reservation before cancelling the order that owns it.

CopyOrchestrator        Order svc        Inventory svc      Payment svc
    |-- CreateOrder -->|                  |                  |
    |<----- ok --------|                  |                  |
    |-- ReserveStock ----------------->|                  |
    |<----- ok ------------------------|                  |
    |-- AuthorizePayment ---------------------------------->|
    |<----- DECLINED ---------------------------------------|  <- failure
    |-- ReleaseStock ----------------->|   (compensate step 2)
    |-- CancelOrder -->|                   (compensate step 1)

Under choreography there is no coordinator. Inventory listens for OrderCreated, payment listens for StockReserved, and a PaymentDeclined event triggers handlers in both inventory and order services. The reverse sequence is never written down in one place. It emerges from subscriptions.

Choreography suits short sagas of two or three steps owned by one team, where the event flow is easy to hold in your head. Choose orchestration once a saga crosses team boundaries, has more than a few steps, or includes pivot steps. In those cases, being able to answer "which compensations ran, and which are still pending?" from a single record is worth the cost of operating the coordinator.

CopyORCHESTRATION: compensation order lives in one saga definition

        +-----------------------------------------------+
        | Orchestrator (durable, highly available)      |
        | forward: 1 CreateOrder      -> ok             |
        |          2 ReserveStock     -> ok             |
        |          3 AuthorizePayment -> DECLINED       |
        | undo:    2' ReleaseStock -> 1' CancelOrder    |
        +-----------------------------------------------+
             | ^              | ^               | ^
             v |              v |               v |
        +-----------+  +---------------+  +-------------+
        | Order svc |  | Inventory svc |  | Payment svc |
        +-----------+  +---------------+  +-------------+
         step 1, 1'        step 2, 2'        step 3

CHOREOGRAPHY: no coordinator; compensation emerges from subscriptions

 +-----------+ OrderCreated  +---------------+ StockReserved +-------------+
 | Order svc |-------------->| Inventory svc |-------------->| Payment svc |
 +-----------+               +---------------+               +-------------+
       ^                             ^                              |
       |      PaymentDeclined        |       PaymentDeclined        |
       +-----------------------------+------------------------------+
  CancelOrder handler         ReleaseStock handler

The same order saga under orchestration, where one coordinator records outcomes and runs ReleaseStock then CancelOrder after the payment decline, and under choreography, where the same compensations are triggered by separate PaymentDeclined subscribers.

Implementing idempotent compensations and durable saga state

Every step and every compensation needs a stable identity, and the natural key is the saga ID plus the step ID. Delivery is at-least-once in practice, so the same command will eventually arrive twice. A participant protects itself with a deduplication table keyed on (saga_id, step_id, kind), where kind is either the action or its compensation. The handler's first write is an insert into that table. A unique-constraint violation means the work has already been done, so the handler returns the recorded outcome and does not redo the side effect.

Cancel of nothing

Messages can arrive out of order. A compensation can reach a participant before the original action, and sometimes the original action never arrives at all. If you reject that compensation, the coordinator will retry it forever. The handler should instead record the compensation as a successful no-op and return success. This record also does a second job: when the late original action finally arrives, its handler checks the deduplication table, finds a compensation already recorded for that key, and turns itself into a no-op too. Without this check, a delayed reservation can land after its cancellation and leave stock held indefinitely.

State that survives a crash

The dedupe insert, the business change, the saga state transition and any outgoing event must commit in one local transaction. You can write the event to a saga log or state table directly, or to a transactional outbox that a relay publishes later. If they commit separately, a crash between them leaves a step that ran but was never recorded, or recorded but never announced. A coordinator that restarts mid-compensation then has to guess. When all four writes commit together, the coordinator reads its state table, finds the steps marked COMPENSATING that are not yet COMPENSATED, and resumes from that recorded point.

Copy-- Participant tables (PostgreSQL).
CREATE TABLE processed_step (
  saga_id uuid, step_id text,
  kind text CHECK (kind IN ('ACTION','COMPENSATION')),
  outcome text NOT NULL,
  PRIMARY KEY (saga_id, step_id, kind));
CREATE TABLE saga_step_state (
  saga_id uuid, step_id text, status text NOT NULL,
  PRIMARY KEY (saga_id, step_id));
CREATE TABLE outbox (
  id bigserial PRIMARY KEY, saga_id uuid NOT NULL,
  event_type text NOT NULL, payload jsonb NOT NULL,
  published_at timestamptz);  -- relay sets this after publishing

-- Compensation handler. :saga_id is a placeholder bound by the application.
BEGIN;
-- Create and lock the step row so the action and compensation for this key serialize.
INSERT INTO saga_step_state VALUES (:saga_id, 'reserve-stock', 'PENDING')
  ON CONFLICT (saga_id, step_id) DO NOTHING;
SELECT status FROM saga_step_state
  WHERE saga_id = :saga_id AND step_id = 'reserve-stock' FOR UPDATE;
-- Dedupe insert. 0 rows affected => already compensated: ROLLBACK, return stored outcome.
INSERT INTO processed_step VALUES (:saga_id, 'reserve-stock', 'COMPENSATION', 'OK')
  ON CONFLICT DO NOTHING;
-- Undo only if the action ran. If it never arrived, 0 rows change ("cancel of nothing").
UPDATE reservation SET status = 'RELEASED'
  WHERE saga_id = :saga_id AND status = 'HELD';
-- Record the transition. The action handler checks this status and becomes a no-op.
UPDATE saga_step_state SET status = 'COMPENSATED'
  WHERE saga_id = :saga_id AND step_id = 'reserve-stock';
INSERT INTO outbox (saga_id, event_type, payload)
  VALUES (:saga_id, 'StockReleased', '{"step":"reserve-stock"}');
COMMIT;
-- The action handler takes the same row lock first. If status = 'COMPENSATED',
-- it records its dedupe row as a no-op and commits without reserving stock.

A PostgreSQL schema and compensation handler that commit the row lock, deduplication insert, business undo, state transition and outbox event in one local transaction, so a compensation that arrives before its action succeeds as a recorded no-op and blocks a late action.

Countermeasures for missing isolation

Recovery keeps a saga consistent eventually, but it does nothing for readers in the meantime. Each local transaction commits on its own, so other requests see intermediate state. A second checkout can read stock that a failing saga is about to release (a dirty read). Two sagas can also read the same balance and each overwrite the other's change (a lost update).

The usual countermeasures:

• Semantic locks. Write an explicit pending state. Other transactions then know the record is in flight and can wait, reject or proceed knowingly.
• Commutative updates. Write stock = stock - 2 instead of stock = 7, so the order in which sagas apply changes does not matter.
• Re-read before acting. Check current values in the step that commits, not the values the saga saw earlier.
• Version checks. Make writes conditional on a row version, and fail or retry on mismatch.

In the order example, inventory moves units to RESERVED with an expires_at timestamp rather than decrementing available stock outright. Payment confirmation flips the reservation to COMMITTED. A payment failure releases it. If the coordinator dies, a reaper job finds reservations past expiry whose saga is not complete and releases them, so stock is not held forever.

That safety has a cost. You add states, timeout tuning and a reaper that needs its own monitoring. Long-running sagas also force a judgment call. If payment is slow, expiring the reservation may compensate a charge that would have succeeded. Waiting holds stock from other buyers. Pick the timeout per step from business tolerance, not a global default.

Copy-- PostgreSQL. Step 1 (inventory service): reserve stock in one local transaction.
BEGIN;
-- Commutative, guarded decrement: never writes an absolute value read earlier.
UPDATE inventory
   SET available = available - 2
 WHERE sku = 'SKU-1042' AND available >= 2;   -- 0 rows => fail this step
-- Semantic lock: the units are visibly in flight, with a deadline.
INSERT INTO reservations (id, saga_id, sku, qty, state, expires_at)
VALUES ('res-7f3a', 'saga-91c2', 'SKU-1042', 2, 'RESERVED',
        now() + interval '15 minutes');
COMMIT;

-- Step 3 (after payment succeeds): commit only if still RESERVED.
-- 0 rows => the reaper already released it; the saga must refund the charge.
UPDATE reservations
   SET state = 'COMMITTED'
 WHERE id = 'res-7f3a' AND state = 'RESERVED';

-- Reaper job (scheduled): release expired reservations and return the stock.
-- The state = 'RESERVED' guard makes it race safely with the commit above:
-- whichever UPDATE takes the row lock first wins; the other matches 0 rows.
BEGIN;
WITH expired AS (
  UPDATE reservations
     SET state = 'RELEASED'
   WHERE state = 'RESERVED' AND expires_at < now()
  RETURNING sku, qty
), totals AS (
  -- Aggregate first: UPDATE ... FROM applies only one match per target row.
  SELECT sku, sum(qty) AS qty FROM expired GROUP BY sku
)
UPDATE inventory i
   SET available = i.available + t.qty   -- commutative increment
  FROM totals t
 WHERE i.sku = t.sku;
COMMIT;

A PostgreSQL sketch of a reservation with an expiry deadline. Guarded, commutative stock updates and state-conditional transitions let the payment commit and the reaper release race without double-releasing stock or committing a reservation that has already expired.

When compensation itself fails and how to test it

Compensations fail too, and the failure is usually not transient. Retry them a bounded number of times with exponential backoff and jitter. Backoff gives a struggling dependency room to recover, and jitter keeps thousands of stalled sagas from retrying in lockstep. When retries are exhausted, move the saga to an explicit COMPENSATION_FAILED state. Persist it, raise an alert, and attach a runbook link to the alert. Never drop the saga or let it loop forever. A silent failure leaves money or inventory inconsistent with no record that anyone owes a fix.

Many of these failures are semantic conflicts with state that changed after the forward step. A refund can target an account closed since the charge. A shipment can be cancelled after the carrier already scanned the parcel. A loyalty-point reversal can hit a balance the customer has already spent. No retry resolves these. A person has to decide, so each saga type needs a named owning team with the authority and tooling to issue a manual credit or write off the loss.

Testing compensation deliberately

Compensation paths almost never run in production until an incident forces them, so failures in them stay hidden. Exercise them on purpose and continuously, in CI and in a staging environment. A useful harness works like this:

• Enumerate every step boundary in the saga definition.
• For each boundary, run the saga with a fault injected there: a crash before the commit, a crash after the commit but before the ack, and a timeout.
• Repeat each run with duplicated messages, reordered delivery, and a coordinator restart mid-compensation.
• Assert that every resource ends either fully committed or fully compensated, and that no saga is lost or stuck without an alert.

Copyimport itertools

import pytest

# Project-local test harness (illustrative names): runs a saga in-process
# against fake services and lets the test inject faults at a chosen step.
from saga_harness import ORDER_SAGA, FaultPlan, run_saga, resource_state

# Enumerate every boundary from the saga definition itself, so new steps
# are covered automatically instead of relying on a hand-maintained list.
STEPS = ORDER_SAGA.step_names()  # e.g. reserve_stock, charge_card, create_shipment
FAULTS = ["crash_before_commit", "crash_after_commit_before_ack", "timeout"]
DELIVERY = ["normal", "duplicate", "reorder", "coordinator_restart"]


@pytest.mark.parametrize(
    "step,fault,delivery", list(itertools.product(STEPS, FAULTS, DELIVERY))
)
def test_saga_always_converges(step, fault, delivery):
    # Fixed seed so a failing combination can be replayed exactly.
    plan = FaultPlan(at_step=step, fault=fault, delivery=delivery, seed=42)
    result = run_saga(ORDER_SAGA, order_id="order-1234", faults=plan)
    state = resource_state(order_id="order-1234")

    # No half-applied outcome: all forward steps stand, or all are undone.
    assert state.all_committed() or state.all_compensated(), state.describe()

    # Duplicates and redelivery must not apply any compensation twice.
    assert all(n <= 1 for n in state.compensation_counts().values())

    # The saga must reach a terminal state; nothing lost or silently stuck.
    assert result.status in {"COMPLETED", "COMPENSATED", "COMPENSATION_FAILED"}

    # A parked saga is acceptable only if a human has been told how to fix it.
    if result.status == "COMPENSATION_FAILED":
        assert result.alert is not None
        assert result.alert.runbook_url

A pytest sketch that runs the saga once for every combination of step boundary, injected fault and delivery anomaly. Each run asserts that resources end fully committed or fully compensated, that no compensation is applied twice, and that any parked saga has raised an alert with a runbook link.

Alternatives and impact on cost and engineering time

Before committing to compensation, check whether a cheaper design covers the workflow.

Sagas are expensive to build. Each compensatable step needs a forward action and an undo action, so a workflow has roughly twice as much logic, and twice as many paths to test. You also pay for deduplication storage, extra round trips that add latency, status columns and pending states in every participant schema, and an on-call rotation that knows how to handle parked sagas.

Sagas also save time in specific ways. When steps are classified explicitly, nobody has to write ad hoc SQL to fix half-finished orders. Durable saga state means investigating an incident is a matter of reading a log instead of reconstructing events from scattered services. Runbook-driven parking gives operators a defined procedure instead of leaving them to guess.

Decision checklist

• Can the data live in one database? Use a local transaction.
• Are all participants XA-capable and few? Consider 2PC.
• Are resources countable? Try escrow first.
• Is temporary drift harmless? Reconcile periodically.
• Otherwise, budget for compensations, deduplication and on-call from day one.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to saga compensating transactions, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• The dependency graph answers "what does this touch?" in seconds instead of a manual audit.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• You can size a change's blast radius before merging, avoiding surprise cross-team firefighting.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.