Saga Compensation in Microservices: Designing Undo Steps That Work

What saga compensation is, and why it is not rollback

A saga is a business workflow split into a sequence of local transactions, each owned by a single service and committed independently. If a later step fails, the saga does not abort the earlier ones. It runs compensating transactions: new actions that counteract the business effect of steps that already committed.

This is why saga compensation in microservices is not rollback. A database rollback discards uncommitted work that nobody else has seen. A saga step has already committed, and other parts of the system may have acted on it. The warehouse may have reserved stock against your order. A notification may have reached the customer. A reporting job may have counted the sale. Deleting the row would not undo any of that and would leave those consumers referring to a record that no longer exists. The compensation therefore has to be a forward business action, such as issuing a refund, cancelling a shipment or releasing an inventory hold, and it should leave an audit trail of its own.

Teams adopt sagas when a distributed transaction across services is unavailable or impractical. Two-phase commit needs every participant to support it, holds locks while waiting on the slowest participant, and ties each service's availability to all the others.

The central claim of this guide is that you design each compensation together with the step it undoes. Every compensation must be idempotent, safe to retry, and correct when messages arrive late or out of order. This guide is for teams whose workflows span several services that each own their data.

How it works: compensatable, pivot, and retryable steps

Take an order saga with four steps: reserve inventory, authorize payment, capture payment, ship. Each falls into one of three categories, and the category decides what happens when something fails.

The forward path runs top to bottom. If authorization fails, the orchestrator runs compensations in reverse order. Here that means releasing the inventory reservation. If capture fails, it voids the authorization and then releases inventory. Once capture succeeds, the saga is committed. There is no reverse path after that point, only forward.

This gives the ordering rule. Steps that can be undone go before the pivot. Steps after it must eventually succeed through retries. A refund is not a compensation for capture, because it is a new business transaction with its own fees, delays and customer-visible effects.

Where you put the pivot is a trade-off. If you place it late, more steps need compensations, and each compensation is code you must write, test and keep correct. If you place it early, more steps land in must-succeed territory. Those steps then need durable retries, idempotency and an escalation path for when retries are exhausted.

CopyCOMPENSATABLE    COMPENSATABLE        PIVOT          RETRYABLE
+-----------+    +-----------+    +-----------+    +-----------+
| Reserve   |--->| Authorize |--->| Capture   |--->| Ship      |<-+
| inventory |    | payment   |    | payment   |    | order     |  |
+-----------+    +-----------+    +-----------+    +-----------+  | retry
                    |                   |              |    |     | with
                    | fails             | fails        |    +-----+ backoff
    +---------------+                   |              | retries exhausted
    |                   +---------------+              |
    v                   v                              v
+-----------+    +-----------+                   +-----------+
| Release   |<---| Void      |                   | Escalate  |
| inventory |    | auth      |                   | to human  |
+-----------+    +-----------+                   +-----------+

FORWARD PATH: left to right along the top row.
REVERSE PATH: compensations run newest-first, and none run once capture
succeeds.

Order saga: failures before the capture pivot trigger compensations in reverse order, while shipping after the pivot is retried with backoff and escalated to a human if retries run out.

Implementing idempotent steps and compensations

Idempotency is what makes those retries safe. Give every forward step and every compensation a key derived from the saga ID and the step name, such as order-123:reserve-stock. Brokers deliver at least once and orchestrators retry on timeout, so a handler must recognise a repeat and return the original outcome instead of reserving stock twice.

Record the key in a processed_messages table (key, saga ID, step, outcome, processed time) in the same local transaction as the state change. A primary-key conflict on insert means the message is a duplicate. If you write the key in a separate transaction, a crash between the two commits leaves the effect applied but unrecorded, and the next retry applies it again.

Outgoing events have the same gap. With a transactional outbox, the handler inserts its reply event into an outbox table in that same transaction, and a separate relay publishes it afterwards. The state change and the event then commit or fail together.

Messages can also arrive out of order. A compensation may arrive before its forward step, typically after the orchestrator timed out and gave up. In that case the handler writes a tombstone row that marks the key as cancelled. If the forward request arrives later, the handler finds the tombstone and rejects the request. Otherwise it would apply an effect that nothing will ever undo.

You cannot keep dedup rows forever, but each one must outlive the longest retry and redelivery window. If you delete them sooner, duplicates get through. Compensation handlers must also handle saga instances that were started under older versions of the forward step. Version the payloads, and keep the old undo logic deployed until those instances finish.

Copy-- PostgreSQL. Dedup rows double as tombstones; outbox rows commit with state.
CREATE TABLE processed_messages (
  idempotency_key text PRIMARY KEY,           -- e.g. 'order-123:reserve-stock'
  saga_id text NOT NULL, step text NOT NULL,
  outcome text NOT NULL,                      -- 'reserved' | 'cancelled' | 'compensated' | 'done'
  processed_at timestamptz NOT NULL DEFAULT now()  -- purge only after max retry window
);
CREATE TABLE outbox (
  id bigserial PRIMARY KEY, saga_id text NOT NULL,
  event_type text NOT NULL, payload jsonb NOT NULL,  -- payload carries a schema version
  published_at timestamptz                    -- set by the relay after publishing
);

-- Forward step (one transaction).
BEGIN;
INSERT INTO processed_messages (idempotency_key, saga_id, step, outcome)
VALUES ('order-123:reserve-stock', 'order-123', 'reserve-stock', 'reserved')
ON CONFLICT (idempotency_key) DO NOTHING;
-- 0 rows inserted: ROLLBACK, then SELECT outcome. 'cancelled' => reject (tombstone);
-- anything else => duplicate, return the stored outcome.
UPDATE stock SET reserved = reserved + 2 WHERE sku = 'SKU-42';
INSERT INTO outbox (saga_id, event_type, payload)
VALUES ('order-123', 'StockReserved', '{"v":2,"sku":"SKU-42","qty":2}');
COMMIT;

-- Compensation (one transaction).
BEGIN;
INSERT INTO processed_messages (idempotency_key, saga_id, step, outcome)
VALUES ('order-123:reserve-stock:undo', 'order-123', 'reserve-stock:undo', 'done')
ON CONFLICT (idempotency_key) DO NOTHING;   -- 0 rows: duplicate undo, ROLLBACK
-- Forward never ran: insert tombstone. Forward ran: mark it compensated.
INSERT INTO processed_messages (idempotency_key, saga_id, step, outcome)
VALUES ('order-123:reserve-stock', 'order-123', 'reserve-stock', 'cancelled')
ON CONFLICT (idempotency_key) DO UPDATE SET outcome = 'compensated'
  WHERE processed_messages.outcome = 'reserved'
RETURNING outcome;
-- Only if RETURNING gave 'compensated': release stock using the qty and version
-- from the original payload, so old saga instances are undone correctly.
UPDATE stock SET reserved = reserved - 2 WHERE sku = 'SKU-42';
INSERT INTO outbox (saga_id, event_type, payload)
VALUES ('order-123', 'StockReleased', '{"v":2,"sku":"SKU-42","qty":2}');
COMMIT;

A PostgreSQL schema and transaction sketch that uses the same table for deduplication and tombstones, so a compensation that arrives first blocks the late forward step, while outbox rows commit atomically with each state change.

Isolation, semantic locks, and compensations that must eventually succeed

A saga commits each local transaction as it goes, so other requests can see intermediate state that may later be undone. Without countermeasures, two failure modes appear. A reader acts on a reservation that gets reversed, which is a dirty read. Two sagas overwrite each other's changes to the same record, which is a lost update.

Semantic locks and reservations

The usual mitigation is to make unconfirmed state explicit. An order sits in PENDING until the pivot step succeeds, and inventory is held rather than decremented. Other code paths check that flag and refuse, queue, or warn. This prevents lost updates on the held resource and stops readers from treating pending data as final. It does not stop readers from seeing pending data, and it does not order concurrent sagas on unrelated records.

Locks have their own cost. A stalled saga keeps stock held indefinitely, so holds need an expiry. Expiry introduces a race: the hold lapses, another customer takes the stock, and then the original saga's confirmation arrives late. The confirm step must check the hold version and fail cleanly, not assume the hold is still valid.

Compensations must eventually succeed

A compensation cannot be allowed to give up quietly. To make that hold:

• Retry it with exponential backoff.
• Persist saga progress after every step, so a restarted coordinator can resume from the last recorded state.
• Define what happens when retries run out: move the instance to a dead-letter queue, alert the on-call engineer, and open a ticket in a human resolution queue.

Some steps cannot be undone at all. A sent email or a shipped package has no reverse operation. A refunded charge may still cost processing fees. For these, agree on a mitigation with the business, such as a correction email, a return label, or absorbing the fee, and implement that as the compensation.

Copyimport random
import time

MAX_ATTEMPTS = 6
BASE_DELAY_S = 0.5
MAX_DELAY_S = 30.0

class PermanentFailure(Exception):
    """Raised by a handler when retrying cannot help (e.g. hold version mismatch)."""

def run_compensation(saga, step, handler, store, dlq, alerts, tickets):
    # Resume the attempt count from durable storage so a restarted coordinator keeps the same retry budget.
    attempt = store.load_attempts(saga.id, step)
    reason = "retry budget exhausted"
    while attempt < MAX_ATTEMPTS:
        try:
            # The handler must be idempotent. A crash after it succeeds but before the result is persisted means it runs again.
            handler(saga, idempotency_key=f"{saga.id}:{step}:compensate")
            store.mark_compensated(saga.id, step)
            return True
        except PermanentFailure as exc:
            reason = f"permanent: {exc}"
            break
        except Exception as exc:
            attempt += 1
            reason = f"transient, {attempt} attempts: {exc}"
            store.record_attempt(saga.id, step, attempt, str(exc))
            if attempt < MAX_ATTEMPTS:
                # Capped exponential backoff with full jitter. A real coordinator would schedule a timer instead of sleeping.
                delay = min(MAX_DELAY_S, BASE_DELAY_S * 2 ** attempt)
                time.sleep(random.uniform(0, delay))

    # The compensation is out of retries or cannot succeed. Escalate so it is never dropped silently.
    store.mark_stuck(saga.id, step, reason)
    dlq.publish({"saga_id": saga.id, "step": step, "reason": reason})
    alerts.page("on-call", f"Saga {saga.id} stuck compensating {step}")
    tickets.open(queue="saga-resolution", saga_id=saga.id, step=step, reason=reason)
    return False

A compensation handler wrapper that retries with persisted, jittered exponential backoff, and on exhaustion or permanent failure marks the saga stuck, dead-letters it, pages on-call and opens a ticket in a human resolution queue.

Orchestration vs choreography for compensation, and other alternatives

With orchestration, one component holds the saga's state and issues each step and each compensation in a defined reverse order. That makes the undo path explicit: you can read it in one place and query which compensations ran. The cost is that the orchestrator becomes critical infrastructure. It must persist its progress before acting, so that after a crash it can resume a half-finished compensation instead of forgetting it.

Choreography removes that component. Each service reacts to events and emits its own. Compensation logic is then spread across every participant, though. To answer "did the inventory release run?" you have to correlate events from several services. Adding a step also means checking that every downstream service listens for its failure event.

Before choosing either, ask whether you need a saga at all:

• Redraw boundaries. If two services always change together, merging them allows one local ACID transaction.
• Two-phase commit. This works only when every participant supports it and you accept blocking while the coordinator is unavailable.
• Reserve, then confirm. Hold stock or authorize a payment without capturing it. A failure releases the hold, and the user never sees a charge reversed.

Testing and observing compensation paths

Compensation code runs rarely in production, so it fails in ways nobody has seen. The only way to trust it is to make it run on purpose.

Inject failures deliberately

Wrap each step's client in a fault injector that can fail before the call, after the remote side commits but before the reply arrives, or on the compensation itself. The "after commit, lost reply" case matters most because it forces a retry against a step that already succeeded. Also replay duplicate messages, deliver them out of order, and kill the coordinator between persisting a state transition and dispatching the next command.

A test sketch:

1. Start an order saga with a fault configured at step k.
2. Restart the coordinator once mid-run.
3. Drain all queues until no messages remain in flight.
4. Assert business invariants, not call sequences. Reserved stock equals stock held by open orders. No customer is charged without a confirmed order. Every saga is either completed or compensated.

Asserting on invariants keeps tests valid when you refactor step order.

Choose interleavings, accept residual risk

With n steps, three fault points per step, and arbitrary reordering, the number of cases grows combinatorially. Test every single-fault position exhaustively, because that set grows only linearly. Add randomized multi-fault runs with recorded seeds so failures can be reproduced. Document the combinations you don't cover as accepted risk.

Observe

Tag every log line, span, and message with the saga ID. Expose a state gauge for running, compensating, compensated, and stuck sagas. Alert when any saga stays in compensating longer than its expected bound, such as the maximum retry window plus margin. A saga in that state is holding inconsistent data until a human resolves it.

Copyimport random
import pytest

STEPS = ["reserve_stock", "charge_payment", "create_shipment"]
FAULTS = ["before_call", "after_commit_lost_reply", "on_compensation"]

def assert_invariants(h, saga_id):
    # Check business outcomes, not call sequences, so refactoring step order keeps tests valid.
    confirmed = h.orders(status="confirmed")
    reserved = h.inventory_snapshot().reserved.get("SKU-42", 0)
    assert reserved == sum(o.qty for o in confirmed if o.sku == "SKU-42")
    charged = {p.order_id for p in h.payments(status="captured")}
    assert charged <= {o.id for o in confirmed}, "customer charged without confirmed order"
    assert h.saga_state(saga_id) in {"completed", "compensated"}

# Exhaustive single-fault matrix: len(STEPS) * len(FAULTS) cases, grows linearly.
@pytest.mark.parametrize("step", STEPS)
@pytest.mark.parametrize("fault", FAULTS)
def test_single_fault(saga_harness, step, fault):
    # saga_harness is a project fixture with fault-injecting clients; faults fire once (transient).
    saga_harness.inject(step=step, fault=fault)
    saga_id = saga_harness.start_order_saga(order_id="ord-1001", sku="SKU-42", qty=2)
    # Crash between persisting a state transition and dispatching the next command.
    saga_harness.restart_coordinator(after_persisted_transitions=1)
    saga_harness.duplicate_all_messages()  # exercise idempotent handlers
    saga_harness.drain_until_quiescent(timeout_s=30)
    assert_invariants(saga_harness, saga_id)

# Randomized multi-fault runs; the seed is in the test ID so any failure can be replayed.
@pytest.mark.parametrize("seed", range(50))
def test_random_multi_fault(saga_harness, seed):
    rng = random.Random(seed)
    for step in rng.sample(STEPS, k=2):
        saga_harness.inject(step=step, fault=rng.choice(FAULTS))
    saga_harness.shuffle_delivery(rng)  # reorder in-flight messages
    saga_id = saga_harness.start_order_saga(order_id=f"ord-{seed}", sku="SKU-42", qty=1)
    saga_harness.drain_until_quiescent(timeout_s=30)
    assert_invariants(saga_harness, saga_id)

A pytest sketch, built on a hypothetical project-specific saga_harness fixture, that runs every single-fault position exhaustively, adds seeded random multi-fault runs, and asserts business invariants after coordinator restarts and duplicate or reordered messages.

When to use sagas, when to avoid them, and the impact on cost and engineering time

A saga is justified when a business transaction spans services that own their data independently, the steps take longer than you can hold a lock, and the business can define what "undo" means for each step. Reserving inventory, charging a card, and booking a carrier is the classic fit.

Avoid one in these cases:

• The compensation is visible to the user. A charge followed by a refund looks to the customer like a mistake, and the refund may take days to post. A slower synchronous flow, such as an authorization hold captured only after fulfillment is confirmed, is often the better experience.
• The workflow fits inside one service. If one database can own the data, a local ACID transaction is simpler and correct by construction. Splitting it into a saga adds failure modes for no benefit.
• Nobody can staff the operational side. Someone has to run dashboards, follow runbooks, and resolve stuck sagas by hand. If no team will own that work, the inconsistent states will build up without anyone noticing.

The design scope includes each compensation, deduplication storage, outbox infrastructure, runbooks, and tooling for human resolution. If these are missing from the estimate, the estimate is too low.

Designing compensations up front costs less than adding undo logic later. Gaps found during incidents get fixed under pressure, one at a time, and often by whoever happens to be on call. When the work is planned:

• Each step has a clear owner for its forward and reverse paths.
• The idempotency and outbox components are built once and reused by later sagas, so the second saga costs much less than the first.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to saga compensation microservices, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• The dependency graph answers "what does this touch?" in seconds instead of a manual audit.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• You can size a change's blast radius before merging, avoiding surprise cross-team firefighting.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.