What is a saga in an order flow, and why are compensations not rollbacks?

A saga splits one business operation into a sequence of local transactions, each committed independently by the service that owns the data. In an order flow, the forward steps are typically: reserve inventory, authorize payment, create shipment, confirm order. Each step that can be undone has a paired compensating transaction.

Forward step Compensating transaction Notes
Reserve inventory Release inventory Safe to retry if keyed by order ID
Authorize payment Void authorization, or refund if captured Refund is a new money movement, not an erasure
Create shipment Cancel shipment Only possible before carrier handoff
Confirm order None Pivot point: commit from here on

A compensation is a semantic undo, not a rollback. A database rollback discards uncommitted writes as if they never happened. A compensation runs after the original step has committed and other parties may have seen it. A refund leaves both the charge and the reversal on the customer's statement. A released reservation may already have caused another customer to see "out of stock."

Because no step is isolated, intermediate states are visible. Model them explicitly with statuses such as PENDING_PAYMENT and RESERVED, or with semantic locks that stop other flows from acting on a half-finished order.

Irreversible actions, such as confirmation emails or handing parcels to a carrier, belong after the pivot. Once the saga passes that point it must only retry forward and never compensate.

Teams use sagas instead of two-phase commit for three reasons. Payment providers and carriers do not take part in your distributed transactions. Holding locks across services couples their availability. Commit coordinators also block when a participant fails.

How does a choreography saga work?

In choreography, no component owns the flow. Each service subscribes to the domain events it cares about, does its local work, and publishes its own event. Order emits OrderPlaced. Inventory reserves stock and emits StockReserved. Payment charges the card and emits PaymentCaptured or PaymentFailed. A failure event is itself the trigger for compensation: Inventory listens for PaymentFailed and releases the reservation, and Order listens for it and marks the order cancelled.

Choreography (event mesh)          Orchestration (hub and spoke)

 Order ───► Inventory                      Inventory
   ▲  ╲        │                               ▲
   │   ╲       ▼                               │
   │    ╲─── Payment          Order ◄──── Orchestrator ────► Payment
   │           │                               │
   └───────── Shipping                         ▼
                                            Shipping

The saga's definition exists only implicitly, spread across subscriptions in several codebases. To answer "where is order 4812 stuck?", you have to reconstruct the sequence from events.

Every step must commit its database change and publish its event atomically. If the service commits and then crashes before publishing, the saga stalls silently. If it publishes before the commit and the commit then fails, downstream services act on something that never happened. The transactional outbox closes this gap. The service writes the event to an outbox table in the same local transaction as the state change, and a separate relay publishes rows from that table to the broker. Delivery becomes at-least-once, so every consumer has to tolerate duplicates.

Choreography also lets cycles form without anyone noticing. Suppose Inventory emits StockAdjusted whenever an order changes, and Order updates the order whenever stock is adjusted. That pair can ping-pong indefinitely. No single repository shows the loop, so it usually surfaces in production as rising event volume.

-- PostgreSQL. The outbox lives in the same database as the service's own state.
CREATE TABLE outbox (
  id           uuid        PRIMARY KEY DEFAULT gen_random_uuid(),
  aggregate_id text        NOT NULL,
  event_type   text        NOT NULL,
  payload      jsonb       NOT NULL,
  created_at   timestamptz NOT NULL DEFAULT now(),
  published_at timestamptz            -- NULL means not yet sent to the broker
);

-- Inventory: the state change and its event commit or roll back together.
-- The CTE writes an outbox row only if the reservation actually happened.
WITH reserved AS (
  UPDATE stock SET reserved = reserved + 2
   WHERE sku = 'SKU-1001' AND available - reserved >= 2
  RETURNING sku
)
INSERT INTO outbox (aggregate_id, event_type, payload)
SELECT 'order-4812', 'StockReserved',
       jsonb_build_object('orderId', 'order-4812', 'sku', sku, 'qty', 2)
  FROM reserved;

-- Relay: claim a batch of unsent rows. SKIP LOCKED lets several relays run.
BEGIN;
SELECT id, event_type, payload
  FROM outbox
 WHERE published_at IS NULL
 ORDER BY created_at
 LIMIT 100
   FOR UPDATE SKIP LOCKED;
-- ...publish each row to the broker here...
-- A crash before this UPDATE means the rows are sent again (at-least-once).
UPDATE outbox SET published_at = now() WHERE id = ANY($1);  -- $1: ids just published
COMMIT;

-- Consumer (Payment): record the event id in the same transaction as its side effect.
-- If 0 rows are inserted, the event is a duplicate. Skip the work.
INSERT INTO processed_events (event_id) VALUES ($1) ON CONFLICT DO NOTHING;

A PostgreSQL transactional outbox: Inventory writes the stock reservation and its StockReserved event in a single statement, a relay publishes unsent rows at-least-once, and a consumer drops duplicate deliveries using a processed-events table.

How does saga orchestration with Temporal work?

Orchestration moves that hidden sequence into a single place. In Temporal, the order flow is an ordinary-looking workflow function: reserve inventory, charge payment, create shipment, in that order, with branching and timeouts written as normal control flow. Each step that touches the outside world is an activity, a separate function that calls the Inventory, Payment or Shipping service and can be retried independently.

The workflow does not keep its progress in memory. The Temporal service records every scheduled activity, result and timer as an event history. If a worker crashes after payment succeeds, another worker picks up the execution, replays the history through the same workflow code, and skips completed activities by reusing their recorded results. It then continues at shipment, so payment is not charged a second time.

Compensations follow the pattern shown in the example below. After each activity succeeds, the workflow appends its undo action to a list, such as "release reservation" or "refund payment." If a later step fails permanently, the workflow runs that list in reverse. Because the list is rebuilt during replay, it survives crashes without any extra storage.

Replay only works if workflow code is deterministic:

  • No direct network or database calls in workflow code.
  • No random numbers.
  • No reads of the system clock.

The SDK provides deterministic substitutes for time and randomness, and anything else with side effects belongs in an activity. For the same reason, changing the step order while executions are still running can break replay. Use the SDK's versioning mechanisms so that old executions keep following the old path.

from datetime import timedelta
from temporalio import workflow

STEP_TIMEOUT = timedelta(seconds=30)

@workflow.defn
class OrderSaga:
    @workflow.run
    async def run(self, order_id: str) -> str:
        # Undo actions, appended only after a step succeeds. On replay this
        # list is rebuilt from event history, so it needs no separate storage.
        compensations: list[str] = []
        try:
            # Each activity call is recorded in history. After a crash, a
            # replaying worker reuses recorded results and does not re-call.
            await workflow.execute_activity(
                "reserve_inventory", order_id, start_to_close_timeout=STEP_TIMEOUT)
            compensations.append("release_inventory")

            await workflow.execute_activity(
                "charge_payment", order_id, start_to_close_timeout=STEP_TIMEOUT)
            compensations.append("refund_payment")

            await workflow.execute_activity(
                "create_shipment", order_id, start_to_close_timeout=STEP_TIMEOUT)
            return "completed"
        except Exception:
            # A step failed after its retries ran out: undo completed steps
            # newest-first. Compensation activities must be idempotent.
            for undo in reversed(compensations):
                await workflow.execute_activity(
                    undo, order_id, start_to_close_timeout=STEP_TIMEOUT)
            raise

# Determinism: no HTTP/DB calls, random() or datetime.now() in this class.
# Use workflow.now() / workflow.random() here, and put side effects in activities.
# Changing the step order? Guard the new path with workflow.patched("patch-id")
# so running executions keep replaying the old sequence.

A Temporal Python workflow runs the order saga as three activities, records an undo action after each success, runs the undo actions in reverse on failure, and notes the determinism and versioning rules that keep replay safe.

How do you implement idempotency, retries and failed compensations in either model?

Both models deliver messages or activity calls at least once, so every downstream step will eventually run twice. The orchestrator cannot prevent duplicates. The downstream service has to absorb them.

Idempotency keys

Derive the key from business identity, such as order-8812:reserve-inventory, not from a random UUID generated per attempt. A per-attempt UUID changes on every retry, so it cannot detect duplicates. The inventory service enforces the key in the same database transaction as the side effect:

  1. Insert the key into a processed_requests table that has a unique constraint.
  2. If the insert succeeds, reserve stock and store the result alongside the key.
  3. If the insert hits the constraint, return the stored result without touching stock.

Compensations need their own keys (order-8812:release-inventory). A release should also succeed as a no-op if the reservation never happened, because a timeout can trigger compensation for a step that never committed.

Timeouts and retries

Set a timeout on every remote call. A hung call is worse than a failed one because it holds the saga in an unknown state. Retry transient failures with exponential backoff and jitter. Classify business rejections, such as "card declined" or "SKU discontinued", as non-retryable. Retrying them only wastes time, so they should go straight to compensation.

When compensation fails

A compensation cannot be abandoned, or the system is left inconsistent. Retry it indefinitely with capped backoff. After a defined threshold, raise an alert and move the order to a dead-letter or manual-review queue. Record enough context there, including the order ID, the failed step, and the last error, for an operator to finish the compensation by hand.

import json

class NonRetryableError(Exception):
    """Business rejection: the caller should compensate, not retry."""

def reserve_inventory(conn, order_id, sku, qty):
    # Key comes from business identity, so every retry of this step reuses it.
    key = f"{order_id}:reserve-inventory"
    with conn:  # psycopg2: commit on success, roll back on exception
        with conn.cursor() as cur:
            # processed_requests.idem_key has a UNIQUE constraint.
            cur.execute(
                "INSERT INTO processed_requests (idem_key) VALUES (%s) "
                "ON CONFLICT (idem_key) DO NOTHING RETURNING idem_key",
                (key,),
            )
            if cur.fetchone() is None:
                # Duplicate delivery: return the stored result, leave stock alone.
                cur.execute(
                    "SELECT result FROM processed_requests WHERE idem_key = %s",
                    (key,),
                )
                return json.loads(cur.fetchone()[0])

            cur.execute(
                "UPDATE inventory SET available = available - %s "
                "WHERE sku = %s AND available >= %s",
                (qty, sku, qty),
            )
            if cur.rowcount == 0:
                # Raising rolls back the key insert too; retries would fail the same way.
                raise NonRetryableError(f"insufficient stock for {sku}")

            result = {"order_id": order_id, "sku": sku, "reserved": qty}
            # Same transaction as the side effect: key, stock and result commit together.
            cur.execute(
                "UPDATE processed_requests SET result = %s WHERE idem_key = %s",
                (json.dumps(result), key),
            )
            return result

A PostgreSQL-backed inventory handler that enforces an order-ID-plus-step idempotency key in the same transaction as the stock reservation, returns the stored result on duplicate delivery, and raises a non-retryable error for business rejections.

Temporal vs choreography: how do observability, testing and development cost compare?

That manual-review queue is also where the two models start to differ in day-to-day cost.

Concern Temporal orchestration Choreography
Debugging one order Open its workflow history: every activity, retry, timer and error in order Join logs and traces across services by correlation ID
"Which orders are stuck in payment?" List workflows filtered by a custom search attribute you set at each step Build it yourself, usually a projection table fed by every service's events
Testing Test environment with time skipping, so multi-day timeouts run in seconds Integration and contract tests across services and the broker
Operating cost Run or pay for a stateful cluster. An outage pauses in-flight orders No central component, but more brokers, consumers and dead-letter queues to watch
Change cost Determinism rules and workflow versioning to learn Changing the flow means coordinated releases across several teams

Tracing in choreography works only if every service propagates the correlation ID, including through retries and dead-letter replays. One consumer that drops it creates a gap you only discover during an incident.

Temporal's fast tests carry a specific risk. Workflow tests usually mock activities, and those mocks can drift from the real payment or inventory service. A green suite then proves that the orchestration logic is correct against behaviour that no longer exists. You still need contract tests at the activity boundary.

Where each saves time:

  • Temporal pays off when flows are long, branch heavily, or change often. Debugging and timeout testing get much cheaper.
  • Choreography costs less when the flow is short and stable and the teams already run event infrastructure well.

When should you use Temporal orchestration, and when should you avoid it?

Pick the model by how hard the compensations are and who owns the flow, not by which tool is newer.

Use Temporal orchestration when the order flow branches (split shipments, partial refunds, backorders), waits on humans or external systems (a manual fraud review that can take hours, a payment provider callback delivered as a signal), or runs for days. Durable timers and workflow state replace the polling jobs and status tables you would otherwise build. It also pays off when there are many compensation paths, because the order of undo steps is ordinary code that you can test.

Avoid it when the flow is short and linear, such as reserve, charge and confirm with one undo each. Avoid it when teams deploy independently and would resist a shared definition of the flow. Avoid it when nobody can operate another stateful platform, or when you are unwilling to depend on one. Watch for the ownership trap: if one team owns the workflow code and it calls every service's API, that team becomes a bottleneck, and every API change needs its review.

Factor Temporal orchestration Choreography
Flow shape Branching, long-running, waits Short, linear, stable
Compensation paths Many, order-sensitive Few, independent
Ownership One team owns the end-to-end flow Each team owns its reactions
Platform capacity Can run or depend on a workflow cluster Existing broker only
Best-suited teams Platform-backed product teams Autonomous domain teams

What are the alternatives and hybrid designs, and how do you migrate between them?

Hybrids: orchestrate inside, publish outside

The most common hybrid gives one bounded context, usually checkout or order management, its own Temporal workflow. The workflow coordinates the steps that context owns, such as payment authorization and inventory reservation. At milestones like "order confirmed" or "order cancelled," it publishes events. Shipping, loyalty and analytics react to those events without knowing a workflow exists. This keeps strict compensation logic where money and stock are at risk. It also avoids making every downstream team adopt a workflow engine.

The main alternative to Temporal is not "no saga." It is choreography with a transactional outbox. Each service writes its state change and its outgoing event in the same local transaction, and a relay publishes the event later. This removes the dual-write gap between database and broker, and it uses infrastructure you probably already run.

Migrating from choreography to orchestration

Orders that are mid-flight under the old model must either drain or be bridged:

  • Draining means new orders start in workflows while old orders finish through the existing event handlers. It is simple, but it only works if old orders complete in hours rather than weeks.
  • Bridging means a workflow adopts an in-flight order by reading its current state and resuming from the matching step.

While both models run, you operate two failure surfaces. Compensations can fire twice, and alerts come from two systems.

A practical sequence for the migration:

  1. Make every handler idempotent first.
  2. Route a small share of new orders through the workflow.
  3. Turn off the old handlers only after the drain completes.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to saga orchestration temporal, this is where it helps:

Saves time

  • Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
  • One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
  • The dependency graph answers "what does this touch?" in seconds instead of a manual audit.

Saves cost

  • Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
  • Less duplicated configuration to maintain: one shared definition inherited by many services.
  • You can size a change's blast radius before merging, avoiding surprise cross-team firefighting.
  • OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.