Saga Pattern Orchestration vs Choreography: Choosing Where State Lives

What a saga is and why distributed transactions push you toward one

A saga is a sequence of local transactions, each of which commits independently inside a single service's database. If step n fails, the saga runs compensating actions for steps n−1 back to 1. These actions semantically undo the earlier work. They do not roll it back physically, because those commits are already durable and possibly visible. A refund reverses a charge. A "release" reverses an inventory reservation. The original rows stay in history.

Compare this with an ACID distributed transaction coordinated by two-phase commit (2PC). Under 2PC, every participant votes to prepare, holds its locks, and waits for the coordinator's commit or abort decision. You get atomicity and isolation across services. The cost is that a slow or crashed participant, or a lost coordinator, leaves everyone else blocked with locks held. 2PC also requires every datastore to support a shared transaction protocol, which many brokers, NoSQL stores, and third-party APIs do not.

Sagas trade those guarantees away. You get eventual consistency and no isolation: other requests can observe the intermediate states. In return, each service owns its own database, nothing blocks waiting for a global decision, and one unavailable dependency degrades a single workflow instead of freezing several services.

This trade-off matters to backend and platform engineers building multi-service workflows such as orders, payments, and infrastructure provisioning. The rest of this article argues one thesis: choosing between orchestration and choreography is a decision about where workflow state and failure-handling logic live.

How choreography and orchestration run the same order flow

Take one checkout: create order, reserve inventory, authorize payment, schedule shipment. Both models run the same four local transactions. What differs is which component knows the sequence and decides what to undo when a step fails.

Choreography

No component holds the workflow. The workflow is the combined set of subscriptions across services. When the card is declined, Payment publishes PaymentFailed. Inventory listens for it and releases the reservation. Order listens for it and marks the order rejected. Compensation flows backward because each service knows which downstream failures should undo its own work.

That knowledge has a cost. Inventory now depends on Payment's event contract. To reconstruct the full flow, you have to read every consumer's subscription list.

Orchestration

A coordinator holds the state machine. It can be a dedicated service or a durable workflow engine. It sends commands, waits for replies, and persists its state after every transition:

• PENDING: send ReserveStock. On StockReserved, go to RESERVED.
• RESERVED: send AuthorizePayment. On PaymentAuthorized, go to PAID. On PaymentDeclined or a timeout, go to COMPENSATING.
• PAID: send ScheduleShipment. On ShipmentScheduled, go to COMPLETED.
• COMPENSATING: send ReleaseStock, then RejectOrder, then go to FAILED.

Because the state is persisted, a crashed coordinator resumes from its last recorded step. The participants never learn about each other.

Copy# Orchestrator state machine for one checkout saga.
# State is persisted after every transition so a restarted coordinator resumes where it left off.

TRANSITIONS = {
    # (current_state, reply): (next_state, commands_to_send)
    ("PENDING", "StockReserved"): ("RESERVED", ["AuthorizePayment"]),
    ("RESERVED", "PaymentAuthorized"): ("PAID", ["ScheduleShipment"]),
    ("RESERVED", "PaymentDeclined"): ("COMPENSATING", ["ReleaseStock"]),
    ("RESERVED", "Timeout"): ("COMPENSATING", ["ReleaseStock"]),
    ("PAID", "ShipmentScheduled"): ("COMPLETED", []),
    # Compensation runs in reverse order: undo inventory, then reject the order.
    ("COMPENSATING", "StockReleased"): ("COMPENSATING", ["RejectOrder"]),
    ("COMPENSATING", "OrderRejected"): ("REJECTED", []),
}


def start(saga_id, store, bus):
    store.save(saga_id, "PENDING")
    bus.send("ReserveStock", saga_id)


def on_reply(saga_id, reply, store, bus):
    state = store.load(saga_id)
    key = (state, reply)
    if key not in TRANSITIONS:
        # Duplicate or late reply (at-least-once delivery): ignore, don't re-run side effects.
        return state
    next_state, commands = TRANSITIONS[key]
    # Persist before sending, so a crash between the two re-sends rather than loses the command.
    # Participants must therefore handle commands idempotently, keyed by saga_id.
    store.save(saga_id, next_state)
    for command in commands:
        bus.send(command, saga_id)
    return next_state

A table-driven orchestrator that holds the saga state, issues commands, and drives compensation (ReleaseStock, then RejectOrder) when payment is declined or times out.

Designing compensating actions that actually work

A compensation does not undo history. Once ChargeCard has succeeded, the charge exists, and other systems may have seen it. RefundPayment is a new forward operation with its own ledger entry, its own failure modes, and sometimes its own fees. The same applies to ReleaseStock after a reservation and CancelShipment after a label is printed. Treat each compensation as a first-class command that the participant owns and documents.

Because the coordinator or event bus will redeliver after crashes and timeouts, every compensation must be idempotent and retryable. Key it on saga ID plus step name, record the outcome, and return the stored result on repeat calls. The handler example below shows this pattern. A refund that runs twice is a worse incident than a refund that runs late.

Order steps around the pivot

Some actions cannot be compensated, such as a sent email or a charge that has already settled with the card network. Classify each step:

• Compensatable steps run first: reserve stock, authorize payment.
• The pivot is the point of no return: capture payment.
• Retryable steps come after the pivot and must eventually succeed: ship, send confirmation.

With this ordering, a failure before the pivot can be fully compensated, and a failure after it only needs a retry.

When compensation fails

Retry with exponential backoff and a cap. When retries are exhausted, raise an alert and move the saga to a manual intervention queue that holds full context. Accept that some effects can only be mitigated. You cannot unsend an email, but you can send a correction.

Copyimport json
import sqlite3

# One row per (saga, step). The primary key makes a redelivered command a lookup, not a second refund.
# CREATE TABLE compensations (saga_id TEXT, step TEXT, status TEXT, result TEXT,
#                             PRIMARY KEY (saga_id, step));

def refund_payment(db: sqlite3.Connection, payments, saga_id: str,
                   payment_id: str, amount_cents: int) -> dict:
    step = "RefundPayment"
    idempotency_key = f"{saga_id}:{step}"

    # Claim the step. A repeat call finds the existing row and reads its state.
    with db:
        db.execute("INSERT OR IGNORE INTO compensations (saga_id, step, status) "
                   "VALUES (?, ?, 'pending')", (saga_id, step))
        status, result = db.execute(
            "SELECT status, result FROM compensations WHERE saga_id = ? AND step = ?",
            (saga_id, step)).fetchone()

    if status == "done":
        return json.loads(result)  # Already compensated: return the stored outcome.

    # A crash between this call and the UPDATE below leads to a retry.
    # Passing the same key lets the payment service deduplicate that retry.
    # Errors propagate so the coordinator can retry with backoff.
    refund = payments.refund(payment_id=payment_id, amount_cents=amount_cents,
                             idempotency_key=idempotency_key)

    outcome = {"refund_id": refund["id"], "amount_cents": amount_cents}
    with db:
        db.execute("UPDATE compensations SET status = 'done', result = ? "
                   "WHERE saga_id = ? AND step = ?",
                   (json.dumps(outcome), saga_id, step))
    return outcome

An idempotent RefundPayment handler keyed on saga ID and step name. It records each outcome and returns the stored result on redelivery, and it passes the same key downstream so a retry after a crash cannot refund twice.

How to implement sagas reliably: outbox, idempotency, and timeouts

Every saga step has the same weak point, whether a coordinator issues commands or services react to events: a service commits a database change and then has to tell someone about it. If it commits and crashes before publishing, the saga stalls with no record of why. If it publishes and then rolls back, downstream services act on a change that never happened. The transactional outbox closes this gap. The service writes the business row and an outbox row in the same local transaction. The outbox row holds a message ID, aggregate ID, event type, payload, creation time, and a nullable published_at. A separate relay reads unpublished rows, sends them to the broker, and marks them published.

The relay can crash between sending and marking, so the system delivers messages at least once. Consumers must therefore be idempotent. A processed_messages table, keyed on consumer name plus message ID, handles deduplication. In one local transaction, the consumer inserts the key, applies the change, and writes its own outbox rows. If the insert fails because the key already exists, the message is a duplicate and the consumer acknowledges it without doing anything else.

Timeouts cover steps that never reply. Give each step a deadline and decide ahead of time what expiry triggers: a retry, a compensation, or escalation to manual intervention.

Choreography adds risks of its own. Events can form cycles, the same event can be processed twice through different paths, and events can arrive out of order. Carry a saga ID and a sequence number on every event so consumers can reject stale ones. These failures often appear only under load or during a partial outage, so test for them under those conditions.

Copy-- PostgreSQL. Outbox rows are written in the same local transaction as the business change.
CREATE TABLE outbox (
  message_id   uuid PRIMARY KEY,
  aggregate_id text NOT NULL,
  event_type   text NOT NULL,
  payload      jsonb NOT NULL,
  created_at   timestamptz NOT NULL DEFAULT now(),
  published_at timestamptz  -- NULL until the relay confirms the broker accepted it
);
-- Lets the relay find unpublished rows without scanning the whole table.
CREATE INDEX outbox_unpublished ON outbox (created_at) WHERE published_at IS NULL;

-- Dedup table: one row per (consumer, message) that has already been applied.
CREATE TABLE processed_messages (
  consumer_name text NOT NULL,
  message_id    uuid NOT NULL,
  processed_at  timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (consumer_name, message_id)
);

-- Consumer pseudocode as one transaction: payment-service handling an OrderCreated delivery.
BEGIN;
INSERT INTO processed_messages (consumer_name, message_id)
VALUES ('payment-service', '7c9e6679-7425-40de-944b-e07fc1f90ae7')
ON CONFLICT DO NOTHING;
-- App checks rows affected: 0 means duplicate -> ROLLBACK, ack the broker, stop.
-- Before applying, the app compares the event's sagaId and seq to state it has stored, and drops stale events.
UPDATE payments SET status = 'AUTHORIZED' WHERE order_id = 'order-1042';
-- The next saga event leaves through this service's own outbox, carrying the saga ID and sequence number.
INSERT INTO outbox (message_id, aggregate_id, event_type, payload)
VALUES (gen_random_uuid(), 'order-1042', 'PaymentAuthorized',
        '{"sagaId": "saga-88", "seq": 2, "orderId": "order-1042"}');
COMMIT;  -- ack the broker only after this commit succeeds

PostgreSQL schemas for a transactional outbox and a processed-messages dedup table, plus a consumer transaction that claims the message ID, applies the change, and writes its own outbox event atomically.

Living without isolation: anomalies and countermeasures

A saga commits each local transaction as it goes, so other requests can see a half-finished workflow. That breaks the isolation in ACID, and it produces three concrete anomalies.

Dirty reads. The order service reserves the last unit of a SKU. The payment step then fails, and the reservation gets compensated. In the window between those two events, a second customer was told the item was sold out.

Lost updates. One saga sets an order's shipping address. A concurrent saga's compensation then restores the old value, silently discarding the change.

Fuzzy reads. A pricing step reads stock at one point and reads it again later in the same saga. A concurrent saga changed stock in between, so the two reads disagree.

Countermeasures

• Semantic locks. Write an explicit in-flight status. For example, UPDATE orders SET status = 'PAYMENT_PENDING' WHERE id = ? AND status = 'CREATED'. Other sagas then treat that order as claimed.
• Commutative updates. Use operations like stock = stock - 1 that give the same result in any order, rather than reading a value and writing it back.
• Re-reading values. Check that the state is still valid immediately before an irreversible step such as charging a card.
• Versioning. Use UPDATE inventory SET qty = ?, version = version + 1 WHERE sku = ? AND version = ?. If zero rows were updated, the version changed underneath you, so retry or abort.

Each countermeasure costs something. Every PENDING state needs a timeout and a recovery path. Conflict retries reduce throughput on hot rows. Intermediate states such as "payment pending" become visible to users and support teams, so the product has to design for them.

Copy-- 1. Semantic lock: claim the order for this saga.
-- The status guard makes the transition atomic. 0 rows updated means another saga
-- already claimed it or it was cancelled, so stop here.
UPDATE orders
   SET status = 'PAYMENT_PENDING',
       pending_since = CURRENT_TIMESTAMP
 WHERE id = 1042
   AND status = 'CREATED';

-- 2. Versioned (optimistic) update: read qty and version first, then write
-- only if nobody changed the row in between.
-- Assume the earlier read returned qty = 5, version = 17.
UPDATE inventory
   SET qty = 4,
       version = version + 1
 WHERE sku = 'SKU-RED-MUG'
   AND version = 17;
-- 0 rows updated: re-read and retry, or abort and run compensations.

-- 2b. Commutative alternative: no read-modify-write, so concurrent sagas
-- can apply it in any order. The guard prevents overselling.
UPDATE inventory
   SET qty = qty - 1
 WHERE sku = 'SKU-RED-MUG'
   AND qty >= 1;

-- 3. Release the semantic lock on success. Guard on the expected state
-- so that a late reply cannot overwrite a cancellation.
UPDATE orders
   SET status = 'PAID', pending_since = NULL
 WHERE id = 1042
   AND status = 'PAYMENT_PENDING';

-- 4. Recovery path: a sweeper finds sagas stuck in PENDING past a timeout
-- and hands them to compensation. Interval syntax is shown in PostgreSQL form.
SELECT id FROM orders
 WHERE status = 'PAYMENT_PENDING'
   AND pending_since < CURRENT_TIMESTAMP - INTERVAL '15 minutes';

A PENDING status used as a semantic lock, a version-checked conditional update, a commutative stock decrement, and the timeout query that every PENDING state needs. In each update, a row count of zero is the signal to retry or compensate.

Orchestration vs choreography: observability, coupling, and cost compared

Ask "what happened to order 123?" and the two models give very different answers. An orchestrator stores saga state in one place, so you query one record and see the current step, retry count, and last error. Choreography has no such record. The answer is spread across the logs of every participant, so you need correlation IDs on every event, distributed tracing that follows asynchronous hops, and often a separately built read model that consumes all saga events just to rebuild the status.

Change cost also falls in different places. In choreography, adding a fraud check between payment and shipping means changing event contracts owned by several teams. Dependencies hide in subscriptions: nobody declares that shipping depends on fraud, and a new subscriber can create a cycle that only shows up in production. In orchestration, the change goes into one service. That same convenience pulls logic toward the orchestrator until it becomes a god service. Its workflow definitions must also be versioned, because instances started under the old flow may still be running weeks later.

Engineering time follows the same pattern. Hand-built orchestrators tend to reimplement retries, durable timers, and state persistence, and they often get them wrong. A workflow engine handles those, but it adds another system to operate and a learning curve for the team. Choreography looks cheap at three steps. At ten, its scattered logic slows onboarding, audits, and incident response.

When to use each, when to mix them, and when to avoid a saga

The choice comes down to where you want the state of a long-running business process to live, and who has to reason about it during an incident.

Most systems end up hybrid. Each bounded context runs its own orchestrator for its internal steps and publishes domain events to other contexts. Branching logic stays visible inside a context, and contexts still don't call each other's internals.

Changing a saga while instances are in flight

Long-lived sagas will outlive at least one deploy. With orchestration, version the workflow definition. Running instances finish on the old version, and new instances start on the new one. With choreography, there is no definition to version, so the event schemas carry the burden. Changes must be backward compatible, which means adding optional fields and never repurposing existing ones. Consumers must also tolerate both old and new shapes until old instances drain.

When not to use a saga

• Redraw boundaries. If two services always change together, merging their data lets one local ACID transaction replace the saga.
• Two-phase commit. Use it where every participant supports it and you can accept blocking while a coordinator is unavailable.
• Eventual consistency with reconciliation. A scheduled job finds and repairs mismatches. This is often enough when brief divergence is harmless.

Choosing deliberately saves engineering time. Debugging an incident in a flow whose state is spread across five consumers costs far more than the design discussion that would have avoided it.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to saga pattern orchestration vs choreography, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Consumers learn about a breaking change at review time rather than from production errors.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.