Saga Orchestration vs Choreography: Handling Failed Steps

What is a saga, and why do microservices need one instead of distributed transactions?

A saga is a sequence of local transactions, each committed independently by one service against its own database. Every step that changes state is paired with a compensating transaction that semantically undoes it. Compensations do not roll anything back. A refund is a new transaction, not an erased charge. If step n fails, the saga runs the compensations for steps n−1 down to 1.

The running example is an order flow with three steps:

1. Inventory reserves the items. The compensation releases the reservation.
2. Payment charges the card. The compensation issues a refund.
3. Shipping schedules a pickup. The compensation cancels it.

Why not use two-phase commit across these services? 2PC requires every participant to hold locks while it waits for a coordinator's decision. A slow or crashed coordinator therefore blocks unrelated work. 2PC also needs every datastore and broker to support a common transaction protocol, which many do not. Finally, it couples the availability of all services together. With 2PC, the order flow can only commit when all three services are up.

The cost of avoiding 2PC is isolation. Each local commit is visible as soon as it happens. That allows anomalies:

• Acting on data that is later compensated. A reporting job could count the charge before it is refunded.
• Lost updates. One saga could overwrite a reservation that another saga then compensates.

The usual countermeasure is a semantic lock. Records carry a pending state such as RESERVATION_PENDING or PAYMENT_AUTHORIZED, and readers treat those states with suspicion. This works, but every consumer must now understand those states and the transitions between them.

How does a saga work at runtime?

Each saga instance follows the same mechanics, whichever coordination model you pick:

1. A saga instance starts with an identifier. Usually this is the business key, such as the order ID. Every message, log line, and database row the saga touches carries it. That is how retries are deduplicated and how operators find the saga later.
2. Each step is one local transaction plus one message. A participant changes its own database and records, in the same transaction, the fact that it did so. The outbox pattern described below is the usual way to do this. A message then tells the next participant, or the orchestrator, that the step succeeded or failed.
3. Progress is tracked somewhere. In orchestration, the orchestrator persists which steps have completed. In choreography, progress is implicit in which events have been published and consumed.
4. On failure, compensations run for completed steps only, in reverse order. A step that never committed has nothing to undo. A step that did commit gets its compensating transaction, which is itself a local transaction and is also retried until it succeeds.
5. The saga ends in one of three states. It completes, it is fully compensated, or it is parked for manual intervention because a step or compensation exhausted its retries.

The guarantee you get is eventual consistency. Each service's data is always locally consistent, and the system as a whole converges to either "all steps applied" or "all applied steps compensated." There is no moment where all three services commit atomically together.

Who should use sagas?

Sagas are aimed at teams whose business operations span several services that each own a separate database, and who cannot or do not want to use a distributed transaction protocol across them. Typical signs that you are in this group:

• A single user action, such as placing an order or opening an account, must update data owned by two or more services.
• At least one step calls an external system, such as a payment processor or carrier API, that cannot take part in your database transaction anyway.
• Partial failure has a business meaning. A charged-but-unshipped order is a real state that someone must resolve, and the business can define what "undo" means for each step.
• Temporary inconsistency between services is acceptable for seconds or minutes, as long as the system converges.

Within that group, the choice between models depends on who owns the process. A team that owns an end-to-end business flow, such as checkout, and is accountable for its failures usually benefits from orchestration. Organizations where independent teams react to shared business events, with no single owner of the whole flow, tend to use choreography for short flows.

How does saga orchestration vs choreography work for the same order flow?

Take one order flow: reserve inventory, charge payment, schedule shipment. Both models run the same three local transactions. They differ in where the sequence is written down and who knows about whom.

Orchestration: one component owns the sequence

An order orchestrator sends ReserveInventory to the inventory service and waits for a reply. On success it sends ChargePayment, then ScheduleShipment. The flow definition lives in one place, the orchestrator's code or workflow definition, and so does the current state of each saga instance.

Participants stay simple. Inventory does not know that payment exists. It accepts a command and returns a result. The orchestrator, however, must know every participant's command contract, reply format, and timeout behavior. Adding a fraud check means changing and redeploying the orchestrator, and any breaking change to a participant's API breaks the orchestrator.

Choreography: the sequence emerges from subscriptions

The order service publishes OrderPlaced. Inventory consumes it, reserves stock, and publishes InventoryReserved. Payment consumes that and publishes PaymentCharged. Shipping consumes that and schedules delivery. No service issues commands. Each one reacts to events it has chosen to subscribe to.

The flow definition is not in any single file. It exists only as the set of subscriptions across all services. Each service knows only which upstream events it consumes and which event it emits. A team can add a consumer without touching anyone else's deployment.

The trade-off is direct. Orchestration couples one component to every participant's API but keeps the flow readable in one place. Choreography keeps services decoupled at deploy time, but answering "what happens after payment?" means reading several codebases.

Copy// Minimal transport interfaces. No specific messaging library is assumed.
type Order = { orderId: string; sku: string; qty: number; amountCents: number };
interface CommandClient { send(target: string, command: string, body: object): Promise<{ ok: boolean }>; }
interface EventBus {
  publish(event: string, body: object): Promise<void>;
  subscribe(event: string, handler: (body: any) => Promise<void>): void;
}

// ORCHESTRATION: the whole sequence lives in this one function.
// The orchestrator must know every participant's name, command, and payload shape.
async function runOrderSaga(client: CommandClient, o: Order) {
  const steps = [
    { target: "inventory", command: "ReserveInventory", body: { orderId: o.orderId, sku: o.sku, qty: o.qty } },
    { target: "payment",   command: "ChargePayment",    body: { orderId: o.orderId, amountCents: o.amountCents } },
    { target: "shipping",  command: "ScheduleShipment", body: { orderId: o.orderId } },
  ];
  for (const step of steps) {
    const reply = await client.send(step.target, step.command, step.body); // wait for each reply
    if (!reply.ok) return { status: "failed", failedAt: step.command }; // compensation comes later
  }
  return { status: "completed" };
}

// CHOREOGRAPHY: no function holds the sequence. Each service declares only
// the event it consumes and the event it emits. The flow is the sum of these subscriptions.
function inventoryService(bus: EventBus) {
  bus.subscribe("OrderPlaced", async (e) => {
    // reserve stock in the local DB, then announce it
    await bus.publish("InventoryReserved", { orderId: e.orderId, amountCents: e.amountCents });
  });
}
function paymentService(bus: EventBus) {
  bus.subscribe("InventoryReserved", async (e) => {
    // charge the card locally. Payment never references inventory directly.
    await bus.publish("PaymentCharged", { orderId: e.orderId });
  });
}
function shippingService(bus: EventBus) {
  bus.subscribe("PaymentCharged", async (e) => { /* schedule delivery for e.orderId */ });
}

The same reserve-charge-ship flow in both models: orchestration keeps the step list in one function that knows every participant's command contract, while choreography spreads the sequence across per-service subscriptions.

What happens when step three fails in each model?

Assume the order flow is: reserve inventory, charge payment, schedule shipment. The carrier API rejects the shipment request.

Orchestration

The orchestrator detects the failure because it made the call and received the error or timeout. It also decides what happens next: retry, or give up and compensate. Compensation order comes from its own record of completed steps, unwound in reverse: refund payment, then release inventory. All of this logic lives in one place, the orchestrator's workflow definition. A reviewer can read the rollback path in the same file as the forward path.

Choreography

The shipping service detects the failure and publishes ShipmentFailed. No component decides to compensate. Payment subscribes and refunds, then emits PaymentRefunded. Inventory might listen for that event or for ShipmentFailed directly. The order of compensation emerges from these subscriptions, which are spread across the consumer code of several repositories.

This makes the path fragile in ways nothing reports. Suppose someone renames orderId to order_id in ShipmentFailed. If payment's consumer drops messages it cannot parse, refunds stop, and no error reaches the shipping team. Now suppose inventory, on ReservationReleased, emits an event that payment treats as a reason to refund again. You get a cycle that only shows up under failure, which is the path that tests exercise least.

Copytype Step = { name: string; run: () => Promise<void>; compensate: () => Promise<void> };

// Orchestration: one function detects the failure, decides to compensate,
// and fixes the order from its own record of completed steps.
async function runSaga(steps: Step[]): Promise<void> {
  const done: Step[] = [];
  for (const step of steps) {
    try {
      await step.run(); // e.g. reserveInventory, chargePayment, scheduleShipment
      done.push(step);
    } catch (err) {
      // scheduleShipment threw: unwind in reverse (refund payment, then release inventory).
      for (const s of [...done].reverse()) await s.compensate();
      throw err;
    }
  }
}

// Choreography: payment's consumer for ShipmentFailed, living in another repo.
// The compensation path depends on this handler understanding the event.
type ShipmentFailed = { orderId: string; reason: string };

function onShipmentFailed(raw: string, refund: (orderId: string) => void): void {
  let evt: Partial<ShipmentFailed>;
  try {
    evt = JSON.parse(raw);
  } catch {
    return; // unparseable message is dropped
  }
  // If shipping renames orderId to order_id, this guard skips every message:
  // no exception, no dead-letter entry, no refund, and shipping is never told.
  if (typeof evt.orderId !== "string") return;
  refund(evt.orderId);
}

The orchestrator unwinds completed steps in reverse from one place, while a choreographed payment consumer silently stops refunding when a field in ShipmentFailed is renamed.

How do you design compensating transactions that actually work?

A compensation is a new business operation that moves the system forward to an acceptable state. It does not undo what happened. If the payment service captured a charge, the compensation issues a refund, and the ledger keeps both entries. If inventory reserved stock, the compensation releases the reservation. It does not delete the row. Deleting history destroys the audit trail and breaks reconciliation with external systems such as card processors, which have no concept of your rollback.

Pivot steps and steps that cannot be undone

Some actions have no compensation. You cannot unsend an email, recall a shipment that has left the warehouse, or reverse a webhook a partner has already processed. Place these steps last, after every step that can still fail. Alternatively, treat one of them as the saga's pivot: the point after which the saga only retries forward and never compensates. Everything before the pivot must be compensable. Everything after it must be retryable until it succeeds.

Idempotency everywhere

Brokers deliver messages at least once, and orchestrators retry on timeouts. Every step and every compensation will eventually run twice. Derive idempotency keys from the saga ID plus the step name, such as order-8812:charge or order-8812:refund. Store each key alongside the result so that a repeat call returns the original outcome.

The outbox underneath both models

A service that commits to its database and then publishes to the broker can crash between the two writes. The transactional outbox pattern avoids this. The service writes the event to an outbox table in the same local transaction as the state change, and a relay publishes it afterward.

Skipping the outbox or idempotency causes the same failures. Retried charges become duplicate charges. Lost compensation events leave reservations orphaned that no process ever releases.

Copy-- PostgreSQL. The refund is a new ledger row; the charge row is never deleted.
CREATE TABLE ledger_entries (
  id              bigserial PRIMARY KEY,
  idempotency_key text NOT NULL UNIQUE,  -- saga ID + step name
  saga_id         text NOT NULL,
  entry_type      text NOT NULL,         -- 'charge' or 'refund'
  amount_cents    bigint NOT NULL,
  created_at      timestamptz NOT NULL DEFAULT now()
);

CREATE TABLE outbox (
  id           bigserial PRIMARY KEY,
  aggregate_id text NOT NULL,
  event_type   text NOT NULL,
  payload      jsonb NOT NULL,
  published_at timestamptz             -- set by the relay after the broker acks
);

-- Compensation for the 'charge' step. The state change and the event commit
-- together, so a crash cannot leave one without the other.
BEGIN;
WITH refund AS (
  INSERT INTO ledger_entries (idempotency_key, saga_id, entry_type, amount_cents)
  VALUES ('order-8812:refund', 'order-8812', 'refund', -4999)
  ON CONFLICT (idempotency_key) DO NOTHING  -- redelivery: no second refund
  RETURNING id, saga_id, amount_cents
)
INSERT INTO outbox (aggregate_id, event_type, payload)
SELECT saga_id, 'PaymentRefunded',
       jsonb_build_object('ledger_entry_id', id, 'amount_cents', amount_cents)
FROM refund;  -- zero rows on a duplicate, so no duplicate event either

-- Return the original outcome whether this was the first call or a repeat.
SELECT id, amount_cents, created_at
FROM ledger_entries WHERE idempotency_key = 'order-8812:refund';
COMMIT;

-- Relay loop: claim unpublished rows; concurrent relays skip locked rows.
-- SELECT id, event_type, payload FROM outbox WHERE published_at IS NULL
--   ORDER BY id LIMIT 100 FOR UPDATE SKIP LOCKED;

A PostgreSQL refund compensation that appends a ledger entry under an idempotency key derived from the saga ID and writes its outbox event in the same transaction, so redelivered messages produce neither a second refund nor a second event.

How do Temporal workflows implement saga orchestration?

A durable workflow engine such as Temporal removes most of that plumbing by making the orchestrator's own state durable. Every step the workflow takes is recorded in an event history kept by the Temporal service: activity scheduled, activity completed, timer fired, signal received. Workers run the workflow code. If a worker crashes, another worker replays the history from the start. Steps that already finished are not re-executed; their recorded results are fed back to the code until it reaches the point where it stopped.

Because state survives crashes, saga logic becomes ordinary code. Retry policies attach to activities. A timer that waits two days is a sleep call that holds no thread and outlives restarts. Compensation follows a simple pattern. After each step succeeds, push its undo action onto a list. If a later step fails, run that list in reverse. The engine records each compensation call too, so a crash in the middle of rollback resumes where it left off and does not start over.

This model has costs:

• Workflow code must be deterministic. Replay only works if the code makes the same decisions every time. Wall-clock reads, random numbers, direct network calls and unordered map iteration belong in activities or the SDK's deterministic equivalents.
• Changing logic needs explicit versioning. If you edit the steps of a workflow that is still running, its next replay no longer matches its recorded history. You must branch on a version marker or let old executions drain first.
• Long-running workflows complicate deploys. Old code paths must stay in place until every execution that uses them has finished.
• The engine is critical infrastructure. If the Temporal service is down, running sagas pause and no new sagas can start.

Copyimport { proxyActivities, sleep } from '@temporalio/workflow';
import type * as activities from './activities';

// Activities do the side effects (DB writes, payment API calls). The engine retries them per this policy.
const { reserveInventory, releaseInventory, chargePayment, refundPayment, createShipment } =
  proxyActivities<typeof activities>({
    startToCloseTimeout: '30 seconds',
    retry: { initialInterval: '1 second', backoffCoefficient: 2, maximumAttempts: 5 },
  });

// Workflow code must be deterministic: no Date.now(), Math.random() or direct I/O here.
export async function orderSaga(orderId: string, amountCents: number): Promise<void> {
  // Undo actions, registered only after the matching step succeeds.
  const compensations: Array<() => Promise<void>> = [];

  try {
    const reservationId = await reserveInventory(orderId);
    compensations.push(() => releaseInventory(reservationId));

    const paymentId = await chargePayment(orderId, amountCents);
    compensations.push(() => refundPayment(paymentId));

    // Durable timer: holds no thread, survives worker restarts, fires after replay.
    await sleep('2 days');

    await createShipment(orderId);
  } catch (err) {
    // Roll back in reverse order. Each call is recorded in history,
    // so a crash mid-rollback resumes here instead of starting over.
    for (const compensate of compensations.reverse()) {
      await compensate();
    }
    throw err;
  }
}

A Temporal TypeScript workflow that registers a compensation after each successful step, uses a durable two-day timer, and runs the compensations in reverse order when a later step fails.

How do you find and recover sagas that are stuck or half-compensated?

In orchestration, the orchestrator holds the saga's state, so "what state is order X in?" is a single lookup. You can see the current step, the last error, and the retry count in one place. Choreography has no such record. You have to reconstruct the answer from evidence. Every event needs a correlation ID (usually the order ID) so you can stitch the flow together across services. Distributed tracing shows the call path, but trace sampling and retention limits mean you can't count on a trace existing for any given order. The more durable option is a projection: a consumer that subscribes to every saga event and maintains a per-order status table. It works, but it is one more service to build, and it is only as accurate as the events it receives.

Both models need explicit handling for sagas that neither complete nor compensate:

• Step timeouts. Without a timeout, a lost reply leaves the saga waiting forever.
• Retries with exponential backoff and a cap. Transient failures recover on their own, and the backoff keeps retries from overloading a struggling dependency.
• Compensations that fail. Retry them too. They must be idempotent, so retrying cannot double-refund a customer.
• A dead-letter or manual-intervention state. When retries run out, park the saga there and alert. Back it with a runbook that says how to inspect the saga, fix the underlying data, and then resume, re-run compensation, or close it out manually.

Centralized visibility shortens incidents because investigation starts from a known state. Engineers don't have to correlate logs across five services before they can reconcile anything.

How can sagas save engineering time?

A saga does not make a distributed flow free to build. The time savings come from replacing ad hoc failure handling with one repeatable structure:

• Failure handling is designed once, not discovered in production. Without a saga, partial failures tend to be handled by one-off scripts and manual data fixes written during incidents. Writing a compensation for each step up front turns "what do we do when shipping fails after payment?" into a code review question.
• Retries stop being dangerous. With idempotency keys and an outbox in place, engineers can retry any step or compensation without first checking whether it already ran. That removes a whole class of investigation from incident response.
• Orchestration shortens debugging. Inspecting one workflow history replaces correlating logs across services. A durable workflow engine also removes the need to hand-write state persistence, timers, and retry loops for each new flow.
• Choreography shortens delivery for small flows. Teams add a consumer for an existing event and ship without coordinating a release with other teams.
• New flows reuse the same pieces. The outbox table, relay, idempotency scheme, and stuck-saga runbook are built once and shared across every saga.

To size the benefit for your own system, compare the cost of building compensations with the cost of handling partial failures manually. As an illustrative estimate, with assumed inputs only: if a flow is assumed to produce 10 partial failures per week, and each one is assumed to take an engineer 30 minutes to investigate and repair by hand, that is about 5 engineer-hours per week. In that scenario, automated compensation that resolves most of those cases pays back its build cost after a number of weeks equal to its build hours divided by 5. Substitute your own failure rate and repair time. If partial failures are rare and cheap to fix, the saga may not pay for itself.

When should you choose orchestration, choreography, or neither?

Choreography fits when a flow has three or four steps, each service owns its reaction to an event, and no single team owns the business process end to end. Each team ships changes without coordinating a release, and nothing new needs to run. The cost shows up later. Once the flow grows past a handful of participants, or compensation must run in strict reverse order, the logic is spread across event handlers that nobody reads together.

Orchestration fits when the flow has many steps, conditional branches, ordered compensation, or operators who need to find a stuck saga and act on it. The orchestrator becomes a dependency every participant integrates with, and you have to run a workflow engine or persistent state store. In return, you get one place to read, test, and debug the flow.

Sometimes neither model is the right call. If every step writes to the same database, use a local ACID transaction. If two services always change together and fail together, the boundary is probably wrong. Merging them is cheaper than coordinating them. If the business can accept a nightly reconciliation job that detects and repairs mismatches, that job is simpler than a saga.

When should sagas be avoided?

Avoid sagas, in either model, in these situations:

• All the data lives in one database. A local transaction gives you atomicity and isolation for free. A saga gives you neither and adds compensation code.
• Readers cannot tolerate intermediate states. Sagas expose each local commit immediately. If a downstream consumer must never see a charge without a matching reservation, and you cannot model pending states with semantic locks, a saga is the wrong tool.
• Steps cannot be compensated and cannot be moved after a pivot. If an irreversible action must happen early in the flow, before steps that can still fail, no compensation design will restore an acceptable state.
• The business cannot define what "undo" means. Compensations are business operations. If product owners cannot say what a refund, release, or cancellation should look like, the engineering team will end up inventing policy in code.
• The team lacks the supporting infrastructure. Without idempotent handlers, an outbox or equivalent, correlation IDs, and a way to find stuck sagas, a saga produces duplicate charges and orphaned reservations that are harder to diagnose than the problem it was meant to solve.
• The service boundary is the real problem. If services always change and fail together, merge them rather than coordinating them.

Avoid choreography specifically when the flow has more than a few steps, needs strict compensation order, or must be inspectable by operators. Avoid orchestration specifically when you cannot operate a workflow engine or durable state store reliably, because its outage pauses every saga that depends on it.

How OneEnv can help

Disclosure: OneEnv is the product behind this blog.

OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.

Applied to saga orchestration vs choreography, this is where it helps:

Saves time

• Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
• One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
• Changing a route's URL or method automatically notifies the services that consume it — no hunting through repos for callers.

Saves cost

• Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
• Less duplicated configuration to maintain: one shared definition inherited by many services.
• Consumers learn about a breaking change at review time rather than from production errors.
• OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.

These are qualitative benefits, not measured results — how much you save depends on how many services share the value.

Want to try it? Start at oneenv.io.