What is the saga pattern, and why do microservices need compensating transactions?
A saga is a sequence of local transactions, each owned by one service and committed independently. If step three of five fails, there is no global rollback. Steps one and two already committed in their own databases. The saga reaches a consistent end state by one of two routes. It can retry its way forward to completion, or it can run compensating transactions that semantically undo the steps that already succeeded.
A compensation is not a rollback. It is a new business action with its own side effects:
- refunding a charge
- releasing a reservation
- cancelling a shipment
A customer whose order failed may see both a charge and a matching refund line on their statement. The history is corrected, not erased. Because compensations are real operations, some are slow, some can fail, and some cannot fully reverse the original effect, such as an email already sent.
The cost is isolation. Between steps, other transactions can observe intermediate state. Inventory may be reserved for an order that will never complete, or an account balance may briefly reflect a charge that is later refunded. Protecting against anomalies here is the application's job. Common countermeasures include:
- semantic locks, such as marking a record as owned by an in-flight saga
- explicit pending statuses that readers know to treat carefully
- updates designed to commute so ordering doesn't matter
Temporal provides none of these.
This tutorial is for backend and platform engineers coordinating writes across services that cannot share a database transaction. It assumes two-phase commit is unavailable or undesirable.
Orchestration vs choreography: what are the alternatives for running a saga?
Before choosing between orchestration and choreography, check whether you need a saga at all. If the writes can live in one database, a single local transaction gives you atomicity with no compensation logic. If one service owns the state change and others only react to it, a transactional outbox is often enough: write the row and an outbox event in the same transaction, then relay the event. Nobody ever has to undo anything.
When several services must each commit and possibly roll back, you have two shapes. In choreography, each service consumes an event, does its step, and emits the next event. Compensations are also triggered by events. No component holds the full sequence, so answering "where is order 812 stuck?" means correlating logs across consumers. In orchestration with Temporal, one workflow function calls each step and registers its compensation. You can read the flow top to bottom, unit-test it, and version it.
The cost is coupling. The orchestrator knows the order of every step, so some teams treat it as a central dependency that every service change must pass through. Choreography keeps that knowledge distributed, but it also makes the knowledge implicit.
| Option | Visibility | Testability | Coupling | Operational cost | Failure handling |
|---|---|---|---|---|---|
| Shared DB transaction | Full | Easy | Shared schema | Lowest | Automatic rollback |
| Outbox, single owner | Good | Easy | Low | Relay process | Retry delivery; no undo |
| Choreography | Fragmented | Hard end-to-end | Implicit, via events | Broker plus tracing | Per-consumer compensations |
| Temporal orchestration | Per-workflow history | Workflow unit tests | Explicit, central | Temporal cluster | Compensations in one function |
Who should use Temporal to run sagas?
Temporal orchestration fits when most of the following are true:
- A business process spans several services, each with its own datastore, and at least one step must be undone if a later step fails. Order placement, account provisioning, money movement between ledgers, and booking flows that reserve several resources are typical examples.
- The process can run longer than a single request. It may wait on retries with backoff, on a human approval, or on an external system that responds minutes or days later. A durable workflow can wait that long without holding a thread or a database lock.
- Someone needs to answer "where is this one stuck, and why?" Support, finance or on-call engineers benefit from a single per-execution history over logs spread across consumers.
- The team can absorb the operational and conceptual cost. That means running a Temporal cluster or paying for a hosted one, and learning to keep workflow code deterministic.
- Several teams build sagas. The cluster and the patterns, such as compensation stacks, idempotency keys and escalation via signals, are shared. The second and third saga cost much less than the first.
It is a poor fit for a single team with one short, two-step flow that an outbox handles. It is also a poor fit for work that fits in one database transaction, or for an organisation that cannot take on another stateful dependency. The last section covers these cases in more detail.
How does Temporal make a saga orchestrator durable?
In Temporal, the workflow function is the orchestrator. You write the saga as ordinary sequential code: call a step, record its compensation, call the next. The Temporal service does not store that function's memory. It stores an append-only event history: workflow started, activity scheduled, activity completed with this result, timer fired. When a worker crashes or a deploy restarts every pod, another worker picks up the workflow and replays the function against that history. Calls that already have recorded results return them without re-executing, so local variables, including your list of pending compensations, are rebuilt exactly. You never write a saga state table or a recovery job.
The moving parts:
- Activities do the real side effects, such as charging a card or reserving stock. The service handles their retries and timeouts.
- Workers are your processes. They host workflow and activity code.
- Task queues connect the two. Workers poll a named queue for workflow tasks and activity tasks, which lets you scale or isolate them independently.
- The Temporal service persists history, dispatches tasks, and runs durable timers.
Replay only works if the workflow code makes the same decisions every time it runs. So network calls, database access, random numbers, UUIDs and reading the wall clock must go into activities or use the SDK's deterministic equivalents for time, sleep and side effects.
History grows with every step, and the service enforces size limits. Long-running or loop-heavy sagas need continue-as-new or child workflows. Both split the history, and that raises a design question. With continue-as-new, you must pass the compensation stack explicitly into the new run. With child workflows, you must decide whether each child compensates its own steps or reports them back to the parent.
+--------------------------- Temporal Service ---------------------------+
| Event History (append-only) Task Queue: order-saga |
| 1 WorkflowStarted [workflow tasks] [activity tasks] |
| 2 ActivityScheduled ReserveStock Durable timers |
| 3 ActivityCompleted ok Retries and timeouts |
| 4 ActivityScheduled ChargeCard |
| 5 ActivityFailed ChargeCard |
+------------------------------------------------------------------------+
| ^ | ^
tasks | | commands, results history | | commands, results
v | v |
+-- Worker A (crashed) -----------+ +-- Worker B (replays) -----------+
| Workflow fn = saga orchestrator | | Replay same workflow fn |
| reserve -> push undoReserve | | recorded results returned, |
| charge -> push refundCharge |--->| not re-executed |
| on failure: run stack LIFO | | compensation stack rebuilt |
| Activities: I/O, side effects | | Deterministic code only |
+---------------------------------+ +---------------------------------+
History limit: continue-as-new must pass the compensation stack explicitly
Workers poll a task queue and report results to the Temporal service, which appends them to the event history. When Worker A crashes, Worker B replays that history through the same workflow function, rebuilding the saga's compensation stack without re-executing completed activities.
How does a Temporal saga work, step by step?
The previous section described the components. This is the sequence they go through for one order:
- Start. An API handler or another service starts the workflow on a task queue with a workflow ID derived from the business key, such as
order-8812. While a workflow with that ID is running, a second start request with the same ID is rejected. Double-submitting the same order therefore does not launch two sagas. - First workflow task. The service records that the workflow started and puts a workflow task on the queue. A worker picks it up and runs the workflow function until it reaches the first activity call. The worker then returns a "schedule activity" command and the function suspends.
- Activity execution. The service records that the activity was scheduled and dispatches an activity task. A worker polling the same queue runs the activity, which reserves stock, and reports success or failure. If the attempt fails with a retryable error or times out, the service schedules another attempt according to the retry policy. The workflow code is not involved in that retry.
- Resume. When the activity completes, the service appends the result to history and creates a new workflow task. A worker, not necessarily the same one, resumes the workflow function. If the worker has no cached state for this workflow, it first replays from the start of history. The function pushes the compensation for the step that just succeeded and calls the next activity.
- Failure and unwind. If a forward activity fails with a non-retryable error or exhausts its retries, the failure is delivered to the workflow as an exception. The
catchblock runs the compensation stack in reverse, and each compensation is an ordinary activity with its own retry policy. - Close. The workflow completes successfully if every forward step succeeded. If it compensated, it fails with the original error. If a compensation is stuck, it stays open waiting for a signal, as shown later. The full history remains available for inspection after the workflow closes, for as long as the namespace's retention period allows.
At every point between these steps, nothing lives only in process memory. Any worker can crash and the saga continues from the last recorded event.
How do you implement the saga pattern in a Temporal workflow?
Take an order saga with three forward steps: reserve inventory, charge payment, then ship. The workflow holds an ordered list of compensations. Each time a forward activity succeeds, it appends the matching undo: release the reservation after a successful reserve, refund the charge after a successful payment. If an activity fails with a non-retryable error, or its retries run out, the workflow catches the failure and runs the list in reverse. Payment is refunded before inventory is released, so each undo runs against the state its forward step left behind.
"Append after success" assumes you know whether the step succeeded. A timeout does not tell you that. If the charge request timed out, the payment service may have committed the charge before the response was lost. For any step where the outcome can be ambiguous, register the compensation before calling the activity. The refund then runs whether or not the charge landed. This only works if the refund is safe to run against a charge that never happened, which the next section covers.
Shipping goes last because it cannot be undone. A package in transit or an email in an inbox has no compensation, so that step acts as the pivot. Every step before it must be reversible, and once it succeeds the saga must complete forward. Put the irreversible step earlier and a later payment failure leaves you with goods shipped and nothing charged.
Temporal makes sure the compensation activities are scheduled and retried, even after a worker crash. It does not make sure they are correct. A refund that targets the wrong amount will still run, reliably and durably.
import { proxyActivities } from '@temporalio/workflow';
import type * as activities from './activities';
const { reserveInventory, releaseInventory, chargePayment, refundPayment, shipOrder } =
proxyActivities<typeof activities>({
startToCloseTimeout: '30 seconds',
retry: { maximumAttempts: 5 },
});
export async function orderSaga(orderId: string, amountCents: number): Promise<void> {
// Stack of undo actions; pushed in forward order, run in reverse.
const compensations: Array<() => Promise<void>> = [];
try {
await reserveInventory(orderId);
// Success is unambiguous here, so register the undo afterwards.
compensations.push(() => releaseInventory(orderId));
// A timeout here is ambiguous: the charge may have committed before the
// response was lost. Register the refund first. It must be a no-op when
// no charge exists for this orderId (idempotent, keyed by orderId).
compensations.push(() => refundPayment(orderId, amountCents));
await chargePayment(orderId, amountCents);
// Pivot step: irreversible, so it goes last and has no compensation.
await shipOrder(orderId);
} catch (err) {
// Reached on a non-retryable failure or once retries are exhausted.
// Unwind in reverse: refund before releasing inventory.
for (const compensate of compensations.reverse()) {
// Temporal schedules and retries these durably across worker crashes.
// It cannot tell whether the refund amount or target is correct.
await compensate();
}
throw err;
}
}
A Temporal TypeScript workflow that pushes compensations onto a stack, registers the refund before the ambiguous charge call, runs the irreversible ship step last, and unwinds compensations in reverse on failure.
How do you make activities and compensations idempotent and safe to retry?
So correctness has to live in the activities. Temporal guarantees at-least-once execution: a worker can complete a charge, crash before reporting the result, and the activity runs again.
Derive idempotency keys from stable identifiers
Build each key from values that are identical on every attempt, such as the workflow ID plus a step name (order-8812:charge-card). Pass it to the downstream service so it can deduplicate. Never use a random UUID generated inside the activity, because each retry would produce a new one. The attempt number is also wrong for this, since it changes on every retry.
Make compensations no-ops when there is nothing to undo
The forward step may have failed before taking effect, or an earlier compensation attempt may already have succeeded. Before acting, the refund activity should look up the charge by its key. If the charge never happened or is already refunded, it should return success without doing anything.
Choose retry policies deliberately
| Activity type | Retries | Non-retryable errors | Timeouts |
|---|---|---|---|
| Forward step | Bounded, with backoff | Business failures such as card declined or out of stock. These trigger compensation. | Start-to-close sized to the call |
| Compensation | Aggressive or effectively unbounded | Very few | Start-to-close, plus heartbeats if the activity is long |
Compensations retry hard because giving up leaves the system partially applied. Long activities should heartbeat so that a dead worker is detected within the heartbeat timeout instead of after the full start-to-close timeout. Heartbeat details can also record progress for the next attempt.
All of this depends on downstream APIs accepting idempotency keys. If you don't control those APIs, you need to add your own deduplication in front of them.
// workflows.ts
import { proxyActivities, workflowInfo } from '@temporalio/workflow';
import type * as activities from './activities';
// Forward steps: bounded retries with backoff. Business failures fail fast and trigger compensation.
const forward = proxyActivities<typeof activities>({
startToCloseTimeout: '30 seconds',
retry: { maximumAttempts: 5, backoffCoefficient: 2, nonRetryableErrorTypes: ['CardDeclined', 'OutOfStock'] },
});
// Compensations: no maximumAttempts, so they retry until they succeed. Heartbeats detect a dead worker early.
const undo = proxyActivities<typeof activities>({
startToCloseTimeout: '5 minutes',
heartbeatTimeout: '30 seconds',
retry: { initialInterval: '1 second', maximumInterval: '2 minutes' },
});
export async function orderSaga(totalCents: number, sku: string): Promise<void> {
// Keys stay the same across retries and replays, e.g. "order-8812:charge-card".
const chargeKey = `${workflowInfo().workflowId}:charge-card`;
const reserveKey = `${workflowInfo().workflowId}:reserve-stock`;
await forward.chargeCard(chargeKey, totalCents);
try {
await forward.reserveStock(reserveKey, sku);
} catch (err) {
await undo.refundCharge(chargeKey); // safe even if the charge never landed
throw err;
}
}
// activities.ts (excerpt; `payments` is your client for the payment service)
import { Context } from '@temporalio/activity';
export async function refundCharge(chargeKey: string): Promise<void> {
Context.current().heartbeat();
const charge = await payments.findChargeByKey(chargeKey);
if (!charge || charge.refunded) return; // nothing to undo, so do nothing and report success
await payments.refund(charge.id, { idempotencyKey: `${chargeKey}:refund` });
}
A Temporal TypeScript saga that builds idempotency keys from the workflow ID plus a step name, gives forward steps bounded retries with non-retryable business errors, and gives the heartbeating refund compensation unbounded retries and an early return when there is nothing to undo.
What happens when a saga is cancelled or a compensation cannot succeed?
Cancellation is where many sagas quietly break. When a Temporal workflow is cancelled, its context is cancelled too, and any activity you schedule from that context fails immediately with a cancellation error. If your compensation loop runs on the same context, the workflow ends in a "cancelled" state, every undo step is skipped, and nothing reports a failure. The inventory stays reserved and the card stays charged. Run compensations in a scope that cancellation cannot reach. In Go, that means workflow.NewDisconnectedContext. In TypeScript, it means CancellationScope.nonCancellable. Other SDKs have an equivalent detached scope.
Some compensations fail because of business logic, not infrastructure. Suppose a refund targets an account that was closed after the charge. Retrying will never fix this, so the activity should return a non-retryable application error. Then the workflow can decide what to do next. For failures that might be transient, cap MaximumAttempts or set an expiration in the retry policy so a broken dependency cannot hold up the saga forever.
When a compensation is out of retries, escalate instead of failing:
- Record the step that failed and the reason in workflow state.
- Expose that state through a query handler so operators can inspect it.
- Raise an alert from an activity.
- Block on a signal such as "resolved manually" or "retry with new target account."
This keeps the workflow open and inspectable until a person makes a decision. Durable retries only keep the problem visible. Someone still has to fix it, so every escalation path needs a runbook that says who acts and how.
import { proxyActivities, CancellationScope, defineQuery, defineSignal, setHandler, condition, ActivityFailure } from '@temporalio/workflow';
import type * as activities from './activities';
// Bounded retries: a broken dependency cannot stall the saga forever.
// refundCard throws a non-retryable ApplicationFailure for logic errors (e.g. account closed).
const { reserveInventory, chargeCard, releaseInventory, refundCard, raiseAlert } = proxyActivities<typeof activities>({
startToCloseTimeout: '30 seconds',
retry: { maximumAttempts: 5 },
});
export const stuckQuery = defineQuery<string | null>('stuckCompensation');
export const resolvedSignal = defineSignal('resolvedManually');
export async function orderSaga(orderId: string): Promise<void> {
const undo: Array<{ name: string; run: () => Promise<void> }> = [];
let stuck: string | null = null; // which undo step is blocked, and why
let resolved = false;
setHandler(stuckQuery, () => stuck);
setHandler(resolvedSignal, () => { resolved = true; });
try {
await reserveInventory(orderId);
undo.unshift({ name: 'releaseInventory', run: () => releaseInventory(orderId) });
await chargeCard(orderId);
undo.unshift({ name: 'refundCard', run: () => refundCard(orderId) });
} catch (err) {
// Without this scope, a cancelled workflow skips every undo step silently.
await CancellationScope.nonCancellable(async () => {
for (const step of undo) {
try { await step.run(); }
catch (e) {
stuck = `${step.name}: ${e instanceof ActivityFailure ? e.cause?.message : String(e)}`;
resolved = false;
await raiseAlert(orderId, stuck); // page a human; the runbook says who acts
await condition(() => resolved); // stay open and queryable until resolved
stuck = null;
}
}
});
throw err;
}
}
A TypeScript Temporal saga that runs compensations in a non-cancellable scope, bounds retries, and escalates a failed undo by alerting, exposing the stuck step through a query, and waiting for a manual-resolution signal.
How can Temporal sagas save engineering time?
The time savings come from infrastructure you no longer write, and from incidents that take less effort to diagnose. They do not come from the saga logic itself, which you still have to design carefully.
- No hand-built state machine. Without a durable orchestrator, a saga usually needs a table recording the current step for each order, code that advances it, and a sweeper job that finds rows stuck in intermediate states and resumes or compensates them. With Temporal, the event history replaces that table and replay replaces the sweeper.
- Retries and timeouts are configuration. Backoff, maximum attempts, non-retryable error types and heartbeat detection are declared in the activity options shown above. You do not reimplement them for each service call.
- Compensation logic lives in one place. The forward steps and their undos sit in a single function. A reviewer can check that every reversible step registers a compensation and that the pivot step comes last, without tracing event handlers across repositories.
- Faster debugging of individual executions. For "where is order 8812 stuck?", you look up the workflow by ID and read its history. It shows which activities ran, their inputs and results, each failed attempt, and any pending compensation exposed through a query. With choreography, answering the same question means correlating logs and traces across every consumer.
- Tests run in seconds. The time-skipping test environment lets you exercise retry backoff and long timers without waiting, so failure-path tests for every step are cheap enough to run on every change.
- Safer changes to in-flight processes. Replay tests and the patching API, both covered in the next section, catch incompatible workflow changes before deploy. They replace ad hoc migrations of a state table.
You give up some of this time elsewhere. Engineers must learn the determinism rules, review workflow code with them in mind, and keep the cluster healthy. The net saving is largest when several sagas share the same cluster and patterns. It is smallest, and can be negative, for a single simple flow that an outbox or one database transaction would handle.
How do you test and version long-running sagas, and when should you avoid them?
Start with a test matrix: one case per step, each forcing that step's activity to fail. Run the workflow in the SDK's test environment with time skipping, so retry backoffs and timers resolve instantly instead of taking hours. Register mocked activities that record their invocation order. For a failure at step n, assert that compensations ran for steps n−1 down to 1, in that order, and that nothing ran for step n or later. Add cases where a compensation fails transiently once, to confirm that retries happen and that the compensation is not skipped.
Unit tests won't catch nondeterminism. For that, export histories from real executions and run them through the SDK's replayer in CI. If a code change produces a different sequence of commands for an existing history, replay fails before the change reaches workers that still have sagas in flight.
When you need to reorder steps or change a compensation, wrap the change in the SDK's patching API (GetVersion in Go and Java, patched in TypeScript and Python). New executions take the new path, while running ones finish on the old path. Remove the old branch only after those executions have drained.
The cost is real. Engineers must learn to keep I/O, clocks and randomness out of workflow code. Someone has to operate a cluster or pay for a hosted one, and that cluster becomes a dependency every saga relies on. In return, you don't hand-build state machines, persistence or crash recovery.
You don't need any of this when a single database transaction covers the work, when steps are naturally idempotent and order doesn't matter, or when a nightly reconciliation job can tolerate the occasional inconsistency.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to saga pattern temporal, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- Reviewers see the exact diff and the list of affected services up front, so approval is a decision rather than an investigation.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- Per-service approve/reject means a bad shared value can be refused where it would break something, before it is live.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.