What is the saga pattern, and why do microservices use it instead of distributed transactions?
A saga breaks one business operation into a sequence of local transactions, each committed by the service that owns the data. If a later step fails, the saga runs compensating transactions to undo the effects of the earlier steps that already committed. In an order flow, you might reserve inventory, charge the card, and then fail to book shipping. The saga then refunds the charge and releases the reservation.
The alternative is a distributed transaction, typically two-phase commit (2PC). It works poorly across independently owned services. 2PC needs every participant to support a shared transaction protocol, which a payment provider's HTTP API or a team's document store usually does not. It also holds locks while waiting on a coordinator, so one slow or unreachable participant blocks all the others. And it ties the release schedules and availability of every service to the coordinator's.
Sagas give up isolation and offer eventual consistency instead. Intermediate states are real and visible: another request can see reserved stock or a pending charge before the saga finishes. You can reduce the impact with semantic locks, such as an order marked PENDING that other operations must respect, but every reader then has to understand those statuses.
Compensation is also not always a true undo. You cannot unsend an email or recall a shipped package. Those cases need business-level corrections, such as an apology message or a return label. Someone has to decide what those corrections are, so they are product decisions, not just code.
Orchestration vs choreography: what are the alternatives for coordinating a saga?
Once the compensations are defined, something has to decide when to run them. Four coordination styles are common.
| Approach | Where saga state lives | Coupling | Debugging a stuck order |
|---|---|---|---|
| Orchestration (Temporal) | Workflow event history | Services expose plain APIs; the orchestrator knows the whole flow | Open one workflow's history and find the failed step |
| Choreography (events) | Spread across services and the broker | Services depend on each other's event schemas | Correlate logs across services by order ID |
| Database state machine | A status table you own | Similar to orchestration | Query the row; retries and timers are your own code |
| Synchronous call chain | The caller's memory | Tight; every service must be up at once | One stack trace, lost if the caller crashes |
Orchestration gives the best visibility. The tradeoff is ownership. The team that owns the workflow becomes a coordination point, because every new step or changed compensation goes through their code and review.
Choreography keeps services autonomous, but no one owns the end-to-end flow. Compensation logic ends up spread across consumers, and the answer to "why wasn't this refunded?" is hard to find.
A database state machine works until you need durable timers, backoff, and safe changes to in-flight flows. At that point you are building a workflow engine yourself.
A synchronous chain is fine for two steps when the caller can undo the first inline and a crash leaves a state you can detect.
Adopt Temporal when several teams share a flow with three or more steps or long waits, and you can operate the cluster or pay for hosting. Skip it for single-team, two-step flows.
How does the saga pattern work in Temporal?
In Temporal, a saga is a single workflow function. Each forward step, such as reserving inventory, charging a card or booking a shipment, is an activity. Each compensation, such as releasing the reservation or refunding the charge, is also an activity. The saga's state is held in ordinary variables inside the workflow: which steps succeeded, the IDs they returned, and a list of compensations to run in reverse order. You don't need a saga log table or a state machine definition, because the code itself is the coordinator.
This works because of durable execution. Temporal records every activity scheduled, completed or failed in the workflow's event history. If a worker crashes after the payment step, another worker picks up the workflow and replays the history from the start. Completed activities are not re-executed; their recorded results are fed back in. The variables are rebuilt exactly as they were, including the list of pending compensations.
Replay only works if workflow code is deterministic. Given the same history, it must make the same decisions. In saga code this means:
- Network calls, database writes and reads of external state belong in activities.
- Current time, timers and random values must come from the SDK's workflow-safe APIs, not the language's standard library.
- Iterating over unordered maps to decide step order can break replay.
Event histories have hard limits on event count and total size. Pass references such as order IDs or object-storage keys between activities instead of full documents. For very long or looping sagas, use continue-as-new to start a fresh history and carry the minimal state forward.
+--------------------------------------------------------------------+
| Saga workflow function (deterministic; replayable on any worker) |
| state: completed steps, returned IDs, compensation stack (LIFO) |
| time, timers, random -> SDK workflow APIs, not stdlib |
+--------------------------------------------------------------------+
| schedule activity ^ result / failure
v |
+--------------------------------------------------------------------+
| Temporal service: appends every event to the workflow's history |
| [Sched][Done: resId][Sched][Done: chargeId][Failed: ship] |
| limits on event count and size: pass IDs/keys, continue-as-new |
+--------------------------------------------------------------------+
| task queue ^ replay: recorded results fed
v | back, not re-executed
+-------------------------+ +-----------------------------------+
| Worker A | | Worker B: replays history, |
| crashes after charge |--->| rebuilds vars, resumes saga |
+-------------------------+ +-----------------------------------+
| runs activities (network
v calls, DB writes live here)
Forward: [Reserve inventory]-->[Charge card]-->[Book shipment]
| fails
|
Undo: [Release reservation]<--[Refund charge]<----+
(popped from compensation stack, newest first)
A Temporal saga as one deterministic workflow whose state is rebuilt from the event history on another worker after a crash, with forward steps and compensations run as activities and compensations executed in reverse order.
How do you implement compensating transactions in a Temporal workflow?
Take an order-booking workflow with three activities: reserve inventory, charge payment, schedule shipping. Before or after each forward step, append its undo (release inventory, refund payment, cancel shipment) to a list held in workflow state. If any step fails with a non-retryable error, or exhausts its retry policy, catch the error, run the list in reverse, and then rethrow so the workflow records the failure. Reverse order matters: a refund that runs while shipping is still booked can leave a package in transit for an unpaid order.
Register before or after the forward activity?
Registering after the activity returns looks cleaner, but it leaves a gap. Suppose the charge activity times out. The payment provider may have processed the charge after the worker lost its connection. If the refund was never registered, nothing undoes it.
Registering before the activity closes that gap. It also means a compensation can run for a step that never took effect. Write every compensation to tolerate this:
- Key the refund by order ID.
- Treat "no such charge" as success.
- Never assume the forward call landed.
Making compensations safe to run against nothing is cheaper than trying to work out which ambiguous steps actually happened.
Compensating on cancellation
When a workflow is cancelled, its context is cancelled too, so compensation activities started from that context would be cancelled immediately. Each SDK provides a way to run cleanup that survives cancellation. In Go, you use a disconnected context. In TypeScript, you use a non-cancellable cancellation scope. The exact API differs by language SDK, so check the one you use. Run the same reverse unwind inside that construct.
import { proxyActivities, CancellationScope, log } from '@temporalio/workflow';
import type * as activities from './activities';
const {
reserveInventory, releaseInventory,
chargePayment, refundPayment,
scheduleShipping, cancelShipment,
} = proxyActivities<typeof activities>({
startToCloseTimeout: '30 seconds',
retry: { maximumAttempts: 5 },
});
export async function bookOrder(orderId: string): Promise<void> {
const compensations: Array<() => Promise<void>> = [];
try {
// Register each undo BEFORE its forward call. A timed-out activity may
// still have taken effect, so each undo must treat "nothing to undo" as success.
compensations.push(() => releaseInventory(orderId));
await reserveInventory(orderId);
compensations.push(() => refundPayment(orderId)); // keyed by order ID
await chargePayment(orderId);
compensations.push(() => cancelShipment(orderId));
await scheduleShipping(orderId);
} catch (err) {
// Activity failures and workflow cancellation both land here. The
// non-cancellable scope lets the undo activities run even after cancellation.
await CancellationScope.nonCancellable(async () => {
for (let i = compensations.length - 1; i >= 0; i--) {
try {
await compensations[i]();
} catch (undoErr) {
// Keep unwinding so one failed undo does not block the earlier ones.
log.error('compensation failed', { orderId, step: i, error: String(undoErr) });
}
}
});
throw err; // rethrow so the workflow records the original failure
}
}
A TypeScript Temporal workflow registers each compensation before its forward activity and unwinds the list in reverse order inside a non-cancellable scope, so cleanup runs after both activity failure and workflow cancellation.
How do you make saga steps idempotent and retry-safe?
Temporal executes activities at least once. If a worker crashes after the payment provider accepts a charge but before the result is recorded in history, Temporal retries the activity and the provider sees the request a second time. Replay does not re-run completed activities, so the risk comes from retries and workflow resets. The fix is an idempotency key that is identical on every attempt. Build it from stable workflow data, such as the workflow ID plus the step name (order-8812:charge-card, order-8812:refund-card), and send it in the downstream API's idempotency header or field. Do not use the run ID, because it changes on reset and continue-as-new. Do not use a random UUID generated inside the activity, because each attempt would get a new one.
Classify failures before you rely on retries:
| Failure | Example | Retry? | Saga action |
|---|---|---|---|
| Infrastructure | Timeout, 503, connection reset | Yes, with backoff | Keep retrying the step |
| Business | Card declined, insufficient stock | No (mark non-retryable) | Start compensation immediately |
| Compensation infrastructure | Refund API unavailable | Yes, for a long time | Keep retrying the compensation |
Raise business failures as non-retryable application errors, or list their error types as non-retryable in the retry policy. Otherwise the workflow spends minutes retrying a decline that will never succeed. Set a Start-To-Close timeout so a hung call fails fast and gets retried.
Compensations need a separate, more persistent policy: a longer Schedule-To-Close window or no attempt limit. Giving up on a refund leaves the system inconsistent, which is worse than waiting. Temporal guarantees retries but not deduplication. Idempotency is the developer's responsibility, and the downstream service has to honor the key.
import { proxyActivities, workflowInfo } from '@temporalio/workflow';
import type * as activities from './activities';
// Forward steps: fail fast on hung calls, retry infra errors with backoff,
// never retry business failures. In the activity, raise a decline with
// ApplicationFailure.create({ type: 'CardDeclined', message: 'declined', nonRetryable: true })
const { chargeCard, reserveStock } = proxyActivities<typeof activities>({
startToCloseTimeout: '10 seconds',
retry: {
initialInterval: '1 second',
backoffCoefficient: 2,
maximumInterval: '30 seconds',
maximumAttempts: 5,
nonRetryableErrorTypes: ['CardDeclined', 'OutOfStock'],
},
});
// Compensations: no maximumAttempts (unlimited), long overall window.
// Giving up on a refund leaves the system inconsistent.
const { refundCard } = proxyActivities<typeof activities>({
startToCloseTimeout: '10 seconds',
scheduleToCloseTimeout: '7 days',
retry: { initialInterval: '5 seconds', backoffCoefficient: 2, maximumInterval: '10 minutes' },
});
export async function orderSaga(order: { customerId: string; sku: string; amountCents: number }) {
// Workflow ID + step name is stable across attempts, replays and resets (unlike runId).
// Activities send this in the provider's idempotency header so retries are deduplicated.
const key = (step: string) => `${workflowInfo().workflowId}:${step}`;
// A non-retryable decline fails here; nothing has been charged, so nothing to compensate.
await chargeCard({ ...order, idempotencyKey: key('charge-card') });
try {
await reserveStock({ sku: order.sku, idempotencyKey: key('reserve-stock') });
} catch (err) {
await refundCard({ ...order, idempotencyKey: key('refund-card') });
throw err;
}
}
A Temporal TypeScript saga that derives idempotency keys from the workflow ID plus step name, uses a bounded retry policy with non-retryable business error types for forward steps, and gives the refund compensation its own unlimited, long-running retry policy.
What happens when a compensation fails?
A compensation that keeps failing usually signals something retries can't fix. The downstream service may have rejected the request permanently, or the resource may already be in a state nobody anticipated. Unlimited retries are a reasonable default for transient faults because Temporal persists the attempt history and the workflow costs nothing while it waits. For errors that retrying cannot resolve, mark them non-retryable and choose an explicit path:
- Alert and keep retrying when you expect the dependency to recover, such as during an outage.
- Park the saga when a human has to decide. The workflow blocks on a signal such as "retry compensation", "mark resolved manually", or "skip". The operator's decision is then recorded in the workflow history next to the failure.
A parked saga is only safe if someone notices it. Set a custom search attribute such as CompensationStatus=Stuck when the workflow enters the waiting state. Operators can then list every affected saga through a visibility query. Add an alert on that count, and another on activity retry and failure metrics from the SDK, so an inconsistent order generates a page instead of a support ticket weeks later.
If you change step order or add a compensation, running workflows will replay their existing histories against the new code. Wrap the change in the SDK's versioning or patching API so old executions keep following the old path. Without that guard, replay produces commands that don't match the recorded history. The worker then reports a non-determinism error, and the workflow task fails repeatedly until you deploy compatible code.
from datetime import timedelta
from temporalio import workflow
from temporalio.common import RetryPolicy, SearchAttributeKey
from temporalio.exceptions import ActivityError
# Keyword search attribute; register it on the namespace before use.
COMP_STATUS = SearchAttributeKey.for_keyword("CompensationStatus")
@workflow.defn
class OrderSaga:
def __init__(self) -> None:
self.decision: str | None = None
@workflow.signal
def resolve(self, decision: str) -> None:
# Operator sends "retry", "resolved_manually" or "skip"; the signal lands in history.
self.decision = decision
async def compensate(self, activity: str, order_id: str) -> None:
while True:
try:
await workflow.execute_activity(
activity, order_id, start_to_close_timeout=timedelta(seconds=30),
retry_policy=RetryPolicy(maximum_attempts=20,
non_retryable_error_types=["RefundRejected"]))
return
except ActivityError:
# Park the saga: make it queryable, then block until a human decides.
workflow.upsert_search_attributes([COMP_STATUS.value_set("Stuck")])
self.decision = None
await workflow.wait_condition(lambda: self.decision is not None)
workflow.upsert_search_attributes([COMP_STATUS.value_set("Resolving")])
if self.decision != "retry":
return
@workflow.run
async def run(self, order_id: str) -> None:
# ... forward steps omitted; assume one failed and we now undo.
if workflow.patched("release-inventory-before-refund"):
await self.compensate("release_inventory", order_id) # new order
await self.compensate("refund_payment", order_id)
else:
await self.compensate("refund_payment", order_id) # histories recorded before the change
await self.compensate("release_inventory", order_id)
A Temporal Python saga that parks a failing compensation behind an operator signal, tags it with a CompensationStatus search attribute so visibility queries can find it, and uses workflow.patched so reordering compensations does not break replay of running executions.
How do you test sagas, and when should you avoid Temporal for them?
Testing compensation paths
Temporal's SDKs ship a test environment that runs workflows in-process with time skipping, so retries, backoff and timers complete in milliseconds. Register mock implementations of each activity in place of the real ones. For a saga with N forward steps, write N test cases, and in each one make a different step's mock return a non-retryable error. Have every compensation mock append its name to a shared list. Then assert two things: the list matches the exact reverse order of the steps that succeeded, and no compensation ran for a step that never executed. That second assertion catches a common bug where the compensation is registered before the forward call instead of after it succeeds.
Mocks can't catch determinism regressions. For that, export histories from real executions, especially ones that are still open, and feed them to the SDK's replayer in CI against your current workflow code. If a change alters the command sequence without a version guard, replay fails in the pipeline instead of on a production worker.
When Temporal is worth the cost
Temporal adds operational load whether you self-host or use a managed service. Someone owns the cluster, its persistence store, worker fleets and upgrades. Versioning also makes long-running workflow code slower to change. In return, you don't write custom state machines, retry bookkeeping or recovery scripts for stuck transactions.
It fits well for flows with many steps, long durations and unreliable dependencies. For a two-step flow, or a team that can't operate another stateful system, an outbox table with idempotent consumers is usually cheaper.
How OneEnv can help
Disclosure: OneEnv is the product behind this blog.
OneEnv (oneenv.io) is a workspace-based governance platform for the things services share — environment variables, database structure, API routes and validation rules. A change to a shared value is staged on a branch, reviewed once, and then every affected service explicitly approves or rejects it before it goes live, so you can see who a change touches before it ships instead of finding out in production.
Applied to saga pattern temporal, this is where it helps:
Saves time
- Every service that inherits a shared value is listed in one place, so nobody searches repos and chats to find out who is affected.
- One review per branch replaces coordinating each team separately; each service still keeps its own approve/reject.
- The dependency graph answers "what does this touch?" in seconds instead of a manual audit.
Saves cost
- Fewer production incidents and rollbacks caused by a shared value, route or column changing under a consumer unannounced.
- Less duplicated configuration to maintain: one shared definition inherited by many services.
- You can size a change's blast radius before merging, avoiding surprise cross-team firefighting.
- OneEnv has a free plan (1 workspace, unlimited services and members, full governance), so trying it costs nothing.
These are qualitative benefits, not measured results — how much you save depends on how many services share the value.
Want to try it? Start at oneenv.io.