Saga Pattern: Orchestration vs Choreography and Compensating Transactions (2026)

Every microservice migration eventually collides with the same sentence from a product manager: “when the customer books the trip, either everything happens or nothing does.” In a monolith with one relational database that sentence costs a BEGIN and a COMMIT. Once payment, inventory and shipping each own a private datastore, it costs a design decision that shapes your reliability, your latency and your on-call rotation for years.

The saga pattern is the mainstream answer. Instead of one atomic transaction spanning services, a saga is a sequence of local transactions, each followed by a defined undo if a later step fails. It trades atomicity and isolation for availability and autonomy, and that trade is only safe if you understand precisely what you gave up. This article walks through why two-phase commit fails across service boundaries, how orchestrated and choreographed sagas differ, how compensating transactions, pivot steps and retriable steps are classified, which countermeasures restore a degree of isolation, and what actually breaks in production.

What this covers: the origin of sagas, why distributed transactions are hard, orchestration vs choreography with a decision matrix, saga step taxonomy, semantic locks and other isolation countermeasures, idempotency and timeouts, persisting saga state, workflow engines such as Temporal, AWS Step Functions and Camunda, observability, a worked booking example with failure injection and code, and the failure modes that bite hardest.

Context and Background

The word “saga” predates microservices by almost four decades. Hector Garcia-Molina and Kenneth Salem introduced it in a 1987 SIGMOD paper titled “Sagas”, aimed at long-lived database transactions. A long transaction that holds locks for minutes or hours starves every other user of the database. Their proposal was to break the long transaction into a sequence of shorter sub-transactions, T1 through Tn, and pair each with a compensating transaction, C1 through Cn-1, that semantically undoes its effect. If the saga fails after Tk, the system runs Ck, Ck-1 down to C1. Crucially, the original paper allowed other transactions to interleave between the sub-transactions: a saga guarantees atomicity in the sense of “all or compensated”, not isolation.

That idea sat mostly in the database literature until the microservices era made it relevant again. When each service owns its data, which is a core tenet of the architecture, a business operation that touches three services is by definition a distributed transaction. Chris Richardson’s catalog at microservices.io popularised the modern formulation, including the two coordination styles and the isolation countermeasures we cover below, and most engineers today meet sagas through that lens.

The incumbent alternative is the classic distributed transaction. The X/Open XA standard defines a resource-manager interface, and two-phase commit (2PC) is the protocol a transaction manager uses to drive it. Java’s JTA, many application servers and several databases support XA. For a single organisation running a few tightly coupled databases on a reliable network, 2PC is still a legitimate tool. The trouble starts when the participants are independently deployed services, some of which expose HTTP APIs or message brokers rather than XA resource managers, and many of which are owned by third parties, such as a card processor, an airline reservation system or a shipping carrier. Those systems will never join your transaction manager’s prepare phase.

If you are building the event plumbing for a saga, two neighbouring patterns matter. Reliable publication of events requires the transactional outbox pattern with change data capture, because a saga step that updates its database and then publishes a message in two separate operations will eventually lose one of them. And if you want a tamper-evident record of what each step did, event sourcing with bitemporal data is a natural storage model for the saga’s own history. Both are prerequisites for taking the pattern from a whiteboard to production.

For the formal background, the original paper is available through the ACM Digital Library (Garcia-Molina and Salem, “Sagas”, SIGMOD 1987), and Richardson’s pattern language is documented at microservices.io.

Why Two-Phase Commit Fails Across Services

Two-phase commit works like this. A coordinator asks every participant to prepare. Each participant durably records that it can commit, acquires and holds the necessary locks, and votes yes or no. If all vote yes, the coordinator writes a commit decision and tells everyone to commit. The protocol delivers atomicity, but only under assumptions that service architectures routinely violate.

Blocking is inherent. After a participant votes yes, it cannot unilaterally decide anything. If the coordinator crashes before delivering the decision, the participant is “in doubt”, holding row locks until the coordinator recovers. Three-phase commit reduces blocking in theory but assumes bounded network delay, which real networks do not provide. In practice, the availability of a 2PC transaction is the product of the availability of every participant plus the coordinator. Five participants each at 99.9% give roughly 99.5% for the transaction (illustrative arithmetic: 0.999 to the power of 5 is about 0.995), before adding the coordinator.

Locks are held across network round trips. A 2PC transaction holds locks for at least two network round trips between prepare and commit, and for as long as the slowest participant takes. Under contention on a hot row, such as a popular product’s stock count, throughput collapses to roughly one over the lock hold time per row. A participant that pauses for garbage collection extends everyone’s hold time.

Heterogeneous participants rarely speak XA. A REST API, a Kafka topic, a SaaS payment gateway and a mainframe queue do not expose prepare and commit. Even where an adapter exists, the CAP trade-off applies: during a partition you must choose between refusing the transaction and risking inconsistency, and 2PC picks consistency at the cost of availability.

Organisational coupling. Joining a shared transaction means sharing a transaction manager, a deployment cadence and a failure domain. That undermines the independent deployability that justified microservices in the first place.

None of this means 2PC is obsolete. Distributed SQL databases such as Spanner-style systems use 2PC internally across shards of one logical database, where the same team controls participants and the network. The point is narrower: 2PC across independently owned services is a poor default, and sagas exist for that gap.

Saga Pattern Reference Architecture: Orchestration and Its Anatomy

Direct answer: a saga is a sequence of local transactions in different services, where each step commits independently and publishes its outcome; if a step fails, previously completed steps are undone by compensating transactions. It provides eventual atomicity without distributed locks, but not isolation. Coordination is either centralised in an orchestrator or distributed through events (choreography).

Figure 1: An orchestrated saga. The orchestrator persists its state in a saga log, sends commands to participants in order, and runs compensations in reverse when a step fails.

The diagram shows the shape of an orchestrated saga for a trip booking. The orchestrator is a state machine. It holds the saga’s identity, its current step and the data accumulated so far. It sends a command to the payment service, waits for a reply, moves to the hotel, then the flight. If the flight service replies with a failure, the orchestrator transitions into a compensating path and issues cancel commands in reverse order for the hotel and then payment. Each participant only knows how to execute its own local transaction and its own compensation; the knowledge of “what comes next” lives in one place.

The orchestrator as an explicit state machine

An orchestrator is easiest to reason about when you draw it as a finite state machine with states such as STARTED, PAYMENT_RESERVED, HOTEL_BOOKED, COMPLETED, COMPENSATING_HOTEL, COMPENSATING_PAYMENT and ABORTED. Transitions are driven by replies. Persisting the current state after every transition is non-negotiable: the orchestrator process will crash, be redeployed or be rescheduled mid-saga, and it must resume exactly where it stopped. That persisted record is the saga log, and it makes the saga itself a durable, queryable entity, which helps both operations and audits.

Commands should be addressed by a stable identifier, typically the saga ID plus the step name. That key becomes the idempotency key on the participant side, which we return to later. Replies should carry the same key so that late or duplicate replies can be correlated and ignored if the saga has already moved on.

Participants own their compensation

A common design mistake is to centralise compensation logic in the orchestrator, for example by having it call a generic “delete record” endpoint. Compensation is business logic and belongs to the owning service. The hotel service knows that cancelling a booking may incur a fee, that a room already checked into cannot be cancelled, and that a cancellation must also release the inventory hold. The orchestrator only knows that a CancelRoom command exists and what replies it may return.

What the orchestrator costs you

An orchestrator introduces a component that must itself be highly available and horizontally scalable, and it concentrates business sequencing in one codebase. Teams sometimes let it grow into a “god service” that embeds domain rules that belong in participants. Mitigate this by keeping the orchestrator thin: it sequences, it does not decide domain outcomes. In exchange you get a single place to read the flow, a single place to add a step, simple cycle-free dependencies (participants never call the orchestrator’s other participants) and a natural spot for timeouts and retries.

Choreography: Sagas Without a Conductor

In a choreographed saga there is no central coordinator. Each service listens for events, performs its local transaction and publishes a new event that triggers the next service. Compensation works the same way: a failure event is published, and services that previously acted subscribe to it and undo their work.

Figure 2: A choreographed saga. Services react to events on a broker; the failure event StockFailed triggers payment to refund and publish PaymentRefunded back to the order service.

The figure shows an order flow. The order service emits OrderCreated. Payment captures funds and emits PaymentCaptured. Inventory attempts to reserve stock and emits either StockReserved or StockFailed. On StockFailed, payment refunds and emits PaymentRefunded, and the order service marks the order cancelled. No component holds the whole flow; the flow exists only as the sum of subscriptions.

Choreography has real strengths. There is no extra infrastructure beyond a broker, no single point of coordination, and loose coupling at deploy time: adding a new subscriber (say, a loyalty service that listens for OrderCompleted) requires no change to existing services. For short sagas with two or three participants and a stable flow, it is lean and fast.

Its weaknesses grow with the number of steps. The flow becomes implicit and hard to see: to answer “what happens after payment succeeds?” you must search every service’s subscriptions. Cyclic dependencies appear easily, since payment must know about stock failures and the order service must know about payment refunds. Testing requires standing up the whole constellation of services and the broker. Most painfully, nobody owns the end-to-end state, so questions like “which sagas are stuck right now?” have no natural answer unless you build a separate read model that consumes all events, which is itself a de facto orchestrator in disguise.

Orchestration vs choreography: decision matrix

Criterion	Orchestration	Choreography
Visibility of the flow	One place, readable as a state machine	Distributed across subscriptions
Coupling	Participants coupled to the orchestrator’s command API	Services coupled to event schemas
Number of steps where it stays manageable	Many (10+ is routine)	Few (roughly 2 to 4)
Cyclic dependencies	Avoided by design	Easy to create
Single point of failure risk	Orchestrator must be made HA	None, but the broker is critical
Adding a step	Edit the orchestrator	Add a subscriber, possibly edit others
Timeouts and retries	Centralised and uniform	Per-service, inconsistent
Observability of in-flight sagas	Native from the saga log	Needs an event-derived read model
Best fit	Complex, regulated or money-moving flows	Simple, stable, high-fanout notifications

A pragmatic rule of thumb, which is an opinion rather than a law: choose choreography when the saga has at most three or four steps, one team owns all participants, and the flow rarely changes. Choose orchestration when money moves, when steps number more than four, when the flow needs timeouts and human intervention, or when auditors will ask you to prove what happened. Many mature systems use both: orchestration inside a business capability and choreography between capabilities.

Compensations, Pivots and Retriable Steps

Not every step in a saga is equal, and the most useful design exercise is to classify them. Richardson’s formulation, which builds on the classic literature, uses three categories.

Compensatable transactions can be undone by a compensating transaction. Reserving funds, holding a seat or creating a pending order are all compensatable. The pivot transaction is the go/no-go point: if it succeeds, the saga will run to completion; if it fails, the saga compensates. It is typically the first step that cannot be undone, or the one that is the last to be able to fail for business reasons, such as charging the card or issuing the ticket. Retriable transactions come after the pivot and are guaranteed to eventually succeed if retried, such as sending a confirmation email or awarding loyalty points. They must never fail the saga; they are retried indefinitely with backoff and parked for human attention if they stall.

Figure 3: Ordering saga steps. Compensatable steps come first, the pivot is the commit point, and retriable steps follow. Dotted edges show compensation running in reverse.

The ordering rule that follows is the central design heuristic: put compensatable steps first, then the pivot, then retriable steps, and where possible put the steps most likely to fail early so that few compensations are needed. If a step has no feasible compensation and cannot be retried to success, it is either your pivot or your design is wrong, and you should have only one of them per saga.

What a compensating transaction really is

A compensating transaction is a semantic undo, not a rollback. A database rollback restores a previous state invisibly. A compensation is a new, forward business action that is visible to other transactions and to customers. Refunding a captured payment is not the same as never charging: the customer’s statement shows both lines, the processor may keep a fee, and the refund can take days to settle. Cancelling a hotel booking may incur a penalty. Releasing a stock reservation is clean, but sending a “your order shipped” email cannot be unsent; the compensation is a second email saying it was cancelled.

This has design consequences. Compensations must be explicitly designed per step, tested as first-class code, and recorded in the audit trail. They also must be commutative with their own forward step when messages arrive out of order: if a CancelRoom command reaches the hotel service before the BookRoom command it is cancelling (possible with retries and parallel delivery), the service must remember the cancellation and refuse the late booking. This “tombstone” or “cancel-first” handling is one of the subtler requirements of the pattern.

Worked arithmetic on abort cost (illustrative)

Suppose a booking saga has three compensatable steps with failure probabilities that are assumed, for illustration, at 1% for payment, 2% for hotel and 8% for flight (sold-out races). Ordered payment, hotel, flight, a hotel failure (probability 0.99 x 0.02, about 0.0198) triggers one compensation, and a flight failure (0.99 x 0.98 x 0.08, about 0.0776) triggers two. The expected compensations per saga are roughly 0.0198 + 0.155, or about 0.18. Reordered flight, hotel, payment, a hotel failure after flight (0.92 x 0.02, about 0.0184) triggers one compensation and a payment failure (0.92 x 0.98 x 0.01, about 0.009) triggers two, for roughly 0.037 in total. Moving the riskiest step first cuts compensation load by almost five-fold in this made-up example. The numbers are hypothetical; the principle, to front-load the likely failures, is general. (In practice you also weigh how expensive each compensation is: a flight hold may be free to release while a captured payment is not.)

Lack of Isolation and Countermeasures

Sagas give you atomicity (eventually) and durability, plus consistency at the level of business invariants, but not isolation. Because each local transaction commits and becomes visible immediately, other transactions can see the intermediate state of a saga that might later be compensated. The ACD of ACID, without the I, is the honest summary.

Three classic anomalies follow. Lost updates: one saga overwrites changes made by another saga that ran between its steps, because it read stale data. Dirty reads: a saga or query reads data written by another saga that is subsequently compensated. Fuzzy (non-repeatable) reads: two steps of the same saga read the same data and see different values because another saga modified it in between.

Richardson and earlier authors describe countermeasures that you apply at the application level. They are not free, and you rarely need all of them.

Semantic lock. A compensatable step sets an application-level flag on the record, such as status = PENDING or locked_by = saga_id, that signals “this record is in the middle of a saga.” Other transactions either fail, wait or treat the record specially depending on the flag. The flag is cleared by the saga’s final retriable step or by its compensation. This is the single most widely used countermeasure. Its cost is deadlock-like risk: two sagas each holding a semantic lock the other needs. Break it with timeouts and a deadlock detection or abort policy.

Commutative updates. Design operations so that order does not matter: debit and credit are commutative, so a compensating credit can run after other debits without conflict. Where possible, model balances as sums of ledger entries rather than overwritten values.

Pessimistic view. Reorder saga steps so the risk of dirty reads is minimised. For example, in a cancel-order saga, raise the available inventory last, so other sagas never see stock that might be re-reserved if the cancellation fails.

Reread value. Before overwriting, reread the record and verify it is unchanged (an optimistic offline lock). If it changed, abort the saga and restart it.

Version file. Record the operations on a record so they can be reordered. This resolves out-of-order arrival of commutative operations by applying them in the canonical order.

By value. Choose concurrency mechanism by the business risk of the request. Low-risk requests run as sagas with weak isolation; high-value requests use a stricter mechanism, such as a distributed transaction with a small participant set or an approval step.

A practical consequence for read paths: if a customer can look at an order while its saga is running, the UI must show the intermediate state honestly (for example, “payment pending”) rather than implying completion. Dirty reads are often a user experience problem as much as a data problem.

Idempotency, Retries and Timeouts

Network delivery in practice is at-least-once. Brokers redeliver on missed acknowledgements, orchestrators retry on timeouts, and clients resend on ambiguous errors. Exactly-once effects are built from at-least-once delivery plus idempotent handlers, not from a magic broker flag. Every command and every compensation in a saga must be idempotent.

The standard mechanism is an idempotency key. The caller sends a stable identifier, for example sagaId:stepName. The participant, in the same local transaction that performs its business effect, inserts the key into a processed-keys table with a unique constraint. If the insert fails because the key exists, it returns the stored result of the previous execution. The uniqueness constraint and the business write must commit atomically; storing the key in a separate cache is a race. Keep keys for longer than the maximum retry window of the longest saga, plus a margin.

Timeouts are the second pillar. A call that times out is ambiguous: the participant may have executed, may be about to execute, or may never have received it. The orchestrator must therefore either retry the same command with the same key (safe because of idempotency) or query the participant’s state before deciding. A common policy is exponential backoff with jitter and a retry budget; for compensations, retry budgets should be effectively unlimited because a saga that stops compensating leaves the business in an inconsistent state. Contrast this with forward steps, where after a bounded number of attempts it is usually better to give up and begin compensating.

Participants should also enforce their own deadlines. A ReserveFunds command that arrives forty minutes late, after the orchestrator has already timed out and begun compensating, should be rejected or its effect cancelled immediately. Carry a deadline or a saga-status check in the command, and check the saga’s current state when in doubt.

Persisting Saga State and Using Workflow Engines

The orchestrator is only as reliable as its persistence. There are three common approaches.

Hand-rolled state table. A saga_instance table with columns for saga ID, type, current state, serialised payload, version and timestamps; a worker loads the row, applies a transition and writes it back with optimistic concurrency. The outgoing command is written to an outbox table in the same transaction, so state transition and message dispatch commit atomically. This is simple, transparent and works well for a handful of flows. It becomes a maintenance burden as you add timers, parallel branches, signals and versioning.

Event-sourced saga. Store each transition as an immutable event and rebuild state by replaying. You get the full history for free, which is attractive for audit, and fits naturally with the event-sourcing approach to financial audit trails described in our article on event sourcing, bitemporal data and CQRS ledgers.

Durable workflow engine. Delegate persistence, timers, retries and recovery to a platform. Three representative families exist, and their models differ meaningfully.

Temporal offers durable execution: you write the workflow as ordinary code in a general-purpose language (Go, Java, TypeScript, Python and others), and the platform persists an event history and replays it to reconstruct state after a crash. That means workflow code must be deterministic: no direct clock reads, random numbers or network calls outside activities, which are the units that perform side effects with retry policies. Temporal’s own documentation describes the saga pattern with compensations registered as the workflow progresses and executed in reverse on failure. The trade-off is operational: you run or buy a Temporal cluster, and you must manage workflow versioning carefully when changing code for in-flight executions.

AWS Step Functions describes workflows in Amazon States Language as a JSON state machine. Standard workflows are durable and can run for up to a year (per AWS documentation at the time of writing), with exactly-once state transitions, and support Retry and Catch blocks that route errors to compensating states. Express workflows are meant for short, high-volume runs, with at-least-once semantics, so compensations must be idempotent. Quotas such as execution history size apply and change over time, so check the current limits before sizing a design. Being a managed service, there is no cluster to operate, but the logic is declarative JSON or generated from tooling rather than normal code.

Camunda (version 8 uses the Zeebe engine) executes BPMN models. BPMN has first-class support for compensation: a compensation boundary event attached to a task names the handler, and a compensation throw event triggers it. The visual model is readable by non-engineers, which helps with regulated processes and human tasks, at the cost of a modelling layer and engine to maintain.

Whatever you choose, keep the same architectural boundaries: the engine sequences and retries; domain services own business rules and compensations; every service call is idempotent. Choosing an engine does not remove the need for the earlier design work, it only removes the plumbing. For a comparison with the economics of event-driven billing flows, see how usage-based billing pipelines handle idempotent events, which faces the same duplicate-delivery problem.

Outbox and exactly-once publication

Whether orchestrated or choreographed, each step must atomically change its local state and emit a message. Writing to the database and then publishing to the broker is a dual write, and a crash between the two loses either the state change or the message. The transactional outbox solves this by writing the message to an outbox table in the same local transaction, with a relay or change data capture (CDC) process publishing it afterwards. Combined with consumer-side idempotency, this yields effectively-once processing. The details, including Debezium-style log tailing and polling publishers, are covered in the article on the transactional outbox pattern with CDC.

Worked Example: Payment and Booking Saga With Failure Injection

Consider a travel booking that reserves funds, books a hotel room and books a flight seat, then issues a confirmation. We use an orchestrated saga because money moves, there are more than three steps, and finance will want an audit trail. Step classification: ReserveFunds and BookRoom are compensatable; BookSeat is the pivot, because once the airline confirms and issues a ticket, cancelling involves airline fare rules rather than a simple undo; CapturePayment and SendConfirmation are retriable after the pivot. (To keep the walk-through short, the sequence below uses the first three steps and the compensation path.)

Figure 4: Failure injection sequence. The flight call times out and is retried with the same idempotency key; when the airline rejects the booking the orchestrator cancels the room and refunds the funds.

The sequence in Figure 4 shows one injected failure. The flight service times out on the first attempt. Because the command carries the key S1:BookSeat, the orchestrator can safely retry. The retry returns a definitive rejection (sold out), the orchestrator moves into compensation, cancels the room and then refunds the funds. Each compensation is acknowledged before the next starts, and the saga ends in ABORTED, a terminal state recorded in the saga log.

The orchestrator in code

The following Python sketch shows the essential mechanics: persisted state, idempotency keys, a step list with compensations, and reverse compensation. It is deliberately minimal; a production version would add an outbox, timers and concurrency control.

from dataclasses import dataclass, field
from enum import Enum
import time

class State(str, Enum):
    RUNNING = "RUNNING"
    COMPENSATING = "COMPENSATING"
    COMPLETED = "COMPLETED"
    ABORTED = "ABORTED"
    NEEDS_HUMAN = "NEEDS_HUMAN"

class Rejected(Exception):
    """Definitive business failure: do not retry, compensate."""

class Unavailable(Exception):
    """Transient failure or timeout: outcome unknown, safe to retry."""

@dataclass
class Step:
    name: str
    action: callable        # (key, data) -> result, idempotent
    compensation: callable  # (key, data) -> None, idempotent, or None
    max_attempts: int = 4

@dataclass
class Saga:
    saga_id: str
    steps: list
    store: object           # persists SagaRecord; load/save with version check
    data: dict = field(default_factory=dict)

    def run(self):
        rec = self.store.load(self.saga_id)  # resumes after a crash
        while rec.state == State.RUNNING and rec.next_index < len(self.steps):
            step = self.steps[rec.next_index]
            key = f"{self.saga_id}:{step.name}"
            try:
                rec.data[step.name] = self._call(step.action, key, rec.data,
                                                 step.max_attempts)
                rec.next_index += 1
            except (Rejected, Unavailable):
                rec.state = State.COMPENSATING
                rec.comp_index = rec.next_index - 1
            self.store.save(rec)  # persist after EVERY transition
        if rec.state == State.RUNNING:
            rec.state = State.COMPLETED
        while rec.state == State.COMPENSATING and rec.comp_index >= 0:
            step = self.steps[rec.comp_index]
            key = f"{self.saga_id}:{step.name}:undo"
            if step.compensation:
                ok = self._retry_forever_with_cap(step.compensation, key, rec.data)
                if not ok:
                    rec.state = State.NEEDS_HUMAN   # never silently drop
                    break
            rec.comp_index -= 1
            self.store.save(rec)
        if rec.state == State.COMPENSATING:
            rec.state = State.ABORTED
        self.store.save(rec)
        return rec.state

    def _call(self, fn, key, data, attempts):
        for i in range(attempts):
            try:
                return fn(key, data)
            except Unavailable:
                if i == attempts - 1:
                    raise
                time.sleep(min(2 ** i, 30) * 0.1)   # add jitter in production

    def _retry_forever_with_cap(self, fn, key, data, cap=50):
        for i in range(cap):
            try:
                fn(key, data)
                return True
            except Unavailable:
                time.sleep(min(2 ** i, 60) * 0.1)
        return False

On the participant side, the hotel service enforces idempotency and the cancel-first rule inside one local transaction:

-- BookRoom, executed in one local transaction
INSERT INTO processed_keys(key, result) VALUES ('S1:BookRoom', NULL)
  ON CONFLICT (key) DO NOTHING;           -- 0 rows => duplicate, return stored result
SELECT 1 FROM tombstones WHERE saga_id = 'S1' FOR UPDATE;  -- cancel arrived first?
-- if a tombstone exists: refuse the booking and return CANCELLED
UPDATE rooms SET status = 'HELD', held_by = 'S1'
  WHERE room_id = $1 AND status = 'FREE';  -- semantic lock via status

The semantic lock here is the HELD status with the saga ID. Other sagas see the room as unavailable; the saga’s pivot or compensation moves it to BOOKED or FREE. A reaper job releases HELD rooms whose owning saga has terminated or exceeded a deadline, which protects against orphaned locks.

Failure injection scenarios worth testing

Injecting failures deliberately is the only way to trust a saga. A useful test matrix covers the following, and each should be automated:

1. Participant timeout, then success on retry. Verify the participant executed once (idempotency key honoured), not twice.
2. Participant rejects. Verify reverse-order compensation, and that the final state is ABORTED.
3. Orchestrator crash after sending a command but before persisting the reply. On restart, the orchestrator resends the same command with the same key; the participant returns the cached result.
4. Compensation fails repeatedly. Verify the saga lands in NEEDS_HUMAN with an alert, rather than looping silently or being marked aborted.
5. Late forward command arrives after cancellation. Verify the tombstone causes it to be refused.
6. Duplicate and reordered replies. Verify a reply for step 2 arriving after the saga moved to step 3 is ignored.
7. Concurrent saga on the same resource. Verify the semantic lock makes the second saga fail fast or wait, not interleave destructively.

Chaos tools can inject these at the network layer, but unit-level fault injection in the participant fakes is cheaper and catches most logic bugs.

Observability and Operating Sagas

An orchestrated saga is a business process, so monitor it as one rather than only as a set of HTTP calls. Four signals cover most needs.

Saga outcome rates. Count completed, aborted and needs-human sagas per type. The aborted rate reflects normal business failure such as sold-out inventory; the needs-human rate reflects defects or dependency outages and should be near zero. Alert on the latter.

Saga duration and stuck sagas. Track the distribution of end-to-end durations and the age of the oldest in-flight saga per state. A saga in COMPENSATING_PAYMENT for an hour is an incident. The saga log makes this a simple query; in a choreographed system you need to build the equivalent from events.

Step-level latency and retry counts. A rising retry rate against one participant is an early warning that a dependency is degrading, often before error-rate alerts fire.

Distributed traces keyed by saga ID. Propagate the saga ID as a trace attribute or baggage item, using the W3C Trace Context standard for propagation across services, so a single trace or a single log query reconstructs the flow, including asynchronous hops through a broker. Add the saga ID, step name and attempt number to every structured log line.

Beyond metrics, plan operational tooling: an admin view to inspect a saga, retry a stuck step, force-compensate or mark a saga resolved after manual intervention. Teams that skip this tooling find the first production incident turns into ad hoc SQL against the saga table.

Trade-offs, Gotchas, and What Goes Wrong

Compensation failure. The most uncomfortable case is a compensation that cannot complete: the payment provider is down for six hours, or the refund is rejected because the card expired. A saga that cannot compensate is stuck in a partially applied state. Design for it: make compensations idempotent and retried far longer than forward steps, escalate to a human queue after a bounded period, and record the exact residual state. Where a business cannot tolerate that residue, reconsider whether the step belongs in the saga at all, or move it behind the pivot.

Out-of-order and duplicate events. Brokers reorder across partitions, retries interleave, and a cancel can overtake the booking it cancels. Defences are the tombstone pattern, per-saga ordering by partitioning on saga ID, sequence numbers or versions on events, and state-machine guards that ignore messages that are invalid in the current state. Treat “message not applicable in this state” as a normal, logged outcome rather than an exception.

Missing isolation leaking into the product. A user refreshes a page mid-saga and sees a charge with no booking. Semantic locks and honest intermediate statuses reduce this, but some leakage is inherent. Decide explicitly which anomalies the business accepts.

Orchestrator as a distributed monolith. If every business rule migrates into the orchestrator, services become dumb CRUD endpoints and every change requires coordinated deployment. Keep decisions in the domain services.

Workflow versioning. Long-running sagas outlive deployments. A saga started under version 1 of the definition may still be running when version 2 ships. Engines such as Temporal provide versioning APIs; hand-rolled systems need explicit schema and definition versioning stored with each instance.

Choreography sprawl. Event-driven flows that began as three services quietly become seven, with implicit cycles and no owner. Revisit the decision when the number of services in a flow grows or when you catch yourself building a dashboard that tracks “where is order 123”.

Not everything needs a saga. If two pieces of data must be strongly consistent, they probably belong in one service and one database. Sagas are a tool for genuinely cross-service business transactions, not a way to justify splitting a tightly coupled domain. A saga also does not eliminate the need for reconciliation: periodic jobs that compare balances across systems catch the residual errors that no pattern prevents.

Latency and complexity cost. A saga turns one local commit into several network hops and several commits. Expect higher end-to-end latency, more states to test and a larger operational surface. That cost is the price of availability and service autonomy; pay it only where you need those.

Practical Recommendations

Start from the business invariant, not the pattern. Write down what must be true when the operation finishes, what customers may see in between, and which failures are acceptable. Then work through the design in this order.

First, classify every step as compensatable, pivot or retriable, and order them so that risky steps come first and retriable steps last. Second, specify each compensation as a business action with its own cost, side effects and failure behaviour. Third, choose the coordination style: default to orchestration for money-moving or long flows, and keep choreography for short, stable, high-fanout flows. Fourth, decide your state store, and prefer a durable workflow engine once you need timers, parallel branches, human tasks or versioning; a simple state table is fine for a handful of linear flows.

Fifth, make everything idempotent and use the outbox for message publication. Sixth, add semantic locks for any record two sagas might touch, and show intermediate statuses in the UI. Finally, build the operational tooling before you need it.

Pre-production checklist

• Every step classified; one pivot per saga; retriable steps after the pivot.
• Every command and compensation idempotent with a stable key and atomic dedup write.
• State persisted after each transition; recovery tested by killing the orchestrator mid-flight.
• Outbox or equivalent for atomic state change plus message emission.
• Tombstone handling for cancel-before-book ordering.
• Timeouts, retry budgets and deadline checks defined per step; unlimited-ish retries for compensation with human escalation.
• Semantic locks with a reaper for orphaned locks.
• Saga ID propagated in traces and logs; metrics for outcome, duration, stuck and needs-human counts.
• Failure-injection tests covering the seven scenarios above.
• Reconciliation job and an admin tool for manual resolution.

If the financial flow touches regulated money movement, remember this article discusses system design only and is not financial or legal advice; consult your compliance team on settlement and refund obligations.

Frequently Asked Questions

What is the saga pattern in microservices?

The saga pattern manages a business transaction that spans several services by splitting it into a sequence of local transactions. Each service commits its own step and signals the outcome. If a later step fails, earlier steps are undone with compensating transactions. It delivers eventual consistency and availability without distributed locks, but it gives up isolation, so you add countermeasures such as semantic locks and idempotent handlers.

What is the difference between orchestration and choreography in a saga?

In orchestration a central coordinator holds the state machine, sends commands and decides what happens next, so the flow is visible in one place. In choreography services react to each other’s events with no coordinator, which is lighter for short flows but harder to trace and test as they grow. Orchestration suits complex or money-moving flows; choreography suits simple, stable ones. Many systems mix both.

What is a compensating transaction?

A compensating transaction is a business action that semantically reverses the effect of an earlier committed step, such as refunding a captured payment or cancelling a reservation. It is not a database rollback: the original effect was visible, and the compensation is a new visible action that may carry costs or side effects. Compensations must be idempotent, retried persistently and designed per step by the owning service.

Why not just use two-phase commit?

Two-phase commit blocks participants while holding locks if the coordinator fails, ties availability to every participant, and requires each system to support a prepare and commit interface, which REST APIs, brokers and third-party SaaS rarely do. It can work inside one organisation across a few XA-capable databases. Across independently owned services, it creates coupling and availability risk, which is why sagas are the usual alternative.

How do you handle failures in a saga when compensation itself fails?

Make compensations idempotent and retry them with backoff for much longer than forward steps. If they still fail, move the saga to an explicit needs-human state, alert, and keep the exact residual state recorded so operators can finish it manually. Prevent the situation by placing hard-to-reverse steps behind the pivot, and by running reconciliation jobs that detect leftover inconsistencies.

Should I use Temporal, Step Functions or Camunda for sagas?

Pick by team fit. Temporal suits teams that want workflows as ordinary code with durable execution and are happy to run or buy the platform. AWS Step Functions suits AWS-centred teams that want a managed, declarative state machine with Retry and Catch for compensation. Camunda suits processes modelled in BPMN with human tasks and business-readable diagrams. All three remove plumbing, but none removes the need for idempotent services and well-designed compensations.

Further Reading

• Transactional outbox pattern with CDC and exactly-once event publishing for atomic state change plus message emission in each saga step.
• Event sourcing, bitemporal data and CQRS ledgers for storing saga history as an auditable event stream.
• Usage-based billing metering architecture with idempotent events for idempotency and duplicate handling in a money-adjacent pipeline.
• Garcia-Molina and Salem, “Sagas” (SIGMOD 1987), the original paper.
• Chris Richardson, Saga pattern on microservices.io, including isolation countermeasures.
• Temporal documentation on durable execution and AWS Step Functions developer guide for engine specifics and current limits.

By Riju — about