The Saga Pattern for Distributed Transactions

A saga manages a business transaction that spans several services, each with its own database, without a distributed lock or global transaction. It's a sequence of local transactions; if a step fails, the saga runs compensating transactions to undo the steps that already succeeded. Sagas trade atomic isolation for availability and loose coupling.

The problem: one booking, three databases

A user books a trip: reserve a flight, book a hotel, rent a car, three services with three databases. The flight books, the hotel fails, and without coordination the user has a flight and no hotel. The system must either finish every step or undo the ones that completed. A single-database transaction would lock rows and commit or roll back everything, but the Order Service can't lock rows in the Payment Service's database, and two-phase commit blocks, adds latency and hurts availability.

A saga splits the work into local transactions, one per service, each committed immediately. If a later step fails, the saga runs compensating transactions for the completed steps, in reverse order.

So every service exposes a forward action and its compensation: Flight has bookFlight / cancelBooking, Hotel has reserveRoom / cancelReservation, Payment has chargePayment / issueRefund, Inventory has decrementStock / returnStock.

Example: placing an order

Placing an order touches Orders, Payments, Inventory and Shipping. Each step commits locally; failures trigger compensations in reverse:

If inventory reservation fails at step 3, the saga refunds the payment and cancels the order. The end state is consistent, but in between, other users could see a charged-but-unconfirmed order: sagas give up isolation.

Orchestration vs choreography

Choreography: Order Service creates the order and publishes OrderCreated; Payment listens, charges, publishes PaymentCharged; Inventory listens, decrements stock, publishes InventoryUpdated. If payment fails, Payment publishes PaymentFailed and Order Service cancels the order. Orchestration: Order Service acts as the coordinator: it calls Payment and waits; on success calls Inventory; if Inventory fails after Payment succeeded, it calls Payment to refund. Orchestration is simpler for most systems (the flow lives in one place, debugging follows one path, state is centralized); choose choreography only when maximum decoupling justifies the complexity.

class CheckoutSaga:
    """Orchestrated saga: run steps in order; on failure run completed steps' compensations in reverse."""

    def __init__(self, order_id, payments, inventory, orders, store):
        self.order_id, self.store = order_id, store          # store persists progress for crash recovery
        self.steps = [
            ("charge",    lambda: payments.charge(order_id, key=f"{order_id}:charge"),
                          lambda: payments.refund(order_id, key=f"{order_id}:refund")),
            ("reserve",   lambda: inventory.decrement(order_id, key=f"{order_id}:reserve"),
                          lambda: inventory.restock(order_id, key=f"{order_id}:restock")),
            ("confirm",   lambda: orders.mark_confirmed(order_id),
                          lambda: None),
        ]

    def run(self):
        done = []
        for name, action, compensate in self.steps:
            try:
                action()                                     # each call is idempotent (keyed)
                done.append((name, compensate))
                self.store.save(self.order_id, completed=[n for n, _ in done])
            except Exception:
                for _, comp in reversed(done):               # undo in reverse order
                    retry_until_success(comp)                # compensations must eventually succeed
                self.store.save(self.order_id, status="FAILED")
                return "FAILED"
        return "CONFIRMED"

Trade-offs

• Temporary inconsistency is visible. After step 1 the user has a booked flight and no hotel yet, and might see the flight in "My Trips" before the hotel books or before compensation cancels it. Most business flows accept this: users understand "processing" states that resolve in seconds, given clear status messages.
• Compensation is not a rollback. It's a new business action that reverses a completed one (refund a charge, release a hold, cancel a booking). The original booking still happened and leaves a record.
• Everything must be idempotent. Retries will repeat steps and compensations; calling cancelBooking twice for the same booking must be safe, and unique idempotency keys prevent duplicate refunds. If the network fails during a compensation, the orchestrator retries until it succeeds. See idempotency.

Interview walk-through (e-commerce checkout): Order Service orchestrates: create the order (PENDING), charge payment, decrement inventory; if inventory fails after payment, refund. Name each service's compensation endpoint, explain retries with idempotency keys for compensations, and say what users see: the order appears in history and moves to "failed" a couple of seconds later if payment fails. Use sagas when operations span services with separate databases, availability matters more than instant consistency, and every step has a real compensation (checkout, travel booking, fulfillment). Avoid them when you need perfect consistency at every moment, when a step can't truly be undone, or when users must never see intermediate states. Most modern distributed systems use sagas.

Making sagas reliable

• Persist saga state so a crash mid-saga resumes rather than leaving it half done (workflow engines handle this).
• Idempotent steps and compensations: every step can be retried, and compensating twice must be safe. See idempotency.
• Reliable messaging: publish step results with the outbox pattern.
• Compensations must be possible: an email already sent can't be unsent. Order steps so irreversible actions come last, or use "pivot" steps after which the saga only moves forward.
• Semantic locks for isolation: mark records as PENDING so other operations treat them carefully until the saga finishes.
• Timeouts: a step that never answers must eventually trigger compensation.

Sagas vs two-phase commit

See two-phase commit for the alternative.