Temporal Durable Workflow Orchestration Tutorial for 2026
Your payment service charged the card, then the process died before it recorded the order. Now there is an angry customer, a reconciliation ticket, and a compensation script nobody trusts. This Temporal durable workflow orchestration tutorial treats that failure as a design problem rather than an operations problem: Temporal makes your business logic durable, so a crash mid-flow resumes exactly where it stopped, with every earlier step intact.
That matters now because hand-rolled retry loops, status columns, and cron sweepers remain the biggest source of correctness bugs in distributed backends, and AI agent pipelines with long, expensive, failure-prone steps have made the problem more visible. The usual introduction stops at “write code, it survives crashes.” This one goes further and argues that the real engineering skill is managing a compatibility contract between code and history across time. You will leave with a mental model of replay, a runnable Python order workflow, correct timeout and retry settings, a plan for history limits and safe deploys, and a realistic cost model.
What this covers: durable execution mechanics, workflow versus activity boundaries, server internals, a complete runnable worker, the four timeouts and retry policy, signals and updates, Continue-As-New, versioning, testing, cost, and the failure modes that bite production teams.
Context and Background
For a decade, the default answer to “run this multi-step process reliably” was a queue plus a state table plus idempotency keys. A consumer pulls a job, performs a side effect, writes progress to a database, and hopes it does not die between the two. When it does, an engineer reconstructs partial state by hand. The saga pattern formalised the compensating actions, and our guide to saga orchestration versus choreography covers that design space, but you still owned the orchestration machinery: the timers, the retries, the “where was I” bookkeeping.
Durable execution engines change what gets persisted. Instead of storing the current state, they store the ordered history of events that produced it, and rebuild state by replaying that history through your code. Temporal descends from Uber’s Cadence project and describes itself as a durable execution platform. It has first-class SDKs for Go, Java, TypeScript, Python, .NET, and others, and it competes with newer engines such as Restate and DBOS. We compare those architectures in durable execution with Temporal, Restate and DBOS, so this tutorial stays focused on Temporal itself.
The vocabulary needs care because the field overlaps with older ideas. “Workflow engine” once meant BPMN diagrams and drag-and-drop designers. In the 2026 sense there is no separate definition language: the workflow is an ordinary function in a general-purpose language, and the runtime makes it crash-proof. That is the appeal. You write sequential code that calls a service, sleeps for an hour, and calls another, and the platform guarantees completion across process crashes, deployments, and network partitions.
The catch, which many introductions skip, is that workflow code must be deterministic so that replay can work. Everything difficult about Temporal flows from that single requirement. A queue-based design fails loudly: a message lands in a dead-letter queue and somebody gets paged. A determinism violation fails quietly and late, when an old workflow wakes up after a deploy and its history no longer matches the code. This is also why Temporal is not free: you accept a new runtime to operate or rent, a programming discipline to learn, and a deployment process that respects running executions.
Versions used here: the Python SDK (temporalio) was at 1.34.0 on PyPI when this was written, requiring Python 3.10 or newer under an MIT license. The server release page showed v1.32.1 as the latest stable build, though I could not confirm its year from the page, so treat server version numbers as something to re-check against the Temporal release notes before you pin anything. For the underlying model, the official Temporal documentation is the primary source throughout.
How Durable Execution Works: The Reference Architecture
Temporal splits your code into two kinds of functions with different rules. A workflow is deterministic orchestration code that decides what happens and in what order, and it never performs I/O directly. An activity is an ordinary function that does the real work, including HTTP calls, database writes, and email, and is allowed to fail. The workflow is the brain that must replay identically forever; the activity is the hand that touches the messy world.
Figure 1: Temporal architecture. The worker you operate hosts both workflow and activity code and long-polls the server, which routes tasks and persists event history.
The diagram shows four roles. Your client starts workflows and sends signals, queries, and updates. Your worker is a process you run that long-polls task queues and executes workflow and activity code. The Temporal Service is a cluster whose frontend is a gRPC gateway, whose history service owns each workflow’s event log, and whose matching service routes tasks to pollers. Persistence sits underneath, commonly PostgreSQL, MySQL, or Cassandra for self-hosted clusters. Your application code never touches that database. It only speaks gRPC to the frontend.
Direct answer: durable execution means a workflow function’s every decision is recorded as an event in an append-only history. When a worker dies, another worker replays that history through the same function. Completed activities are not re-run; their recorded results are fed back in. The code fast-forwards to the exact point of failure with all local variables restored.
The event history is the source of truth
Every workflow execution has an ordered event history: WorkflowExecutionStarted, ActivityTaskScheduled, ActivityTaskStarted, ActivityTaskCompleted, TimerStarted, TimerFired, and so on. When workflow code calls an activity, the SDK does not block a thread. It emits a command to the server, which records the scheduling event and suspends the workflow. The worker may evict that workflow from memory entirely. An hour-long wait costs no worker resources.
When the activity finishes, the server appends the completion event and schedules a new workflow task. Some worker picks it up, fetches the history, and re-executes your function from the top. This time the call to the activity returns the recorded result instead of scheduling anything. The mechanism is replay, and it explains the central rule: if your code issues a different sequence of commands on replay than it did originally, history and code disagree, and the task fails with a non-determinism error.
Figure 2: The replay loop. Commands that match recorded events consume the recorded result. Commands past the end of history are new, and become fresh events.
Walk through the order example. History begins with the start event. The code calls charge_payment, producing scheduled, started, and completed events with a charge ID. Then it calls reserve_inventory, adding the same trio. If the worker dies between the charge completing and the reservation being scheduled, recovery is simple: a new worker replays the recorded charge result, never calls the payment gateway for that step again, and schedules the reservation. The history is simultaneously the audit log and the recovery mechanism, with no separate checkpoint to manage.
Why this is stronger than checkpointing
A reasonable engineer might ask why not simply snapshot the process. Snapshots capture memory but not the world. If the process crashed after calling a payment API but before the snapshot, restoring an older snapshot replays the side effect. Temporal sidesteps this by making the boundary explicit: only activity results cross it, and each result is durably recorded before the workflow proceeds. The orchestration state is therefore a pure function of the history, which is exactly what makes it reproducible on any machine.
There is a subtle consequence. Because the activity result is recorded only after the activity finishes, there is a window where the activity has run in the outside world but the server has not yet recorded success: the worker could die after the payment API returned but before it reported back. In that window Temporal will retry the activity, so activities have at-least-once semantics. The platform gives you exactly-once orchestration, not exactly-once side effects. Closing that last gap is your job, with idempotency keys, a theme we return to below and in our guide to idempotency keys for payment APIs.
Task queues, stateless workers, and scaling
The task queue name is the routing key that connects a client’s start call to the workers able to serve it. Clients and workers never address each other directly. Because durable state lives in the server’s history, a worker holds nothing precious: kill it, redeploy it, or run ten copies. Scaling out means starting more worker processes on the same queue, and the matching service load-balances tasks across them. Workers also maintain a sticky cache of recently executed workflows, so most workflow tasks deliver only the new events rather than the whole history; a full replay happens on cache miss, such as after eviction or a worker restart.
That cache has an operational implication worth knowing. Replay cost grows with history length, so workflows with thousands of events are measurably more expensive to recover than small ones. The next sections show how to keep histories bounded, and why the platform sets hard limits on them.
Determinism: The Compatibility Contract Between Code and History
Most tutorials present determinism as a list of forbidden function calls. A better framing is that every running workflow has a contract with the code that will replay it: “given this recorded history, you will emit the commands I recorded.” Any code change, library upgrade, or environment difference that alters those commands breaks the contract. The forbidden-call list is just the obvious way to break it. The deployment-time way is subtler, and we cover it in the versioning section.
What you cannot do inside workflow code
Workflow code is re-executed on every replay, so it must emit identical commands every time. That excludes wall-clock reads, so use workflow.now() instead of datetime.now(). It excludes standard-library randomness and UUIDs, so use workflow.random() and workflow.uuid4(). It excludes network and disk I/O, which belongs in activities. It excludes reading mutable global state, and iterating over unordered collections whose order can differ between processes. The Python SDK runs workflows inside a sandbox that restricts many of these, but the sandbox is a safety net, not a guarantee.
The payoff is blocking-looking code that is genuinely durable. await workflow.sleep(timedelta(days=30)) sleeps for a month. The worker can restart a hundred times in between, because the sleep is a timer event waiting for a fired event. That is the qualitative difference from a job queue with a delayed message: the whole call stack waits, not a row.
# WRONG: non-deterministic inside workflow code
@workflow.run
async def run(self, order: OrderInput) -> str:
deadline = datetime.now() + timedelta(hours=1) # wall clock
token = str(uuid.uuid4()) # random
if random.random() < 0.1: # random branch
...
# RIGHT: primitives whose values are pinned in history
@workflow.run
async def run(self, order: OrderInput) -> str:
deadline = workflow.now() + timedelta(hours=1) # replay safe time
token = str(workflow.uuid4()) # replay safe id
# Genuine randomness or external reads belong in an activity.
The wrong version passes tests and works the first time in production. It fails weeks later, on replay, when the clock returns a different value than it did originally and the function takes a branch contradicting the recorded history. The right version reads time and randomness from primitives whose values are captured in history, so replay reproduces the original execution.
The four non-obvious ways teams break replay
Beyond the standard list, four patterns cause most real incidents. First, editing the order or number of awaited steps in a workflow that has open executions: inserting an activity call between two existing ones shifts every later command. Second, changing a timer duration or activity options in place, which can alter the commands emitted. Third, upgrading a dependency whose iteration order or default behaviour changes inside workflow code. Fourth, catching broad exceptions in a way that swallows the SDK’s internal control-flow exceptions, such as those used to evict or cancel a workflow.
None of these show up in a unit test against a fresh execution, because a fresh execution has no history to disagree with. The only reliable detector is replaying recorded histories against the new code. The Python SDK includes a Replayer that feeds a captured history through your current workflow definition and raises if they diverge. Export histories of representative open and recently closed workflows from production (the CLI and Web UI can both download them), store them as fixtures, and run the replayer in CI. That converts a quiet runtime failure into a deterministic build failure.
A Complete Runnable Worker Setup
Let us build an order-processing workflow: charge a payment, reserve inventory, send a confirmation, with retries and timeouts on each step, plus compensation if a later step fails. This is the shape of flow described in our patterns for transactional outbox and reliable event publishing, except here the orchestration is code rather than infrastructure. The complete setup is four small files.
Install the SDK and start a local dev server in a second terminal. The temporal server start-dev command in the Temporal CLI runs an in-memory-backed development server with the gRPC endpoint on localhost:7233 and the Web UI on port 8233 by default.
pip install temporalio
# In another terminal, with the Temporal CLI installed:
temporal server start-dev
Activities: where side effects live
Activities are ordinary functions that may do real I/O. Note the idempotency key in the payment call: because delivery is at-least-once, the external system must be told that a retry is the same logical request. We derive the key from the order ID, which is stable across retries.
# activities.py
import asyncio
from dataclasses import dataclass
from temporalio import activity
from temporalio.exceptions import ApplicationError
@dataclass
class OrderInput:
order_id: str
customer_id: str
amount_cents: int
@activity.defn
async def charge_payment(order: OrderInput) -> str:
if order.amount_cents <= 0:
# Bad input will never succeed, so do not retry it.
raise ApplicationError("amount must be positive",
type="ValidationError", non_retryable=True)
idem_key = f"charge-{order.order_id}" # stable across retries
activity.logger.info("Charging %s cents key=%s", order.amount_cents, idem_key)
await asyncio.sleep(0.2) # replace with a gateway call that accepts idem_key
return f"charge_{order.order_id}"
@activity.defn
async def refund_payment(charge_id: str) -> None:
activity.logger.info("Refunding %s", charge_id)
await asyncio.sleep(0.1)
@activity.defn
async def reserve_inventory(order: OrderInput) -> str:
await asyncio.sleep(0.2)
return f"reservation_{order.order_id}"
@activity.defn
async def send_confirmation(order: OrderInput) -> None:
await asyncio.sleep(0.1)
The workflow: orchestration only
The workflow performs no I/O. It schedules activities with explicit timeouts and a retry policy, and registers a compensation after each successful step. If a later step fails permanently, it runs the compensations in reverse order. This is the saga pattern expressed as plain try/except.
# workflow.py
from datetime import timedelta
from temporalio import workflow
from temporalio.common import RetryPolicy
from temporalio.exceptions import ActivityError
with workflow.unsafe.imports_passed_through():
from activities import (OrderInput, charge_payment, refund_payment,
reserve_inventory, send_confirmation)
RETRY = RetryPolicy(
initial_interval=timedelta(seconds=1),
backoff_coefficient=2.0,
maximum_interval=timedelta(seconds=30),
maximum_attempts=5,
non_retryable_error_types=["ValidationError"],
)
@workflow.defn
class OrderWorkflow:
@workflow.run
async def run(self, order: OrderInput) -> str:
compensations = []
try:
charge_id = await workflow.execute_activity(
charge_payment, order,
start_to_close_timeout=timedelta(seconds=10),
retry_policy=RETRY)
compensations.append((refund_payment, charge_id))
reservation = await workflow.execute_activity(
reserve_inventory, order,
start_to_close_timeout=timedelta(seconds=10),
retry_policy=RETRY)
await workflow.execute_activity(
send_confirmation, order,
start_to_close_timeout=timedelta(seconds=5),
retry_policy=RETRY)
return f"order {order.order_id} ok {charge_id} {reservation}"
except ActivityError:
for fn, arg in reversed(compensations):
await workflow.execute_activity(
fn, arg,
start_to_close_timeout=timedelta(seconds=10),
retry_policy=RETRY)
raise
Two details deserve attention. The compensation activities use the same retry policy because a refund that fails once must still eventually succeed; that is the whole point of running compensation inside a durable function. And the raise at the end re-surfaces the original failure after compensation completes, so the workflow is marked failed and visible in the UI rather than silently “succeeding” after a rollback.
The worker and the client
The worker registers workflow and activity types against a named task queue and polls forever. It is the only long-lived process you operate.
# worker.py
import asyncio
from temporalio.client import Client
from temporalio.worker import Worker
from activities import (charge_payment, refund_payment,
reserve_inventory, send_confirmation)
from workflow import OrderWorkflow
async def main() -> None:
client = await Client.connect("localhost:7233")
worker = Worker(
client,
task_queue="order-tq",
workflows=[OrderWorkflow],
activities=[charge_payment, refund_payment,
reserve_inventory, send_confirmation],
)
await worker.run()
if __name__ == "__main__":
asyncio.run(main())
# starter.py
import asyncio
from temporalio.client import Client
from activities import OrderInput
from workflow import OrderWorkflow
async def main() -> None:
client = await Client.connect("localhost:7233")
result = await client.execute_workflow(
OrderWorkflow.run,
OrderInput(order_id="A-1001", customer_id="C-42", amount_cents=4999),
id="order-A-1001",
task_queue="order-tq",
)
print(result)
if __name__ == "__main__":
asyncio.run(main())
Run python worker.py in one terminal and python starter.py in another. Kill the worker mid-run and restart it. The workflow resumes without re-charging the card, because the completion event is already recorded. Open the Web UI at localhost:8233 and you can read the full event history of order-A-1001, including every timer and command the workflow emitted. The workflow ID is also your deduplication handle: while an execution with that ID is running, a second start with the same ID is rejected by default, and the Workflow ID reuse and conflict policies control what happens after it closes.
Figure 3: One order from start to finish. Each arrow crossing the server becomes a durable history event.
Notice what you did not write: no state table, no manual “where was I” recovery logic, no dead-letter plumbing in the orchestration layer. Notice also what you still must write: the idempotent payment call. The Python SDK recommends threaded activities as a starting point for blocking libraries, with async def activities appropriate when every call inside is non-blocking. Mixing a blocking client into an async def activity stalls the worker’s event loop and will surface as unexplained timeouts.
Timeouts, Retries, and Long-Lived Workflows
Four timeouts that are not interchangeable
Activities have four timeout settings, and choosing them well is most of the skill. Schedule-to-start limits how long a task waits in the queue before a worker picks it up, which makes it a worker-starvation detector. Start-to-close limits a single attempt’s execution time, and it is the one you almost always set. Schedule-to-close caps total wall-clock time across all retries. Heartbeat timeout applies to long activities that call activity.heartbeat(); if the server does not receive a heartbeat within the window, the attempt is considered failed and is retried. An activity needs at least one of start-to-close or schedule-to-close set, and the official guidance is to set start-to-close in nearly every case.
A common mistake is setting only schedule-to-close and expecting per-attempt protection. A single hung attempt can then consume the entire budget before anything retries. Set start-to-close to a realistic single-attempt budget, and let the retry policy plus an optional schedule-to-close govern the total. Be aware that Temporal relies on start-to-close to detect lost activity tasks, so omitting it removes the mechanism that recovers from a worker dying mid-task.
For slow work such as a multi-minute media conversion or a large import, add heartbeats. A worker that dies silently is detected at the heartbeat interval instead of after the whole start-to-close window. Heartbeats can also carry a progress payload, so the retried attempt can read the last heartbeat details and resume rather than start from zero. Note that SDKs throttle heartbeats, and on Temporal Cloud only heartbeats that actually reach the server are billed as actions.
Figure 4: Each attempt is bounded by start-to-close and heartbeat timeouts. Failures consume retry budget, and backoff waits are timers recorded in history.
Retry policy: the default is unlimited
The retry policy has five fields: initial interval, backoff coefficient, maximum interval, maximum attempts, and a list of non-retryable error types. The documented activity defaults are a one-second initial interval, a coefficient of 2.0, a maximum interval of 100 times the initial interval, and unlimited attempts. That last default surprises people. An activity that fails because a downstream dependency is permanently broken will retry indefinitely, quietly, with backoff capped at 100 seconds. For many systems that is the right default, because transient outages self-heal and the workflow simply waits. But you should choose it deliberately, and pair any unbounded policy with alerting on activity attempt counts.
Workflows themselves have no retry policy by default, and the guidance is to retry at the activity level because it is cheaper and finer-grained. Retries of workflow tasks (the replay step) do happen automatically with backoff, which is why a determinism error makes a workflow appear “stuck” rather than failed: the task keeps failing and retrying until you deploy a fix. Use non-retryable errors sparingly, mainly for bad input where a retry cannot help, as the ValidationError in our example does.
Retry timing is computed by the server and the waiting happens off the worker. A worker can crash and restart during a 30-second backoff with no effect on the schedule. That is why Temporal can promise retries that survive deploys: retry state is durable in the same way the workflow is.
Signals, queries, and updates
A running workflow is addressable. Signals push data in asynchronously, for example “customer cancelled”. Queries read state synchronously without mutating it or adding history events. Updates are validated, synchronous mutations that return a result, with an optional validator that rejects bad requests before they are written to history. Together they make a workflow a durable, addressable entity closer to an actor than a script.
@workflow.defn
class ApprovalWorkflow:
def __init__(self) -> None:
self._cancelled = False
self._approved: str | None = None
@workflow.signal
def cancel(self) -> None:
self._cancelled = True
@workflow.query
def is_cancelled(self) -> bool:
return self._cancelled
@workflow.update
async def approve(self, approver: str) -> str:
self._approved = approver
return f"approved by {approver}"
@approve.validator
def _check(self, approver: str) -> None:
if not approver:
raise ValueError("approver required")
@workflow.run
async def run(self) -> str:
# Wait up to 7 days for approval or cancellation, durably.
try:
await workflow.wait_condition(
lambda: self._cancelled or self._approved is not None,
timeout=timedelta(days=7))
except TimeoutError:
return "expired"
return "cancelled" if self._cancelled else f"approved {self._approved}"
The seven-day wait costs no worker memory and survives any number of deploys, which is exactly what a queue-plus-cron implementation struggles to match. Be careful with handlers that run activities concurrently with the main run: if an update handler can interleave with the main function across awaits, guard shared state with asyncio.Lock and let in-flight handlers finish before the workflow returns. Child workflows are the decomposition tool: one workflow starts and awaits another, each with its own history and failure scope.
History limits and Continue-As-New
Event history is not free, because replay cost grows with it. The documentation states that a workflow execution’s event history is limited to 51,200 events or 50 MB, with warnings starting at 10,240 events or 10 MB. Exceeding the hard limit terminates the execution. There are additional per-execution caps, for example on the number of signals and updates, and a default maximum of 2,000 pending activities, child workflows, signals, and cancellation requests per workflow. Every loop that can run unbounded must therefore be designed with a way out.
Continue-As-New is that way out. The workflow atomically closes its current run and starts a fresh execution with the same workflow ID and an empty history, carrying forward only the state you pass. In the Python SDK you can ask workflow.info().is_continue_as_new_suggested() and roll over when the server hints that history is large, rather than guessing a magic iteration count.
@workflow.defn
class BillingLoop:
@workflow.run
async def run(self, cycles_done: int) -> None:
while True:
await workflow.sleep(timedelta(days=30))
await workflow.execute_activity(
charge_subscription,
start_to_close_timeout=timedelta(seconds=30),
retry_policy=RETRY)
cycles_done += 1
if workflow.info().is_continue_as_new_suggested():
workflow.continue_as_new(cycles_done)
Two details trip people up. Pending signals and updates must be drained or handled before you continue as new, or they can be lost to the old run. And a Continue-As-New counts as a workflow start for billing on Temporal Cloud, so rolling over every few events is wasteful in both replay cost and actions. A monthly billing loop produces tiny histories; a workflow that processes a stream of signals can reach the limit in hours.
Versioning: Deploying Code While Workflows Are Running
Versioning is where the compatibility contract becomes operational. If you deploy a change that alters the command sequence while executions are open, replay fails for exactly those executions. Short-lived workflows (seconds to minutes) barely notice, because they finish before the old code disappears. Workflows that live for days, weeks, or years are the real problem, and they are also the main reason to use Temporal in the first place.
The official documentation describes three approaches. Patching uses workflow.patched("my-patch") to branch: new executions take the new code path, while open executions replay the old path. The call records a marker in history, and replay fails if code expecting the marker doesn’t find it. Once no open execution uses the pre-patch path, you switch to workflow.deprecate_patch("my-patch"), and later remove it after pre-patch workflows have aged out of retention. Workflow cutover copies the workflow under a new name such as OrderWorkflowV2 and registers both types. It is simple and safe for new starts but does nothing for running ones.
Worker Versioning is the approach the current docs recommend as the default for most production deployments. You tag workers with a Deployment Version, which is a deployment name plus a build ID, and the server routes tasks so a given workflow keeps running on workers that carry its code. Each deployment has one Current Version and optionally a Ramping Version that receives a percentage of new starts, so you can canary a release. Workflows choose a behaviour: Pinned executions stay on the version where they began and need no patching, which suits short workflows or long ones that use Continue-As-New. Auto-Upgrade executions move to the target version as you roll out code, which means you must still keep the code replay-safe with patching. Pinned workflows imply running several worker versions at once (“rainbow” deployments), so budget the operational overhead. An older experimental Worker Versioning scheme from before 2025 was slated for removal from the server in March 2026, so ignore tutorials written around it.
My opinionated recommendation: use Pinned plus Continue-As-New for entity-style workflows you can periodically roll over, and Auto-Upgrade plus patching only for workflows that genuinely need to pick up new logic mid-flight. Whichever you choose, the replay test in CI remains the safety net.
Testing and Observability
Temporal’s testing story is a strength. The Python SDK ships a time-skipping test environment: WorkflowEnvironment.start_time_skipping() fast-forwards timers, so a test of a 30-day sleep completes in milliseconds. Activities can be mocked by registering stub functions with the same name and signature on the test worker, and ActivityEnvironment lets you unit-test an activity in isolation, including its heartbeats and cancellation. Add the Replayer against stored histories and you cover logic, timing, and compatibility.
# test_order.py (pytest-asyncio)
from temporalio.testing import WorkflowEnvironment
from temporalio.worker import Worker, Replayer
from temporalio import activity
@activity.defn(name="charge_payment")
async def charge_payment_mock(order) -> str:
return "charge_test"
async def test_happy_path():
async with await WorkflowEnvironment.start_time_skipping() as env:
async with Worker(env.client, task_queue="t", workflows=[OrderWorkflow],
activities=[charge_payment_mock, ...]):
result = await env.client.execute_workflow(
OrderWorkflow.run, sample_order, id="t-1", task_queue="t")
assert "ok" in result
async def test_replay_prod_histories():
await Replayer(workflows=[OrderWorkflow]).replay_workflow(history_from_prod)
For observability, the Python SDK supports OpenTelemetry tracing and metrics. Watch four signals: schedule-to-start latency (are workers keeping up), workflow task failure rate (determinism problems show here), activity attempt counts (silent retry storms), and history length of the longest-running executions (approaching the limit). A dashboard with those four catches most production surprises before customers do. Our guide to retries, backoff and jitter explains how to size the policies you feed into those activities so a retry storm does not take down the dependency you are protecting.
Cost and Capacity: Self-Hosted Versus Temporal Cloud
You have two ways to run the server. Self-hosting is free as software, but you operate the frontend, history, matching, and worker services plus a database that carries every history write. Temporal Cloud is billed on usage. Per the pricing documentation at the time of writing, the Developer plan has no monthly minimum beyond 10 percent of usage spend, Business has a $500 per month minimum or 10 percent of usage spend (whichever is greater), and Enterprise and Mission Critical are annual contracts priced through sales. Actions on pay-as-you-go cost $50 per million at low volume, with Business and higher tiers stepping down in bands to $25 per million between 100 million and 200 million. Storage is billed per GB-hour: $0.042 for active and $0.00105 for retained. Verify these against the live pricing page before budgeting, since they change.
What counts as an action matters more than the headline rate. According to the Cloud documentation, starts, activity starts and each retry, timers (including implicit timeout timers), signals, queries, accepted or rejected updates, and child workflow starts each bill, and a Continue-As-New counts as a workflow start. Local activities within one workflow task collapse into a single action. The practical implication: a chatty design with many small activities, aggressive retries, and polling queries costs more than a coarse one.
Illustrative worked example (my estimate, not a benchmark): suppose the order workflow above costs about 5 actions per successful order: one start, three activities, and roughly one more for timers and overhead, with a few percent retries on top. At 1 million orders per month that is about 5 million actions. At $50 per million that is roughly $250 of action spend, which is below the $500 Business minimum, so the minimum binds. At 20 million orders the same arithmetic gives 100 million actions, and tiered pricing applies. Use this as a template: count the actions in your workflow from a real history, multiply, then add storage for your retention window.
Capacity on a self-hosted cluster is dominated by persistence write throughput, since each command and event is a durable write, plus history-shard sizing set at cluster creation, which is hard to change later. I have no verified public benchmark for a specific throughput figure to quote here, and any number you read without a stated workload and hardware should be treated sceptically. Load-test your own workflow shape. For small teams the break-even usually favours Cloud until action volume is large or data-residency rules force self-hosting.
Trade-offs, Gotchas, and What Goes Wrong
Temporal is not the right answer for everything, and the failure modes are specific. Non-determinism errors after deploys are the signature problem, covered above. They present as stuck workflows, not crashes. The mitigations are replay tests, Worker Versioning, and patching discipline.
Payload and history bloat. Passing large objects between activities writes them into history, and they are replayed on cache miss. Pass references (an object-store key) instead of blobs, and enable a payload codec or data converter if you need encryption or compression. The Python SDK page marks external storage support as experimental, so do not build on it blindly.
Activities are at-least-once. Teams assume Temporal makes side effects exactly-once and double-charge a customer after a worker crash. It does not. Every non-idempotent external call needs an idempotency key derived from stable identifiers, as in our payment example.
Hot workflows. A single workflow ID that receives very high signal rates becomes a serial bottleneck, because its history is appended one event at a time. Shard hot entities or batch signals. The documented caps on signals and updates per execution are a hint that this design is outside the intended envelope.
Silent unlimited retries. The default retry policy never gives up. A broken credential then turns into a growing pile of retrying activities and, on Cloud, a growing action bill, with no failure to alert on. Add attempt-count alerts or a finite cap.
Using Temporal as a queue or a database. It is a coordination engine for stateful processes. Fire-and-forget fan-out of millions of independent messages is better served by a log or queue; see Kafka versus RabbitMQ versus SQS for task queues. Likewise, do not query workflow state for analytics; project it into a real store.
Learning-curve cost. Engineers must internalise that workflow code is replayed. Code review checklists, a shared “what is allowed in a workflow” document, and the sandbox restrictions reduce but never eliminate mistakes. If your processes are three steps that finish in a second, a transactional outbox and a retry policy may be simpler.
Practical Recommendations
Start with the smallest real workflow that spans a failure you care about, such as order fulfilment or an onboarding flow with a human approval. Keep workflow functions thin: orchestration, branching, and waiting only. Put every network call, database write, and clock-dependent decision in an activity, and write each activity as if it will run twice.
Set start-to-close on every activity, add heartbeats to anything longer than a minute, and choose a retry policy on purpose, including a ceiling or an alert. Use workflow IDs derived from business keys, such as order-A-1001, so starts deduplicate naturally. Roll long-lived loops over with Continue-As-New before history approaches the documented warning thresholds rather than the hard limit.
Decide your versioning strategy before the first production deploy, not after the first stuck workflow. Wire the replay test into CI with histories exported from production. Instrument schedule-to-start latency, workflow task failures, activity attempt counts, and maximum history length.
A short checklist before go-live:
- Workflow code contains no I/O, wall-clock reads, or unseeded randomness.
- Every activity has a start-to-close timeout and an idempotency strategy.
- Retry policies are deliberate, and non-retryable errors are classified.
- Unbounded loops use Continue-As-New, and payloads carry references not blobs.
- A replay test runs in CI against exported histories.
- A versioning approach (Pinned, Auto-Upgrade with patching, or cutover) is documented per workflow type.
- Alerts exist on task failures, attempt counts, and history size.
Frequently Asked Questions
What is Temporal durable execution in simple terms?
Durable execution means your function’s progress is recorded as an append-only history of events, so a crash does not lose its place. When a worker dies, another worker replays the history through the same code, skips steps whose results are already recorded, and continues from the point of failure. You write ordinary sequential code, and the platform provides retries, timers, and recovery. The trade is that workflow code must be deterministic so replay reproduces the same commands.
What is the difference between a Temporal workflow and an activity?
A workflow is deterministic orchestration code that decides what happens next and is replayed from history after failures. It must not perform I/O or read nondeterministic values. An activity is a normal function that performs side effects such as API calls and database writes, and it may fail and be retried. The workflow schedules activities and receives their recorded results. Keeping that boundary clean is what keeps replay safe and side effects controlled.
Does Temporal guarantee exactly-once execution?
It guarantees exactly-once orchestration of workflow logic through its event history, but activities are effectively at-least-once. If a worker dies after an activity’s side effect but before the result is recorded, the activity retries. To get an exactly-once outcome in the outside world, pass an idempotency key derived from stable identifiers such as the order ID, and make the external system honour it, as with payment gateways.
What are Temporal’s history limits and how do I avoid them?
The documentation states that a workflow execution’s event history is limited to 51,200 events or 50 MB, with warnings beginning at 10,240 events or 10 MB. Exceeding the limit terminates the execution. Avoid it by calling Continue-As-New, which closes the run and starts a new one with empty history, carrying forward only needed state. In Python, check is_continue_as_new_suggested() and roll over when it returns true.
How do I deploy new code without breaking running workflows?
Use Worker Versioning, patching, or a workflow cutover. Worker Versioning pins running workflows to the build that started them or lets them auto-upgrade. Patching with workflow.patched() branches old and new logic by recording a marker. A cutover registers a new workflow type for new starts only. Whichever you choose, replay stored production histories against the new code in CI before release.
Is Temporal free, and when should I choose Cloud?
The server is open source and free to self-host, but you pay to operate a cluster and its database. Temporal Cloud bills per million actions plus storage, with a Business plan minimum of $500 per month as of the pricing page I checked. Choose Cloud when you want to avoid running persistence and upgrades, and self-host when volume is very high or data residency demands it. Confirm current prices first.
Further Reading
- Durable execution with Temporal, Restate and DBOS: how the architectures differ.
- Saga pattern: orchestration versus choreography: compensation design that Temporal workflows implement as code.
- Retries, backoff, jitter and retry budgets: sizing the policies your activities use.
- Payment API idempotency keys: closing the at-least-once gap.
References
- Temporal Documentation, “Workflow Execution Limits and Event History” and “Retry Policies”: https://docs.temporal.io/
- Temporal Documentation, “Versioning (Python)” and “Worker Versioning”: https://docs.temporal.io/develop/python/workflows/versioning
- Temporal Cloud, “Pricing” and “Actions”: https://docs.temporal.io/cloud/pricing
- temporalio on PyPI (Python SDK, MIT license): https://pypi.org/project/temporalio/
- Temporal server releases: https://github.com/temporalio/temporal/releases
Last Updated: October 2026
By Riju — about
