What Is the Orchestrator Pattern? Workflow and Recovery Examples

Key takeaways

• An orchestrator owns workflow transitions while participating services retain their domain rules.
• Durable state must include enough information to resume and reconcile work after interruption.
• A timeout can leave an outcome unknown; retries need stable operation identity and participant support.
• Compensation, manual intervention, and workflow-version changes are part of the design.

The orchestrator pattern assigns coordination of a multi-step process to an explicit controller. The orchestrator tracks progress, requests operations from participants, and decides what should happen next when work succeeds, fails, or remains unresolved.

Imagine an order that reserves inventory, authorises payment, and creates a fulfilment request. The last request times out. The coordinator now has a harder problem than simply choosing whether to retry: fulfilment may already have created the request and lost its response.

The pattern becomes useful when those situations have a defined representation and recovery path. A sequence of service calls is only the beginning.

Separate Workflow Decisions From Domain Decisions

The orchestrator decides that payment should follow a successful reservation. Inventory decides whether a reservation can be made. Payment decides whether an authorisation is valid. Fulfilment decides whether it can accept the requested work.

The coordinator does contain business logic: the rules of the process itself. The boundary is between that cross-service process and the detailed rules owned by each participant. Copying stock-allocation logic into the coordinator would create two places that need to agree on inventory behaviour.

The same workflow can use synchronous calls, queued commands, or a mixture. Compared with choreography, the defining property is that one logical coordinator owns the next-step decisions.

Model States That Reflect What Is Known

A state machine can make the order’s transitions explicit:

State	Meaning	Possible next action
Awaiting reservation	Inventory has not yet confirmed a result.	Process the result or apply the timeout policy.
Awaiting payment	Reservation exists; authorisation is pending.	Accept the result or reconcile an uncertain attempt.
Awaiting fulfilment	Payment is authorised; fulfilment is unresolved.	Await, reconcile, or retry safely.
Compensating	Recovery actions remain outstanding.	Continue or escalate recovery.
Accepted	Required acceptance conditions are satisfied.	Publish the outcome.
Cancelled	Required cancellation conditions are satisfied.	Close the workflow.
Needs review	Automation cannot safely choose the next action.	Request an authorised decision.

These are illustrative states, not a universal order model. An accepted order may still need later shipping and delivery workflows.

Store the workflow identifier, definition version, operation identifiers, participant references, deadlines, and relevant results. A state label alone is insufficient if recovery cannot identify the reservation or payment it needs to inspect.

Design for a Crash Between Any Two Steps

Suppose the payment service authorises the payment, then the coordinator crashes before recording the response. After restart, its stored state still says payment is pending.

Repeating the call blindly can create another effect. Recording success before making the call creates the opposite problem: the workflow may claim that work happened when it never did. A local state write cannot atomically cover an arbitrary remote operation.

The design needs stable operation identity, participant-side duplicate handling where supported, and a way to query or reconcile uncertain results. A workflow engine can manage durable scheduling and history, but its guarantees must be understood at the boundary with external effects.

For messaging, record the intent to dispatch a command reliably and handle repeat delivery. Also protect workflow transitions from concurrent workers, using mechanisms such as conditional updates or engine-managed execution ownership.

Make Retry Identity Match the Intended Operation

AWS’s discussion of idempotent APIs explains how a caller-provided identifier can let a service recognise repeated intent. For the fulfilment request, retries should retain the identifier of that same logical operation.

A new transport attempt is not necessarily a new business operation. Generating a fresh idempotency key on every timeout can defeat the protection. Conversely, reusing the same key for a genuinely different request can conflict with the provider’s rules.

Check the participant’s retention window, parameter-matching requirements, and response behaviour. Keep retries bounded by both attempts and the workflow deadline. A rejected address needs correction; repeatedly sending it does not turn it into a transient problem.

Treat Compensation as Work That Can Fail

If fulfilment definitively rejects the order, the workflow may need to cancel payment authorisation and release the inventory reservation. Microsoft’s compensating transaction guidance treats such recovery as application-specific work that may itself need retries or intervention.

Compensation does not erase history. Cancelling an authorisation differs from refunding a captured payment. An instruction sent to a warehouse may need a cancellation before physical dispatch becomes irreversible. Recovery order should follow those business constraints rather than mechanically reversing every step.

Keep the workflow in a recovery state until the required outcomes are confirmed. If releasing inventory succeeds but cancelling payment remains uncertain, a simple “cancelled” label conceals unfinished responsibility.

Record compensation identifiers and results with the same care as forward operations. A restart during recovery should not discard the knowledge of which corrective actions already completed.

Make Timeouts and Late Results Agree

A deadline can expire while a participant is still working. The coordinator must define what happens if success arrives after it has begun cancellation.

For the order, a late fulfilment success might require cancelling the newly created fulfilment request. Ignoring the response could leave goods moving for an order the customer believes was cancelled.

Use explicit transition rules for late and duplicate results. Identify the operation and workflow version before applying an incoming response. Avoid allowing a stale callback to move a completed cancellation back into the normal acceptance path.

Give Operators Safe Actions

A support view should show the current state, the last confirmed result, pending work, deadlines, and the reason progress stopped. Link diagnostic evidence using correlation identifiers, while keeping the business record independent of whether a sampled trace survives.

Manual actions need defined preconditions. “Resume” should mean continuing from a known state under specific rules, not starting the entire order again. Record who requested the action and what decision justified it. Prefer guarded commands over direct database edits that bypass transition checks.

The coordinator is also infrastructure that needs availability, capacity planning, and recovery. Logical centralisation does not require one process, but adding workers must not allow them to issue conflicting transitions for the same workflow.

Plan for Workflows That Outlive a Deployment

A workflow created yesterday may resume under today’s software. If the sequence or meaning of a state has changed, decide whether old instances finish on their original definition or undergo an explicit migration.

Include pending timers, callbacks, and compensation in that decision. Deploying a new handler is not enough if existing participant responses no longer match what it expects.

Before release, the workflow’s evidence should cover crashes after remote success, repeat messages, late replies, failed compensation, and recovery from stored state. These cases reveal whether the coordinator can preserve meaning across interruption.

The orchestrator pattern earns its place when an unfinished order remains understandable after the successful path has broken down. It gives the process a durable owner, while making uncertainty and recovery visible enough for the system or an operator to act responsibly.

A durable workflow needs to explain what it knows, what remains uncertain, and which action is safe to attempt next.