Workflow Recovery#

Workflow recovery is the mechanism by which DBOS re-executes workflows that were left in a PENDING state — typically because a process crashed or restarted before the workflow completed. Recovery re-runs the workflow from its last committed step, relying on idempotency stored in the system database.

Startup Recovery#

During DBOS._launch(), DBOS queries the system database for all PENDING workflows belonging to the current executor ID and application version, then submits a startup_recovery_thread to the background thread pool. This thread retries recovery for each pending workflow until it succeeds or encounters a non-retryable error.

This path only runs for local deployments — when neither a Conductor key nor the DBOS Cloud flag is set 1.

The public API recover_pending_workflows(executor_ids) provides the same capability on demand and returns WorkflowHandle objects for each recovered workflow.

Core Recovery Execution: execute_workflow_by_id#

All recovery paths converge on execute_workflow_by_id() in _core.py. Its steps:

1. Fetch status from DB — retrieves the stored workflow_status row, including name, inputs, class_name, config_name, and queue_name 2.
2. Validate workflow ID — checks that the workflow ID is not empty or whitespace-only. If invalid, immediately marks the workflow ERROR with a DBOSRecoveryError stating "Cannot recover a workflow with an empty or whitespace-only ID" and raises the error. This guard prevents poisoned rows (workflows with invalid IDs that may have been inserted before validation was added) from proceeding through the recovery process.
3. Deserialize inputs — deserializes the original arguments; if deserialization fails, immediately marks the workflow ERROR rather than leaving it stuck in PENDING 3.
4. Look up the workflow function — resolves status["name"] against dbos._registry.workflow_info_map; raises DBOSWorkflowFunctionNotFoundError if not found 4.
5. Inject class context — prepends the correct class instance or class object to args (see below) 5.
6. Dispatch — calls start_workflow (or its async variant) with is_recovery=True.

For queued workflows, _recover_workflow() first clears the queue assignment and returns a handle without re-dispatching 6.

Instance Lookup via Composite Key (class_name/config_name)#

Workflows belonging to a DBOSConfiguredInstance store both class_name and config_name in the workflow_status table. During recovery, execute_workflow_by_id constructs the composite key "{class_name}/{config_name}" and looks it up in dbos._registry.instance_info_map 7. If the key is absent, DBOSWorkflowFunctionNotFoundError is raised.

For non-configured class methods (static/class methods without a config_name), the class object is resolved from dbos._registry.class_info_map by class_name alone, and is only prepended to args when the function type is not DBOSFuncType.Static 8.

Automatic Instance Registration#

Instances are registered automatically when a DBOSConfiguredInstance subclass is instantiated. The chain is:

1. DBOSConfiguredInstance.__init__ calls DBOS.register_instance(self).
2. DBOSRegistry.register_instance computes the composite key and stores the instance in instance_info_map.

A warning is logged if registration happens after DBOS.launch(), since recovery may already be in progress 9. Registering two different objects under the same key raises an exception 10.

All DBOSConfiguredInstance objects must be created before DBOS.launch() is called.

Retry Logic for Unresolved Functions#

startup_recovery_thread uses a retry loop with a key distinction 11:

This retry behavior accommodates late registration — for example, when a class instance is constructed on a background thread shortly after launch().

Once a workflow exceeds its max_recovery_attempts (default: 100, configurable via @DBOS.workflow(max_recovery_attempts=N)) 12, its status is set to MAX_RECOVERY_ATTEMPTS_EXCEEDED and it will no longer be picked up for recovery.

Key Files#