Workflow Run State Management#

Dify workflow runs transition through a defined set of states managed entirely by an event-driven persistence layer. Understanding these transitions — and where they can fail to complete — is essential for diagnosing "stuck running" incidents.

State Machine#

Workflow executions use the WorkflowExecutionStatus enum (from the graphon package) with these values:

A WorkflowRun DB record is created with status=running as soon as GraphRunStartedEvent fires 1. All subsequent status transitions are driven by events handled in WorkflowPersistenceLayer.on_event(): GraphRunSucceededEvent → succeeded 2, GraphRunFailedEvent → failed 3, GraphRunAbortedEvent → stopped 4, GraphRunPausedEvent → paused 5. When a workflow terminates with an error, the persistence layer also marks all in-flight node executions as failed via _fail_running_node_executions().

How Runs Get Stuck in running#

A run remains in running if no terminal graph event reaches the persistence layer. There are several documented root causes:

1. Pre-execution exception (SSE gap)
If an exception is raised in WorkflowAppGenerator.generate() after the WorkflowRun record is created but before the worker thread enters its try/except, no error event is published to the SSE queue and no terminal graph event fires. The client hangs until timeout and the DB record stays running. This gap was tracked in 6. Several PRs addressed different manifestations:

• PR #36973 — adds StreamEvent.ERROR to the terminal event set so the stream closes on error 7
• PR #37100 — publishes a workflow_finished event after early-failure error payloads 8
• PR #37244 — wraps conversation/message lookup in try/except to publish errors instead of silent hangs 9

2. Legacy stop signal interception
PR #37129 identified that legacy Redis stop flags could interrupt GraphEngine runs before terminal events were emitted, leaving runs in running 10.

3. Celery worker restart (warm shutdown)
Workflow execution is dispatched as Celery tasks via tier-specific tasks in async_workflow_tasks.py. If a Celery worker is restarted before an in-flight task completes, the task is lost and the WorkflowRun record stays running indefinitely — there is no built-in startup recovery scan.

The warm-shutdown handler in workflow_warm_shutdown.py mitigates this for graceful restarts: it hooks worker_shutting_down and worker_shutdown Celery signals and uses the process-local registry in active_workflow_tasks.py to track in-flight task IDs. On warm shutdown, it sets a _celery_warm_shutdown_started flag 11 that execution code checks to abort cleanly. However, this only works when Celery receives a warm-shutdown signal — an abrupt kill or OOM leaves no chance for cleanup 12.

Key Files#

Active Task Tracking#

active_workflow_tasks.py maintains a thread-safe set[str] of active Celery task IDs using an RLock 13. The active_workflow_task(task_id) context manager registers a task on entry and removes it on exit 14. The warm-shutdown handler reads get_active_workflow_task_count() to decide whether to wait 15. This registry is process-local and is reset at worker initialization via reset_active_workflow_tasks() 16 — it has no cross-process visibility and does not survive a crash.

Pause / Resume#

The resume_workflow_execution Celery task in async_workflow_tasks.py handles resumption of paused runs, reloading serialized GraphRuntimeState from the pause record and re-invoking WorkflowAppGenerator.resume(). PR #39706 (open at time of writing) targets extending this to full resume-across-rolling-updates durability 17.

Operational Guidance#

• Stuck runs after normal restarts: If warm-shutdown is configured and the worker receives SIGTERM, in-flight runs should abort and the DB records should update. If records remain running, check logs for whether _on_worker_shutdown logged remaining tasks 12.
• Stuck runs after crash/OOM: There is no automatic recovery. Orphaned running records must be identified and manually marked as failed (or via a migration/script) after the fact.
• SSE stream hangs on startup errors: If the UI shows a workflow stuck "running" without a visible error, look for exceptions raised in the generate path before the worker thread starts — these may leave both the DB record and the SSE stream in a terminal-unreachable state.