Celery Task Resilience#

Dify runs workflow execution on Celery workers spread across multiple queues. Resilience behavior varies significantly by task type: storage tasks (saving run/node records to the database) have explicit Celery retry logic, while workflow trigger tasks (the execute_workflow_* functions that actually run a workflow) treat failures as terminal and write the error state directly to the database. A warm-shutdown mechanism ensures in-flight workflows are aborted gracefully rather than killed mid-run.

Workflow Storage Tasks: Retry with Exponential Backoff#

Two storage tasks persist workflow execution results to the DB asynchronously, both on the workflow_storage queue:

• save_workflow_execution_task — saves/updates a WorkflowRun record
• save_workflow_node_execution_task — saves/updates a WorkflowNodeExecutionModel record

Both are configured with bind=True, max_retries=3, default_retry_delay=60 1 2. On any exception, they re-raise using self.retry(exc=e, countdown=60 * (2**self.request.retries)) — meaning retries are spaced at 60 s, 120 s, and 240 s 3 4. The upsert logic (SELECT before INSERT) makes these tasks idempotent across retries 5.

Workflow Trigger Tasks: No Celery Retry#

The three tier-specific trigger tasks — execute_workflow_professional, execute_workflow_team, and execute_workflow_sandbox — do not use ack_late, autoretry_for, reject_on_worker_lost, or max_retries. They differ only in the queue they target (PROFESSIONAL_QUEUE, TEAM_QUEUE, SANDBOX_QUEUE) 6.

On any exception, _execute_workflow_common catches it, marks the WorkflowTriggerLog as FAILED with an error message, and commits — no retry is attempted 7. This is an explicit design choice noted in a code comment: "Final failure — no retry logic (simplified like RAG tasks)" 8.

Because Celery defaults apply (ack_early, no reject_on_worker_lost), a worker crash mid-task means the message is acknowledged before the task finishes, so Celery does not automatically requeue it 9. The WorkflowTriggerLog row would remain in RUNNING status until a manual intervention or a future cleanup job.

Warm Shutdown: Graceful In-Flight Abort#

workflow_warm_shutdown.py wires two Celery signal handlers on worker startup via setup_workflow_warm_shutdown_handler() 10 (called from ext_celery.py line 165) 11:

• worker_shutting_down → _on_worker_shutting_down: sets a process-level threading.Event flag (_celery_warm_shutdown_started) if the shutdown is a warm shutdown 12. In-flight workflow runners poll celery_warm_shutdown_started() via CelerySignalCommandChannel and emit an AbortCommand to stop gracefully.
• worker_shutdown → _on_worker_shutdown: logs a warning if any workflow tasks are still active at final shutdown.

The abort reason written to the run is "Workflow stopped because the worker is shutting down." 13. This only activates for warm shutdowns (e.g., celery worker --shutdown-timeout); a SIGKILL bypasses it.

HITL/Pause Resume: Separate Celery Task#

For workflows paused at a Human-in-the-Loop (HITL) node, resume_workflow_execution is a separate @shared_task (no explicit retry or ack_late) that:

1. Loads the persisted WorkflowResumptionContext from the DB 14
2. Restores GraphRuntimeState and ResponseStreamFilter 15
3. Re-fetches live ORM objects (app, workflow, user) 16
4. Calls WorkflowAppGenerator.resume() 17

If WorkflowResumptionContext.loads() fails, the exception is re-raised and the task fails with no retry 18. See Workflow Resume Architecture for full pause/resume state serialization details.

Reference: Ops Trace Tasks (Explicit Retry Pattern)#

For contrast, process_trace_tasks on the ops_trace queue is an example of the explicit Celery retry pattern — max_retries and default_retry_delay are configurable via dify_config.OPS_TRACE_RETRYABLE_DISPATCH_MAX_RETRIES and OPS_TRACE_RETRYABLE_DISPATCH_DELAY_SECONDS 19. Retries only fire on RetryableTraceDispatchError; non-retryable failures increment a Redis counter at {OPS_TRACE_FAILED_KEY}_{app_id} 20.

Global Celery Configuration#

Dify's Celery setup in ext_celery.py sets task_ignore_result=True and allows per-task overrides via CELERY_TASK_ANNOTATIONS (an env-configurable dict), but sets no global task_acks_late, worker_prefetch_multiplier, or task_reject_on_worker_lost. These remain at Celery defaults.

Key Files#