Parallel Workflow Execution#

Dify's workflow engine (graphon) runs parallel branches by spawning worker threads from a configurable thread pool. Three distinct classes of runtime hazards have produced production bugs: Flask application context loss in worker threads, in-place credential mutation that corrupts shared ToolRuntime state, and variable pool / run-context propagation from parent graphs to child sub-graphs.

Key entry points:

• WorkflowEntry.__init__ — bootstraps the GraphEngine, captures the current execution context, and layers on execution limits and observability.
• WorkflowAppRunner.run — constructs the VariablePool, delegates graph initialization to _init_graph(), then drives WorkflowEntry.
• DifyNodeFactory — creates every node in the graph; child engines for iteration/loop sub-graphs call DifyNodeFactory.with_runtime_state() to obtain a factory scoped to the child's state.

Flask Application Context Loss in Worker Threads#

Symptom: When an Iteration or parallel branch node runs LLM or HTTP-request sub-nodes on worker threads, those threads raise "Working outside of application context." 1. The error is reproducible with is_parallel: true and multiple concurrent workers. 2

Root cause: Python's contextvars and Flask's app context are thread-local; worker threads spawned by the graph engine's thread pool start with a blank slate.

How Dify solves it:

1. capture_current_context() is called on the parent thread at engine construction time. It calls contextvars.copy_context() and — via a registered Flask adapter — captures the Flask app context and flask_login user proxy.
2. The snapshot is stored on GraphRuntimeState._execution_context immediately before the engine starts 3.
3. Graphon's worker pool restores that snapshot inside each worker thread, replaying all ContextVar values via var.set(val) and pushing a new Flask app_context(). This pattern was systematized in PR #30607 4.

The legacy helper preserve_flask_contexts(flask_app, context_vars) does the same work for the non-Celery worker-thread path used by AdvancedChatAppGenerator 5.

Ongoing gaps: Issues #38299 1 and #39977 2 indicate that sub-nodes inside deeply nested parallel iterations on v1.15–1.16 can still lose context. The workaround is to disable parallel mode (is_parallel: false) until thread-context restoration is fully recursive.

Database sessions: A related fix (PR #35855) migrated node construction DB queries from Flask-SQLAlchemy's current_app-bound session to session_factory.create_session(). The fetch_memory() function in node_factory.py explicitly documents this requirement 6.

Credential Mutation: Bearer Prefix Accumulation#

Symptom: A custom API tool called multiple times (agent loop, workflow retry, or parallel branch) sends an escalating Authorization header — Bearer my-token on the first call, Bearer Bearer my-token on the second, etc. — causing 401/403 failures on every call after the first. 7 8

Root cause: ApiTool.fork_tool_runtime() passes the ToolRuntime object by reference, so the parent and all forked tools share the same credentials dict. The old assembling_request() wrote the prefixed value back into that shared dict:

# buggy — mutates the shared dict
credentials["api_key_value"] = f"Bearer {credentials['api_key_value']}"

Because the dict is never reset between invocations, each call finds an already-prefixed value and prepends again. 9

Fix (merged in PR #38616): 10 Use a local variable; leave credentials unchanged:

api_key_value = credentials["api_key_value"]
if api_key_header_prefix == "bearer" and api_key_value:
    api_key_value = f"Bearer {api_key_value}"
headers[api_key_header] = api_key_value # credentials dict never touched

Earlier attempts: PR #35997 11 and PR #36350 12 proposed the same pattern; PR #38616 is the one that merged. The current implementation is in api/core/tools/custom_tool/tool.py.

Broader lesson: Any mutable dict or Pydantic model passed by reference through fork_tool_runtime() is shared state. Avoid in-place mutation; prefer constructing a new value and assigning it to a local variable before use.

Variable Pool and Run-Context Propagation to Child Graphs#

Parallel branches and container nodes (Iteration, Loop) spawn independent child GraphEngine instances. Two objects must reach every child correctly:

VariablePool#

The pool is bootstrapped once in WorkflowAppRunner.run() from build_bootstrap_variables() and held on GraphRuntimeState 13. Child engines share the parent's pool by default, so outputs written by a child node with variable_pool.add((node_id, key), value) are immediately visible to downstream sibling nodes in the parent graph. 14

Known hazard: before PR #30059, loop-iteration outputs were never cleared between cycles, so a subsequent iteration could read stale prior-cycle values. The fix added _clear_loop_subgraph_variables() at the start of each cycle. 15

DifyRunContext / run_context#

User identity (tenant_id, app_id, user_id, user_from, invoke_from, trace_session_id) is assembled into a DifyRunContext by build_dify_run_context(), then wrapped in a DifyGraphInitContext frozen dataclass 16. This is translated to Graphon's GraphInitParams via to_graph_init_params() and stored on every DifyNodeFactory instance.

Child engines inherit run_context directly from the parent — no reconstruction needed — via DifyNodeFactory.with_runtime_state(child_graph_runtime_state) 17. All child nodes access identity through resolve_dify_run_context(run_context) in node_runtime.py, which raises ValueError on a missing key rather than silently producing empty state. 18

Bug: empty user_id in single-node debug runs — PR #34044 fixed a path where _prepare_single_node_execution() hardcoded user_id="", causing DatasetQuery.created_by to fail UUID validation inside iteration subgraphs. 19

Mutable account state (PR #39283): A refactor removed user.set_tenant_id(tenant_id) calls from Celery worker paths. Repository factory constructors now receive an explicit tenant_id: str argument, eliminating reliance on mutable ORM state that does not survive thread/process boundaries. 20

Architecture sketch#

WorkflowAppRunner.run()
  └─ build_dify_run_context() → DifyGraphInitContext → DifyNodeFactory
       └─ GraphEngine (parent)
            ├─ VariablePool ─────────────────── shared with children
            └─ parallel branch / iteration node
                 └─ child GraphEngine
                      └─ DifyNodeFactory.with_runtime_state(child_state)
                           └─ same run_context ── same tenant/user identity