Plugin Daemon Architecture#

Overview#

The plugin daemon is a separate Go-based service (default port 5002) that handles all plugin execution in Dify. The API never runs plugin code directly — every model dispatch, tool call, agent strategy execution, credential validation, and datasource operation is proxied over HTTP to the daemon 1 2.

Dify API ──(HTTP / SSE)──► Plugin Daemon :5002
             headers: X-Api-Key, X-Plugin-ID
             path: plugin/{tenant_id}/...

The daemon maintains its own PostgreSQL database (dify_plugin, separate from the main dify DB) and its own filesystem tree for installed plugin packages 1.

Entry points in the API codebase:

Request Routing: Headers and URL Structure#

All requests flow through BasePluginClient in api/core/plugin/impl/base.py. Two complementary identification mechanisms are used together on every call.

Headers#

X-Api-Key is the shared secret between the API and daemon for authentication. X-Plugin-ID scopes the dispatch to the specific installed plugin within the tenant.

URL Patterns#

All URLs follow plugin/{tenant_id}/{segment}. The tenant_id in the path scopes the request to a specific workspace; the X-Plugin-ID header identifies which plugin within that workspace handles it.

1

Transport and HTTP Client#

Two Transport Modes#

BasePluginClient offers two methods 3:

• _request() — synchronous, blocking httpx.Client.request(). Used for non-streaming operations (list plugins, validate credentials, etc.).
• _stream_request() — synchronous generator using httpx.Client.stream(), consumed line-by-line as SSE. Used for LLM inference, embeddings, and rerank.

Key constraint: Both methods are fully synchronous — they block the calling Gunicorn worker thread for the entire duration of the daemon call, including all LLM inference time. Agent nodes are especially vulnerable because they serialize multiple tool + LLM calls in a single worker hold 2.

HTTP Client Pool#

A module-level, process-shared httpx.Client is created once at import time via get_pooled_http_client(), capped at 50 keep-alive / 100 max connections.

Response Envelope#

Non-streaming responses are wrapped in PluginDaemonBasicResponse. A non-zero code field triggers _handle_plugin_daemon_error(). Streaming responses are newline-delimited JSON; each line is validated against the same envelope before being yielded.

Timeout Control#

The default timeout is PLUGIN_DAEMON_TIMEOUT (600 s). A context manager use_plugin_daemon_request_timeout() allows scoped per-request shortening without mutating global state.

Error Handling#

Exception Hierarchy#

Defined in api/core/plugin/impl/exc.py:

PluginDaemonError
├── PluginDaemonInternalError
│ ├── PluginDaemonInternalServerError
│ ├── PluginDaemonUnauthorizedError
│ ├── PluginDaemonNotFoundError
│ └── PluginRuntimeError
└── PluginDaemonClientSideError
    ├── PluginDaemonBadRequestError
    ├── PluginInvokeError (also ValueError)
    ├── PluginUniqueIdentifierError
    ├── PluginNotFoundError
    └── PluginPermissionDeniedError

⚠️ Do not rename exception classes in exc.py. _handle_plugin_daemon_error() dispatches by error_type string equality against class __name__. Renaming breaks the mapping silently 4.

Error Dispatch#

_handle_plugin_daemon_error() uses match/case on the error type name from the daemon response. The PluginInvokeError branch further matches nested invoke_error_type to re-raise typed invoke errors such as InvokeRateLimitError, InvokeAuthorizationError, InvokeServerUnavailableError, PluginRuntimeError, etc.

Runtime-Unavailability Bug (→ PR #41696)#

When the plugin daemon restarts, Redis has no plugin runtime state yet. The daemon returns HTTP 404 with message "no available node, plugin runtime not found". Before the fix 5:

1. HTTP 404 was classified as PluginDaemonClientSideError by status code alone.
2. Streaming paths converted PluginDaemonInnerError → bare ValueError in invoke_llm.
3. handle_value_error in external_api.py mapped any ValueError → HTTP 400 invalid_param.

A valid in-flight Chatflow appeared to have bad input, not a transient infrastructure failure 6.

PR #41696 introduces PluginDaemonUnavailableError and maps it to HTTP 503 plugin_daemon_unavailable, so clients can distinguish retryable daemon outages from genuine bad-request errors.

Configuration#

Defined in PluginConfig (api/configs/feature/__init__.py) 7:

Additional Docker-specific variables 8:

Plugin daemon container storage binds ./volumes/plugin_daemon → /app/storage inside the container.

Version Synchronization#

The plugin daemon is versioned in lockstep with the Dify API. Protocol changes between releases cause complete communication failure when versions diverge 1.

Version Pairings#

When mismatched, the daemon logs "failed to find the version of the plugin sdk" and the API returns connection errors 1.

Post-Upgrade Checklist#

After any major version bump 8:

1. Schema migration — docker compose exec api flask upgrade-db (applies Alembic migrations to main dify DB; auto-runs on startup if MIGRATION_ENABLED=true)
2. Plugin backfill — docker compose exec api flask backfill-plugin-auto-upgrade (normalizes per-tenant plugin auto-upgrade strategy rows after identifier changes)
3. Verify daemon image tag matches the Dify API release
4. Clean stale plugin dirs from ./volumes/plugin_daemon/plugin/ for any record not found entries in daemon logs

Common Upgrade Failures#