External Knowledge Integration#

Dify's external knowledge integration lets tenants connect any retrieval system (e.g. RagFlow, Bedrock, custom RAG stacks) to Dify's knowledge pipeline without ingesting documents into Dify itself. The feature is built around three concepts: a fixed API contract that the external service must implement, a credential/API template scoped to a tenant, and a binding that links a Dify dataset to an external knowledge source ID.

Architecture Overview#

• Dataset 1 — when provider = "external", the dataset carries no documents; retrieval is fully delegated outward.
• ExternalKnowledgeApis 2 — tenant-scoped API template storing endpoint and api_key as a JSON blob in the settings column.
• ExternalKnowledgeBindings 3 — join table mapping a dataset_id to an external_knowledge_api_id and an external_knowledge_id (the ID used in the remote system).

API Contract#

The external service must expose POST {endpoint}/retrieval 4. Dify constructs this URL by appending /retrieval to the configured endpoint 5 and validates the endpoint is reachable before saving a new API template 6.

Request Body#

See full request schema including all supported metadata_condition operators.

Response Body#

HTTP 200 with {"records": [...]} 7. Each record:

Any non-200 response raises an ExternalKnowledgeRetrievalError that surfaces as 502 external_knowledge_failed to callers 8.

Authentication#

Dify sends the configured API key as Authorization: Bearer {api_key} 9. During retrieval this header is assembled directly from the stored settings 10. The service layer also supports bearer, basic, and custom header schemes for generic HTTP tool calls via assembling_headers.

Tenant-Scoped Credential and Dataset Management#

Creating an API Template#

ExternalDatasetService.create_external_knowledge_api stores name, description, and settings (JSON with endpoint + api_key) in external_knowledge_apis, scoped by tenant_id.

check_endpoint_and_api_key 6 fires a live POST {endpoint}/retrieval probe before the record is saved, rejecting on 400, 403, 404, or 502.

When updating, if the caller sends api_key = HIDDEN_VALUE, the service preserves the stored key 11 to avoid leaking credentials in round-trip API responses.

Binding a Dataset#

create_external_dataset atomically:

1. Creates a Dataset row with provider = "external" and the retrieval model config.
2. Creates an ExternalKnowledgeBindings row linking dataset_id → external_knowledge_api_id + external_knowledge_id.

All lookups are scoped by tenant_id to prevent cross-tenant access 12.

external_knowledge_api_use_check 13 counts binding references within the same tenant before allowing deletion.

Retrieval Flow#

fetch_external_knowledge_retrieval is the main call site:

1. Resolves the binding for (tenant_id, dataset_id).
2. Resolves the API template for the binding's external_knowledge_api_id, scoped to tenant_id.
3. Builds the request body with knowledge_id, query, retrieval_setting, and optional metadata_condition.
4. POSTs via ssrf_proxy (SSRF-safe HTTP client).
5. Returns records[] on HTTP 200; raises ExternalKnowledgeRetrievalError on any other status.

DatasetRetrieval._retriever and single_retrieve both call this service when dataset.provider == "external" 14 15. Retrieved records are tagged with provider="external" and merged with Dify-native results before scoring and ranking.

REST API Entry Points#

Managed by api/controllers/console/datasets/external.py:

Key Source Files#