Elasticsearch Integration#

Dify ships Elasticsearch support as a VDB plugin (dify-vdb-elasticsearch) under api/providers/vdb/vdb-elasticsearch/. The plugin registers two backends via Python entry points 1:

Both variants are declared in VectorType and share the same elasticsearch==8.14.0 client library 2. Elasticsearch 8.0.0 or later is required; the factory enforces this at init time 3.

Index Schema and Metadata Mappings#

When an index does not yet exist, create_collection creates it with a fixed mapping 4:

Mapping doc_id and document_id as keyword (rather than letting ES auto-map them as text) is intentional: it avoids tokenization and makes term-level filters efficient 5.

The collection name is lowercased and derived from the dataset ID via Dataset.gen_collection_name_by_id. Index creation is guarded by a Redis lock to prevent race conditions during concurrent indexing 6.

Search Behavior#

Vector search (search_by_vector) uses the native ES KNN query. num_candidates is set to ceil(top_k * 1.5). When a document_ids_filter is present, it becomes a terms filter on metadata.document_id inside the KNN body — keeping scoring in ES rather than doing post-hoc filtering 7. Scores come directly from hit["_score"] (cosine similarity). A score_threshold kwarg can prune low-scoring results 8.

Full-text search (search_by_full_text) issues a standard match query against page_content. With a document_ids_filter it wraps the match in a bool + filter clause 9. No score threshold is applied on the full-text path.

See Vector Store Integration for cross-provider score behavior comparisons, and the Hybrid Search article for parallel vector/keyword retrieval and result fusion.

Japanese Variant (elasticsearch-ja)#

ElasticSearchJaVector extends ElasticSearchVector and only overrides create_collection 10. It adds an index-level settings block with a custom analyzer (ja_analyzer) that chains:

• Char filters: icu_normalizer, kuromoji_iteration_mark
• Tokenizer: kuromoji_tokenizer
• Token filters: kuromoji_baseform, kuromoji_part_of_speech, ja_stop, kuromoji_number, kuromoji_stemmer

The page_content field is then mapped with "analyzer": "ja_analyzer" and "search_analyzer": "ja_analyzer" 11, ensuring both index-time and query-time tokenization use Japanese morphological analysis. All other methods (vector search, full-text search, CRUD) are inherited unchanged from ElasticSearchVector.

Note: The Japanese factory (ElasticSearchJaVectorFactory) does not support the Elastic Cloud path — it reads only ELASTICSEARCH_HOST/PORT/USERNAME/PASSWORD 12.

Configuration and Deployment#

Set VECTOR_STORE=elasticsearch (or elasticsearch-ja) in the environment. The standard factory reads from Flask config 13:

A reference env file lives at docker/envs/vectorstores/elasticsearch.env.example. Docker Compose spins up docker.elastic.co/elasticsearch/elasticsearch:8.14.3 with xpack.security.enabled=true and single-node discovery 14.

Entry points (from pyproject.toml) wire the factories into Dify's plugin discovery system so no code changes are needed to swap backends 1.