Retrieval Filtering and Scoring#

Dify's knowledge retrieval pipeline applies quality gates in three distinct phases: pre-retrieval SQL metadata filtering, concurrent vector/keyword search with deferred threshold application, and post-fusion scoring via reranking. The result is a quality-first model where top_k is a ceiling, not a floor.

Pipeline Overview#

Query
  │
  ▼
┌─────────────────────────────────────────────┐
│ Stage 1 — SQL Metadata Filter (pre-search) │
│ get_metadata_filter_condition() │
│ → SELECT doc_id WHERE doc_metadata... │
│ → metadata_filter_document_ids (allowlist) │
│ Short-circuit: 0 docs matched → return [] │
└──────────────────┬──────────────────────────┘
                   │ document_ids_filter
                   ▼
┌─────────────────────────────────────────────┐
│ Stage 2 — Concurrent Search │
│ embedding_search() ← score_threshold=0.0 │
│ full_text_index_search() (hybrid only) │
│ keyword_search() │
│ ← top_k caps candidate count │
└──────────────────┬──────────────────────────┘
                   │ raw candidates
                   ▼
┌─────────────────────────────────────────────┐
│ Stage 3 — Fusion / Reranking + Threshold │
│ WeightRerankRunner (weighted score fusion) │
│ OR RerankModelRunner (external model) │
│ score_threshold applied here │
│ top_n = top_k (max, not guaranteed) │
└─────────────────────────────────────────────┘

Stage 1: Pre-Retrieval Metadata Filtering#

DatasetRetrieval.get_metadata_filter_condition() runs before any vector or keyword search. It evaluates metadata conditions against DatasetDocument.doc_metadata (a JSON column) via SQLAlchemy and returns a dict[str, list[str]] allowlist keyed by dataset ID.

Three modes 1:

process_metadata_filter_func() translates each Condition into a SQLAlchemy expression against DatasetDocument.doc_metadata. It supports string ops (LIKE/NOT LIKE), equality, numeric comparisons, in/not in, empty/not empty, and time operators (before/after). An empty in [] short-circuits to literal(False) 2.

Short-circuit: If a metadata filter is active but produces zero matching documents, retrieval returns [] immediately — no vector query is issued 3 4.

The document_ids_filter list is threaded through all search methods: embedding_search(), full_text_index_search(), and keyword_search().

Stage 2: Concurrent Search with Deferred Threshold#

RetrievalService._retrieve() fans out to up to three search paths in parallel using a ThreadPoolExecutor 5.

Critical design: score threshold deferral. For HYBRID_SEARCH, embedding_search forces score_threshold=0.0 at vector retrieval time 6. The code comment explains the reason directly:

Applying the user score threshold at vector retrieval time uses embedding similarity, which is not comparable to reranked or fused scores and incorrectly drops high-quality chunks.

This ensures that chunks which score poorly on raw embedding similarity but rank highly after fusion are not prematurely discarded.

Stage 3: Fusion, Reranking, and Final Threshold#

After all parallel futures complete, hybrid results are deduplicated via _deduplicate_documents() (highest-score-wins by doc_id, first-seen by content key), then passed to DataPostProcessor 7.

Two fusion strategies:

WEIGHTED_SCORE → WeightRerankRunner
Computes vector_weight × cosine_similarity + keyword_weight × TF-IDF_cosine 8. Vector scores reuse the stored retrieval score if available; otherwise cosine similarity is recomputed from the raw document vector 9. Score threshold is applied inline: any document below the threshold is dropped before top_n truncation 10.

RERANKING_MODEL → RerankModelRunner
Deduplicates by doc_id, then calls an external rerank model via invoke_rerank / invoke_multimodal_rerank 11. Score threshold and top_n are applied after receiving the model's scores 12.

Fallback threshold: If no rerank runner is active (e.g., single-dataset hybrid with no reranker), a final _filter_documents_by_vector_score_threshold() pass is applied to the fused list 13.

top_k as a Maximum#

top_k is passed as top_n into every rerank runner and governs the final [:top_n] slice 14 15. Because the score threshold filter runs before the truncation, the actual returned count can be anywhere from 0 to top_k. The default is top_k=4 16. Callers should never assume len(results) == top_k.

For single-dataset retrieval (calculate_vector_score), the same pattern holds: threshold filtering runs first, then [:top_k] truncation 17.

Key Files#