Knowledge Base Metadata Filtering#

Overview#

Metadata filtering is a pre-retrieval stage that narrows which documents are eligible before any vector, keyword, or full-text search is performed. It operates in three modes — disabled, manual, and automatic — and uses a two-stage architecture:

1. Stage 1 — SQL filter: evaluate metadata conditions against DatasetDocument.doc_metadata to produce a metadata_filter_document_ids map (keyed by dataset ID).
2. Stage 2 — Constrained search: pass those document IDs as document_ids_filter into RetrievalService.retrieve(), restricting vector/keyword search to matching documents only.

The central orchestration method is DatasetRetrieval.get_metadata_filter_condition(), called before both single-dataset and multi-dataset retrieval paths. Its return value — a (metadata_filter_document_ids, metadata_condition) tuple — is then threaded through single_retrieve() and multiple_retrieve().

Filtering Modes#

disabled (default)#

Returns (None, None) immediately with no database queries 1. No overhead is added to retrieval.

manual#

Conditions are defined statically in the node or app configuration as a MetadataFilteringCondition. At runtime:

• Variable interpolation: {{variable}} placeholders in string condition values are resolved against workflow inputs by _replace_metadata_filter_value(). In the workflow node, the variable pool performs this resolution earlier via KnowledgeRetrievalNode._resolve_metadata_filtering_conditions(), which handles both scalar and sequence values.
• Logical combination: conditions are combined with AND or OR based on logical_operator 2.

automatic#

Conditions are derived at request time by an LLM:

1. All DatasetMetadata field names for the target datasets are queried from the database 3.
2. A few-shot prompt (see template_prompts.py) instructs the LLM to return a {"metadata_map": [...]} JSON structure containing metadata_field_name, metadata_field_value, and comparison_operator for each extracted filter.
3. The LLM response is parsed with parse_and_check_json_markdown(), and only recognized field names are kept 4.
4. Auto mode defaults the logical operator to "or" 5.

The LLM to use for automatic mode is configured separately via metadata_model_config on the node/request.

Data Model#

Core types live in api/core/rag/entities/metadata_entities.py:

Operator groups 6:

• String/array: contains, not contains, start with, end with, is, is not, empty, not empty, in, not in
• Numeric: =, ≠, >, <, ≥, ≤
• Time: before, after

At the workflow node level, KnowledgeRetrievalNodeData (in entities.py) holds metadata_filtering_mode ("disabled" | "automatic" | "manual"), metadata_model_config, and metadata_filtering_conditions. Note: the conditions field on MetadataFilteringCondition is marked deprecated=True in the Pydantic model 7 but is still actively used at runtime.

SQL Condition Evaluation#

DatasetRetrieval.process_metadata_filter_func() translates each Condition into a SQLAlchemy filter expression against DatasetDocument.doc_metadata, a JSON column. Key behaviors:

• JSON access: DatasetDocument.doc_metadata[metadata_name].as_string() for text comparisons; .as_float() for numeric and time operators 8.
• String ops (contains, start with, end with, not contains): use LIKE / NOT LIKE with escape_like_pattern() to prevent injection 9.
• Equality (is / =): dispatches on value type — string uses ==, numeric uses .as_float() == 10.
• in / not in: accepts a comma-separated string, a list, or a tuple. An empty list short-circuits to literal(False) (in) or literal(True) (not in) 11.
• empty / not empty: maps to IS NULL / IS NOT NULL on the JSON key 12.

After all filter expressions are built, the final SQL query restricts to documents that are indexed (indexing_status == "completed"), enabled, and not archived, then applies the combined AND/OR filter 13. Results are grouped into metadata_filter_document_ids: dict[str, list[str]] keyed by dataset ID.

Two-Stage Retrieval Architecture#

Query / Attachment
       │
       ▼
┌─────────────────────────────────────────────┐
│ Stage 1 – SQL Metadata Filter │
│ get_metadata_filter_condition() │
│ → SELECT doc_id FROM DatasetDocument │
│ WHERE doc_metadata conditions... │
│ → metadata_filter_document_ids │
└──────────────────┬──────────────────────────┘
                   │
       ┌───────────┴─────────────┐
       │ No matching docs? │ Has docs?
       │ Return [] immediately │
       └─────────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────────┐
│ Stage 2 – Constrained Vector/Keyword Search│
│ RetrievalService.retrieve( │
│ document_ids_filter=doc_ids │
│ ) │
│ → embedding_search() / keyword_search() │
│ / full_text_index_search() │
└─────────────────────────────────────────────┘

Short-circuit behavior: if any metadata condition is active but produces zero matching document IDs, retrieval returns an empty result immediately — no search query is issued 14 15.

External knowledge bases: the metadata_condition object is forwarded verbatim to external dataset providers via ExternalDatasetService.fetch_external_knowledge_retrieval(), letting external backends apply their own filtering logic.

Multi-dataset parallelism: in _multiple_retrieve_thread(), each dataset's allowed document IDs are looked up from metadata_filter_document_ids before spawning per-dataset retrieval threads, so datasets with zero matching documents are skipped entirely 16.

Key Files and Entry Points#

Call path (workflow):
KnowledgeRetrievalNode._run() → _fetch_dataset_retriever() 17 → DatasetRetrieval.knowledge_retrieval() 18 → get_metadata_filter_condition() → single_retrieve() or multiple_retrieve() → RetrievalService.retrieve().