Metadata Filtering#

Metadata filtering is a pre-retrieval step that narrows the document search space to a doc_ids whitelist before vector/BM25 search runs. It is applied across all retrieval entry points — the REST API, Agent Retrieval node, chat completions, and the Dify-compatible endpoint — and resolves through a shared implementation in common/metadata_utils.py 1.

Metadata Storage#

User-defined metadata is stored as a flexible dictionary (meta_fields) in a per-tenant ES/Infinity index named ragflow_doc_meta_{tenant_id} 2. Each document record has three fields:

The index is created lazily on first write and dropped when it becomes empty 3. On the ES backend, user metadata keys are namespaced under meta_fields.<key> via dynamic object mapping 4. On the Infinity backend, chunks use a per-KB table but the metadata index remains per-tenant, matching the ES layout 5.

API validation: meta_fields values must be str, int, float, or a list of those types. Nested dicts are rejected 6. When LLM extraction produces combined string values (e.g. "关羽、孙权"), DocMetadataService splits them on common delimiters before storage 7.

Source of truth: ES/Infinity is the sole authoritative store. MySQL meta_fields column has been removed 8.

Filter Modes#

apply_meta_data_filter() supports three modes 9:

In auto and semi_auto modes, if no documents match the generated filter the function returns None (allowing the retrieval to proceed unfiltered). In manual mode, a no-match returns the sentinel ["-999"], which guarantees zero retrieval results rather than an unfiltered fallback 10.

Operators#

14 filter operators are supported 11:

String comparisons are case-insensitive. Date comparisons require strict YYYY-MM-DD format on both sides; mismatched types skip the record rather than raising 12. The logic key ("and" / "or") controls how multiple conditions are combined, defaulting to "and" 13.

The Dify integration normalizes operator names before evaluation — "is" → "=", "not is" → "≠", ">=" → "≥", etc. — via convert_conditions() 14.

Push-Down vs. In-Memory Evaluation#

Filtering has two execution paths 15:

1. ES push-down (_filter_doc_ids_by_metadata_es()) — translates conditions into an ES bool query via common/metadata_es_filter.py 16. Operators ≠ and not in are routed to in-memory fallback for multi-valued fields because ES must_not term has different semantics than the per-bucket Python evaluation 17.
2. Infinity push-down (_filter_doc_ids_by_metadata_infinity()) — builds a SQL WHERE clause via common/metadata_infinity_filter.py.
3. In-memory fallback (meta_filter()) — loads all metadata for the KB set into Python via get_flatted_meta_by_kbs() and evaluates conditions in a {field: {value: [doc_ids]}} structure 18. This path is used when kb_ids is not supplied, the ES client is unavailable, or push-down returns None. Manual-mode manual with kb_ids skips the expensive get_flatted_meta_by_kbs() round-trip entirely 19.

If push-down returns more matches than the limit cap (default 10,000), it falls back to in-memory to avoid silent truncation 20.

Common Use Cases#

Location-based retrieval: Store a region or location field as meta_fields on each document. Use a manual filter with {"key": "region", "op": "=", "value": "{{user_region}}"} in the Agent Retrieval node; the manual_value_resolver substitutes the runtime value before evaluation. The resulting doc_ids whitelist is intersected with the semantic search results.

Semi-auto categorization: Configure semi_auto with ["category", "date"] in the retrieval node. The LLM generates filter conditions from those keys based on the user's query, narrowing the doc pool before vector search without hard-coding conditions.

Key Source Files#