Chunk Metadata Extraction#

RAGFlow's chunk metadata extraction pipeline enriches each parsed document chunk with LLM-generated signals — structured metadata fields, keywords, synthetic questions, and content tags — then aggregates those signals into a single document-level record stored in a per-tenant Elasticsearch or Infinity index. MySQL is not involved; ES/Infinity is the sole source of truth for document metadata 1.

Pipeline Overview#

Post-chunking enrichment runs across four async functions, all defined in chunk_post_processor.py. They are invoked on the list of chunk dicts produced by the parser:

Each function calls LLMBundle with the tenant's configured chat model 2, dispatches one async task per chunk, and await asyncio.gather()s them all. A chat_limiter semaphore bounds concurrent LLM calls.

The same logic also exists in the original task_executor.py; the chunk_post_processor.py version is the refactored extraction of those handlers.

Metadata Config Construction#

Before dispatching per-chunk tasks, build_metadata_config merges two sources from parser_config:

• metadata — user-defined schema, either as a JSON schema dict or a list of {key, description, enum} items.
• built_in_metadata — system-defined fields (e.g., update_time, file_name) injected from apply_built_in_metadata 3.

The merged config is passed through turn2jsonschema to normalize it into a JSON schema dict before the LLM prompt is constructed 4.

LLM Cache (Redis)#

Every LLM call is gated by a Redis cache keyed on xxhash64(llm_name + content + task_type + config) with a 24-hour TTL 5. Cache hits skip the LLM entirely:

cached = get_llm_cache(chat_mdl.llm_name, chunk["content_with_weight"], "metadata", metadata_conf)
if not cached:
    cached = await gen_metadata(chat_mdl, schema, content)
    set_llm_cache(...)

The same get_llm_cache / set_llm_cache helpers from rag/graphrag/utils.py are shared by keyword extraction, question generation, and tagging. Cache hits for tags also use get_tags_from_cache / set_tags_to_cache to avoid re-fetching the full tag list from the KB.

Per-Chunk → Document Aggregation#

After all chunk tasks complete, generate_metadata reduces chunk-level metadata_obj dicts into a single document dict via update_metadata_to 6:

1. Iterates chunks, calling update_metadata_to(accumulator, chunk["metadata_obj"]).
2. update_metadata_to merges list values (deduplicating via dedupe_list) and overwrites scalar values.
3. Fetches any existing document metadata via DocMetadataService.get_document_metadata(doc_id) and merges again, so re-runs append rather than overwrite.
4. Writes the final dict via DocMetadataService.update_document_metadata(doc_id, metadata).

Before writing, DocMetadataService post-processes list values by splitting on common delimiters (、,，;；|) to fix LLM outputs that combine multi-valued fields into a single string 7.

DocMetadataService and the ragflow_doc_meta_{tenant_id} Index#

DocMetadataService manages a per-tenant index named ragflow_doc_meta_{tenant_id} 8. Each document is a record with three fields:

The index is created on first write (create_doc_meta_idx) and dropped when it becomes empty 9. Key operations:

• update_document_metadata — ES backend uses a scripted replace_meta_fields for full overwrite semantics (avoids deep-merge stale key leakage); Infinity backend falls back to delete+insert 10.
• filter_doc_ids_by_meta_pushdown — pushes metadata filter conditions directly into ES DSL or Infinity SQL, returning matching doc IDs without loading all metadata into Python 11.
• get_flatted_meta_by_kbs — paginates through all doc-meta records for a set of KBs and returns {field: {value: [doc_ids]}} for UI faceting 12.

Index Architecture#

Two separate ES/Infinity index types exist per tenant 13:

On the Infinity backend, chunks use a per-KB table (ragflow_{tenant_id}_{kb_id}), but the metadata index remains per-tenant, matching the ES layout.

Key Source Files#