Keyword Extraction#
Overview#
RAGFlow extracts keywords from each document chunk using an LLM call during the post-chunking pipeline. The results are stored in two fields per chunk:
important_kwd— array of keyword strings, mapped as Elasticsearchkeywordtype (exact-match,similarity: boolean)important_tks— tokenized form of the joined keywords, mapped astextfor BM25 ranking
The important_kwd^30 BM25 boost means correctly parsed keywords have an outsized impact on retrieval relevance. Mis-parsing — treating the entire LLM response as one keyword instead of many — silently kills this boost.
Entry Point#
Keyword extraction is triggered when parser_config.auto_keywords > 0. The async function extract_keywords in chunk_post_processor.py fans out one async task per chunk using asyncio.gather, bounded by a chat_limiter semaphore. Results are cached in Redis (keyed on xxhash64(llm_name + content + "keywords" + {topn})) with a 24-hour TTL via get_llm_cache / set_llm_cache .
LLM Prompt#
keyword_extraction() in rag/prompts/generator.py renders keyword_prompt.md with Jinja2. The prompt instructs the model to:
- Output the top
Nkeywords/phrases (N =auto_keywordssetting) - Match the language of the source content
- Delimit keywords by ENGLISH COMMA
The function strips <think>...</think> reasoning tokens from the response and returns an empty string on **ERROR** responses .
Delimiter Normalization#
The raw LLM response is split with a regex before being stored :
re.split(r"[,,;;、\r\n]+", cached)
This covers:
| Token | Meaning |
|---|---|
, | ASCII comma |
, | Chinese full-width comma |
; | ASCII semicolon |
; | Chinese full-width semicolon |
、 | Japanese/Chinese enumeration comma |
\r\n | Newlines |
Empty/whitespace-only tokens are filtered out. The broadened delimiter set was introduced in PR #14540 (merged May 2026) because Chinese LLMs (Qwen, GLM, etc.) frequently emit , instead of , even when prompted for English commas — causing the full output to be stored as one concatenated keyword string instead of N individual ones.
Earlier, PR #12618 (merged January 2026) fixed a related serialization bug in the Infinity connector: important_kwd was round-tripped through space-separated serialization, which fragmented multi-word phrases like "Senior Fund Manager" into three separate tokens.
Elasticsearch 32,766-Byte Keyword Limit#
important_kwd is mapped as Elasticsearch keyword type, which enforces a hard 32,766-byte UTF-8 limit per term . When the LLM returns a response with no recognized delimiters, the entire string passes through the regex unsplit and is stored as a single oversized term. Elasticsearch rejects the document with:
document_parsing_exception: Document contains at least one immense term
in field="important_kwd" (whose UTF8 encoding is longer than the max
length 32766)
This is a known open issue (#17074) . RAGFlow currently has no pre-insertion byte-length validation for important_kwd values. The failure is non-deterministic — the same document may succeed on retry because LLM outputs vary.
Workaround: Delete the document and re-parse. If it consistently fails, use a model that reliably follows the English comma delimiter instruction.
Key Source Files#
| File | Purpose |
|---|---|
rag/svr/task_executor_refactor/chunk_post_processor.py | extract_keywords() — async per-chunk extraction, Redis caching, delimiter split |
rag/prompts/generator.py | keyword_extraction() — LLM call, reasoning-token stripping |
rag/prompts/keyword_prompt.md | Jinja2 prompt template |
conf/mapping.json | ES dynamic template mapping *_kwd → keyword type |
rag/svr/task_executor.py | Legacy path with equivalent doc_keyword_extraction inline |