Keyword Extraction#

Overview#

RAGFlow extracts keywords from each document chunk using an LLM call during the post-chunking pipeline. The results are stored in two fields per chunk:

• important_kwd — array of keyword strings, mapped as Elasticsearch keyword type (exact-match, similarity: boolean) 1
• important_tks — tokenized form of the joined keywords, mapped as text for BM25 ranking 2

The important_kwd^30 BM25 boost means correctly parsed keywords have an outsized impact on retrieval relevance. Mis-parsing — treating the entire LLM response as one keyword instead of many — silently kills this boost.

Entry Point#

Keyword extraction is triggered when parser_config.auto_keywords > 0. The async function extract_keywords in chunk_post_processor.py fans out one async task per chunk using asyncio.gather, bounded by a chat_limiter semaphore. Results are cached in Redis (keyed on xxhash64(llm_name + content + "keywords" + {topn})) with a 24-hour TTL via get_llm_cache / set_llm_cache 3.

LLM Prompt#

keyword_extraction() in rag/prompts/generator.py renders keyword_prompt.md with Jinja2. The prompt instructs the model to:

• Output the top N keywords/phrases (N = auto_keywords setting)
• Match the language of the source content
• Delimit keywords by ENGLISH COMMA

The function strips <think>...</think> reasoning tokens from the response and returns an empty string on **ERROR** responses 4.

Delimiter Normalization#

The raw LLM response is split with a regex before being stored 5:

re.split(r"[,，;；、\r\n]+", cached)

This covers:

Empty/whitespace-only tokens are filtered out. The broadened delimiter set was introduced in PR #14540 (merged May 2026) because Chinese LLMs (Qwen, GLM, etc.) frequently emit ， instead of , even when prompted for English commas — causing the full output to be stored as one concatenated keyword string instead of N individual ones.

Earlier, PR #12618 (merged January 2026) fixed a related serialization bug in the Infinity connector: important_kwd was round-tripped through space-separated serialization, which fragmented multi-word phrases like "Senior Fund Manager" into three separate tokens.

Elasticsearch 32,766-Byte Keyword Limit#

important_kwd is mapped as Elasticsearch keyword type, which enforces a hard 32,766-byte UTF-8 limit per term 1. When the LLM returns a response with no recognized delimiters, the entire string passes through the regex unsplit and is stored as a single oversized term. Elasticsearch rejects the document with:

document_parsing_exception: Document contains at least one immense term
in field="important_kwd" (whose UTF8 encoding is longer than the max
length 32766)

This is a known open issue (#17074) 6. RAGFlow currently has no pre-insertion byte-length validation for important_kwd values. The failure is non-deterministic — the same document may succeed on retry because LLM outputs vary.

Workaround: Delete the document and re-parse. If it consistently fails, use a model that reliably follows the English comma delimiter instruction.

Key Source Files#