Multilingual Search Tokenization#

RAGFlow's BM25 index uses language-aware tokenization to produce accurate stems and tokens for non-English knowledge bases. Without this, the default English Porter stemmer leaves non-English words un-stemmed or incorrectly reduced, degrading BM25 recall. The system threads a language parameter from the KB configuration through the entire chunking pipeline into the core tokenizer.

Architecture#

The tokenization stack has two layers:

1. rag/nlp/rag_tokenizer.py — A thin Python wrapper around infinity.rag_tokenizer.RagTokenizer 1. When the Infinity document engine is active, tokenize() and fine_grained_tokenize() pass through to the Infinity SDK; otherwise they delegate to the parent class. The set_language() method — which selects the appropriate Snowball stemmer — is provided by the Infinity SDK parent class, not defined in this file 2.

2. rag/nlp/__init__.py — Higher-level functions that call set_language() before tokenizing and propagate the language keyword argument throughout:

   • tokenize(d, txt, eng, language="English") — sets language on the tokenizer singleton, then populates content_ltks (coarse) and content_sm_ltks (fine-grained).
   • tokenize_chunks(..., language="English") — iterates chunks and delegates to tokenize().
   • tokenize_table(..., language="English") — also adjusts row delimiters: Chinese and Japanese use "；" instead of ";" 3.
   • tokenize_chunks_with_images() and doc_tokenize_chunks_with_images() mirror the same pattern.

All defaults are "English", preserving backward compatibility for existing KBs 4.

Language Parameter Threading#

rag/svr/task_executor.py is the entry point where the KB language field is picked up and propagated:

• build_chunks() calls rag_tokenizer.tokenizer.set_language(task["language"]) at line 422 before tokenizing chunks, and also passes lang=task_language to the chunker 5.
• run_dataflow() calls set_language(task.get("language", "English")) at line 771.
• run_raptor_for_kb() calls set_language(row.get("language", "English")) at line 1042.
• do_handle_task() calls set_language(task_language) at line 1406.

All 14 parser modules under rag/app/ (audio, book, email, laws, manual, naive, one, paper, picture, presentation, qa, resume, table, tag) were updated to pass language=lang into the tokenization calls as part of PR #14140 4.

Supported Languages#

The KB language selector (LanguageList in web/src/constants/common.ts) exposes 16 languages:

The NLTK Snowball stemmer (bundled with NLTK, no new dependency) is selected by set_language() for the 16 languages that have a Snowball mapping. Languages without a mapping (Chinese, Japanese, Korean, etc.) fall back to the Porter stemmer — this is acceptable because those languages use character-based tokenization anyway 4.

The cross-language search form (web/src/components/cross-language-form-field.tsx) exposes a curated subset of 11 languages: English, Chinese, Spanish, French, German, Japanese, Korean, Vietnamese, Arabic, Turkish, and Dutch.

RTL Text Support (Arabic, Hebrew, etc.)#

RTL language support has a nuanced history tracked in issue #4675. The core problem was in DeepDoc OCR: it did not correctly handle RTL text extraction from PDFs, causing Arabic/Hebrew chunks to appear in LTR order — breaking both chunk display and downstream BM25 retrieval 6.

• Workaround (still valid for OCR-heavy PDFs): Convert the PDF to .docx first (e.g., via Microsoft Word), then upload the .docx. The DOCX parser preserves Arabic character order correctly 6.
• Current status: Issue #4675 was closed as RTL support was added. If problems persist with the latest version, open a new issue with a sample file and logs.

Arabic is included in LanguageList and LanguageMap 7 and the cross-language form field 8. The BM25 tokenizer path is configured for Arabic; the historical limitation was specific to OCR-based PDF extraction.

Key Source Files#