Elasticsearch Index Management#

RAGFlow employs a per-tenant index architecture for chunk and document storage. All knowledge bases (KBs) belonging to the same tenant share a single Elasticsearch index, ensuring tenant-level isolation while avoiding index proliferation. Two index types exist: ragflow_{tenant_id} stores all RAG chunks, while ragflow_doc_meta_{tenant_id} holds per-document metadata fields .

Index Creation & Lifecycle#

Index creation occurs during document ingestion via init_kb() in rag/svr/task_executor.py. The helper calls search.index_name(row["tenant_id"]) to generate the chunk index name, then invokes settings.docStoreConn.create_idx() with the tenant index name, KB ID, and vector size. The low-level create_idx() method in ESConnectionBase delegates to the Elasticsearch Indices API, applying settings and mappings from conf/mapping.json. Because the method checks index_exist() first, multiple KBs under the same tenant safely share the index without overwriting it .

Index deletion is tenant-scoped: delete_idx() only drops the index when dataset_id is empty, preserving the shared index when a single KB is removed .

mapping.json: Shard/Replica & Field Templates#

conf/mapping.json defines the authoritative schema for every new tenant chunk index.

Resource settings :

number_of_shards: 2
number_of_replicas: 0
refresh_interval: 1000ms

Custom similarity : A scripted BM25 variant (scripted_sim) applies boolean-style IDF with capped TF (min(doc.freq, 1)), used by *_tks text fields.

Dynamic field templates — fields are typed by suffix convention :

Suffix	ES Type	Notes
`*_tks`	`text` + `scripted_sim`	Token fields for BM25 ranking
`*_ltks`	`text` (whitespace)	Light/lower-weight text
`_kwd`, `_id`, `*_uid`	`keyword`	Exact-match / filter fields
`*_flt`	`float`	Numeric floats
`_int`, `_long`, `_ulong`, `_short`	integer variants	Numeric integers
`_dt`, `_time`, `*_at`	`date`	Date fields
`*_nst`	`nested`	Nested objects
`*_obj`	`object` (dynamic)	Dynamic sub-objects
`_fea` / `_feas`	`rank_feature` / `rank_features`	PageRank / tag boosting
`_512_vec`, `_768_vec`, `_1024_vec`, `_1536_vec`	`dense_vector` + cosine	Embedding vectors at 512/768/1024/1536 dimensions
`*_bin`	`binary`	Binary blobs
`_with_weight`, `_list`	`text` (not indexed)	Stored but not searchable

One explicit mapping falls outside dynamic templates: lat_lon → geo_point .

Dense vector templates use "similarity": "cosine" and "index": true for KNN search.

doc_meta_es_mapping.json#

conf/doc_meta_es_mapping.json governs the ragflow_doc_meta_{tenant_id} index for per-document metadata. It reuses the same shard/replica defaults (2 shards, 0 replicas, 1000ms refresh) and defines three explicit fields :

id — keyword (document ID)
kb_id — keyword (KB filter)
meta_fields — dynamic object (arbitrary user metadata)

The outer mapping sets "dynamic": "runtime" so unknown top-level fields become runtime fields rather than persisted mappings .

Implementation Classes#

Class / File	Role
`ESConnectionBase` (`common/doc_store/es_conn_base.py`)	Loads `mapping.json`, owns `create_idx`, `delete_idx`, `refresh_idx`, `count_idx`
`ESConnection` (`rag/utils/es_conn.py`)	Singleton; implements `search`, `insert`, `update`, `delete`; scopes all writes with `kb_id`
`DocMetadataService` (`api/db/services/doc_metadata_service.py`)	Creates / queries `ragflow_doc_meta_{tenant_id}` via `create_doc_meta_idx`; `_get_doc_meta_index_name()` constructs the index name

ESConnectionBase is initialized with the mapping file path and connection pool (es_conn_pool), making it straightforward to swap in alternate schemas.

All write and query operations inject kb_id into filters to enforce KB-level isolation within the shared tenant index .

Multi-Tenancy & Isolation#

Tenant isolation by index: Each tenant's chunks live exclusively in ragflow_{tenant_id}. Cross-tenant queries are structurally impossible because the index name is tenant-scoped.
KB isolation within a tenant: Achieved via the kb_id field stored on every document. All search and delete operations inject kb_id into the query filter , ensuring KB-level boundaries.
Infinity backend differs: For the Infinity vector store, the naming pattern is ragflow_{tenant_id}_{kb_id} — each KB gets a separate table rather than sharing one index per tenant .