Infinity Document Engine#
RAGFlow supports multiple document store backends โ Elasticsearch, OceanBase, SeekDB, SereneDB, and Infinity โ all behind the DocEngine interface. The Infinity backend (internal/engine/infinity/) maps three logical store types to concrete Infinity tables:
| Store | Table name pattern | Key files |
|---|---|---|
| Chunk store | {baseName}_{datasetID} (or just {baseName} for skill tables) | chunk.go |
| Metadata store | ragflow_doc_meta_{tenantID} | metadata.go |
| Memory store | memory_{tenantID}_{memoryID} (prefix-aware) | user_delete.go |
Table name builders live in common.go: buildChunkTableName and buildMetadataTableName.
Idempotent Deletion and Absent-Table Handling#
The core asymmetry: existence checks vs. direct drops#
Infinity does not automatically tolerate a missing table the way Elasticsearch silently accepts a missing index. The shared dropTable helper:
- Calls
tableExistsWithDBโ which maps Infinity SDK errors containing"not found"or"doesn't exist"to(false, nil). - Returns an error if the table does not exist (
table '<name>' does not exist).
This means DropChunkStore and DropMetadataStore are not idempotent on their own โ callers must guard against missing tables themselves.
Admin / user-deletion path#
The admin user-deletion path in internal/admin/user_delete.go uses two different patterns:
Direct drops (owned tenant)
deleteExternalData drops the owned tenant's chunk store, memory store, and metadata store directly without a prior existence check. If the tables never existed (e.g. tenant never ingested a document), the dropTable error will surface.
Existence-guarded record deletion (non-owned datasets/documents)
For datasets belonging to another tenant, or for individual documents whose dataset is not being deleted, the code checks ChunkStoreExists and MetadataStoreExists before calling DeleteChunks/DeleteMetadata . This prevents a DeleteChunks call from hitting a non-existent table.
Infinity-specific per-table fan-out โ dropUserChunkStore iterates over each dataset/memory ID and drops one Infinity table per ID. For Elasticsearch/other engines it issues a single DropChunkStore with an empty dataset ID .
How to achieve idempotent drops#
Pattern used elsewhere in the codebase (e.g. SkillIndexerService.ReindexAll):
exists, err := docEngine.ChunkStoreExists(ctx, baseName, datasetID)
if err != nil { ... }
if exists {
if err := docEngine.DropChunkStore(ctx, baseName, datasetID); err != nil { ... }
}
The same applies to metadata stores via MetadataStoreExists + DropMetadataStore.
Filter Construction and Deletion Safety#
Beyond table existence, the Infinity engine enforces a second safety layer on delete and update paths:
DeleteChunks(internal/engine/infinity/chunk.go) aborts ifconditionis non-empty butbuildFilterFromConditionreturns an empty string or"1=1", preventing unintended match-all deletes .deleteMetadataWithTable(internal/engine/infinity/metadata.go) applies the same guard .buildFilterFromConditionincommon.goreturns"1=1"when all condition values are nil or blank, making the guard trigger correctly.
Keyword and JSON-list filter quirks#
Infinity's filter handling differs from Elasticsearch in two ways that affect deletion correctness:
- Keyword columns (
*_kwd,source_id): Multi-token values use exact equality (col = 'value'); single-token values usefilter_fulltext(...). This prevents Infinity's tokenizer from silently matching nothing for cluster/navigation names . - JSON-list columns (
source_chunk_ids,source_doc_ids, etc.): Filtered viajson_contains(col, '<literal>')on JSON-typed columns, with fallback tofilter_fulltexton legacy Varchar columns . - Deep OR chains:
joinBalancedwraps OR clauses as a balanced binary tree to avoid Infinity's Thrift depth limit (which aborts with error 7018 on flat chains of ~22+ clauses).
Historical Context#
| PR | What changed |
|---|---|
| #14153 | Added TABLE_NOT_EXIST (error 3022) handling to update() in both rag/utils/infinity_conn.py and memory/utils/infinity_conn.py; insert() and search() were already tolerant. |
| #17686 | Added unconstrained-delete guards to DeleteChunks and deleteMetadataWithTable; fixed json_contains vs. filter_fulltext routing for legacy Varchar columns; fixed metadata get routing for ragflow_doc_meta_* tables. |
Key Source Files#
| File | Purpose |
|---|---|
internal/engine/engine.go | DocEngine interface โ all backend-agnostic operations |
internal/engine/infinity/common.go | dropTable, tableExists, filter builders, table-name helpers |
internal/engine/infinity/chunk.go | DropChunkStore, ChunkStoreExists |
internal/engine/infinity/metadata.go | DropMetadataStore, MetadataStoreExists |
internal/admin/user_delete.go | Admin user-deletion path โ direct drops and existence-guarded record deletes |