Infinity Database Stability#

RAGFlow uses Infinity as its document/vector store engine. Infinity has a history of stability issues — segmentation faults, table disappearance, and catalog corruption — that affect document parsing and retrieval. RAGFlow mitigates these at the application layer through a connection pool with startup health checks, RocksDB metadata retry logic, and multi-worker connection management.

Known Stability Issues#

Segmentation faults during document parsing
Infinity crashes during SELECT execution with the message "This task should not be scheduled if the source queue is empty" (in src/executor/operator_state.cpp:50), preceded by a null pointer dereference (txn_id: nullptr). The crash generates a core dump in the container root directory. Recovery requires a container restart, but the crash recurs upon resuming parsing. Reported in Docker (CentOS7) and Kubernetes deployments processing 80–100+ PDFs. 1

Table disappearance / error 3022
Infinity surfaces InfinityException: error_code=3022 … Table X doesn't exist during chunk-list operations on tables that were previously created. This is a known Infinity engine bug — tables can disappear or fail to be created correctly. Workaround: clear Infinity data, restart, and re-ingest files; or switch to Elasticsearch. 2

RocksDB catalog contention (error 9003 / "Resource busy")
Concurrent CREATE TABLE / DROP TABLE operations race on Infinity's RocksDB-backed catalog counters (e.g., db|1|next_table_id). Instead of blocking on a lock, Infinity raises error 9003. This is reproducible when two users create a knowledge base simultaneously, during batch onboarding, or in multi-replica deployments. 3

Connection Pool Health Management#

InfinityConnectionPool (common/doc_store/infinity_conn_pool.py) is a singleton that wraps the Infinity SDK ConnectionPool.

• Pool size: Controlled by INFINITY_POOL_MAX_SIZE env var, defaulting to 4 4. This was reduced from 32 in PR #12006 after 16 workers × 32 connections = 512 simultaneous connections exceeded Infinity's 128-connection limit and hung all workers.
• Startup health check: On initialization, the pool retries up to 24 times (2-second sleep each, 120s total) calling show_current_node() and requiring server_status in ["started", "alive"] before accepting the pool as ready 5. Raises an exception if Infinity remains unhealthy.
• refresh_conn_pool(): Called on per-connection errors during InfinityConnectionBase.__init__. Checks liveness; on failure, destroys and recreates the underlying ConnectionPool 6.
• health(): Exposes a {"type": "infinity", "status": "green"|"red", "error": …} dict via show_current_node() for health endpoints 7.

Multi-worker startup (PR #12006): rag/svr/task_executor.py delays each worker's startup by worker_num × 2.0s + random jitter [0, 0.5s], where the worker number is parsed from CONSUMER_NAME. This prevents the thundering-herd startup pattern.

RocksDB Metadata Retry Logic#

_retry_on_meta_contention() in infinity_conn_base.py wraps all CREATE TABLE, CREATE INDEX, and DROP TABLE calls. It retries on error 9003 with exponential backoff + ±50% jitter 8:

• Default: 5 attempts, 50ms base delay → worst-case ~1.5s total budget
• Tunable via INFINITY_META_RETRY_MAX and INFINITY_META_RETRY_BASE_DELAY_MS
• Non-9003 exceptions are re-raised immediately; all wrapped operations use ConflictType.Ignore, making retries idempotent 9

Detection handles both SDK exceptions with an error_code attribute and older SDKs that surface plain Exception((9003, "...")) tuples, plus substring matching on "Resource busy" + "rocksdb" as a fallback 10.

Key Files#

Related: Issue #5809 (segfault) · Issue #8624 (table missing) · PR #12006 (pool size + startup delay)