Dosu LogoDosu Logo
Ask
Join our Discord
ragflowPublic
InfiniFlow
Documentsragflow
Internal Compilation Artifact Indexing
Internal Compilation Artifact Indexing
Type
Topic
Status
Published
Created
Jul 21, 2026
Updated
Jul 21, 2026

Internal Compilation Artifact Indexing (available_int=0)#

available_int is an integer field stored on every chunk/row in the RAGFlow document store (Elasticsearch or Infinity). A value of 1 means the row is a normal, user-facing chunk that participates in hybrid BM25 + KNN retrieval. A value of 0 flags the row as an internal compilation artifact — it is persisted to the same index for pipeline bookkeeping but is invisible to the standard retriever.

Why it exists#

RAGFlow's knowledge compilation pipelines (TOC extraction, RAPTOR summarization, wiki MAP/REDUCE/PLAN/REFINE, knowledge-graph structure compilation, dataset navigation trees) produce large numbers of intermediate or structural rows that must survive restarts and enable incremental re-runs. Storing them in the same document store index avoids a separate persistence layer. The available_int=0 flag is the mechanism that prevents those rows from polluting user-facing search results.

How the standard retriever enforces it#

Dealer.retrieval() in rag/nlp/search.py always passes "available_int": 1 in the search request, so only normal chunks are returned. The lower-level get_filters() method propagates available_int as a first-class filter condition, allowing callers to explicitly request rows with available_int=0 for internal queries.

On the Infinity backend, available_int required a special fix: because Python treats 0 as falsy, an earlier generic filter-builder was silently dropping the condition. PR #14416 patched infinity_conn_base.py to use explicit if v == 0 / elif v == 1 comparisons instead of if not v: continue, ensuring available_int=0 queries are preserved.


Artifact Types Marked available_int=0#

TOC entries#

When the Extractor component detects field_name == "toc", _build_TOC() creates a single synthetic chunk containing the full JSON table of contents. It sets available_int=0 and toc_kwd="toc" so the TOC row is excluded from free-text retrieval; it is only read back by the toc_enhance re-ranking step (merged via retrieval_by_toc()).

Wiki pipeline intermediate rows (artifact_map_extract)#

The MAP phase of the wiki/artifact compilation pipeline uses _wiki_build_resume_doc() to write one per-chunk extract row per source chunk. The function explicitly omits embedding vectors and tokenized text fields and sets available_int=0. The docstring states the dual intent: "Intentionally omits q_<dim>_vec / content_ltks / content_sm_ltks so retrievers cannot surface this row; also sets available_int=0 which most ragflow retrievers already filter on." These rows act as a resume cache — the REDUCE phase reads them back by filtering on compile_kwd rather than available_int.

Knowledge-graph / structure compilation rows#

_struct_upsert_graph_json() in rag/advanced_rag/knowlege_compile/structure.py stores the serialized hypergraph JSON for each document with knowledge_graph_kwd="graph" and available_int=0. This makes the raw graph data available for the canvas graph view without exposing it to chunk retrieval.

Dataset navigation trees#

_make_nav_doc_row() and _make_nav_cluster_row() in rag/advanced_rag/knowlege_compile/dataset_nav.py write document-leaf and cluster-node rows with available_int=0. The dedicated query function search_dataset_nav() is the "sanctioned read seam" (its docstring's words) — callers query nav rows explicitly by compile_kwd filter, bypassing the standard available_int=1 gate, and use the returned document IDs to scope a subsequent chunk retrieval.


How Internal Queries Bypass the Filter#

Pipelines that need to read back available_int=0 rows bypass the Dealer.retrieval() path entirely. They call lower-level document store methods (search, get, update) directly with an explicit compile_kwd filter and no available_int restriction — or explicitly pass available_int=0 as a filter condition. This pattern is visible in:

  • _wiki_load_resume_map() reading back artifact_map_extract rows
  • search_dataset_nav() reading nav-tree nodes
  • Graph-JSON read-back in rebuild_structure_graph_json()

Key Files#

FileRole
rag/nlp/search.pyStandard retriever — enforces available_int=1 gate
common/doc_store/infinity_conn_base.pyInfinity filter builder — explicit zero-safe handling of available_int
rag/flow/extractor/extractor.pyTOC builder — sets available_int=0 + toc_kwd="toc"
rag/advanced_rag/knowlege_compile/wiki.pyWiki MAP resume rows — available_int=0
rag/advanced_rag/knowlege_compile/structure.pyGraph JSON rows — available_int=0, knowledge_graph_kwd="graph"
rag/advanced_rag/knowlege_compile/dataset_nav.pyNav-tree nodes — available_int=0; search_dataset_nav() for targeted reads
Documents
Agent Import and DSL Compatibility
Agent Prompt Processing
Agent Retrieval
API Authorization
API Error Codes
Ascend Inference Pipeline
Authentication
Azure OpenAI Integration
Canvas Architecture
Chat Assistant Configuration
Chat Completion API
Chrome for Testing ARM64 Support
Chunk Metadata Extraction
Chunker Pipeline
Compilation Template Management
Component Variable Propagation
Connection and Resource Management
Connector Architecture
Connector Document Sync
Conversation Session Management
What is the complete API flow for building a frontend UI with RAGFlow, covering dialogs, conversations, message history, streaming responses, and deletion?
Database Migrations
Dataflow Pipeline Execution
Dataset Access Control
Dataset Configuration UI
Dataset Parsing Mode
Dataset Parsing Status
DeepDoc Model Distribution
DeepDoc Model Path Resolution
DeepDoc Module
Dify External Knowledge Integration
Docker Build Configuration
Document Parsing Pipeline
Elasticsearch Index Management
Embedding Pipeline
Embedding Vector Validation
Encrypted Storage
Figure Description Language Propagation
Frontend Build and Deployment
GPU and Accelerator Support
Hybrid Search and Retrieval
Infinity Database Stability
Internal Compilation Artifact Indexing
Keyword Extraction
Knowledge Compilation Pipeline
Knowledge Graph
Knowledge Graph Retrieval
Knowledge Graph Visualization
Layout Element Overlap Detection
LLM Configuration and Selection
LLM Driver Integration
LLM Provider Integration
MCP Server Integration
Media Context Configuration
Metadata Filtering
MinerU Configuration and Provider Resolution
MinerU PDF Parsing
Model Provider Architecture
Model Selection UI
Model Thinking and Reasoning
Multi-Architecture Docker Support
Multi-Backend Object Storage
Multi-Page Table and Element Handling
Multilingual Search Tokenization
Native Library Build and Linking
OCR Backend and Model Loading
Parser Configuration
Parser Output Lifecycle
Parser-Chunk Contract
PDF Chunk Position Rendering
Picture Chunker Media Processing
Pipeline Canvas Architecture
Provider Configuration Persistence
Provider Model Discovery
Python Dependency Management
RAGFlow Python SDK
Redis Cache Architecture
Retrieval API
Retrieval Pipeline
SSRF Protection
Table Column Field Normalization
Table Structure Parsing
Task Cancellation
Tenant Model Resolution
Text2SQL
TSR Coordinate System Alignment