Internal Compilation Artifact Indexing (available_int=0)#

available_int is an integer field stored on every chunk/row in the RAGFlow document store (Elasticsearch or Infinity). A value of 1 means the row is a normal, user-facing chunk that participates in hybrid BM25 + KNN retrieval. A value of 0 flags the row as an internal compilation artifact — it is persisted to the same index for pipeline bookkeeping but is invisible to the standard retriever.

Why it exists#

RAGFlow's knowledge compilation pipelines (TOC extraction, RAPTOR summarization, wiki MAP/REDUCE/PLAN/REFINE, knowledge-graph structure compilation, dataset navigation trees) produce large numbers of intermediate or structural rows that must survive restarts and enable incremental re-runs. Storing them in the same document store index avoids a separate persistence layer. The available_int=0 flag is the mechanism that prevents those rows from polluting user-facing search results.

How the standard retriever enforces it#

Dealer.retrieval() in rag/nlp/search.py always passes "available_int": 1 in the search request, so only normal chunks are returned. The lower-level get_filters() method propagates available_int as a first-class filter condition, allowing callers to explicitly request rows with available_int=0 for internal queries.

On the Infinity backend, available_int required a special fix: because Python treats 0 as falsy, an earlier generic filter-builder was silently dropping the condition. PR #14416 patched infinity_conn_base.py to use explicit if v == 0 / elif v == 1 comparisons instead of if not v: continue, ensuring available_int=0 queries are preserved.

Artifact Types Marked available_int=0#

TOC entries#

When the Extractor component detects field_name == "toc", _build_TOC() creates a single synthetic chunk containing the full JSON table of contents. It sets available_int=0 and toc_kwd="toc" so the TOC row is excluded from free-text retrieval; it is only read back by the toc_enhance re-ranking step (merged via retrieval_by_toc()).

Wiki pipeline intermediate rows (artifact_map_extract)#

The MAP phase of the wiki/artifact compilation pipeline uses _wiki_build_resume_doc() to write one per-chunk extract row per source chunk. The function explicitly omits embedding vectors and tokenized text fields and sets available_int=0. The docstring states the dual intent: "Intentionally omits q_<dim>_vec / content_ltks / content_sm_ltks so retrievers cannot surface this row; also sets available_int=0 which most ragflow retrievers already filter on." These rows act as a resume cache — the REDUCE phase reads them back by filtering on compile_kwd rather than available_int.

Knowledge-graph / structure compilation rows#

_struct_upsert_graph_json() in rag/advanced_rag/knowlege_compile/structure.py stores the serialized hypergraph JSON for each document with knowledge_graph_kwd="graph" and available_int=0. This makes the raw graph data available for the canvas graph view without exposing it to chunk retrieval.

Dataset navigation trees#

_make_nav_doc_row() and _make_nav_cluster_row() in rag/advanced_rag/knowlege_compile/dataset_nav.py write document-leaf and cluster-node rows with available_int=0. The dedicated query function search_dataset_nav() is the "sanctioned read seam" (its docstring's words) — callers query nav rows explicitly by compile_kwd filter, bypassing the standard available_int=1 gate, and use the returned document IDs to scope a subsequent chunk retrieval.

How Internal Queries Bypass the Filter#

Pipelines that need to read back available_int=0 rows bypass the Dealer.retrieval() path entirely. They call lower-level document store methods (search, get, update) directly with an explicit compile_kwd filter and no available_int restriction — or explicitly pass available_int=0 as a filter condition. This pattern is visible in:

• _wiki_load_resume_map() reading back artifact_map_extract rows 1
• search_dataset_nav() reading nav-tree nodes 2
• Graph-JSON read-back in rebuild_structure_graph_json() 3

Key Files#