Knowledge Compilation Pipeline#

Overview#

Introduced in v0.27.0 (PR #16515), the Knowledge Compilation Pipeline replaced the older GraphRAG engine as RAGFlow's primary mechanism for producing structured knowledge artifacts from a Knowledge Base (KB). It lives under rag/advanced_rag/knowlege_compile/ and is a template-driven, multi-phase pipeline that processes all eligible documents in a KB to produce wiki pages, entity/relation graphs, topics, and other artifact types.

The pipeline is not limited to Knowledge Graphs. The active template determines what artifact is built 1:

Templates are YAML files stored under api/db/init_data/compilation_templates/ and managed via REST endpoints at api/apps/restful_apis/compilation_template_api.py 1.

Pipeline Phases: MAP → REDUCE → PLAN → REFINE#

The public entry point is run_wiki() in rag/svr/task_executor_refactor/dataset_wiki_generator.py. It orchestrates four sequential phases implemented in rag/advanced_rag/knowlege_compile/wiki.py 2.

MAP (per doc, parallel batches) → REDUCE (KB-wide dedup) → PLAN (page reconciliation) → REFINE (page writing)

MAP — Per-document extraction#

wiki_map_from_chunks() processes document chunks in batches of 64 3 across all eligible documents. For each batch, it calls the LLM via gen_json() to extract entities, concepts, claims, relations, and topics as JSON.

Incremental logic: Each chunk is fingerprinted by hash. MAP compares current chunk hashes against stored artifact_map_extract ES rows: UNCHANGED chunks are skipped; NEW and CHANGED are re-extracted; DELETED rows are purged 4. Extracts are persisted to ES as non-searchable artifact_map_extract rows.

Progress is reported at 5%–65% of overall task progress 5.

REDUCE — KB-wide canonicalization#

wiki_reduce_from_extracts() deduplicates entities and concepts across all documents in three sub-phases:

1. Exact dedup — normalize by (normalize(name), type) for entities, normalize(term) for concepts.
2. Embedding dedup — cosine similarity; pairs ≥ 0.95 auto-merge; borderline pairs escalate.
3. LLM disambiguation — batched LLM calls resolve ambiguous pairs.

A resume cache (artifact_reduce_result) is keyed by an input hash fingerprint of the MAP state — re-runs skip REDUCE if MAP produced no delta 6.

PLAN — Page reconciliation#

wiki_plan_from_reduction() compares the canonicalized entities/concepts against existing artifact_page ES rows using top-1 KNN embedding search. Each item is classified as UPDATE (cosine ≥ 0.95), MAYBE, or CREATE. MAYBE items go to an LLM call for final classification. A single planning LLM call then produces {pages: [...], estimated_page_count, compilation_notes}.

Resume cache: returns cached plan if the stored input hash equals the current REDUCE output hash 7.

REFINE — Page writing#

wiki_refine_from_plan() writes the final wiki pages in parallel (bounded by asyncio.Semaphore). For each page:

• Assembles evidence from claims matching entity names, with fallback to chunk_ids from the entity/concept itself.
• Builds a context budget capped at 60k chars.
• Calls the writer LLM; for UPDATE actions, merges new and existing content with an LLM, falling back to new content if merged output shrinks below 70% of the max input.
• Transforms [[slug]] wikilinks to artifact/{kb_id}/{slug} URLs.

Final pages land in ES as searchable artifact_page rows 8. Entity and relation rows (artifact_entity, artifact_relation) are also materialized for the canvas graph view via build_wiki_page_graph().

Document Eligibility: parser_config Template Assignment#

A document participates in the wiki/artifact pipeline only if its parser_config.compilation_template_group_id resolves to at least one template with kind == "artifacts" 9.

Resolution chain 10:

1. Read parser_config.compilation_template_group_id (or parser_config.ext.compilation_template_group_id).
2. Call CompilationTemplateGroupService.resolve_template_ids(group_id, tenant_id) for each group_id.
3. Load each template via CompilationTemplateService.get_saved() and check config.kind.
4. Only templates where kind == "artifacts" make the document eligible.

First-template-wins: The first eligible template's llm_id and example field are used as the canonical chat model and writer example for the KB-wide REDUCE / PLAN / REFINE phases 11. Documents without a matching template are silently skipped.

Compilation progress is tracked on the Knowledgebase model via artifact_task_id / artifact_task_finish_at fields 1.

Multi-level Error Handling and Fault Isolation#

The pipeline applies graceful degradation at three scopes:

Chunk/batch level (MAP): Individual batch failures are caught and logged; the loop continues to the next batch. Aggregate MAP results remain usable 12.

LLM call level: Timeouts or failures return empty extracts / empty JSON rather than crashing. Evidence fallback ensures REFINE can still attribute pages to source chunks via entity chunk_ids when claims are sparse 4.

Phase level (REDUCE / PLAN / REFINE): All three are wrapped in a single try/except 13. Failure stops the pipeline at that phase and sets progress to -1 (error state), but prior phases' ES rows remain intact for the next run's resume caches.

ES I/O: Batch search failures fall back to per-ID get(). Insert failures for artifact_page, artifact_entity, and artifact_relation rows log the exception but do not fail the task — the user still receives a success message 14. Progress callbacks are also wrapped in try/except to prevent UI feedback from crashing the task 15.

Model resolution: Chat model resolution errors fall back to the tenant default model 16.

Key Source References#