Knowledge Base Document Processing#

Document ingestion in Dify follows a four-stage async pipeline: Extract → Clean → Segment → Index. The final outputs are DocumentSegment rows in PostgreSQL and entries in a vector or keyword store. No intermediate extracted files are persisted — only processed text chunks. 1

Celery entry point: _document_indexing() in api/tasks/document_indexing_task.py dispatches work to the dataset or priority_dataset queue.
Orchestrator: IndexingRunner in api/core/indexing_runner.py drives _extract → _transform → _load_segments → _load in sequence.

Stage 1 — Extract#

ExtractProcessor.extract() dispatches to per-format extractors based on file extension and the ETL_TYPE config (dify vs Unstructured). All extractors implement BaseExtractor and return list[Document]. 2

4

URL extraction: load_from_url() downloads the resource to a temp directory, sniffs the content-type or Content-Disposition header to determine the suffix, and routes to the matching extractor. Supported URL content types: application/pdf, text/plain, application/json. 5

Null safety: upload_file may be None when extracting from a URL. PdfExtractor accepts optional tenant/user context and skips image persistence when context is absent. 6 7

After extraction the document status transitions to SPLITTING.

Stage 2 — Clean#

CleanProcessor.clean() sanitizes text before splitting.

Always applied:

• Strips <| / |> tokens
• Removes ASCII control chars (0x00–0x08, 0x0B–0x0C, 0x0E–0x1F, 0x7F) and U+FFFE

Configurable via pre_processing_rules:

8 9

Encoding detection uses charset_normalizer (replacing chardet as of PR #29022).

Stage 3 — Segment (Transform)#

Segmentation is driven by the process_rule.mode and the selected index processor. 10

Chunking Modes#

11 12

Index Processor Selection#

IndexProcessorFactory creates processors based on the doc_form field:

13

Each chunk receives a UUID doc_id and SHA content hash doc_hash. Multimodal image references are extracted into SegmentAttachmentBinding rows. An optional LLM-generated summary can be produced per chunk (up to 10 workers, vision-model aware). 13

Parent-Child (Hierarchical) Mode#

Two parent modes: paragraph (document split into paragraph-sized parents, each further split into children) or full-doc (entire document is one parent). Only child chunks enter the vector store. Parents are stored in document_segments unembedded; children go into the child_chunks table with a segment_id FK. At query time: child doc_id → ChildChunk.segment_id → parent DocumentSegment. 14 15

Stage 4 — Index (Load)#

IndexingRunner._load() indexes chunks based on indexing_technique: 16

DocumentSegment.enabled is set false during ingestion and flipped to true only after each chunk is successfully indexed.

Segment storage: DatasetDocumentStore.add_documents() writes to the document_segments table. Key DocumentSegment fields: content, index_node_id (vector store key), index_node_hash, position, tokens, word_count, enabled, status, answer (Q&A mode), keywords (ECONOMY), hit_count. 17

Token counting: A dedicated token_counter tracks tokens per chunk. A known issue (tracked in #39560) sends all chunks to the embedding plugin unbatched during counting, potentially causing HTTP 413 errors for large documents, while the actual embedding call in cached_embedding.py is correctly batched. 18 19

Async Scheduling & State Transitions#

Document state machine: waiting → parsing → cleaning → splitting → indexing → completed 20

Poll GET /datasets/{dataset_id}/documents/{batch}/indexing-status to track progress.

Celery queues: dataset (standard) and priority_dataset (paid/cloud plans). Both tasks delegate to _document_indexing_with_tenant_queue() for tenant isolation. 21

Per-tenant Redis FIFO queue: TenantIsolatedTaskQueue enforces that documents for the same tenant are processed serially, preventing resource contention.

Key Redis guards 22:

Task dispatch always happens after session.commit() to prevent Celery tasks from running on uncommitted data. 23

Known issue: Deleting a knowledge base while documents are actively indexing can leave orphaned document_segments, child_chunks, and pgvector tables if the cleanup task (clean_document_task) races with an in-progress IndexingRunner. 24 25

MinerU Plugin Integration#

MinerU is an optional tool plugin — not a built-in extractor. Users invoke it in workflow or agent nodes to pre-process documents, then feed the Markdown output into a knowledge base ingestion step.

Plugin location: tools/mineru/ in langgenius/dify-official-plugins

Supported inputs: PDF, DOC, DOCX, PPT, PPTX, PNG, JPG, JPEG 26

Outputs: Markdown text, JSON content list, extracted images, optional HTML/DOCX/LaTeX exports 27

Deployment modes 28:

Parsing backends (backend parameter, default pipeline) 29:

Known issue: Non-pipeline backends may fail in certain self-hosted Docker deployments due to backend compatibility constraints. 30

Integration flow: MineruTool._invoke() → parser_file() → local (_parse_local_v2) or remote (upload + poll); extracted images are re-uploaded via Dify's session file API and their Markdown paths replaced with Dify-hosted preview URLs. 31

Key Source Files#