Knowledge Base Document Processing#

Document ingestion in Dify follows a four-stage async pipeline: Extract → Clean → Segment → Index. The final output is DocumentSegment rows in PostgreSQL plus entries in a vector or keyword store. No intermediate extracted files are persisted — only processed text chunks.

The Celery entry point is _document_indexing() in api/tasks/document_indexing_task.py. The runner orchestrates the pipeline in IndexingRunner calling _extract → _transform → _load_segments → _load in sequence.

Stage 1: Extract#

ExtractProcessor.extract() dispatches to per-format extractors based on file extension. All extractors implement BaseExtractor and return list[Document].

File-type routing differs by the ETL_TYPE environment variable (dify vs Unstructured). Key extractors:

• PDF: PdfExtractor uses pypdfium2 to iterate pages and extracts embedded images as markdown links. Images are uploaded to storage and saved as UploadFile records 1.
• DOCX: WordExtractor; DOC (Unstructured mode only): UnstructuredWordExtractor
• Web: FirecrawlWebExtractor, WaterCrawlWebExtractor, JinaReaderWebExtractor 2
• Notion: NotionExtractor

For URL extraction 3, the processor downloads content to a temporary directory, detects content-type from headers, and routes to the appropriate extractor.

Null safety: upload_file may be None when extracting from a URL. The PDF extractor accepts optional tenant/user context and skips image persistence if context is absent 4 5. After extraction, document status transitions to SPLITTING.

Stage 2: Clean#

CleanProcessor.clean() sanitizes text before splitting.

Always applied:

• Strip <| / |> tokens
• Remove ASCII control chars (0x00–0x08, 0x0B–0x0C, 0x0E–0x1F, 0x7F) and U+FFFE

Configurable via pre_processing_rules:

• remove_extra_spaces: collapse 3+ newlines → 2, multiple spaces → one
• remove_urls_emails: strip bare emails/URLs; preserve Markdown ![alt](url) / [text](url) syntax

Stage 3: Segment (Transform)#

ParagraphIndexProcessor.transform() splits cleaned text into chunks, assigns IDs, and extracts multimodal attachments.

Three chunking modes 6:

• automatic: EnhanceRecursiveCharacterTextSplitter, defaults: delimiter \n, max 500 tokens, 50 overlap
• custom: FixedRecursiveCharacterTextSplitter with user-defined separator + fallback chain (\n\n, \n, 。, . , , "")
• hierarchical: same splitter produces parent+child chunks

Each chunk receives a UUID doc_id and SHA content hash doc_hash. Multimodal: image references extracted into SegmentAttachmentBinding rows. Optional LLM-generated summary per chunk (up to 10 workers, 5-min cap, vision-model aware).

Stage 4: Index (Load)#

IndexingRunner._load() indexes chunks based on indexing_technique:

• HIGH_QUALITY: embed via dataset embedding model → vector store (10 parallel threads, distributed by content hash to avoid deadlocks). Calls Vector(dataset).create(documents) 7.
• ECONOMY: Jieba keyword index only. Calls Keyword(dataset).add_texts(documents).

Document status → COMPLETED after all chunks indexed.

Segment Storage#

DatasetDocumentStore.add_documents() writes to the document_segments table. DocumentSegment model 8 key fields: content, index_node_id (vector store key), index_node_hash, position, tokens, word_count, enabled (false during ingestion → true after indexing), status, answer (Q&A mode), keywords (ECONOMY), hit_count.

Parent-Child Chunking (Hierarchical Mode)#

Two parent modes 9:

• paragraph: document → paragraph-sized parents → child chunks
• full-doc: entire document is one parent; children are subsets

Only child chunks enter the vector store. Parents are stored in document_segments unembedded; children in child_chunks table with segment_id FK. At query time: child doc_id → ChildChunk.segment_id → parent DocumentSegment → max-score aggregation across matched children.

Implemented in ParentChildIndexProcessor. Cleanup resolves child index_node_ids via segment_id join; deletes from vector store first; race-condition-safe via precomputed_child_node_ids kwarg 9.

MinerU Plugin Integration#

MinerU is an optional tool plugin — not a built-in extractor. Users invoke it in workflow/agent nodes before feeding results into a knowledge base.

Plugin: tools/mineru/ in langgenius/dify-official-plugins v0.5.7
Supported inputs: PDF, DOC, DOCX, PPT, PPTX, PNG, JPG, JPEG
Outputs: Markdown text, JSON content list, extracted images, optional HTML/DOCX/LaTeX exports

Deployment:

• Local: set base_url to self-hosted server (e.g., http://127.0.0.1:8888)
• Official API: base_url=https://mineru.net + API token

Parsing backends (backend param, default: pipeline): pipeline (CPU-compatible), VLM-based backends (vlm-transformers, vlm-sglang-engine/client, etc.), hybrid backends (v2.7: hybrid-auto-engine recommended). Known issue: non-pipeline backends may fail in certain self-hosted Docker deployments 10.

Async Scheduling & State Transitions#

Document state machine: waiting → parsing → cleaning → splitting → indexing → completed 11

Celery queues: dataset (standard) and priority_dataset (paid plans, cloud edition).
Per-tenant Redis FIFO queue: TenantIsolatedTaskQueue enforces tenant isolation.
Task dispatch happens only after session.commit() to prevent tasks running on uncommitted data 12.

Key Redis guards 12:

• document_{id}_is_retried (600s lock) — retry idempotency
• document_{id}_is_paused (no TTL) — pause signal to running runner
• document_{id}_indexing (600s post-commit) — in-flight guard for enable/disable

Known issue: deleting a knowledge base while documents are actively indexing can leave orphaned document_segments, child_chunks, and pgvector tables 13 14.

Key Source Files#