Knowledge Base Document Processing#
Document ingestion in Dify follows a four-stage async pipeline: Extract β Clean β Segment β Index. The final outputs are DocumentSegment rows in PostgreSQL and entries in a vector or keyword store. No intermediate extracted files are persisted β only processed text chunks.
Celery entry point: _document_indexing() in api/tasks/document_indexing_task.py dispatches work to the dataset or priority_dataset queue.
Orchestrator: IndexingRunner in api/core/indexing_runner.py drives _extract β _transform β _load_segments β _load in sequence.
Stage 1 β Extract#
ExtractProcessor.extract() dispatches to per-format extractors based on file extension and the ETL_TYPE config (dify vs Unstructured). All extractors implement BaseExtractor and return list[Document].
| Source | Extractor(s) |
|---|---|
.pdf | PdfExtractor (pypdfium2; extracts embedded images as Markdown links) |
.docx | WordExtractor |
.doc (Unstructured only) | UnstructuredWordExtractor |
.xlsx / .xls | ExcelExtractor |
.csv | CSVExtractor |
.md / .mdx | MarkdownExtractor or UnstructuredMarkdownExtractor |
.html / .htm | HtmlExtractor |
.ppt / .pptx | UnstructuredPPTExtractor / UnstructuredPPTXExtractor |
.txt (fallback) | TextExtractor |
| Notion datasource | NotionExtractor |
| Web (Firecrawl / WaterCrawl / Jina) | FirecrawlWebExtractor / WaterCrawlWebExtractor / JinaReaderWebExtractor |
URL extraction: load_from_url() downloads the resource to a temp directory, sniffs the content-type or Content-Disposition header to determine the suffix, and routes to the matching extractor. Supported URL content types: application/pdf, text/plain, application/json.
Null safety: upload_file may be None when extracting from a URL. PdfExtractor accepts optional tenant/user context and skips image persistence when context is absent.
After extraction the document status transitions to SPLITTING.
Stage 2 β Clean#
CleanProcessor.clean() sanitizes text before splitting.
Always applied:
- Strips
<|/|>tokens - Removes ASCII control chars (0x00β0x08, 0x0Bβ0x0C, 0x0Eβ0x1F, 0x7F) and U+FFFE
Configurable via pre_processing_rules:
| Rule | Effect |
|---|---|
remove_extra_spaces | Collapses 3+ consecutive newlines β 2; multiple horizontal spaces β one |
remove_urls_emails | Strips bare URLs and email addresses; preserves Markdown  / [text](url) syntax |
Encoding detection uses charset_normalizer (replacing chardet as of PR #29022).
Stage 3 β Segment (Transform)#
Segmentation is driven by the process_rule.mode and the selected index processor.
Chunking Modes#
Mode (process_rule.mode) | Splitter | Default params |
|---|---|---|
automatic | EnhanceRecursiveCharacterTextSplitter | delimiter \n, max 500 tokens, 50 overlap |
custom | FixedRecursiveCharacterTextSplitter | User-defined separator + fallback chain (\n\n, \n, γ, . , , "") |
hierarchical | FixedRecursiveCharacterTextSplitter | Separate parent and child chunk settings |
Index Processor Selection#
IndexProcessorFactory creates processors based on the doc_form field:
doc_form | Processor | Description |
|---|---|---|
text_model | ParagraphIndexProcessor | Standard text chunking |
qa_model | QAIndexProcessor | LLM-generates Q&A pairs from each chunk |
hierarchical_model | ParentChildIndexProcessor | Parent + child chunks; only children are embedded |
Each chunk receives a UUID doc_id and SHA content hash doc_hash. Multimodal image references are extracted into SegmentAttachmentBinding rows. An optional LLM-generated summary can be produced per chunk (up to 10 workers, vision-model aware).
Parent-Child (Hierarchical) Mode#
Two parent modes: paragraph (document split into paragraph-sized parents, each further split into children) or full-doc (entire document is one parent). Only child chunks enter the vector store. Parents are stored in document_segments unembedded; children go into the child_chunks table with a segment_id FK. At query time: child doc_id β ChildChunk.segment_id β parent DocumentSegment.
Stage 4 β Index (Load)#
IndexingRunner._load() indexes chunks based on indexing_technique:
| Technique | Behavior |
|---|---|
HIGH_QUALITY | Embeds chunks via the dataset's embedding model β writes to vector store (10 parallel threads, distributed by content hash to avoid deadlocks) |
ECONOMY | Jieba keyword index only; no embeddings |
DocumentSegment.enabled is set false during ingestion and flipped to true only after each chunk is successfully indexed.
Segment storage: DatasetDocumentStore.add_documents() writes to the document_segments table. Key DocumentSegment fields: content, index_node_id (vector store key), index_node_hash, position, tokens, word_count, enabled, status, answer (Q&A mode), keywords (ECONOMY), hit_count.
Token counting: A dedicated token_counter tracks tokens per chunk. A known issue (tracked in #39560) sends all chunks to the embedding plugin unbatched during counting, potentially causing HTTP 413 errors for large documents, while the actual embedding call in cached_embedding.py is correctly batched.
Async Scheduling & State Transitions#
Document state machine: waiting β parsing β cleaning β splitting β indexing β completed
Poll GET /datasets/{dataset_id}/documents/{batch}/indexing-status to track progress.
Celery queues: dataset (standard) and priority_dataset (paid/cloud plans). Both tasks delegate to _document_indexing_with_tenant_queue() for tenant isolation.
Per-tenant Redis FIFO queue: TenantIsolatedTaskQueue enforces that documents for the same tenant are processed serially, preventing resource contention.
Key Redis guards :
| Key | TTL | Purpose |
|---|---|---|
document_{id}_is_retried | 600 s | Retry idempotency lock |
document_{id}_is_paused | None | Pause signal to running runner |
document_{id}_indexing | 600 s | In-flight guard for enable/disable operations |
Task dispatch always happens after session.commit() to prevent Celery tasks from running on uncommitted data.
Known issue: Deleting a knowledge base while documents are actively indexing can leave orphaned document_segments, child_chunks, and pgvector tables if the cleanup task (clean_document_task) races with an in-progress IndexingRunner.
MinerU Plugin Integration#
MinerU is an optional tool plugin β not a built-in extractor. Users invoke it in workflow or agent nodes to pre-process documents, then feed the Markdown output into a knowledge base ingestion step.
Plugin location: tools/mineru/ in langgenius/dify-official-plugins
Supported inputs: PDF, DOC, DOCX, PPT, PPTX, PNG, JPG, JPEG
Outputs: Markdown text, JSON content list, extracted images, optional HTML/DOCX/LaTeX exports
Deployment modes :
| Mode | base_url | Auth |
|---|---|---|
| Local (self-hosted) | e.g. http://127.0.0.1:8888 | token optional |
| Official API | https://mineru.net | API token required |
Parsing backends (backend parameter, default pipeline) :
| Backend | Notes |
|---|---|
pipeline | CPU-compatible; broadest deployment support |
vlm-transformers, vlm-sglang-*, vlm-vllm-* | VLM-based; version-specific (v2.0βv2.6) |
hybrid-auto-engine (v2.7, recommended) | Combines pipeline + VLM |
hybrid-http-client | Hybrid HTTP variant |
Known issue: Non-pipeline backends may fail in certain self-hosted Docker deployments due to backend compatibility constraints.
Integration flow: MineruTool._invoke() β parser_file() β local (_parse_local_v2) or remote (upload + poll); extracted images are re-uploaded via Dify's session file API and their Markdown paths replaced with Dify-hosted preview URLs.
Key Source Files#
| File | Role |
|---|---|
api/core/rag/extractor/extract_processor.py | Format dispatch; file/URL/Notion/web routing |
api/core/rag/extractor/pdf_extractor.py | PDF text + image extraction via pypdfium2 |
api/core/rag/cleaner/clean_processor.py | Text sanitization |
api/core/indexing_runner.py | Orchestrates Extract β Clean β Segment β Index |
api/tasks/document_indexing_task.py | Celery task entry points |
api/core/rag/index_processor/index_processor_base.py | BaseIndexProcessor; splitter selection via _get_splitter() |
api/core/rag/index_processor/processor/paragraph_index_processor.py | Standard chunking + loading |
api/core/rag/index_processor/processor/parent_child_index_processor.py | Hierarchical chunking + loading |
api/models/dataset.py | DocumentSegment, ChildChunk DB models |
api/services/dataset_service.py | DocumentService β lifecycle ops (retry, pause, batch enable/disable) |
api/core/rag/pipeline/queue.py | TenantIsolatedTaskQueue β per-tenant Redis FIFO |
api/core/rag/embedding/cached_embedding.py | Batched embedding with caching |
api/tasks/clean_document_task.py | Teardown: vector index + PG rows + object storage |