Chunker Pipeline#

The RAGFlow chunker pipeline is a stage in the document-processing flow (rag/flow/) that takes parser output and splits it into indexable chunks. Two chunker implementations exist under rag/flow/chunker/:

• TokenChunker — token-size/delimiter-based splitting with optional overlap and media context.
• TitleChunker — heading-aware chunking, dispatching to HierarchyTitleChunker or GroupTitleChunker based on configured method.

Both extend ProcessBase / ProcessParamBase from rag/flow/base.py 1, which wraps _invoke() with timing, error handling, and canvas callback integration.

Upstream Data Contract#

Chunkers receive a single validated Pydantic model, TokenChunkerFromUpstream, containing:

The schema uses Pydantic field aliases so upstream components can write json, markdown, etc. directly as output keys.

Format Branching in TokenChunker#

TokenChunker._invoke() reads from_upstream.output_format and branches on three paths 2:

1. Flat text (markdown / text / html): splits via delimiter regex or falls back to naive_merge for token-size merging. Optional children_delimiters apply a secondary split for parent–child chunk relationships (mom field) 3.

2. Structured JSON (json): normalized through _build_json_chunks() 4, which maps doc_type_kwd to ck_type (text / table / image) and extracts PDF position metadata into an internal PDF_POSITIONS_KEY field. Media chunks then receive surrounding text context via _attach_context_to_media_chunks() 5.

3. One-shot mode (delimiter_mode = "one"): emits a single merged chunk regardless of size .

The chunker always sets output_format = "chunks" before returning 7, so the next stage always receives the chunks format.

Format Branching in TitleChunker#

BaseTitleChunker.extract_line_records() 8 normalizes all upstream formats into a uniform list of dicts:

{"text": str, "doc_type_kwd": str, "img_id": str|None, "layout": str, PDF_POSITIONS_KEY: list}

• Flat text formats (markdown / text / html): split on \n, filtered, and converted to text-only records with empty positions 9.
• Structured formats (chunks / json): each item's positions are extracted via extract_pdf_positions() for coordinate-aware downstream processing.

build_chunks_from_record_groups() 10 materializes each group into an output chunk: for flat-text input it concatenates text; for structured input it calls RAGFlowPdfParser.remove_tag() and merge_pdf_positions() to union bounding boxes across records.

PDF Metadata Lifecycle#

PDF coordinate metadata (PDF_POSITIONS_KEY) travels through the pipeline as an internal field and is finalized only at the last step:

1. Extraction: extract_pdf_positions() normalizes any of four raw formats (position tag string, positions list, _pdf_positions, or individual fields) into a canonical list.
2. Accumulation: positions are carried on every internal chunk dict and merged when records are grouped.
3. Preview generation: restore_pdf_text_previews() 11 12 generates thumbnail crops for structured-input chunks and caches them by position key.
4. Finalization: finalize_pdf_chunk() 13 14 converts accumulated positions into the indexed position_int, page_num_int, and top_int fields consumed downstream.

This deferred finalization ensures coordinates are only serialized once, at the chunker's output boundary.

Level Resolution in TitleChunker#

BaseTitleChunker.resolve_title_levels() 15 attempts outline-based level assignment first (matches text against PDF bookmark outlines) and falls back to frequency-based regex matching when outline coverage is below 3%. The hierarchy method selects a target level from the resolved set via resolve_target_level(); the group method groups records into sections by title boundaries and token constraints.

Key Files#