Document Chunking#

Docling's chunking system converts a DoclingDocument into a sequence of DocChunk objects, each bounded by a configurable token limit and enriched with provenance metadata. The pipeline is multi-stage: a structural pass follows the document hierarchy, a splitting pass enforces token constraints, and an optional merge pass consolidates undersized adjacent chunks. The primary entry point is HybridChunker, which orchestrates all three passes.

See also: PDF-to-RAG Pipeline (end-to-end workflow) and Document Chunk Metadata and Provenance (data model detail).

Pipeline Architecture#

DoclingDocument
      │
      ▼
HierarchicalChunker ← Stage 1: document-structure split
(doc_chunk.py / hierarchical_chunker.py)
      │ DocChunk[]
      ▼
_split_by_doc_items() ← Stage 2a: split oversize multi-item chunks
      │
      ▼
_split_using_plain_text() ← Stage 2b: semchunk / LineBasedTokenChunker
      │
      ▼
_merge_chunks_with_matching_metadata() ← Stage 3 (merge_peers=True)
      │
      ▼
Iterator[DocChunk]

The full pipeline is wired in HybridChunker.chunk().

Stage 1 — HierarchicalChunker#

HierarchicalChunker walks the document tree using ChunkingDocSerializer to serialize each item to text, grouping content under the current heading context. It emits DocChunk objects at three points:

1. Content chunks 1 — the main path; doc_items is populated from ser_res.spans, headings from the current heading stack, and origin from dl_doc.origin.
2. Heading-only chunks 2 — emitted when always_emit_headings=True and a heading goes out of scope without body content.
3. Trailing headings 3 — any unmatched headings remaining at document end.

The default serializer is ChunkingDocSerializer, which extends MarkdownDocSerializer and uses TripletTableSerializer for tables by default. A custom ChunkingSerializerProvider can substitute any BaseTableSerializer.

Stage 2 — Token-Aware Splitting#

After the hierarchical pass, HybridChunker applies two token-enforcement steps 4:

2a. Split by doc items#

_split_by_doc_items() uses a sliding window over doc_items in an oversize chunk, greedily expanding the window until adding one more item would exceed max_tokens. The result preserves headings and origin in each sub-chunk's DocMeta 5.

2b. Semantic / line-based splitting#

_split_using_plain_text() handles single items that are still too large. For tables (when repeat_table_header=True), it delegates to LineBasedTokenChunker which re-prefixes each body chunk with the header row. For all other text, it uses semchunk — a semantic text splitter — with the available token budget calculated by subtracting the heading/caption overhead from max_tokens.

Stage 3 — Peer Merging#

When merge_peers=True (the default), _merge_chunks_with_matching_metadata() consolidates consecutive undersized chunks that share the same headings into a single DocChunk, accumulating their doc_items and joining their text with self.delim. Merging stops when the token count would exceed max_tokens or the heading context changes 6.

Output: DocChunk / DocMeta#

Every emitted chunk is a DocChunk(text, meta: DocMeta). The DocMeta fields are:

excluded_embed and excluded_llm class-level lists ensure that doc_items, schema_name, version, and origin are omitted when contextualize() builds the embedding string — only headings flows into the embedding payload by default 11.

For the full provenance chain (DocItem.prov → ProvenanceItem → page_no / bbox / charspan), see Document Chunk Metadata and Provenance.

HybridChunker Configuration#

HybridChunker is importable as from docling.chunking import HybridChunker or from docling_core.transforms.chunker.hybrid_chunker import HybridChunker 12. Requires pip install docling-core[chunking] 13.

Tokenizer options: HuggingFaceTokenizer (default) and OpenAITokenizer (requires [chunking-openai]) both implement BaseTokenizer (count_tokens, get_max_tokens, get_tokenizer) 15.

Table Serialization#

The default TripletTableSerializer emits flat "**Column**, row = value" strings that lose column-to-value bindings. For table-heavy documents, switch to MarkdownTableSerializer via a custom provider 16:

from docling_core.transforms.chunker.hierarchical_chunker import (
    ChunkingDocSerializer, ChunkingSerializerProvider,
)
from docling_core.transforms.serializer.markdown import MarkdownParams, MarkdownTableSerializer

class MDTableSerializerProvider(ChunkingSerializerProvider):
    def get_serializer(self, doc):
        return ChunkingDocSerializer(
            doc=doc,
            table_serializer=MarkdownTableSerializer(),
            params=MarkdownParams(compact_tables=True),
        )

chunker = HybridChunker(
    tokenizer=tokenizer,
    repeat_table_header=True,
    serializer_provider=MDTableSerializerProvider(),
)

See Table Export and Serialization for a fuller comparison.

Key Source Files#