Table Chunking#

Table chunking refers to the process of splitting large tables across token-bounded chunk boundaries while preserving enough header context for each resulting chunk to be independently usable. It is a sub-problem within HybridChunker's broader document chunking pipeline, activated when a serialized table item exceeds max_tokens.

How Table Splitting is Triggered#

HybridChunker.chunk() runs three passes in sequence 1:

1. HierarchicalChunker — serializes the document, grouping content under heading context.
2. _split_by_doc_items() — sliding window over multi-item chunks that still exceed max_tokens.
3. _split_using_plain_text() → segment() — handles single items that are still too large.

Tables reach the segment() method as a single-item DocChunk containing one TableItem. The table path inside segment() is activated when repeat_table_header=True and the serializer is a ChunkingDocSerializer and the chunk contains exactly one TableItem 2.

LineBasedTokenChunker — The Core Splitting Mechanism#

When the table path is active, segment() calls table_serializer.get_header_and_body_lines() to split the serialized table into header rows and body rows 3. It then constructs a LineBasedTokenChunker with:

• prefix = the header rows (plus any preamble/caption text before the first row)
• omit_prefix_on_overflow = self.omit_header_on_overflow

LineBasedTokenChunker.chunk_text() iterates through body lines and greedily packs them into chunks, prepending the prefix (header) to each new chunk 4. When a line is too large to share a chunk with the prefix, the chunker checks omit_prefix_on_overflow:

• False (default): the line is split by token limit using binary search 5, keeping the prefix intact across all chunks.
• True: the prefix is dropped for that overflowing line, preserving the line's integrity at the cost of inconsistent formatting. A UserWarning is emitted when this occurs 6.

Binary Search Split#

split_by_token_limit() binary-searches over character indices to find the longest head that stays within token_limit, then optionally snaps back to the nearest whitespace boundary.

Key Configuration Parameters#

All parameters are set on HybridChunker 7:

Serialization Format: Triplet vs. Markdown#

HybridChunker's default serializer is ChunkingDocSerializer, which uses TripletTableSerializer by default. Triplet format emits flat "**Column**, row = value" strings that lose cell-to-column-header bindings — the core semantic payload of structured tables 8.

For RAG pipelines with significant table content (in one evaluation, 53% of chunks from HTML documents were table chunks 8), switch to MarkdownTableSerializer via a custom provider:

from docling_core.transforms.chunker.hierarchical_chunker import (
    ChunkingDocSerializer, ChunkingSerializerProvider,
)
from docling_core.transforms.serializer.markdown import MarkdownParams, MarkdownTableSerializer

class MDTableSerializerProvider(ChunkingSerializerProvider):
    def get_serializer(self, doc):
        return ChunkingDocSerializer(
            doc=doc,
            table_serializer=MarkdownTableSerializer(),
            params=MarkdownParams(compact_tables=True),
        )

chunker = HybridChunker(
    tokenizer=tokenizer,
    repeat_table_header=True,
    serializer_provider=MDTableSerializerProvider(),
)

See the advanced chunking and serialization docs for the canonical example. The maintainers recommend this ChunkingSerializerProvider-based approach over post-processing workarounds, since post-processing requires round-tripping back to chunk.meta.doc_items which may be lost after persistence to a vector store 8.

Known Issue: Missing | - | Separator on Continuation Chunks#

When using MarkdownTableSerializer with repeat_table_header=True, continuation chunks (all chunks after the first) may be missing the | - | header separator line, causing Markdown parsers to treat the block as plain text 9. This was reported against docling-core v2.85.0.

Key Source Files#