Code Chunking#

Docling's code chunking system splits source code files into semantically meaningful units (functions, classes, preambles) using Tree-sitter AST parsing. It is implemented in docling-core as a strategy pattern that plugs into HierarchicalChunker.

Architecture Overview#

HierarchicalChunker
  └── code_chunking_strategy: BaseCodeChunkingStrategy (optional)
        └── StandardCodeChunkingStrategy
              └── _INNER_CHUNKERS_BY_LANG (per-language dispatch)
                    ├── Python → _PythonFunctionChunker
                    ├── TypeScript → _TypeScriptFunctionChunker
                    ├── JavaScript → _JavaScriptFunctionChunker
                    ├── C → _CFunctionChunker
                    └── Java → _JavaFunctionChunker

When HierarchicalChunker encounters a CodeItem during document traversal, it delegates to code_chunking_strategy.chunk_code_item() instead of treating the code as plain text. Without a strategy set, code items are serialized and chunked like any other text node.

Key Components#

BaseCodeChunkingStrategy#

BaseCodeChunkingStrategy is the abstract interface any code chunking strategy must implement. Its sole abstract method is chunk_code_item(*, item: CodeItem, doc: DoclingDocument, **kwargs) -> Iterator[CodeChunk].

StandardCodeChunkingStrategy#

StandardCodeChunkingStrategy is the built-in implementation. It lazily instantiates and caches a per-language _CodeChunker from the map 1:

Python, TypeScript, JavaScript, C, Java

For unsupported languages (e.g. Go, Rust), it falls back to emitting a single CodeChunk with chunk_type=CodeChunkType.CODE_BLOCK covering the entire code item 2.

Language Chunkers (_CodeChunker subclasses)#

Each language chunker in _language_code_chunkers.py uses Tree-sitter to parse the AST and extract:

• Functions/methods — one CodeChunk per function, with relevant imports and enclosing class context prepended 3
• Classes without methods — emitted as a class-level chunk 4
• Preamble — code not covered by any other chunk (imports, module-level variables, orphan code) is merged into a single PREAMBLE chunk 5

Large function/method chunks are further split by _ChunkSizeProcessor to stay within token limits, with each sub-chunk named <function>_part_N 6.

CodeChunk / CodeDocMeta#

CodeChunk extends BaseChunk and carries a CodeDocMeta with:

HierarchicalChunker Integration#

HierarchicalChunker accepts code_chunking_strategy: Optional[BaseCodeChunkingStrategy] = None. When set, CodeItem nodes are routed to the strategy instead of the default serializer path 7. HybridChunker (the token-budget-aware wrapper over HierarchicalChunker) inherits this parameter transparently.

All components are re-exported from docling_core.transforms.chunker.

DocumentConverter Limitation#

DocumentConverter (in the main docling package) only accepts formats with registered backends (PDF, DOCX, HTML, etc.) — it does not accept raw source code files (.py, .ts, .java, etc.). You must build a DoclingDocument manually:

from docling_core.types.doc.document import DoclingDocument, DocumentOrigin
from docling_core.types.doc.labels import CodeLanguageLabel
from docling_core.transforms.chunker import HierarchicalChunker, StandardCodeChunkingStrategy

doc = DoclingDocument(name="my_file.py")
doc.origin = DocumentOrigin(filename="my_file.py", mimetype="text/x-python", binary_hash=12345)

with open("my_file.py") as f:
    doc.add_code(text=f.read(), code_language=CodeLanguageLabel.PYTHON)

strategy = StandardCodeChunkingStrategy()
chunker = HierarchicalChunker(code_chunking_strategy=strategy)
chunks = list(chunker.chunk(doc))

See test_code_chunking_strategy.py for worked examples including multi-file repository processing.

Source Files#