Document Chunk Metadata and Provenance#

The DocChunk / DocMeta / ProvenanceItem system connects document chunks produced by the chunking pipeline back to the exact locations in the original source document. Each chunk carries a DocMeta object that holds the source DocItem objects, contextual headings, and a DocumentOrigin reference. Every DocItem in turn carries a prov list of ProvenanceItem objects encoding the page number, bounding box, and character span of that element in the source file.

This three-layer design—DocChunk → DocMeta.doc_items → DocItem.prov—makes it possible for downstream consumers (RAG pipelines, UI viewers, citation generators) to trace any chunk of text back to a specific page region in the original document.

Data Model#

ProvenanceItem#

ProvenanceItem is a lightweight pointer into the original document:

It is attached directly to DocItem.prov as a list, allowing a single logical item to span multiple regions (e.g., a paragraph that breaks across columns).

Note: For media formats (audio/video), the analogous mechanism is TrackSource via the source field, not prov. The code comment confirms ProvenanceItem should be used for documents with layout (PDF, DOCX, PPTX, HTML, Markdown) 1.

DocumentOrigin#

DocumentOrigin captures file-level metadata:

• mimetype — validated against known MIME types
• binary_hash — Uint64 hash of the original file bytes
• filename — original filename (no path)
• uri — optional URI (https://, file://, s3://, …)

DocMeta#

DocMeta is the metadata payload attached to every DocChunk:

Two class-level lists, excluded_embed and excluded_llm, omit schema_name, version, doc_items, and origin from the embedding and LLM context strings respectively — so only headings flows into embedding/LLM payloads by default.

BaseChunk and BaseMeta (from docling_core/transforms/chunker/base.py) define the text + meta contract and the contextualize() method used to build embedding strings.

How Chunks Are Built (HierarchicalChunker)#

HierarchicalChunker (in docling_core/transforms/chunker/hierarchical_chunker.py) is the primary chunker that produces DocChunk objects. Its chunk() method emits DocChunk at three points:

1. Heading-only chunks 2 — emitted when always_emit_headings=True and a heading goes out of scope without body content. doc_items and headings are populated from heading_by_level; origin is omitted.

2. Content chunks 3 — the main path. doc_items comes from ser_res.spans (items touched during serialization); headings from heading_by_level; origin set to dl_doc.origin from the input DoclingDocument.

3. Trailing headings 4 — any remaining unmatched headings at document end. Same as case 1, no origin.

The captions field is never populated by HierarchicalChunker and is marked deprecated in DocMeta 5.

Format-Specific Provenance Handling#

PDF#

PDF processing goes through the Reading Order model (docling/models/stages/reading_order/readingorder_model.py), which assembles the final DoclingDocument from parsed page elements. It creates ProvenanceItem objects with real page_no and bbox values derived from the PDF parser output 6, then passes them to doc.add_text(), doc.add_table(), doc.add_picture(), etc. The lower-level DoclingParseDocumentBackend handles raw PDF parsing but does not itself create ProvenanceItem objects.

DOCX#

The DOCX backend (msword_backend.py) does not set ProvenanceItem objects on document items. Word documents lack a fixed page layout during parsing, so no page_no or bbox are recorded; DocItem.prov remains an empty list for DOCX-sourced items.

Markdown#

The MarkdownDocumentBackend produces layout-less output — it adds text, headings, tables, and pictures without prov entries. It does set DocumentOrigin at the document level with mimetype="text/markdown" and the file's binary_hash 7, so DocMeta.origin is populated for downstream consumers but per-item location is unavailable.

HTML#

HTMLDocumentBackend centralises provenance creation in helper methods _make_prov() 8 and _make_text_prov() 9 that construct ProvenanceItem objects from rendered layout data before calling document-addition methods.

ODP, PPTX, XLSX#

All three backends create ProvenanceItem objects inline when adding document items:

• OpenDocument (opendocument_backend.py) — text, table, and picture items 10
• PowerPoint (mspowerpoint_backend.py) — shape text and slide notes 11
• Excel (msexcel_backend.py) — singleton cells, tables, and images 12

In all these backends, page_no corresponds to the slide/sheet index (1-based).

Key Source References#