Building a Modular IDP Pipeline with Docling Components#

Context and Goals#

Modern document processing workloads rarely fit into a single model. A scanned medical intake form, a dense financial report, and a multi-column technical specification all share one property: each page contains multiple distinct content regions, and no single processing engine handles all of them optimally. A table requires structure-aware parsing. Printed text benefits from standard OCR. Handwritten annotations demand Handwritten Text Recognition (HTR). Signatures and stamps need visual classification. Running the wrong tool on a region wastes computation and degrades accuracy.

This guide describes an architecture for addressing this challenge using a scatter-gather pattern:

1. A layout detector identifies all content regions on a page and assigns each a structural label (text, table, picture, form, etc.)
2. Each region is scattered — routed to the most appropriate specialized processing tool based on its label, confidence, and domain-specific logic
3. Results are gathered and assembled into a single unified document

The critical design decision in this guide is that Docling's components are used as standalone building blocks, not as a monolithic pipeline. Rather than feeding documents through DocumentConverter and receiving a completed DoclingDocument, this architecture extracts the individual components — LayoutPredictor, LayoutPostprocessor, and DoclingDocument — and wires them together with custom orchestration logic.

This approach is appropriate when:

• You need routing decisions to occur before specialized processing runs (not after)
• You need to integrate tools outside the Docling ecosystem (external HTR engines, proprietary classifiers, domain-specific table parsers)
• You require different SLA tiers or parallel execution strategies for different region types
• The overhead of running the full pipeline (OCR, table parsing) on regions that will be discarded is unacceptable

The guide covers each component in isolation — its interface, limitations, and correct usage — before assembling them into the complete pipeline architecture.

Section 1: Docling's Two Pipeline Modes#

Docling provides two distinct pipeline implementations for document conversion, each designed for different use cases. This section provides factual background on what each pipeline does — engineers building modular IDP systems will extract individual components from these pipelines rather than using them directly.

StandardPdfPipeline#

StandardPdfPipeline is Docling's production PDF conversion pipeline, implementing a multi-threaded staged architecture where each stage runs in its own worker thread 1. Models are initialized once at construction time and shared read-only across worker threads 2.

Stage Topology#

Pages flow through five threaded stages in strict sequence, followed by sequential cross-page assembly 2:

Preprocess → OCR → Layout → Table → Assemble → Reading Order → Heading Hierarchy → Enrichment

Critical ordering note: OCR runs before layout analysis, not after 2. This means text cells are already available when the layout model executes. When using layout components standalone, this ordering constraint disappears — but it clarifies why the pipeline's design decisions differ from a typical modular architecture.

Stage Descriptions#

After all pages complete, the pipeline runs sequential cross-page models:

• ReadingOrderModel: Determines element ordering, resolves caption/footnote attachments 4
• HeadingHierarchyModel: Infers heading levels, optionally uses PDF bookmarks extracted before pipeline start 5 2
• enrichment_pipe: Post-document enrichment models run after assembly — CodeFormulaVlmModel and DocumentPictureClassifier are disabled by default (do_code_enrichment=False, do_formula_enrichment=False, do_picture_classification=False) 6

Layout Model Selection#

The default layout model is DOCLING_LAYOUT_HERON 7. Alternative models in the Egret family offer different accuracy/speed tradeoffs:

• DOCLING_LAYOUT_EGRET_MEDIUM 8
• DOCLING_LAYOUT_EGRET_LARGE 9
• DOCLING_LAYOUT_EGRET_XLARGE 10

PDF Backends#

The pipeline supports two PDF backends for page parsing:

• pypdfium2: Pure Python binding to PDFium 11
• docling-parse: Rust-accelerated parser with streaming capabilities 12

The pipeline does not use qpdf — the available backends are pypdfium2 and docling-parse only.

VlmPipeline#

VlmPipeline processes each page with a single Vision-Language Model rather than composing results from separate layout/OCR/table models 13. It produces a DoclingDocument directly via structured token formats — no separate layout, OCR, or table stages.

Processing Flow#

initialize_page → VlmConvertModel → _assemble_document → enrichment_pipe

1. initialize_page: Loads page backend and optionally fetches segmented page for text override when force_backend_text=True
2. VlmConvertModel: Scales page image, builds prompt, calls engine.predict_batch(), stores raw model response
3. _assemble_document: Parses response format (DocTags, DOCLANG, Markdown, HTML, etc.) into DoclingDocument
4. enrichment_pipe: Reserved for future post-assembly stages (currently empty)

Key Models and Presets#

The pipeline defaults to the granite_docling preset, which uses IBM's Granite-Docling-258M model at ibm-granite/granite-docling-258M 13. Alternative presets include:

• smoldocling: SmolDocling model, produces DocTags format with structural annotations + bounding boxes
• granite_docling: Default preset, IBM Granite-Docling-258M, DocTags format

Both models output structured tokens (<title>, <text>, <loc_N> bounding-box tokens) that the pipeline parses into DoclingDocument format 13.

When to Use VlmPipeline#

VlmPipeline is appropriate for documents with complex layouts (forms, mixed content) where the staged pipeline produces fragmented output. The single-model approach trades per-stage optimization for holistic page understanding.

Implications for Modular Architecture#

This guide uses neither pipeline directly. Instead, it extracts individual components:

• From docling-ibm-models: LayoutPredictor (standalone layout detection)
• From docling/utils: LayoutPostprocessor (confidence thresholding, NMS, overlap resolution)
• From docling-core: DoclingDocument builder methods (programmatic assembly)

The StandardPdfPipeline stage order clarifies why components need special handling when used standalone: OCR cells won't be present when using LayoutPredictor independently, so cell-to-cluster assignment and orphan cell recovery (normally handled by LayoutPostprocessor in the pipeline) become the orchestrator's responsibility.

Section 2: The Modular Component Architecture#

Docling's architecture is deliberately separated into three independent packages that can be composed at different levels of abstraction. This separation enables both end-to-end document conversion via the full pipeline and selective use of individual components for custom workflows. Understanding this modularity is the key to building a scatter-gather IDP system that routes document regions to specialized tools before any pipeline overhead runs.

Package Separation#

Docling's functionality is distributed across three packages, each with a distinct scope:

docling-ibm-models — Pure ML models with no pipeline dependency. This package contains standalone model implementations including LayoutPredictor (layout detection), table structure models, and reading order models. These models accept standard Python objects (PIL Images, numpy arrays) and return simple data structures (dictionaries, lists). They have no dependency on the main docling package and can be imported and invoked independently 14.

docling — Pipeline orchestration that wraps these models into a staged document conversion workflow. The DocumentConverter class coordinates preprocessing, OCR, layout analysis, table parsing, and document assembly. The BaseLayoutModel class in this package is an adapter that wraps standalone models like LayoutPredictor for use within the pipeline—it is not the only way to use those models 15.

docling-core — Data types and DoclingDocument schema implemented as pure Pydantic models. This package has no dependency on either docling or docling-ibm-models and defines the output format for assembled documents. It can be used independently to validate or construct documents from any source.

This separation is intentional: the model package provides perception capabilities, the pipeline package provides orchestration, and the core package provides the data schema. Each can be used without the others.

LayoutPredictor: Designed for Standalone Use#

The LayoutPredictor class lives in docling-ibm-models/docling_ibm_models/layoutmodel/layout_predictor.py and is explicitly designed for standalone use outside the Docling pipeline 14. Its constructor accepts only model-intrinsic parameters with no dependency on pipeline state or ConversionResult objects:

LayoutPredictor(
    artifact_path: str, # path to model directory
    device: str = "cpu", # "cpu", "cuda", "mps", "xpu"
    num_threads: int = 4, # CPU only
    base_threshold: float = 0.3,
    blacklist_classes: Set[str] = set(),
)

16

Input: PIL Image.Image or numpy array. The predictor converts input to RGB internally 17.

Output: Generator yielding dictionaries with keys label, confidence, l, t, r, b. Coordinates are in pixel space with origin at the top-left corner of the image 18.

Methods:

• predict(image) — Single image inference, yields bounding box dicts 19
• predict_batch(images) — Batch inference for multiple pages, returns List[List[dict]]. More efficient than repeated predict() calls 20

Thread safety: The predictor uses a global lock during initialization to prevent threading issues during model loading 21. Inference runs under @torch.inference_mode() 22.

Filtering: The base_threshold parameter is applied at the RT-DETR post-process step, providing initial coarse filtering before any spatial post-processing 23. The blacklist_classes parameter suppresses specific labels at inference time 24.

The official demo confirms standalone usage: demo_layout_predictor.py instantiates LayoutPredictor, loads images from disk, calls predict(), and processes results without constructing any Docling pipeline 25.

Model Family#

All Docling layout models share a common LayoutPredictor API and output the same 17-label taxonomy, making them interchangeable. Model specifications are defined as LayoutModelConfig instances and registered in the LayoutModelType enum 26.

27

All variants support CPU, CUDA, MPS, and XPU accelerators 28.

The 17-Label Taxonomy#

All model variants output the same canonical label set defined in LayoutLabels 29:

Core structural labels (0-10):

• Caption, Footnote, Formula, List-item, Page-footer, Page-header, Picture, Section-header, Table, Text, Title

Extended labels (11-16):

• Document Index, Code, Checkbox-Selected, Checkbox-Unselected, Form, Key-Value Region

The Text label covers all text blocks—both printed and handwritten—without distinction. This means LayoutPredictor cannot differentiate handwritten annotations from printed text. Any routing logic that requires this distinction must apply a secondary classifier after layout detection 15.

Example: Standalone Usage#

from docling_ibm_models.layoutmodel.layout_predictor import LayoutPredictor
from huggingface_hub import snapshot_download
from PIL import Image

# Download model once and cache locally
artifact_path = snapshot_download(
    repo_id="docling-project/docling-layout-egret-xlarge"
)

# Initialize predictor (no pipeline required)
predictor = LayoutPredictor(
    artifact_path=artifact_path,
    device="cuda",
    num_threads=4,
    base_threshold=0.3,
    blacklist_classes={"Document Index", "Code"} # filter unwanted labels
)

# Single page prediction
page_image = Image.open("page_001.png")
regions = list(predictor.predict(page_image))

# Output format:
# [
# {"label": "Text", "confidence": 0.95, "l": 50, "t": 100, "r": 400, "b": 200},
# {"label": "Table", "confidence": 0.91, "l": 50, "t": 250, "r": 600, "b": 500},
# ...
# ]

# Multi-page batch processing (more efficient)
page_images = [Image.open(f"page_{i:03d}.png") for i in range(10)]
all_regions = predictor.predict_batch(page_images)
# Returns List[List[dict]] — one list per page

15

Why This Separation Matters#

This modular architecture enables a fundamental advantage for custom IDP pipelines: routing decisions can happen immediately after detection, before any OCR or pipeline stages run.

In a scatter-gather architecture:

1. LayoutPredictor detects regions and their structural types (Table, Text, Picture, etc.)
2. A routing layer examines each region and decides which specialized tool to invoke
3. Each region type dispatches to its appropriate expert: printed text to OCR, tables to TableFormer, pictures to VLM classifiers
4. Results assemble into the final document

This is impossible when using the full pipeline, which runs OCR on all text regions before you can inspect results. By using LayoutPredictor standalone, you gain full control over what happens after layout detection—no unnecessary OCR on handwritten regions, no wasted compute on regions that route to specialized tools, and the ability to run L2 experts in parallel across region types.

The BaseLayoutModel in the main docling package is an adapter that wraps LayoutPredictor for pipeline use 15. Using the predictor standalone is not a workaround—it is a legitimate, fully supported usage pattern that Docling's architecture explicitly enables.

Section 3: Using LayoutPostprocessor Standalone#

LayoutPostprocessor transforms raw layout predictions into clean, structured Cluster objects through overlap resolution, confidence filtering, and cell assignment. The class lives in docling/utils/layout_postprocessor.py and is used internally by LayoutModel.predict_layout() immediately after LayoutPredictor.predict_batch() returns raw detections. While designed for the pipeline, the postprocessor can run standalone in image-only workflows where you control layout detection and routing but still need spatial cleanup.

Location and API Stability#

LayoutPostprocessor is an internal class in docling.utils, not a public API 30. Use the Adapter pattern and pin your docling version (e.g., docling==2.x.y) to prevent unexpected breakage. Confidence thresholds and overlap parameters are hardcoded class constants — to change them, you must subclass or fork.

Constructor Signature#

LayoutPostprocessor(
    page: Page,
    clusters: list[Cluster],
    options: LayoutOptions | LayoutObjectDetectionOptions,
)

31

The main entry point is postprocess() → tuple[list[Cluster], list[TextCell]] 32.

Required Objects#

You must provide three objects from Docling's datamodel:

Page#

From docling.datamodel.base_models, the Page object carries per-page state 33. Required fields:

• page_no (int) — 1-indexed page number
• size (Size) — page dimensions (Size(width=..., height=...) from docling_core.types.doc)
• parsed_page (SegmentedPdfPage | None) — optional for standalone use; only required if skip_cell_assignment=False (see below)

Cluster#

From docling.datamodel.base_models, each Cluster represents a detected region 34:

Cluster(
    id: int, # sequential index within the page
    label: DocItemLabel, # e.g., TEXT, TABLE, PICTURE
    bbox: BoundingBox, # from docling_core.types.doc
    confidence: float, # model confidence
    cells: list[TextCell], # initially empty, filled by postprocessor
    children: list[Cluster], # initially empty, populated for wrappers
)

LayoutOptions#

From docling.datamodel.pipeline_options, LayoutOptions or LayoutObjectDetectionOptions control postprocessing behavior 35. Key fields:

• keep_empty_clusters (default False) — retain clusters with no assigned text cells
• skip_cell_assignment (default False) — bypass spatial cell→cluster assignment entirely
• create_orphan_clusters (default True on LayoutOptions, False on LayoutObjectDetectionOptions) — unassigned text cells become new TEXT clusters

Converting Raw LayoutPredictor Output to Cluster Objects#

LayoutPredictor.predict_batch() returns plain dicts with keys label, confidence, l, t, r, b (pixel coordinates, top-left origin). Three steps convert each dict to a Cluster 36:

1. Label normalization: lowercase the string, replace spaces and hyphens with underscores
   • "Section-header" → "section_header", "Key-Value Region" → "key_value_region"
2. BoundingBox construction: BoundingBox.model_validate(pred) reads the l/t/r/b pixel fields directly
3. Cluster instantiation: id=idx, label=DocItemLabel(norm_label), bbox=bbox, confidence=pred["confidence"]

Example bridge code:

from docling.datamodel.base_models import Cluster, Page, Size
from docling.utils.layout_postprocessor import LayoutPostprocessor
from docling.datamodel.pipeline_options import LayoutOptions
from docling_core.types.doc import BoundingBox, DocItemLabel

def raw_preds_to_clusters(raw_preds):
    clusters = []
    for idx, pred in enumerate(raw_preds):
        norm_label = pred["label"].lower().replace(" ", "_").replace("-", "_")
        try:
            label = DocItemLabel(norm_label)
        except ValueError:
            continue # skip unknown labels
        bbox = BoundingBox.model_validate(pred)
        cluster = Cluster(
            id=idx,
            label=label,
            bbox=bbox,
            confidence=pred["confidence"],
        )
        clusters.append(cluster)
    return clusters

What the Postprocessor Provides#

Per-Label Confidence Thresholds#

Detections below these hardcoded class constants are filtered 37:

Label Remapping#

After thresholding, TITLE → SECTION_HEADER 38. This means DocItemLabel.TITLE predictions never surface in the output.

Overlap Resolution via Union-Find#

The postprocessor groups overlapping clusters using a Union-Find data structure and selects one winner per group 39. Three parameter sets control aggressiveness 40:

Pictures and wrappers require larger area ratios and confidence gaps before removal.

Cross-Type Conflict Handling#

Before overlap resolution, two TABLE-wins rules apply 41:

1. Wrapper vs TABLE: If a KEY_VALUE_REGION, FORM, or other wrapper overlaps a TABLE by >90% (intersection_over_self) and the wrapper's confidence advantage is <0.1, the wrapper is dropped 23.
2. Picture vs TABLE: If a PICTURE has IoU >0.8 against a TABLE, the picture is dropped.

Full-Page False Positive Filtering#

Clusters with bbox area >90% of page area are removed 42.

Wrapper Containment#

Regular clusters ≥80% contained within a TABLE, FORM, KEY_VALUE_REGION, or DOCUMENT_INDEX bbox become children of that wrapper and are removed from the top-level list 43. For FORM and KEY_VALUE_REGION only, the wrapper's bbox is re-fit to the tight union of its children 44.

What It Does NOT Do When parsed_page=None#

When using the postprocessor standalone without PDF text cells, you must set skip_cell_assignment=True in LayoutOptions. If skip_cell_assignment=False and parsed_page=None, the postprocessor will raise an AssertionError at line 256 45.

Operations skipped when skip_cell_assignment=True:

• Cell-to-cluster assignment (spatial matching of text cells to clusters)
• Orphan cell recovery (unassigned cells becoming new TEXT clusters)
• Bbox refinement based on text cells
• Sorting cells within clusters

Use skip_cell_assignment=True for image-only workflows where you have no PDF text cells.

When to Use LayoutPostprocessor Standalone#

• You're working with page images (not PDFs) and need overlap resolution and threshold filtering but control your own pipeline
• You've replaced LayoutPredictor with a different detector but still want Docling's spatial postprocessing rules
• You're building a modular IDP pipeline where layout detection is one stage and you want consistent cleanup across multiple detector outputs

The postprocessor provides the spatial intelligence layer — it cannot fix semantic errors (e.g., text mis-classified as handwriting) but it reliably handles overlaps, duplicates, and low-confidence false positives.

Section 4: Error Propagation — Two Levels#

In a modular IDP architecture, error management operates at two distinct levels. The first level handles spatial and geometric issues within the layout detection and postprocessing stage — overlapping boxes, duplicate detections, false positives. The second level addresses semantic and routing concerns — whether a detected region should actually go to a specific expert tool. Conflating these two levels leads to brittle systems; recognizing them as separate layers with separate responsibilities is crucial.

Level 1: Spatial Post-Processing (LayoutPostprocessor)#

The LayoutPostprocessor in docling/utils/layout_postprocessor.py 46 performs spatial cleanup on raw layout predictions. Its responsibility is strictly geometric: resolving overlaps, filtering low-confidence detections, removing false positives, and handling cross-type conflicts between structural labels.

What it handles:

• Overlapping detections: When multiple clusters cover the same region — for example, two TEXT clusters overlapping at 80% intersection — the postprocessor uses Union-Find to group overlapping regions and select the best representative 47. Selection rules consider area ratios, confidence differences, and label-specific heuristics 48.

• Duplicate regions: The model may predict essentially identical boxes for the same region at slightly different confidences. NMS (Non-Maximum Suppression) logic removes these duplicates 49.

• Low-confidence detections: Per-label confidence thresholds filter out unreliable predictions. TEXT, TABLE, PICTURE, CAPTION, FOOTNOTE, FORMULA, LIST_ITEM, PAGE_HEADER, and PAGE_FOOTER require confidence ≥ 0.5; SECTION_HEADER, TITLE, CODE, CHECKBOX types, FORM, and KEY_VALUE_REGION require ≥ 0.45 37. These thresholds are hardcoded class constants, not runtime-configurable.

• False positive full-page pictures: Regions classified as PICTURE that cover more than 90% of the page area are filtered out, as they are typically scan artifacts rather than genuine figures 50.

• Cross-type conflicts: When KEY_VALUE_REGION proposals nearly coincide with TABLE detections (overlap > 90% and confidence difference < 0.1), the KEY_VALUE_REGION is removed in favor of the structured TABLE 41. Similarly, when PICTURE and TABLE regions have high intersection-over-union (> 0.8), the PICTURE is dropped to preserve the structured table 51.

• Label remapping: TITLE labels are automatically remapped to SECTION_HEADER 38.

What it does NOT handle:

The postprocessor operates purely on geometric and structural information. It cannot assess semantic correctness:

• Semantic label correctness: If the model predicts "TABLE" for what is actually a hand-drawn grid, the postprocessor cannot detect this error. It has no visual understanding of the region content.

• Printed vs handwritten distinction: A TEXT region may contain printed or handwritten text — the postprocessor has no mechanism to distinguish these modalities. The model's output contains only structural labels, not writing style 52.

• Document context: The postprocessor cannot determine whether a layout prediction makes sense given the document type. A high-confidence FORM detection on a research paper might be spurious, but this requires document-level knowledge the postprocessor does not possess.

• Confidence scores for postprocessing decisions: When the postprocessor selects one cluster over another due to area or confidence rules, it does not emit a new confidence score reflecting the reliability of that decision 53.

Level 2: Routing Validation (User-Built)#

Level 2 is where domain knowledge and business logic live. After spatial cleanup, the orchestrator must decide whether each region should actually go to a specific expert tool. This is a semantic and strategic decision, not a geometric one.

Key routing decisions the orchestrator must make:

• TABLE at 0.56 confidence: Should this region go to TableFormer? Or is the confidence too low, warranting VLM confirmation first? Or should a second specialized classifier verify the table structure before expensive parsing begins?

• TEXT region with uniform ink: Is this printed text (route to OCR) or handwriting (route to HTR)? The layout model cannot distinguish these, so the orchestrator must apply a secondary classifier 15.

• Low-confidence PICTURE: Is this a genuine figure worth sending to a figure classifier, or is it an OCR noise artifact that should be ignored entirely?

• FORM region overlapping TABLE: If both labels appear for the same spatial region, which tool gets priority? Should the FORM be processed as a structured table, or does the FORM label indicate unstructured field extraction is needed?

• High-confidence CODE region: Should this go to a syntax highlighter or a code-aware OCR engine that preserves indentation and special characters?

This layer is where the user's innovation lives. Docling provides raw detections; you bring domain expertise to route them intelligently.

What standalone LayoutPredictor provides at the raw level:

The LayoutPredictor in docling-ibm-models 14 offers minimal but essential controls for raw detection:

• Confidence scores per detection: Each prediction includes a confidence field 52.

• base_threshold filtering: A coarse pre-filter (default 0.3) applied during inference. Predictions below this threshold are never emitted .

• blacklist_classes filtering: Suppress specific labels at inference time. For example, blacklist_classes={"Form", "Key-Value Region"} prevents those labels from appearing in the output .

• Coordinate clamping to image dimensions: Bounding box coordinates are clamped to [0, width] for horizontal and [0, height] for vertical to prevent out-of-bounds boxes 58.

That is the full extent of what the raw predictor provides. Everything else — confidence thresholds per region type, secondary classification, routing logic, tool selection — is the orchestrator's responsibility.

Practical Routing Strategy Patterns#

Three common patterns emerge in production IDP systems:

1. Hard threshold routing

• Confidence ≥ 0.8: route to primary tool immediately
• 0.5 ≤ confidence < 0.8: route to VLM or secondary classifier for confirmation
• Confidence < 0.5: flag for manual review or fallback tool

2. Label-based routing

• TABLE → TableFormer
• PICTURE → figure classifier (stamp/signature detector)
• TEXT → OCR/HTR branching based on secondary handwriting classifier

3. Multi-classifier approach

Use a secondary specialized model for ambiguous cases. For example, if the layout model predicts TEXT at 0.62 confidence, apply a handwriting classifier to the cropped region before routing. If the handwriting classifier returns "printed" at 0.90 confidence, route to OCR; if "handwritten" at 0.85, route to HTR.

Why This Separation Matters#

Conflating Level 1 (spatial cleanup) with Level 2 (semantic routing) creates fragile systems where geometric heuristics are expected to solve semantic problems. For example:

• Trying to use overlap thresholds to distinguish printed from handwritten text does not work — they may have identical bounding boxes.
• Adjusting confidence thresholds in the postprocessor cannot fix a model that labels handwriting as "TEXT" without distinguishing modality — the label itself is ambiguous.

The LayoutPostprocessor correctly handles spatial NMS and overlap resolution, freeing you from reimplementing that logic. You bring domain knowledge for routing: what confidence thresholds make sense for your document types, which secondary classifiers to apply, and which tools are appropriate for ambiguous detections.

Detailed analysis of routing timing and its impact on compute efficiency — particularly when routing decisions happen before versus after OCR — is provided in 15. The key insight: routing at Level 2 (after detection, before specialized processing) maximizes efficiency and flexibility by dispatching regions to the right tools immediately, avoiding wasted computation on inappropriate tools.

Section 5: Document-Level Classification vs. Layout Analysis#

A common design question when building IDP pipelines: "If I'm already doing layout analysis to identify tables, forms, and text regions, do I also need document-level classification?"

The answer is yes, and the reason is that they operate on orthogonal axes.

What Each Layer Answers#

Layout analysis answers: "What content types exist on this page, and where are they?"

The output is a spatial map of structural elements: this region is a TABLE, that region is TEXT, this region is a PICTURE. This map drives per-region tool selection — it determines which specialized processor handles each extracted bounding box.

Document classification answers: "What type of document is this?"

The output is a document-level category: invoice, medical intake form, legal contract, financial statement. This category drives pipeline configuration — it determines which schema to apply to extracted data, which validation rules to enforce, which compliance requirements govern processing, and which SLA tier the job belongs to.

Why They Are Not Redundant#

Consider a TABLE region detected by the layout model. The layout analysis tells you it's a table and routes it to TableFormer. But document classification tells you whether that table is:

• A line-item table in an invoice → extract to invoice data schema, validate against supplier catalog
• A medication dosage table in a medical form → apply HIPAA-compliant handling, route to clinical NLP
• A financial data table in an SEC filing → extract to XBRL schema, validate numerical consistency

The same layout element, the same structural label, the same TableFormer output — but entirely different downstream behavior based on document type.

Recommended Execution Order#

Document classification should run before or in parallel with layout detection, not after. Both operate on the page image, not on extracted content. Running them concurrently avoids serial latency. Running classification first allows the pipeline configuration to be established before layout results are returned, so routing decisions can incorporate document-type context from the start.

Page Image
    ├──→ LayoutPredictor (structural detection)
    └──→ Document Classifier (type detection)
           ↓
Both results merged by orchestrator
    ↓
Routing decisions: (label + confidence + document_type) → expert tool

The Configuration Profile Pattern#

A clean pattern for this separation is a configuration profile keyed on document type. Each profile specifies:

• Which expert tools are enabled for which labels
• Confidence thresholds for routing decisions (may differ from LayoutPostprocessor's thresholds)
• Semantic schemas for validation
• SLA requirements (e.g., synchronous vs. async processing)
• Compliance handling (data retention, PII masking, audit logging)

PROFILES = {
    "invoice": {
        "TABLE": {"tool": "tableformer", "schema": InvoiceLineItemSchema},
        "TEXT": {"tool": "ocr", "schema": InvoiceHeaderSchema},
    },
    "medical_form": {
        "TABLE": {"tool": "tableformer", "schema": ClinicalDataSchema, "compliance": "hipaa"},
        "TEXT": {"tool": "htr_or_ocr", "schema": PatientDataSchema, "compliance": "hipaa"},
    },
}

Document classification enables this profile selection. Layout analysis populates the regions that get processed against it. Neither can substitute for the other.

Section 6: Assembly Schema — DoclingDocument#

Once a scatter-gather pipeline has routed regions to expert tools and collected results, the final step is assembly. DoclingDocument serves as the ideal assembly target for modular IDP pipelines.

Why DoclingDocument as the Assembly Target#

DoclingDocument is not tied to Docling's full pipeline — it can be built entirely from scratch 59. As a schema-versioned Pydantic model, it offers native serialization to JSON, YAML, Markdown, and HTML. The document already carries all the fields needed for comprehensive document representation: texts, tables, pictures, pages, and provenance.

Downstream tools such as LangChain, LlamaIndex, and HybridChunker already speak DoclingDocument. Using it as the assembly schema avoids inventing a proprietary format that will require adapters everywhere. The document schema evolves with versioning, preserving backward compatibility while enabling forward progress.

Creating DoclingDocument from Scratch#

Create an empty document with a single constructor call:

from docling_core.types.doc.document import DoclingDocument

doc = DoclingDocument(name="my-processed-document")

The document is now ready to receive items from expert tools 60.

Builder Methods#

DoclingDocument provides builder methods for adding content programmatically. Each method returns the created item, enabling chaining or later reference 61.

add_text(label, text, prov=None, parent=None)#

Adds text items with a specified label. Supports TEXT, CAPTION, FOOTNOTE, and other text-based labels 62:

from docling_core.types.doc.labels import DocItemLabel

item = doc.add_text(
    label=DocItemLabel.TEXT,
    text="Extracted paragraph text",
    prov=provenance_item,
)

For section headers and captions, use the same method with the appropriate label, or use the specialized methods described below.

add_table(data, prov=None)#

Adds a table with structured data 63:

from docling_core.types.doc.items.table.table_data import TableData

table_item = doc.add_table(
    data=table_data, # TableData with rows/cols/cells
    prov=provenance_item,
)

The TableData object must contain the table structure with rows, columns, and cell contents.

add_picture(annotations=None, image=None, prov=None)#

Adds a picture element 64:

picture_item = doc.add_picture(
    annotations=None,
    image=None,
    prov=provenance_item,
)

Images and annotations are optional; the provenance preserves the spatial location from the source document.

add_heading(text, level=1, prov=None)#

Adds a section header with hierarchical level 65:

heading = doc.add_heading(
    text="Section Title",
    level=2,
    prov=provenance_item,
)

add_code(text, code_language=None, prov=None)#

Adds a code block with optional language annotation 66:

code_item = doc.add_code(
    text="def example():\n return 42",
    code_language=CodeLanguageLabel.PYTHON,
    prov=provenance_item,
)

add_list_item(text, enumerated=False, prov=None)#

Adds a list item to a list group 67:

list_item = doc.add_list_item(
    text="First item",
    enumerated=True,
    prov=provenance_item,
)

Custom Provenance from External Tools#

ProvenanceItem preserves the source coordinates from whichever external tool produced the output 68. This is not Docling's pipeline provenance — it is your tool's provenance:

from docling_core.types.doc.common.reference import ProvenanceItem
from docling_core.types.doc.base import BoundingBox, CoordOrigin

prov = ProvenanceItem(
    page_no=1,
    bbox=BoundingBox(
        l=50, t=100, r=400, b=200,
        coord_origin=CoordOrigin.TOPLEFT
    ),
    charspan=(0, 42),
)

doc.add_text(
    label=DocItemLabel.TEXT,
    text="Extracted text from custom OCR",
    prov=prov,
)

The bounding box uses the coordinate origin you specify — top-left or bottom-left — ensuring compatibility with different tool outputs.

Custom Metadata via BaseMeta#

Each item carries a meta field supporting custom metadata through a namespaced key convention 69. The double underscore delimiter separates namespace from field name:

# On any item's .meta field
item.meta.set_custom_field("my_pipeline", "tool_used", "tesseract-4.1")
item.meta.set_custom_field("my_pipeline", "region_confidence", 0.92)
item.meta.set_custom_field("my_pipeline", "routing_decision", "printed_ocr")

The key format is "namespace__fieldname". This approach records which expert tool processed each region and with what confidence, enabling full traceability and auditing.

Polars as Scatter-Gather Coordination Tracker#

Polars DataFrame serves as an ideal tracker for parallel processing state. Define a column schema that captures the scatter-gather lifecycle:

import polars as pl

# Schema: region_id, page_no, label, bbox_l, bbox_t, bbox_r, bbox_b,
# confidence, tool_assigned, status, result_ref

Use cases for the Polars tracker:

• Dispatch: Fan out regions to parallel workers based on tool assignment
• Status tracking: Track state transitions from pending → processing → complete/failed
• Reading-order sorting: Sort by (page_no, bbox_t, bbox_l) before assembly to preserve document flow
• Analytics: Count regions by type, compute latency per tool, identify bottlenecks

Final Assembly#

The final assembly step iterates the sorted DataFrame and calls the appropriate doc.add_*() methods:

def assemble_document(df: pl.DataFrame, results: dict) -> DoclingDocument:
    doc = DoclingDocument(name="assembled")

    # Sort by reading order
    ordered = df.sort(["page_no", "bbox_t", "bbox_l"])

    for row in ordered.iter_rows(named=True):
        result = results[row["region_id"]]

        prov = ProvenanceItem(
            page_no=row["page_no"],
            bbox=BoundingBox(
                l=row["bbox_l"], t=row["bbox_t"],
                r=row["bbox_r"], b=row["bbox_b"],
            ),
            charspan=(0, len(result.get("text", ""))),
        )

        label = row["label"]
        if label in ("TEXT", "SECTION_HEADER", "CAPTION"):
            doc.add_text(
                label=DocItemLabel(label.lower()),
                text=result["text"],
                prov=prov,
            )
        elif label == "TABLE":
            doc.add_table(data=result["table_data"], prov=prov)
        elif label == "PICTURE":
            doc.add_picture(prov=prov)
        # ... handle other labels

    return doc

Key Advantage#

Each expert tool produces results in its own format. Polars tracks and normalizes the coordination state. DoclingDocument unifies everything into a single schema with standard export paths to JSON, Markdown, or HTML. Provenance is preserved from the original tool output, enabling full traceability from source page coordinates through tool selection to final assembled output.

Section 7: Offline Deployment#

All three components used in this architecture support fully offline operation. No runtime network access is required once models are downloaded.

Component-by-Component Offline Profile#

docling-core — Pure Pydantic data types (DoclingDocument, BoundingBox, ProvenanceItem, BaseMeta). No ML models, no network calls. Install and use air-gapped with no special configuration.

LayoutPostprocessor — Pure computational logic (spatial indexing, Union-Find, threshold filtering). No models, no network. Works fully offline by design 46.

LayoutPredictor — Loads model weights from a local artifact_path. Once the model is present on disk, inference is entirely offline. The directory must contain 16:

• model.safetensors — model weights
• preprocessor_config.json — image preprocessing configuration
• config.json — model architecture configuration

Downloading Models for Offline Use#

Via the Docling CLI (downloads to a managed local cache):

docling-tools models download layout -o /path/to/models/egret-xlarge \
    --model docling_layout_egret_xlarge

Via HuggingFace Hub (for explicit directory control):

from huggingface_hub import snapshot_download

artifact_path = snapshot_download(
    repo_id="docling-project/docling-layout-egret-xlarge",
    local_dir="/path/to/models/egret-xlarge",
)

Both approaches produce a directory that can be used directly as artifact_path for LayoutPredictor.

Available Layout Models#

All models are hosted under the docling-project organization on HuggingFace 26:

All variants share the same LayoutPredictor API and the same 17-label output taxonomy — swapping models requires only changing artifact_path 29.

Deployment Checklist#

• Download chosen layout model to a persistent local directory
• Set artifact_path to that directory at LayoutPredictor initialization
• Verify the three required files are present (model.safetensors, preprocessor_config.json, config.json)
• Pin docling, docling-ibm-models, and docling-core versions in your dependency lockfile
• Test inference with a sample page image before deploying to production
• For GPU deployments, verify CUDA version compatibility with your PyTorch install

Section 8: The Complete Architecture#

Combining all components described in this guide yields a modular IDP pipeline where perception, filtering, routing, and assembly are cleanly separated across well-defined layers.

Architecture Diagram#

┌─────────────────────────────────────────────────────────────┐
│ INPUT │
│ PDF page or scanned image │
└───────────────────────────┬─────────────────────────────────┘
                             ↓
┌─────────────────────────────────────────────────────────────┐
│ LayoutPredictor (docling-ibm-models) │
│ Standalone — no pipeline dependency │
│ Input: PIL Image Output: [{label, confidence, l,t,r,b}] │
└───────────────────────────┬─────────────────────────────────┘
                             ↓ raw dicts
┌─────────────────────────────────────────────────────────────┐
│ Cluster Conversion Wrapper │
│ label.lower().replace(" ", "_").replace("-", "_") │
│ BoundingBox.model_validate(pred) │
│ Cluster(id, label, bbox, confidence) │
└───────────────────────────┬─────────────────────────────────┘
                             ↓ list[Cluster]
┌─────────────────────────────────────────────────────────────┐
│ LayoutPostprocessor (docling.utils) │
│ ⚠ Internal API — use Adapter pattern │
│ · Confidence thresholding (per-label: 0.45–0.5) │
│ · NMS / overlap resolution (Union-Find) │
│ · Cross-type conflict handling │
│ · TITLE → SECTION_HEADER remapping │
│ · Full-page false-positive filtering │
│ Note: set skip_cell_assignment=True for image-only input │
└───────────────────────────┬─────────────────────────────────┘
                             ↓ cleaned list[Cluster]
┌─────────────────────────────────────────────────────────────┐
│ Routing Decision Layer (user-built) │
│ ← Your innovation lives here → │
│ · Document classifier (invoice? form? contract?) → │
│ configuration profile selection │
│ · Per-region confidence routing: │
│ conf ≥ 0.8 → primary tool │
│ conf 0.5–0.8 → confirm with VLM │
│ conf < 0.5 → flag for review │
│ · Label-type dispatching: │
│ TABLE → TableFormer (or VLM) │
│ TEXT (printed) → OCR │
│ TEXT (handwritten) → HTR │
│ PICTURE → Image Classifier │
│ · Polars DataFrame for parallel state tracking │
└──┬──────────────┬──────────────┬────────────────────────────┘
   ↓ ↓ ↓ ↓
[TableFormer] [OCR Engine] [HTR Tool] [VLM / Classifier]
   ↓ ↓ ↓ ↓
└──────────────────────────────────────────────────────────────┘
                             ↓ gathered results
┌─────────────────────────────────────────────────────────────┐
│ DoclingDocument Assembly (docling-core) │
│ Pure Pydantic — no pipeline dependency │
│ · Sort by reading order: (page_no, bbox_t, bbox_l) │
│ · doc.add_text() / add_table() / add_picture() etc. │
│ · ProvenanceItem preserves source tool coordinates │
│ · meta.set_custom_field() records tool metadata │
└───────────────────────────┬─────────────────────────────────┘
                             ↓
┌─────────────────────────────────────────────────────────────┐
│ DoclingDocument │
│ export_to_markdown() / export_to_json() / HybridChunker │
└─────────────────────────────────────────────────────────────┘

Responsibilities by Layer#

The Key Insight#

Docling provides perception capabilities — detection, spatial post-processing, and document representation. Your orchestrator provides decision intelligence — routing, expert selection, and ambiguity management.

This division is not a limitation. It is the correct separation of concerns for building production IDP systems. Docling's layout models are trained on millions of document images and produce high-quality structural predictions. The routing layer is where domain knowledge, compliance requirements, and business logic live. Keeping these two concerns separate means you can upgrade the Docling models without touching your routing logic, and you can evolve your routing logic without retraining models.

Implementation Checklist#

1. Install dependencies:

   ⧉

   pip install docling-ibm-models docling docling-core huggingface-hub polars pillow
   

2. Download layout model:

   ⧉

   python -c "from huggingface_hub import snapshot_download; snapshot_download('docling-project/docling-layout-egret-xlarge', local_dir='./models/egret-xlarge')"
   

3. Build the Cluster conversion wrapper — label normalization + BoundingBox.model_validate() 36

4. Wrap LayoutPostprocessor behind an Adapter — pin docling version in your lockfile

5. Implement routing logic — start simple (label-based dispatch), add confidence-tiered routing as needed

6. Wire expert tools — each tool returns structured output + source bounding boxes

7. Assemble via DoclingDocument — use ProvenanceItem to preserve per-region source coordinates 70

8. Export or chunk — doc.export_to_markdown(), doc.export_to_json(), or feed to HybridChunker for RAG pipelines