PDF Document Pipeline#

StandardPdfPipeline is Docling's production PDF conversion pipeline 1. It extends ConvertPipeline and implements a multi-threaded stage topology where each stage runs in its own worker thread, enabling pipeline-level parallelism across pages. Models are initialized once at construction time and shared read-only across worker threads 2.

The pipeline is driven by ThreadedPdfPipelineOptions and is the entry point for all PDF conversion invoked through DocumentConverter with pdf_backend set to docling_parse, threaded_docling_parse, or equivalent. 3

Stage Topology#

Pages flow through five threaded stages wired in sequence, followed by sequential cross-page assembly 4:

A dedicated PageProducer thread feeds ThreadedItem envelopes into the first stage. Each stage pulls batches from its input queue, calls the model, and emits results to the next stage. The main thread drains the output queue and collects completed pages 5.

Key infrastructure types:

Per-Stage Behavior#

1. Preprocess (PreprocessThreadedStage, batch size 1)#

PagePreprocessingModel validates the page backend and renders a page image at the configured images_scale. Invalid pages are marked failed before the model is invoked 6. Options: PagePreprocessingOptions(images_scale=...) 7.

2. OCR (ThreadedPipelineStage)#

Configured via PdfPipelineOptions.ocr_options; enabled only when do_ocr=True (default). The model is created through get_ocr_factory(), which supports pluggable engines: EasyOCR, Tesseract (CLI and Python), RapidOCR, OcrMac, Nemotron, and KServe v2 8 9.

3. Layout (ThreadedPipelineStage)#

LayoutModel.predict_layout() batches page images through LayoutPredictor.predict_batch() from docling-ibm-models, then immediately runs LayoutPostprocessor.postprocess() on each page:

• Confidence thresholding per label 10
• Label remapping (e.g., TITLE → SECTION_HEADER) 11
• Cell-to-cluster assignment: PDF/OCR text cells are spatially matched to layout clusters using an R-tree spatial index (SpatialClusterIndex) to efficiently find candidate clusters that may overlap with table cells 12
• Orphan cell recovery: unassigned cells become new TEXT clusters 13
• Overlap resolution: competing clusters are deduplicated by area/confidence ratios, with separate thresholds for regular, picture, and wrapper cluster types 14

The LayoutModel defines label sets used downstream by PageAssembleModel: TEXT_ELEM_LABELS, TABLE_LABELS, FIGURE_LABEL, and CONTAINER_LABELS 15.

4. Table Structure (ThreadedPipelineStage)#

Runs TableFormer V1, V2, or Granite Vision depending on table_structure_options, when do_table_structure=True. 16

5. Assemble (ThreadedPipelineStage, batch size 1, with postprocess hook)#

PageAssembleModel.__call__() iterates page.predictions.layout.clusters and converts each cluster to a typed element:

• TEXT_ELEM_LABELS → TextElement — text is assembled from cells via sanitize_text(), which handles dehyphenation across lines, Unicode normalization, and ligature expansion (FB00–FB06 range)
• TABLE_LABELS → Table (from page.predictions.tablestructure, or empty fallback)
• PICTURE → FigureElement
• FORM / KEY_VALUE_REGION → ContainerElement

Hyperlinks are resolved per-cluster via _match_hyperlink(), which requires ≥50% spatial overlap between a hyperlink annotation rect and the cluster bbox. The assembled result is stored in page.assembled 17.

After assemble, the _release_page_resources() postprocess hook frees the image cache and unloads the page backend (unless enrichment stages require them) 18.

Cross-Page Assembly (_assemble_document)#

After all pages are collected, _assemble_document() runs sequentially 19:

1. Collects all page.assembled elements into a single AssembledUnit.
2. ReadingOrderModel 20: converts assembled elements to ReadingOrderPageElement objects, calls ReadingOrderPredictor.predict_reading_order() (from docling-ibm-models), then resolves caption/footnote attachments and text merges (soft-hyphen handling). Builds the final DoclingDocument, assigning ContentLayer.FURNITURE to PAGE_HEADER and PAGE_FOOTER 21.
3. HeadingHierarchyModel: infers heading levels; optionally uses PDF bookmarks extracted before pipeline stages start 22 23.
4. enrichment_pipe: post-document enrichments (code/formula, picture classification/description, chart extraction) run from BasePipeline._enrich_document(). 24

Confidence scores are aggregated across pages: layout_score (mean), ocr_score (mean), table_score (mean), parse_score (10th percentile) 25.

Configuration Reference#

Options class: ThreadedPdfPipelineOptions 26.

Key Source Files#