Picture Detection and Segmentation#

Overview#

Docling's layout pipeline detects pictures as PICTURE-labeled Cluster objects. These clusters go through several picture-specific stages in LayoutPostprocessor — confidence gating, cross-type conflict resolution, full-page filtering, and intra-picture overlap removal via Union-Find — before being assembled into FigureElement/PictureItem outputs. For scanned PDFs, embedded visuals are pure pixel regions inside a full-page raster; they only become separate PictureItems if the layout model detects them as PICTURE clusters.

Detection: From Raw Detections to PICTURE Clusters#

Raw detections come from the object-detection inference engine via LayoutObjectDetectionModel.predict_layout(). The _predictions_to_clusters() method maps each engine output (label ID, confidence score, pixel bbox) to a Cluster. Bounding boxes are clamped to page bounds during this step 1, which matters for downstream area-ratio calculations in the postprocessor.

Label mapping from model config strings to DocItemLabel.PICTURE happens in _build_label_map() (e.g., the model config string "Picture" → enum PICTURE).

PICTURE-Specific Postprocessing#

LayoutPostprocessor (in docling/utils/layout_postprocessor.py) routes PICTURE clusters through a dedicated path in _process_special_clusters(), separate from regular text/structural clusters.

1. Confidence Thresholding#

PICTURE detections require confidence ≥ 0.5 2, applied before any overlap resolution 3.

2. Cross-Type Conflict Resolution#

_handle_cross_type_overlaps() removes PICTURE clusters that have IoU > 0.8 against a TABLE cluster 4. IoU (not containment) is used deliberately so a small genuine figure fully inside a large table region is not removed — only near-coincident detections are dropped.

3. Full-Page Picture Filtering#

Clusters with bbox.area() / page_area > 0.90 are unconditionally removed 5. These are typically scan artifacts — the full-page raster misclassified as a picture — not genuine embedded figures.

4. Intra-Picture Overlap Removal via Union-Find#

Surviving picture clusters are deduplicated by _remove_overlapping_clusters(clusters, "picture"). This uses UnionFind (with path compression and union by rank) to group all mutually overlapping picture clusters, then selects one winner per group via _select_best_cluster_from_group().

The picture-type overlap parameters are hardcoded class-level constants 6:

A candidate picture cluster is dropped only when area_ratio ≤ 2.0 AND conf_diff > 0.3. This is substantially more lenient than regular clusters, making picture detections much harder to eliminate.

5. Spatial Indexing#

Overlap candidate search uses a dedicated picture SpatialClusterIndex 7, separate from the regular and wrapper indexes. It combines an R-tree and two interval trees (one per axis): find_candidates() casts a wide net via union of all three index hits, while check_overlap() filters to true overlaps using IoU or per-cluster containment (default overlap_threshold=0.8, containment_threshold=0.8) 8.

Segmentation in Scanned PDFs#

In scanned PDFs, pictures are not separate PDF image objects — they are pixel regions inside a full-page raster. Docling can only segment them if the layout model detects the region as a PICTURE cluster 9. If the model misses the region, no pipeline option forces detection; the fallback is to manually crop the rendered page image.

This limitation is tracked in GitHub issue #3367.

Assembly: PICTURE Cluster → PictureItem#

After postprocessing, PageAssembleModel converts each PICTURE cluster into a FigureElement. It first checks page.predictions.figures_classification.figure_map for a pre-classified result (from the figure classification stage); if absent, it creates a fallback FigureElement from the cluster data. FigureElement maps to PictureItem in the DoclingDocument, accessible via result.document.pictures.

Hardcoded Thresholds Reference#

All postprocessor constants are class-level on LayoutPostprocessor and are not runtime-configurable through PdfPipelineOptions. Changing them requires subclassing or forking 10.

Key Source Files#