Picture Classification Filtering#

Picture classification filtering is a mechanism in Docling's enrichment pipeline that controls which PictureItems receive VLM-based descriptions. Before sending an image to a vision-language model (VLM), the description model checks the picture's pre-computed classification predictions and confidence scores against three configurable options — an allow-list, a deny-list, and a minimum confidence threshold — and skips the VLM call for any picture that doesn't pass.

This filter is a cost-control and precision tool: VLM description is expensive, so restricting it to, say, only charts or diagrams avoids burning compute on logos, barcodes, or decorative photographs.

Prerequisites#

Classification filtering requires that picture classification has already run, which populates PictureItem.meta.classification. Classification is enabled separately via the do_picture_classification flag on ConvertPipelineOptions 1. The description model reads the already-populated metadata at inference time — no coupling at the model-wiring level. If classification hasn't run, the filter falls back gracefully (see Logic below).

Configuration#

All three filter parameters live on PictureDescriptionBaseOptions, which is the shared base for every picture-description backend (PictureDescriptionApiOptions, PictureDescriptionVlmOptions, PictureDescriptionVlmEngineOptions):

These fields are set directly on whichever picture_description_options you pass to the pipeline:

from docling.datamodel.pipeline_options import (
    PdfPipelineOptions,
    PictureDescriptionVlmEngineOptions,
)
from docling_core.types.doc import PictureClassificationLabel

opts = PdfPipelineOptions(
    do_picture_classification=True,
    do_picture_description=True,
    picture_description_options=PictureDescriptionVlmEngineOptions.from_preset(
        "smolvlm",
        classification_allow=[
            PictureClassificationLabel.BAR_CHART,
            PictureClassificationLabel.LINE_CHART,
        ],
        classification_min_confidence=0.7,
    ),
)

Filter Logic#

The filtering is implemented in _passes_classification in picture_description_base_model.py. The rules, in order:

1. No filters set (allow=None and deny=None) → always passes. 5
2. No classification data on the picture (model wasn't run, or meta is absent) → passes if allow is None; blocked if an allow-list exists. 6
3. Deny check: any prediction that meets min_confidence and is in the deny-list → picture is blocked. 7
4. Allow check: if an allow-list is set, at least one prediction must meet min_confidence and be in the allow-list. 8

Confidence gating uses _meets_confidence: when min_confidence <= 0, every prediction passes regardless of its score. A prediction with confidence=None is treated as failing confidence gating when min_confidence > 0.

The filter is called inside the __call__ loop of PictureDescriptionBaseModel, after the area-threshold check but before image rendering — so a filtered picture simply isn't added to the images batch at all 9.

Available Labels#

Labels come from the PictureClassificationLabel enum in docling-core. The active v2 model (DocumentFigureClassifier-v2.0) recognises 25 labels, grouped as:

• Charts: bar_chart, box_plot, flow_chart, line_chart, pie_chart, scatter_plot, table, other_chart
• Images: full_page_image, page_thumbnail, photograph
• Documents/Company: bar_code, icon, logo, qr_code, signature, stamp
• Specialised: chemistry_structure, engineering_drawing, screenshot_from_computer, screenshot_from_manual, geographical_map, topographical_map, calendar, crossword_puzzle, music, other

Legacy labels from earlier model versions remain in the enum for backward compatibility 10.

Key Source Files#