VLM Picture Description#

Docling's picture description feature uses Vision Language Models (VLMs) to generate natural-language captions for PictureItem elements extracted during document conversion. Descriptions are stored in PictureItem.meta.description as a DescriptionMetaField. It is activated by setting do_picture_description=True on PdfPipelineOptions.

The implementation is split across three layers:

• PictureDescriptionBaseModel — shared enrichment loop, area-threshold gating, classification filtering, and output writing
• Options classes in pipeline_options.py — backend-specific configuration
• Model specs / presets in stage_model_specs.py — named, pre-wired model+engine combinations

Built-in Presets#

The recommended entry point is PictureDescriptionVlmEngineOptions.from_preset(preset_id). Four presets are registered out of the box:

All four default to AUTO_INLINE engine type and a picture_area_threshold of 0.05 1. VLM models are not included in the default docling-tools models download set and must be explicitly downloaded 2.

You can also define and register a custom preset via PictureDescriptionVlmEngineOptions.register_preset(StageModelPreset(...)) 3.

Backend / Engine Support#

All presets go through the unified VLM inference engine layer. The backend is selected via engine_options on from_preset():

AUTO_INLINE on Apple Silicon with MLX: SmolVLM uses moot20/SmolVLM-256M-Instruct-MLX; Pixtral uses mlx-community/pixtral-12b-bf16; Qwen uses mlx-community/Qwen2.5-VL-3B-Instruct-bf16 4. Granite Vision 3.3-2B does not have an MLX export and falls back to CPU Transformers on Apple Silicon 5.

The API backend requires enable_remote_services=True and an OpenAI-compatible chat/completions endpoint 6.

Output Quality Configuration#

Generation parameters (VlmModelSpec)#

VlmModelSpec — embedded inside every preset — controls generation directly 7:

Override these at from_preset() time by mutating the returned object or by passing overrides as keyword args:

opts = PictureDescriptionVlmEngineOptions.from_preset(
    "smolvlm",
    prompt="Describe this chart for a blind reader.",
)
opts.model_spec.max_new_tokens = 512
opts.model_spec.temperature = 0.3

Known issue (fixed in ≥ 2.105.0 + PRs #3279 / #3322): In earlier releases generation_config on PictureDescriptionVlmEngineOptions was defined but silently ignored, causing max_new_tokens to be hardcoded to 200 and temperature to be fixed at a default value. These PRs forward max_new_tokens, temperature, and extra_generation_config from VlmModelSpec into the engine's VlmEngineInput via a new get_runtime_input_extra_config() method 8 9.

Image resolution#

scale on the preset (default 2.0) multiplies base image resolution before the VLM sees it 10. Higher values increase detail but use more memory and time. picture_area_threshold (default 0.05) skips pictures whose bounding box is less than 5% of the page area 11.

Legacy option: PictureDescriptionVlmOptions (deprecated) uses a plain generation_config dict defaulting to {"max_new_tokens": 200, "do_sample": False} 12. Prefer PictureDescriptionVlmEngineOptions for new code.

Classification-Based Filtering#

Before making any VLM call, PictureDescriptionBaseModel checks three filter fields inherited by all options classes from PictureDescriptionBaseOptions:

The filter is implemented in _passes_classification() and called after the area check but before image rendering 13. Requires do_picture_classification=True on pipeline options to populate PictureItem.meta.classification; if classification hasn't run, pictures pass when allow=None and are blocked when an allow-list is set 14.

Available labels come from PictureClassificationLabel in docling-core (25 labels covering charts, photographs, barcodes, engineering drawings, etc.).

Example — describe only charts with ≥ 70% confidence:

from docling.datamodel.pipeline_options import PdfPipelineOptions, PictureDescriptionVlmEngineOptions
from docling_core.types.doc import PictureClassificationLabel

opts = PdfPipelineOptions(
    do_picture_classification=True,
    do_picture_description=True,
    picture_description_options=PictureDescriptionVlmEngineOptions.from_preset(
        "smolvlm",
        classification_allow=[
            PictureClassificationLabel.BAR_CHART,
            PictureClassificationLabel.LINE_CHART,
        ],
        classification_min_confidence=0.7,
    ),
)

Key Source Files#