VLM Extraction Pipeline#

Overview#

The VLM Extraction Pipeline is Docling's subsystem for pulling specific structured fields from document pages using a Vision-Language Model. It is distinct from VlmPipeline (which performs full document conversion) — this pipeline produces typed, schema-driven output rather than a converted document representation.

The entry point is DocumentExtractor in docling/document_extractor.py. It accepts PDF and IMAGE inputs only 1 and exposes two public methods:

• extract(source, template, ...) — single-document, returns ExtractionResult
• extract_all(source, template, ...) — batch, returns Iterator[ExtractionResult]

Pipeline instances are cached by (pipeline_class, options_hash), so repeated calls with the same configuration reuse the loaded model 2.

Install requirement: pip install docling[vlm]

Configuration: VlmExtractionPipelineOptions#

VlmExtractionPipelineOptions in docling/datamodel/pipeline_options.py has two fields:

The docstring explicitly lists the two supported model/style combinations 3:

• NU_EXTRACT_2B_TRANSFORMERS + ExtractionPromptStyle.NUEXTRACT (default)
• GRANITE_VISION_4_1_TRANSFORMERS + ExtractionPromptStyle.GRANITE_VISION

Model Presets#

Both extraction-specific presets live in docling/datamodel/vlm_model_specs.py:

NU_EXTRACT_2B_TRANSFORMERS 4: numind/NuExtract-2.0-2B, pinned to a specific revision to avoid MPS issues; response_format=PLAINTEXT; prompt is empty (template is injected by the pipeline, not the model spec).

GRANITE_VISION_4_1_TRANSFORMERS 5: ibm-granite/granite-vision-4.1-4b; also pinned by revision; requires trust_remote_code=True; prompt is likewise empty.

Both use InferenceFramework.TRANSFORMERS, torch_dtype="bfloat16", and support CPU/CUDA/MPS/XPU 6.

Prompt Style: ExtractionPromptStyle#

ExtractionPromptStyle (in docling/datamodel/extraction_options.py) is a two-value enum:

• NUEXTRACT — uses Qwen's chat template with qwen-vl-utils vision processing and applies torch.compile() at model init
• GRANITE_VISION — uses a standard HuggingFace chat format with an explicit system prompt that instructs the model to return valid JSON

The style must match the chosen model; mixing them (e.g., GRANITE_VISION_4_1_TRANSFORMERS + NUEXTRACT) will produce incorrect output.

ExtractionFormatOption and the Required backend Field#

ExtractionFormatOption wraps a pipeline_cls + pipeline_options pair. It inherits from BaseFormatOption, which declares backend: Type[AbstractDocumentBackend] as a required field with no default.

Common Pydantic validation error: Omitting backend when constructing ExtractionFormatOption raises:

pydantic_core._pydantic_core.ValidationError: 1 validation error for ExtractionFormatOption
backend
  Field required [type=missing, ...]

The default helper _get_default_extraction_option() handles this automatically (PDF → PyPdfiumDocumentBackend, IMAGE → ImageDocumentBackend). When building a custom ExtractionFormatOption, always supply backend explicitly:

from docling.document_extractor import DocumentExtractor, ExtractionFormatOption
from docling.datamodel.base_models import InputFormat
from docling.datamodel.pipeline_options import VlmExtractionPipelineOptions
from docling.datamodel.vlm_model_specs import GRANITE_VISION_4_1_TRANSFORMERS
from docling.datamodel.extraction_options import ExtractionPromptStyle
from docling.pipeline.extraction_vlm_pipeline import ExtractionVlmPipeline
from docling.backend.pypdfium2_backend import PyPdfiumDocumentBackend

extractor = DocumentExtractor(
    extraction_format_options={
        InputFormat.PDF: ExtractionFormatOption(
            pipeline_cls=ExtractionVlmPipeline,
            pipeline_options=VlmExtractionPipelineOptions(
                vlm_options=GRANITE_VISION_4_1_TRANSFORMERS,
                extraction_prompt_style=ExtractionPromptStyle.GRANITE_VISION,
            ),
            backend=PyPdfiumDocumentBackend, # required — no default
        )
    }
)

Template Serialization#

The pipeline's _serialize_template() converts the template argument passed to extract() into a JSON string for the VLM prompt. Four input forms are accepted:

When passing a Pydantic class, populate Field(examples=[...]) on your fields so the generated schema hint is meaningful to the VLM.

Key Source Files#