VLM Pipeline#

VlmPipeline is Docling's holistic page-understanding pipeline — it runs a single Vision Language Model over each page image rather than composing results from separate layout, OCR, and table-structure models. It is selected via --pipeline vlm on the CLI or by passing VlmPipelineOptions to DocumentConverter. 1

VlmPipeline inherits from PaginatedPipeline and lives in docling/pipeline/vlm_pipeline.py. Its constructor branches on the type of vlm_options:

• New path (recommended): VlmConvertOptions → uses VlmConvertModel, the preset-based runtime system. 2
• Legacy path (deprecated): InlineVlmOptions or ApiVlmOptions → wires up one of HuggingFaceTransformersVlmModel, HuggingFaceMlxModel, VllmVlmModel, or ApiVlmModel. A DeprecationWarning is raised; migrate to VlmConvertOptions.from_preset(...). 3

VlmPipelineOptions sets generate_page_images = True by default and exposes a force_backend_text flag: when enabled (DOCTAGS format only), bounding boxes from the VLM are used for PDF text extraction, replacing model-predicted text with the native PDF layer. 4

Processing Flow#

initialize_page → build_pipe (VlmConvertModel) → _assemble_document → enrichment_pipe

1. initialize_page — loads the page backend, reads page size, and (if force_backend_text) fetches a segmented page for later text override. 5

2. VlmConvertModel.__call__ — the sole build_pipe stage. For each page it scales the page image (get_image(scale, max_size)), fetches the model's prompt template, builds VlmEngineInput objects, and calls engine.predict_batch(). The raw model text is stored in page.predictions.vlm_response. 6

3. _assemble_document — reads response_format from model_spec (or legacy vlm_options) and dispatches to the appropriate parser (see Response Format Parsers). After assembly, generate_picture_images triggers per-picture image cropping from the page image. 7

4. enrichment_pipe — currently empty; reserved for future post-assembly enrichment stages. 8

5. _determine_status — overrides the base pipeline's status logic to detect VlmStopReason.LENGTH (truncated output) or CONTENT_FILTERED, both of which downgrade the result to PARTIAL_SUCCESS and append an ErrorItem. 9

Note: ThreadedDoclingParseDocumentBackend is explicitly unsupported — VlmPipeline requires ordered/random page access via load_page(). Use StandardPdfPipeline if you need streamed page delivery. 10

Response Format Parsers#

ResponseFormat is a str enum in pipeline_options_vlm_model.py. _assemble_document switches on its value: 11

DOCTAGS specifics: DocTags is the XML-like token format (<title>, <text>, <loc_N> bounding-box tokens) emitted by models such as SmolDocling and Granite-Docling. The pipeline assembles all pages as (tokens_str, PIL.Image) pairs, then delegates to DoclingDocument.load_from_doctags in docling-core. When force_backend_text=True, every TextItem's text is overwritten with the string extracted from the native PDF backend at the predicted bounding box. 12

DOCLANG specifics: Each page's raw VLM text is scanned for a <doclang>…</doclang> fragment via _extract_doclang_fragment. Missing or malformed fragments produce an empty stub page and a PARTIAL_SUCCESS status instead of a hard failure. 13

Markdown / HTML specifics: The pipeline strips triple-backtick code fences (via _extract_code_block) before passing the text to the backend. Provenance bounding boxes are faked as (0,0,0,0) since these formats carry no spatial coordinates. 14

Document Assembly#

All parsers produce per-page DoclingDocument instances. The shared helper _add_page_metadata_and_concatenate attaches the page image and size metadata to each per-page doc, then calls DoclingDocument.concatenate(docs=page_docs) to merge them into a single document with renumbered pages. (The DOCTAGS path bypasses this helper and calls DoclingDocument.load_from_doctags directly, which handles multi-page concatenation internally.)

For DOCLANG, deserialization is fault-tolerant: exceptions during deserializer.deserialize_str() produce an empty DoclingDocument for that page rather than aborting the whole document. 15

The Markdown/HTML assembly iterates items with included_content_layers=set(ContentLayer) (all layers), ensuring no content is silently dropped before the final document is returned to the caller. 16 17

Model Presets#

VlmConvertOptions uses the preset system from stage_model_specs.py. Call VlmConvertOptions.from_preset("<name>") to configure a model. The default for VlmPipelineOptions is granite_docling. 4

Additional presets include got_ocr, nanonets_ocr2, gemma_12b, gemma_27b, dolphin, glm_ocr, falcon_ocr, lightonocr. 27

Engine can be overridden independently of the preset:

from docling.datamodel.pipeline_options import VlmConvertOptions, VlmPipelineOptions
from docling.datamodel.vlm_engine_options import ApiVlmEngineOptions

options = VlmPipelineOptions(
    vlm_options=VlmConvertOptions.from_preset(
        "smoldocling",
        engine_options=ApiVlmEngineOptions(url="http://localhost:11434"),
    )
)

Key Files#

Related topics: VLM Inference Engine (backend abstraction) · VLM Formula and Code Extraction (post-assembly enrichment stage) · DocLang Format · DoclingDocument Builder API