VLM Inference Engine#

Docling's VLM inference engine layer is a unified abstraction for running vision-language model inference across multiple runtime backends. It decouples pipeline stages (document elements, page images) from backend-specific inference details, letting the same high-level stages drive HuggingFace Transformers, Apple MLX, vLLM, or any OpenAI-compatible API endpoint without code changes.

Key files:

Core Interfaces#

VlmEngineType#

VlmEngineType is a str enum with two class methods (is_api_variant, is_inline_variant) for grouping:

VlmEngineInput / VlmEngineOutput#

VlmEngineInput is a Pydantic model holding:

• image — a PIL Image
• prompt — text prompt string
• temperature (default 0.0), max_new_tokens (default 4096)
• stop_strings — list of generation-stop triggers
• extra_generation_config — backend-specific pass-through dict

VlmEngineOutput holds text, optional stop_reason, and a metadata dict.

BaseVlmEngine#

BaseVlmEngine is the ABC all backends implement. Key contract points:

• initialize() — load model/connect once; implementations set self._initialized = True 1
• predict_batch(input_batch) — the single required method; takes a List[VlmEngineInput] and returns List[VlmEngineOutput] 2
• predict() — convenience wrapper that calls predict_batch([input_data])[0]; engines must NOT override it 3
• __call__() — dispatches single inputs to predict() and lists to predict_batch() 4
• cleanup() — optional resource release hook 5

Lazy initialization: initialize() is called automatically on the first predict() or __call__() invocation, not at construction time.

BaseVlmEngineOptions#

BaseVlmEngineOptions is a Pydantic model with a _registry class-var that auto-registers concrete subclasses by their engine_type literal. This powers the VlmEngineOptionsMixin deserialization: when engine_options is provided as a dict (e.g., from JSON config), the mixin looks up the concrete class by engine_type and validates into it.

Factory and Backend Instantiation#

create_vlm_engine() is the single entry point for constructing any backend. It receives:

• options: BaseVlmEngineOptions — determines which branch to take
• model_spec: VlmModelSpec — used to generate the engine-specific EngineModelConfig (repo_id, revision, torch dtype, generation settings) via model_spec.get_engine_config(engine_type) 6
• enable_remote_services, artifacts_path, accelerator_options — forwarded to inline engines

All backend modules are lazy-imported inside each branch, so unused backends don't require their optional dependencies to be installed 7.

For API-type engines, model_spec.get_api_params(engine_type) is called to inject provider-specific parameters (model name aliases, etc.) into model_config.extra_config["api_params"] 8.

Concrete Backend Classes#

All concrete option classes are in vlm_engine_options.py. Key per-backend settings:

• Transformers — device, 8-bit loading, quantization, torch dtype, model compilation 9
• MLX — trust_remote_code only; Apple Silicon exclusive 10
• vLLM — tensor parallelism, GPU memory utilization, cudagraph_mode (PIECEWISE by default) 11
• API — base URL, headers, timeout, max concurrency; supports OpenAI, Ollama, LM Studio 12

AUTO_INLINE Backend Selection#

AUTO_INLINE is the default for all built-in presets. AutoInlineVlmEngine defers backend selection to initialize() via an internal _select_engine() call 13:

1. macOS + Apple Silicon (MPS detected): Selects MLX if the model has an explicit MLX export declared and mlx_vlm is importable. Otherwise logs a warning and falls back to Transformers. 14
2. CUDA + prefer_vllm=True: Selects VLLM if the model's supported_engines includes it and vllm is importable. 15
3. Default: Selects TRANSFORMERS. 16

AutoInlineVlmEngineOptions exposes a prefer_vllm: bool flag (default False) to opt into the VLLM path on CUDA 17.

Note: The Transformers VLM engine explicitly excludes MPS from its supported_devices ([CPU, CUDA, XPU] only). If you run on Apple Silicon without an MLX model export, the engine falls back to CPU via Transformers. 18

Usage in Pipeline Stages#

Pipeline stages call create_vlm_engine() to get a backend instance and then drive it entirely through predict_batch(). The code/formula enrichment stage (CodeFormulaVlmModel) illustrates the typical pattern:

1. Construct VlmEngineInput objects — one per cropped region, carrying the PIL image, a sentinel prompt (<code> or <formula>), temperature=0.0, max_new_tokens=2048, and model-defined stop strings. 19
2. Call engine.predict_batch(batch) in groups of 5 elements. 19
3. Post-process the VlmEngineOutput.text results (strip special tokens, extract language tags, etc.). 19

The engine is fully decoupled from document data structures: it only sees PIL images and strings, making it reusable across different pipeline stages (VLM-based table structure, caption enrichment, code/formula extraction, etc.) without modification.