VLM Formula and Code Extraction#

Docling's VLM code/formula enrichment stage runs after the main layout analysis pass and enriches already-detected CodeItem and FORMULA TextItem elements with accurate text extracted via a Vision Language Model — converting formulas to LaTeX and code blocks to their original source text with language tag. It is off by default and must be enabled explicitly via PdfPipelineOptions.do_code_enrichment and/or do_formula_enrichment 1.

The central implementation lives in CodeFormulaVlmModel, which inherits from BaseItemAndImageEnrichmentModel. It is instantiated in StandardPdfPipeline._init_models() and prepended to the enrichment_pipe list, running after per-page assembly completes.

Activation flags in PdfPipelineOptions:

2

Preset System and Available Models#

Model selection uses a named-preset system defined in stage_model_specs.py. Each StageModelPreset bundles a VlmModelSpec (HuggingFace repo ID, prompt template, response format, engine-specific overrides) with default inference engine and image scaling configuration 3.

Two presets are registered for CODE_FORMULA 4:

CodeFormulaV2 uses stop strings </doctag> and <end_of_utterance> 5. granite_docling shares the GRANITE_DOCLING_MODEL_SPEC_BASE with the VLM-convert stage (max 8192 new tokens, Ollama alias ibm/granite-docling:258m) 6.

The default options instance is constructed at import time: _default_code_formula_options = CodeFormulaVlmOptions.from_preset("codeformulav2") 7.

To switch presets or add a custom engine:

from docling.datamodel.pipeline_options import CodeFormulaVlmOptions, PdfPipelineOptions

# Use the granite_docling preset instead
code_formula_options = CodeFormulaVlmOptions.from_preset("granite_docling")

# Or use a specific engine (e.g., MLX on Apple Silicon)
from docling.datamodel.vlm_engine_options import MlxVlmEngineOptions
code_formula_options = CodeFormulaVlmOptions.from_preset(
    "codeformulav2",
    engine_options=MlxVlmEngineOptions()
)

pipeline_options = PdfPipelineOptions(
    do_code_enrichment=True,
    do_formula_enrichment=True,
    code_formula_options=code_formula_options,
)

See the full preset comparison example for side-by-side output comparison.

Inference Backends (VlmEngineType)#

The VlmEngineType enum in vlm/base.py defines the available backends:

VlmModelSpec.engine_overrides lets presets specify per-engine model checkpoints, torch dtypes, and generation options. For example, the granite_docling preset maps VlmEngineType.MLX → ibm-granite/granite-docling-258M-mlx and VlmEngineType.API_OLLAMA → model name ibm/granite-docling:258m 6.

Engine resolution happens in StagePresetMixin.from_preset(). If no engine_options are provided, it infers them from preset.default_engine_type. Auto-inline selects MLX only when the model has an explicit MLX export declared 8.

Processing Flow#

1. Element filtering — is_processable() checks if an item is a CodeItem (with extract_code=True) or a TextItem with label == FORMULA (with extract_formulas=True).

2. Image cropping — inherited from BaseItemAndImageEnrichmentModel; each element's bounding box is cropped from the page image at 1.67× scale (≈120 DPI) with an 18% expansion factor 9.

3. Prompt construction — _get_prompt() returns <code> for code items and <formula> for formulas. These sentinel prompts signal extraction intent to the model.

4. Batch inference — elements are batched in groups of 5 and passed as VlmEngineInput objects to engine.predict_batch(), with temperature=0.0 and max_new_tokens=2048 10. The BaseVlmEngine abstract interface (base.py) hides backend differences.

5. Post-processing — _post_process() removes </code>, </formula>, <loc_0><loc_0><loc_500><loc_500> tokens, and truncates at <end_of_utterance>.

6. Language detection (code only) — _extract_code_language() parses a leading <_language_> tag (e.g., <_javascript_>) from the model output. The matched value is converted to a CodeLanguageLabel enum via _get_code_language_enum(), defaulting to UNKNOWN if unrecognized 11.

7. Document update — item.text is set to the cleaned output; CodeItem.code_language is also updated 12.

Key Files and Entry Points#