Torch Compile Optimization#

torch.compile() is applied at model initialization across Docling's Transformers-based inference engines to improve throughput via TorchDynamo/Inductor graph compilation. It is on by default (compile_torch_models = True) and controlled by a global flag that can be overridden per-engine or globally.

Global Control#

The top-level switch is settings.inference.compile_torch_models, defined in docling/datamodel/settings.py. It defaults to True and can be overridden via the DOCLING_INFERENCE_COMPILE_TORCH_MODELS environment variable or the scoped() context manager 1.

Per-engine, the flag surfaces as compile_model: bool in each engine options dataclass (e.g., TransformersObjectDetectionEngineOptions, TransformersImageClassificationEngineOptions), with its default wired to default_compile_model() 2.

Python Version and PyTorch Version Gating#

All Transformers-based engines follow the same two-tier version gate before calling torch.compile():

• Python < 3.14: compile with any PyTorch 2.x
• Python ≥ 3.14: compile only if torch >= 2.10; otherwise emit a warning and skip

This pattern is consistently applied in:

• VLM engine — transformers_engine.py L266–278
• Object detection engine — transformers_engine.py L156–170
• Image classification engine — transformers_engine.py L132–146

The Python 3.14 cutoff was introduced because torch._dynamo was incompatible with CPython 3.14's new internals at the time (see PR #2530).

NuExtract: Simpler Gate, Always-On#

The legacy extraction path in transformers_extraction_model.py L99–105 uses a simpler (earlier) gate: torch.compile() is applied unconditionally when prompt_style == NUEXTRACT and sys.version_info < (3, 14). There is no compile_model flag here — compilation is always attempted for NuExtract on supported Python versions. If the condition fails (Python ≥ 3.14 or non-NuExtract style), the model is placed in .eval() mode instead.

⚠️ A known failure mode on macOS with Transformers 5.x is ImportError: cannot import name 'ConvertFrameReturn' from 'torch._dynamo.types', triggered by this torch.compile() call 3.

Scope: Model Only (Post-Processing Excluded)#

torch.compile() is applied to the model object alone. Post-processing steps (e.g., image_processor.post_process_object_detection()) run outside the compiled graph. In the object-detection engine, inference runs inside torch.inference_mode() and results are passed to the HF processor for post-processing separately 4. This avoids graph breaks from Python-heavy post-processing code.

vLLM Backend: No torch.compile()#

The vLLM engine (vllm_engine.py) does not call torch.compile() at all. Instead, it uses vLLM's native CompilationConfig with CUDAGraphMode for GPU, and enforce_eager=True on CPU 5.

The separation is intentional: running vLLM 0.19+ with PyTorch 2.10+ and torch.compile() simultaneously triggers a duplicate TritonTemplate assertion. The workaround adopted in PR #3404 was to disable torch.compile in the vLLM path and use spawn-based multiprocessing. Additionally, vLLM is not yet available on Python 3.14 6, providing a natural boundary between the two compilation systems.

LayoutPredictor (docling-ibm-models)#

The standalone LayoutPredictor in docling-ibm-models does not use torch.compile(). It loads via AutoModelForObjectDetection, calls .eval(), and runs inference under @torch.inference_mode() 7. Compilation for layout prediction is handled upstream by the TransformersObjectDetectionEngine in docling when that engine wraps the model.

Key Files#