GPU Accelerator Support#

Docling supports hardware-accelerated inference across four device backends: CUDA (NVIDIA), MPS (Apple Silicon via Metal Performance Shaders), XPU (Intel GPU), and CPU. Device selection is centralized in two files:

• accelerator_options.py — defines AcceleratorDevice and AcceleratorOptions
• accelerator_utils.py — implements decide_device(), the runtime resolution function

Configuration#

AcceleratorOptions is a Pydantic BaseSettings model, meaning fields can be set programmatically or via environment variables (prefix: DOCLING_) 1:

Valid device values: auto, cpu, cuda, cuda:N (specific GPU index), mps, xpu 2.

Flash Attention 2 requires the flash-attn package and a compatible NVIDIA Ampere-or-newer GPU 3.

Device Resolution (decide_device)#

decide_device(accelerator_device, supported_devices) is called by each model at initialization. It:

1. Probes availability: checks torch.cuda.is_available(), torch.backends.mps.is_available(), and torch.xpu.is_available() 4.
2. Filters by supported_devices: if a model passes a supported_devices list, unavailable-or-excluded accelerators are silently removed from consideration 5.
3. Resolves in AUTO mode: priority order is CUDA → MPS → XPU → CPU 6.
4. Raises AcceleratorDeviceNotAvailableError if an explicitly requested device is unavailable or not in the model's supported_devices 7.

Recommendation: Use device="auto" unless targeting a specific GPU index. Explicit device values will raise hard errors if the hardware is absent.

Per-Component Accelerator Support Matrix#

Support is inconsistent across Docling's inference components:

Key distinction on Apple Silicon: VLM workloads use MLX (via MlxVlmEngine) rather than raw MPS. The AUTO_INLINE VLM engine selects MLX automatically on macOS when the model has an MLX export and mlx-vlm is installed; otherwise it falls back to Transformers on CPU 10.

CUDA-Specific Options#

• Multi-GPU: Use cuda:N (e.g., cuda:1) to target a specific GPU. decide_device validates the index against torch.cuda.device_count() 13.
• Flash Attention 2: Enable via AcceleratorOptions(cuda_use_flash_attention2=True) or DOCLING_CUDA_USE_FLASH_ATTENTION2=true. Requires Ampere+ GPU and flash-attn 3.
• vLLM for VLM inference: On CUDA, AUTO_INLINE can route to vLLM if prefer_vllm=True is set in AutoInlineVlmEngineOptions and the model supports it 10.
• Nemotron OCR: Requires CUDA 13.x, Linux x86_64, Python 3.12 exactly. Install with pip install "docling[feat-ocr-nemotron]" 12.

Known Limitations#

• TableFormer MPS disabled: Both V1 and V2 override MPS → CPU at init. The reason is documented as "until we know why it makes things slower" — this is a known open issue, not a resolved one 9.
• Transformers VLM + MPS: Setting device=mps with a VLM preset that has no MLX export raises AcceleratorDeviceNotAvailableError. Use device=auto for safe fallback 9.
• MLX thread safety: MlxVlmEngine and MLX Whisper serialize on a process-wide _MLX_GLOBAL_LOCK. Avoid MLX-based models in high-concurrency scenarios 9.
• Third-party models with hardcoded .cuda() will fail on MPS/CPU without upstream changes 9.

Quick Reference: Environment Variables#

DOCLING_DEVICE=auto|cpu|cuda|cuda:N|mps|xpu
DOCLING_NUM_THREADS=4 # also reads OMP_NUM_THREADS as fallback
DOCLING_CUDA_USE_FLASH_ATTENTION2=true

14