HuggingFace Transformers Integration#

Docling's structured extraction pipeline uses TransformersExtractionModel to run vision-language models (VLMs) locally via HuggingFace Transformers. The default preset is numind/NuExtract-2.0-2B, with ibm-granite/granite-vision-4.1-4b as an alternative 1. The class loads the model via AutoModelForImageTextToText.from_pretrained and the processor via AutoProcessor.from_pretrained 2.

This integration has a history of breaking API changes across Transformers versions, requiring explicit version exclusions in pyproject.toml.

Known Compatibility Issues#

1. dtype vs torch_dtype (active bug)#

The model init call passes dtype= to AutoModelForImageTextToText.from_pretrained 3. Certain Transformers 5.x builds dropped dtype as a valid kwarg for Qwen2VLForConditionalGeneration, causing:

TypeError: Qwen2VLForConditionalGeneration.__init__() got an unexpected keyword argument 'dtype'

This was first reported against Docling 2.58.0 4 and remains unresolved as of July 2026, with users still hitting it on Transformers 5.12.1 4. The correct kwarg for modern Transformers is torch_dtype.

2. AutoModelForVision2Seq removed (Transformers 5.x)#

Transformers 5.x removed AutoModelForVision2Seq. PR #3200 migrated all model specs to AutoModelForImageTextToText (AUTOMODEL_IMAGETEXTTOTEXT). Any code or serialized config still referencing the old class will fail on Transformers ≥5.0.

3. processor.tokenizer attribute missing (TokenizersBackend)#

In Transformers 5.x, AutoProcessor.from_pretrained for some models returns a TokenizersBackend that exposes _tokenizer rather than tokenizer. This breaks:

• TransformersExtractionModel.__init__() — accesses self.processor.tokenizer.padding_side
• build_nuextract_inputs() — calls processor.tokenizer.apply_chat_template(...)

PR #3276 partially addressed this in TransformersEngine via a _get_tokenizer() helper, but the prompt_utils.py direct access remains 5.

4. torch._dynamo import error (macOS + Transformers 5.x)#

ImportError: cannot import name 'ConvertFrameReturn' from 'torch._dynamo.types' appears on certain macOS Torch + Transformers 5.x combinations. It is triggered by the torch.compile() call applied to the NuExtract model at init 6. This call is guarded to skip on Python ≥ 3.14 7.

Version Constraints (models-vlm-inline extra)#

The pyproject.toml models-vlm-inline extra encodes the known-working ranges:

macOS carries a tighter upper bound (<5.9.0) due to an unresolved Transformers issue (huggingface/transformers#46159) 8. Transformers 5.0–5.3 are excluded on all platforms due to the Vision2Seq removal and TokenizersBackend regressions.

⚠️ Still unresolved: As of July 2026, Transformers 5.12.1 still breaks the dtype kwarg path 4. Users hitting this outside the pinned range should downgrade to a version within the constraints above.

Install via:

pip install "docling[vlm]"

qwen-vl-utils Dependency#

qwen-vl-utils>=0.0.11 is a hard runtime dependency for the NUEXTRACT prompt style 9. It provides process_vision_info and fetch_image, which are called inside build_nuextract_inputs(). The GRANITE_VISION prompt style does not require it 5.

Key References#