VLM Token Limits and Truncation#

Token limits cap how many tokens a VLM can generate per page. When a limit is hit, the pipeline detects it via the VlmStopReason enum and surfaces a PARTIAL_SUCCESS result rather than silently returning truncated output.

VlmStopReason enum#

Defined in docling/datamodel/base_models.py, this str enum has five values:

The stop_reason field lives on VlmPrediction (the per-page prediction object) and defaults to UNSPECIFIED.

Configuring the token budget#

max_new_tokens is set on InlineVlmOptions with a default of 4096. Override it per model spec:

• GRANITEDOCLING_TRANSFORMERS / GRANITEDOCLING_2STAGE_TRANSFORMERS — 8192
• NANONETS_OCR2_TRANSFORMERS / NANONETS_OCR2_MLX — 15000
• LIGHTONOCR_TRANSFORMERS / LIGHTONOCR_MLX / LIGHTONOCR_VLLM_API — 4096

For API-based models (ApiVlmOptions), the limit is passed through the params dict (e.g., max_tokens=4096) because the API server controls generation; there is no max_new_tokens field on ApiVlmOptions.

stop_strings on both InlineVlmOptions and ApiVlmOptions provides early stopping before the token budget is exhausted (e.g. ["</doctag>", "<|end_of_text|>"] for GraniteDocling) 1. When a stop string fires, the reason becomes STOP_SEQUENCE.

How backends populate stop_reason#

Inference backends produce a string value which is later promoted to the enum in _prediction_from_engine_output:

Note: The MLX engine produces string values "stop_string" and "custom_criteria" that do not match any VlmStopReason value string (which would need "stop_sequence"), so they fall back to UNSPECIFIED after the enum conversion in _prediction_from_engine_output.

Truncation detection and pipeline consequences#

Standard VLM pipeline (vlm_pipeline.py): when stop_reason is LENGTH or CONTENT_FILTERED, the pipeline appends an ErrorItem with category=INFERENCE_FAILURE and sets ConversionStatus.PARTIAL_SUCCESS for that page 7.

Extraction VLM pipeline (extraction_vlm_pipeline.py): uses the same logic — LENGTH or STOP_SEQUENCE stop reasons on a page's VlmPrediction set ext_res.status = ConversionStatus.PARTIAL_SUCCESS 8. The raw VLM text is still stored in ExtractedPageData.raw_text even when JSON parsing fails 9.

The final _determine_status method in ExtractionVlmPipeline preserves PARTIAL_SUCCESS if it was set during the per-page loop 10.

Key source files#