VLM Inference Configuration#

When running Docling's DocTags-based VLM models (e.g., granite-docling-258M, smoldocling) through vLLM or any OpenAI-compatible /v1/chat/completions endpoint, generation parameters must be explicitly configured to preserve structural DocTags tokens. Without the correct settings, the model silently returns an empty document even though ConversionStatus.SUCCESS is reported — making the failure invisible to callers 1.

The Critical Parameter: skip_special_tokens=False#

DocTags structural markup (<section_header_level_1>, <text>, <otsl>, <doctag>, etc.) are special tokens in the model's vocabulary. Most inference servers strip special tokens by default during decoding. When that happens, only <loc_*> location coordinates survive; the element tags disappear entirely, and DocTags parsing yields zero document elements 1.

The fix: pass "skip_special_tokens": False in the params dict of ApiVlmEngineOptions when connecting to vLLM or a generic OpenAI-compatible endpoint 2.

The official vLLM example shows the required params:

vlm_options = VlmConvertOptions.from_preset(
    "granite_docling",
    engine_options=ApiVlmEngineOptions(
        runtime_type=VlmEngineType.API,
        url="http://localhost:8000/v1/chat/completions",
        params={
            "model": "ibm-granite/granite-docling-258M",
            "temperature": 0.0,
            "max_tokens": 8192,
            "skip_special_tokens": False, # ← required
        },
        timeout=90,
    ),
)

Note: For the TRANSFORMERS engine (local/inline), skip_special_tokens=False is set automatically inside GRANITE_DOCLING_MODEL_SPEC_BASE — no manual override needed. The issue is specific to API and vLLM backends.

Why skip_special_tokens Behaves Differently Per Engine#

The parameter has different mechanics across engines:

For vLLM served as an external API, passing the parameter in params sends it to the server in the JSON payload 5 6. Whether the server honors it depends on its implementation — vLLM does, but LM Studio (as of testing) ignores it even when passed because its chat template or decoding pipeline strips the tokens regardless 1.

Granite-Docling Generation Parameters#

The GRANITE_DOCLING_MODEL_SPEC_BASE defines the canonical generation config for both the VLM_CONVERT and CODE_FORMULA stages:

When using the vLLM API, pass max_tokens (not max_new_tokens, which is a Transformers convention) and skip_special_tokens: False explicitly in params.

Pre-Configured API Types vs. Generic API#

Docling provides three pre-configured VlmEngineType variants that set the correct endpoint URL automatically:

• API_OLLAMA → http://localhost:11434/v1/chat/completions (model auto-set to ibm/granite-docling:258m)
• API_LMSTUDIO → http://localhost:1234/v1/chat/completions
• API_OPENAI → https://api.openai.com/v1/chat/completions

For generic vLLM or other servers, use VlmEngineType.API and provide url and params manually — including skip_special_tokens: False.

The key distinction: Ollama and LM Studio's pre-configured types do not automatically inject skip_special_tokens=False into API params. For endpoints known to strip special tokens (including LM Studio), running the model locally via AUTO_INLINE/Transformers is the most reliable workaround 1.

Silent Empty-Document Failure#

A confirmed sharp edge: when DocTags structural tokens are stripped, conversion completes with ConversionStatus.SUCCESS and md_chars=0 — zero warning, zero error 1. Callers must add an explicit check if operating in automated pipelines:

result = doc_converter.convert(source)
assert result.status == ConversionStatus.SUCCESS
md = result.document.export_to_markdown()
if len(md) == 0:
    # DocTags tokens may have been stripped — check skip_special_tokens config
    raise ValueError("Conversion produced empty document")

Key Files#