VL Mode (VLM Backend) in MinerU#

VL (Vision-Language) mode is a document-parsing backend (-b vlm-engine) that delegates structural understanding to an end-to-end vision-language model — typically Qwen2-VL — rather than the traditional pipeline of layout detection + OCR + specialized models. The model processes page images directly and returns structured block data containing bounding boxes, block types, and extracted content in a single inference pass 1.

Architecture#

The entry point is vlm_analyze.py, which:

1. Renders PDF pages into PIL images in windowed batches (default window: 64 pages) 2
2. Calls predictor.batch_two_step_extract() on each window via a MinerUClient wrapper
3. Passes raw model output to model_output_to_middle_json.py to produce the middle.json intermediate artifact

The ModelSingleton caches loaded model instances keyed by (backend, model_path, server_url). Supported inference backends include transformers, vllm-engine, vllm-async-engine, lmdeploy-engine, and mlx-engine 3.

Output format validation happens in vlm_magic_model.py. Each block must have a bbox (x1,y1,x2,y2) and type field. Malformed blocks are caught, logged as warnings, and skipped rather than raising 4. Text blocks with null content are normalized to "" to prevent downstream failures 5. The recognized block types are: text, title, image, table, chart, code, algorithm, equation, ref_text, phonetic, header, footer, page_number, aside_text, page_footnote, list 6.

VLM-Specific Output Types (v2.5+)#

The VLM backend adds block types not present in the pipeline backend: code/algorithm, ref_text, phonetic, header/footer/page_number (routed to discarded_blocks), aside_text, page_footnote 7. The VLM backend output is not backward-compatible with the pipeline backend format.

Deprecation: VLM → Hybrid Mode#

VLM mode is actively being deprecated in favor of hybrid mode. A maintainer confirmed in June 2026:

"2605 的 merge_prev 标识被训废了，因此 vlm 模式不能合并都是正常的。建议使用 hybrid 模式，hybrid 模式硬件要求与 vlm 模式完全一致，且支持使用行 det 信息进行跨栏与跨页的合并。在线 saas 的 vlm 实际上已经是 hybrid 了，我们后面也会考虑直接移除 vlm backend。"
(The merge_prev marker in model v2605 was "training wasted," so VLM mode's inability to merge is expected. Hybrid mode has identical hardware requirements and supports cross-column/cross-page merging via line detection. The online SaaS VLM is already hybrid; we will consider removing the VLM backend entirely.)

8

The official online service (opendatalab.com) already uses hybrid mode by default, confirmed by a maintainer reply in July 2026 9.

Migration path: Replace -b vlm-engine with -b hybrid-engine (or hybrid-auto-engine). Hardware requirements are identical 8.

Known Quality and Hallucination Issues#

A maintainer stated: "如果严格要求无幻觉，推荐使用 pipeline 模式" — If strict hallucination-free output is required, use pipeline mode 9. For occasional dropped characters (often caused by abnormal fonts), enabling the OCR switch may help 9.

VLM vs. hybrid trade-offs: VLM handles complex/borderless tables best and is immune to PP-DocLayoutV2 detection failures (since the VLM processes layout end-to-end). It is approximately 40% slower than hybrid 10. For dense layouts where hybrid misses text blocks, VLM correctly detects all records 10.

VRAM / Memory Management#

OOM Pattern on High-VRAM Devices (NPU/GPU)#

The primary OOM trigger is PyTorch memory fragmentation, not actual exhaustion. After many alloc/free cycles, reserved memory far exceeds allocated memory, leaving no contiguous block for the next allocation. A representative error from a 60.96 GiB Ascend 910B 11:

RuntimeError: NPU out of memory. Tried to allocate 20.00 MiB
(total capacity: 60.96 GiB; allocated: 729.95 MiB; free: 10.81 MiB; reserved: 764.00 MiB)

The clean_vram() utility only fires when total VRAM ≤ 8 GB, so high-VRAM devices skip cache cleanup entirely unless MINERU_VIRTUAL_VRAM_SIZE overrides the threshold 12.

Key Mitigations#

Device exclusivity: Hybrid/VLM engine modes require the GPU/NPU to be dedicated to MinerU. Sharing the device with other processes exacerbates fragmentation 11.

ModelSingleton leak: The VLM ModelSingleton never evicts cached model instances across requests. Unlike the pipeline/hybrid singletons, vlm_analyze.py registers atexit shutdown hooks 13 — but these only fire on process exit. Periodic service restart or running mineru-vllm-server as a separate process remains the most reliable mitigation 12.

Key Files and References#