PDF Layout Detection#

Layout detection is the stage in MinerU's pipeline that identifies and classifies structural regions — text blocks, titles, tables, images, formulas, and more — in each PDF page image before any content extraction occurs. Because all downstream steps (OCR, formula recognition, reading-order assignment) consume the bounding boxes emitted here, any region missed by the layout model is permanently lost from the output. No later stage can recover content from an undetected region.

The core model is PP-DocLayoutV2 (mineru/model/layout/pp_doclayoutv2.py), an RT-DETR-based object detector that predicts 25 layout classes including text, paragraph_title, table, image, display_formula, inline_formula, and vertical_text 1. It also predicts reading order via a transformer decoder with bidirectional attention masks 2.

Three backends consume this model differently:

How PP-DocLayoutV2 Works#

The PPDocLayoutV2LayoutModel wraps the PPDocLayoutV2ForObjectDetection detector. The batch_predict method is the main entry point used by both Hybrid and Pipeline backends.

Detection pipeline per batch:

1. Preprocess each image into a tensor via _preprocess_single_image()
2. Run self.model(pixel_values=batch_tensor) for object detection 6
3. Post-process via _post_process_object_detection — applies confidence thresholding, IoU-based deduplication, formula box merging, and header/footer boundary detection 7

Two-level thresholding controls which boxes are kept:

• Per-class thresholds (0.4–0.5) applied inside the model's forward pass using max_probs >= thresholds 8. These are defined in DEFAULT_CLASS_THRESHOLDS and registered as adjustable tensor buffers 9.
• Global confidence threshold applied per prediction in post-processing: keep = score >= self.conf 10.

The text and paragraph_title classes use a threshold of 0.4 — lower than most other classes, making them the most sensitive detections. A dense layout that confuses the model can still yield probabilities below this floor, causing entire regions to drop.

Backend Integration#

Hybrid Backend#

The MineruHybridModel initializes the layout model as AtomicModel.Layout 11. Layout prediction is wrapped by run_layout_inference(), a thread-safe lock guard used across backends.

The document is processed in configurable page windows (default: 64 pages) 12. Per window, the call chain is:

doc_analyze / aio_doc_analyze → _predict_layout_for_window → _predict_layout_for_title_split → run_layout_inference(layout_model.batch_predict, ...)

The layout results feed para_blocks in the middle.json intermediate file, which all subsequent extraction steps read from.

Pipeline Backend#

batch_analyze.py follows the same pattern — __call__ invokes run_layout_inference(self.model.layout_model.batch_predict, ...) 13.

VLM Backend#

The VLM backend does not call PPDocLayoutV2 directly. It delegates to predictor.batch_two_step_extract() (or the async variant), where the vision-language model handles structural understanding end-to-end 14. This is the architectural reason VLM is immune to PP-DocLayoutV2's detection failures.

Known Issue: Hybrid Backend Misses Text Blocks in Dense Layouts#

Affected version: MinerU v3.4.0 Hybrid backend 15

Symptom: In pages with dense, multi-record layouts — e.g., Chinese admissions documents with tightly packed short-line entries under a professional group header — the Hybrid backend's Layout Predict phase fails to generate bounding boxes for some text regions. Those records are completely absent from the final output. The middle.json para_blocks shows a corresponding vertical gap (≈92 units) where the missing content visually appears 15.

Root cause: The PP-DocLayoutV2 model produces confidence scores below its detection thresholds for the affected regions. The issue is in the layout model's bbox generation, not in character extraction or OCR — confirmed by 15:

• mineru.utils.pdf_text_tool.get_page_chars() returns all expected characters (2,048 chars in the test case)
• Forcing --mode txt (bypasses OCR entirely) still reproduces the omission
• The issue is consistent across --effort levels

Scope: 5+ records missed across 20 sampled pages of a 600-page Chinese university admissions PDF. Dense layouts common in military academy and directed-recruitment sections are especially affected 15.

Workaround: Use the VLM backend (-b vlm-engine). It correctly detected all missing records in comparative testing. Trade-off: approximately 40% slower than the Hybrid backend 15.

Debugging Layout Detection Failures#

Step 1 — Isolate the page. Rerun on a single page using -s <page_index> -e <page_index> to get a clean intermediate file.

Step 2 — Inspect middle.json. Check para_blocks for bbox gaps. A gap in the y coordinate range where content should appear is the primary signal that layout detection dropped a region 15.

Step 3 — Verify the text layer. Call mineru.utils.pdf_text_tool.get_page_chars() on the same page. If characters are present in the text layer but absent from para_blocks, the layout model is the culprit 15.

Step 4 — Confirm with --mode txt. Force text-extraction mode (-m txt) to rule out OCR as a factor. If the same records are still missing, the failure is definitively in the Layout Predict phase 15.

Tuning option. The per-class detection thresholds in DEFAULT_CLASS_THRESHOLDS are registered as adjustable tensor buffers 9. Lowering the text class threshold below 0.4 may recover borderline detections but risks increasing false positives.

Switch backend. If the layout model consistently underperforms on a document type, switching to vlm-engine is the most reliable fix and requires no model tuning 15.