Pipeline Streaming Architecture#

MinerU's pipeline backend uses a callback-driven streaming pattern where GPU inference and file I/O output run concurrently. The entry point is doc_analyze_streaming() in pipeline_analyze.py. In contrast, the VLM and hybrid backends return results synchronously and write output immediately after each inference call 1.

Data Flow#

The flow is orchestrated by _process_pipeline() in common.py:

1. A ThreadPoolExecutor(max_workers=1) is created for output writing 2.
2. An on_doc_ready callback is defined: when called, it submits run_output_task() to the executor and appends the resulting Future to output_futures 3.
3. doc_analyze_streaming() is called with the callback, running the windowed inference loop.
4. After the loop, _process_pipeline() blocks on each future via future.result() to propagate any errors 4.

┌─────────────────────────────────────────────────────────┐
│ _process_pipeline() │
│ │
│ ThreadPoolExecutor (max_workers=1) │
│ ┌──────────────────────────────┐ │
│ │ run_output_task ◄──────────┼── on_doc_ready() │
│ │ (_process_output, write MD, │ (callback, called │
│ │ JSON, images…) │ per completed doc) │
│ └──────────────────────────────┘ │
│ │
│ doc_analyze_streaming() │
│ ┌─────────────────────────────────────────────────────┐│
│ │ Window loop (64 pages/window default) ││
│ │ fill → batch_image_analyze() → append results ││
│ │ → _finalize_processing_window_context() ││
│ │ → on_doc_ready(doc_index, model_list, …) ││
│ └─────────────────────────────────────────────────────┘│
└─────────────────────────────────────────────────────────┘

doc_analyze_streaming() Internals#

The function batches all input documents into fixed-size processing windows 5:

• Window size: default 64 pages, overridable via MINERU_PROCESSING_WINDOW_SIZE 6.
• Multi-doc batching: pages are drawn from each doc_context in order until the window capacity is filled 7. A single window can span multiple documents.
• Per-window steps: load images → batch_image_analyze() (runs layout, formula, OCR-det, OCR-rec) → append_batch_results_to_middle_json() → check if any doc is complete.
• When a document's last page is processed, _finalize_processing_window_context() runs post-processing and fires on_doc_ready. A context['closed'] guard ensures the callback fires exactly once per document 8.

The on_doc_ready Callback#

The callback is the seam between inference and output. It receives (doc_index, model_list, middle_json, ocr_enable) and submits the output task to the background thread 3. This means:

• The inference loop continues to the next window immediately, without waiting for output to finish.
• Output writing (Markdown, JSON, image extraction, layout visualization) proceeds in parallel on a single background thread.
• All output futures are joined at the end of _process_pipeline(), so errors in output are not silently swallowed 4.

Contrast: VLM / Hybrid Backends#

The VLM backend (_process_vlm, _async_process_vlm) and hybrid backend use a straightforward synchronous return pattern 9:

• vlm_doc_analyze() or aio_vlm_doc_analyze() returns (middle_json, infer_result) directly.
• _process_output() is called immediately after inference returns — no callback, no executor.
• Documents are processed one at a time in a simple for loop.

The pipeline backend's callback pattern exists because:

1. GPU utilization: multi-model inference across large document batches is the bottleneck. Overlapping I/O output with the next window's inference avoids leaving the GPU idle.
2. Multi-doc batching: the pipeline windows across multiple input documents simultaneously; output can only begin once a document's last page window is complete, making a push-based callback the natural fit.

Note: The pipeline backend has no true async variant. Even when called from aio_do_parse(), it runs via the synchronous _process_pipeline() path (the comment reads: "pipeline 模式暂不支持异步，使用同步处理方式") 10.

Key Source Files#