Windows Multiprocessing#

MinerU uses a persistent ProcessPoolExecutor for PDF image rendering. On Windows, this executor requires extra care due to the platform's spawn-only process model, slower worker startup, connection-reset failures on broken pools, and GBK console encoding. This article covers the design of the executor, known failure modes, and the environment variables that control its behavior.

Persistent Executor Design#

A single module-level ProcessPoolExecutor (_pdf_render_executor) is shared across requests and created lazily on first use 1 2. Capacity is the minimum of available CPUs, MINERU_PDF_RENDER_THREADS, and the hard ceiling MAX_PDF_RENDER_PROCESSES = 3 3 4.

Windows vs. Non-Windows Executor Creation#

On Windows, the executor is created with a plain ProcessPoolExecutor(max_workers=...) — no custom mp_context is needed because Windows always uses spawn 5. On Linux/macOS with a non-spawn start method (e.g., fork), the code explicitly forces spawn to ensure process isolation 6.

See _create_pdf_render_executor for the full implementation.

Process Spawn Delay (Windows cold-start throttle)#

When the pool is starting up and hasn't reached its max_workers count yet, successive submit() calls can race each other and corrupt worker initialization. To prevent this, _submit_pdf_render_task serializes submissions and inserts a 100 ms sleep (PDF_RENDER_PROCESS_SPAWN_DELAY_SECONDS = 0.1) between each new-worker submit 7 8. This is particularly relevant on Windows where process spawning is significantly slower than on Linux.

Timeout and BrokenProcessPool Handling#

Each render call uses concurrent.futures.wait with a timeout driven by MINERU_PDF_RENDER_TIMEOUT (default 300 s) 9. If the deadline is exceeded, or if a BrokenProcessPool exception is raised (e.g., a worker process crashes or is killed), the shared executor is recycled — cleared from the module-level singleton and forcibly terminated 10.

Recycling calls _recycle_pdf_render_executor, which:

1. Clears _pdf_render_executor so the next request gets a fresh pool.
2. Calls _terminate_executor_processes — terminate() → grace period join → kill() if still alive.
3. Calls executor.shutdown(wait=False, cancel_futures=True).

Connection reset errors (ConnectionResetError) typically appear on Windows when a worker process exits abruptly while the main process still has an open pipe to it; they manifest as a BrokenProcessPool and are handled by the same recycle path.

FastAPI Lifecycle Integration#

fast_api.py imports and calls shutdown_pdf_render_executor during the FastAPI shutdown sequence via shutdown_runtime_resources. The executor is also registered with atexit directly in pdf_image_tools.py for non-server usage 11.

The is_main_multiprocessing_process() guard in fast_api.py 12 prevents worker sub-processes from executing top-level server initialization code, which is critical on Windows where the entire module is re-imported in each spawned worker.

GBK Encoding Failure (Windows Console)#

On Windows systems with a Chinese (GBK/CP936) locale, fast_api.py lines 1449–1450 contain two bare print() calls that write to sys.stdout without specifying an encoding 13. If the output contains characters outside GBK (e.g., the copyright symbol © / \xa9), Python raises a UnicodeEncodeError, which surfaces as an API error during PDF parse 14:

'gbk' codec can't encode character '\xa9' in position 593: illegal multibyte sequence

Workarounds (apply before mineru server start):

The permanent fix is to replace the two print() calls with logger.info() (loguru is already imported and configured) or to add sys.stdout.reconfigure(encoding='utf-8') at the CLI entry point. This is tagged [NEXT] in the upstream tracker 14.

Environment Variables Reference#