vLLM Worker Health and Recovery#

Overview#

When mineru-router manages local GPU workers, it relies on each worker's /health endpoint to decide whether to restart that worker. A vLLM EngineCore fatal failure (EngineDeadError) breaks this loop: the GPU process crashes, the active parse task fails, but the worker's HTTP server remains alive and /health continues returning HTTP 200. The router never sees a non-200 response, so it never triggers its restart logic. Every subsequent task routed to that worker fails immediately until the container is manually restarted. This is tracked in issue #5171 1.

Architecture: How Recovery Is Supposed to Work#

mineru-router mineru-api (local GPU worker)
     | |
     |-- POST /tasks -----------------------> |
     | [AsyncTaskManager._process_task]
     | [vLLM EngineCore dies]
     | task.status = TASK_FAILED
     | last_worker_error = None ← gap here
     | |
     |-- GET /health -----------------------> |
     |<-- 200 OK (is_healthy() = True) ------ |
     | |
     | (router does NOT restart worker) |
     |-- POST /tasks -----------------------> | ← fails immediately

The router polls /health every 2 seconds and restarts a local worker only after WORKER_HEALTH_FAILURE_RESTART_THRESHOLD = 5 consecutive non-200 responses 2 — meaning recovery takes up to ~10 seconds once the health signal is correct. The /health endpoint in fast_api.py returns 503 only when AsyncTaskManager.is_healthy() returns False 3. is_healthy() returns False only when last_worker_error is non-None 4.

The Gap: _process_task Does Not Write last_worker_error#

AsyncTaskManager._process_task catches all exceptions in a bare except Exception block, marks the task failed, and signals the task event — but does not set self.last_worker_error. Because last_worker_error stays None, is_healthy() returns True, /health returns 200, and the router keeps routing to the dead worker 1.

The last_worker_error field is already read by both is_healthy() and the /health endpoint, and it is set correctly when the dispatcher loop itself crashes 5. The missing piece is writing it for engine-fatal exceptions that surface inside individual task processing.

Root Cause: PIL Race Condition Triggering EngineDeadError (PR #4979)#

The most common production trigger for EngineDeadError is a race condition in the hybrid-auto-engine backend: PIL images are passed by reference to async predictor threads, then closed in the finally block of aio_doc_analyze before those threads finish. Under concurrent load (≥2 PDFs), the in-flight thread calls .resize() on a closed image, raises ValueError: Operation on closed image, which propagates into the vLLM event loop and kills EngineCore.

PR #4979 (merged May 19, 2026) addresses this root cause by copying PIL images before passing them to the predictor in hybrid_analyze.py — decoupling the image lifetimes from the close-on-exit finally block. This prevents the most common EngineDeadError trigger but does not address the general health-reporting gap.

Fix: Classify Fatal Errors in _process_task (PR #5188)#

PR #5188 (open as of 2026-07-15) adds fatal-error detection to _process_task in mineru/cli/fast_api.py:

• FATAL_WORKER_ERROR_MARKERS: direct string patterns ("engine dead", "engine is dead", "enginecore", "async llm engine has failed", etc.)
• FATAL_WORKER_ERROR_NORMALIZED_MARKERS: alphanumeric-normalized forms ("enginedead", "enginecore", "unrecoverableengine") to handle vLLM version variations.
• A new _is_fatal_worker_error(exc) helper checks the exception's module, class hierarchy, and message against both marker sets.

When a fatal error is detected, _process_task sets self.last_worker_error and calls self._wake_waiters() (instead of the narrower _signal_task_event()), which causes is_healthy() to return False, /health to return 503, and the router to restart the worker after 5 consecutive failures (~10 seconds at default poll rate). Non-fatal task errors (document-specific parsing failures) continue to be task-local and do not affect worker health.

Key Files and Entry Points#

Workaround (Until PR #5188 Is Merged)#

Restart the Pod or container running the failed worker. The router will detect the process exit and respawn a new worker with a fresh vLLM engine. To reduce blast radius, limit concurrency with --gpu-memory-utilization and avoid heavy concurrent PDF loads until PR #4979 is confirmed in your deployed version.