Docling Serve Worker Architecture#

Docling-serve has two orthogonal worker layers that operate independently and should not be confused:

1. HTTP-layer: uvicorn worker processes (OS-level, handle HTTP I/O)
2. Processing-layer: orchestrator workers inside each uvicorn process (handle ML inference)

The orchestration backend is selected by DOCLING_SERVE_ENG_KIND (local | rq | ray, default: local), wired up by orchestrator_factory.py.

┌──────────────────────────────────────────────────────────┐
│ uvicorn process (1..N, set by UVICORN_WORKERS) │
│ │
│ FastAPI app │
│ └── LocalOrchestrator (lru_cache singleton per proc) │
│ ├── asyncio.Queue (task_queue) │
│ ├── AsyncLocalWorker 0 ──► asyncio.to_thread() │
│ ├── AsyncLocalWorker 1 ──► asyncio.to_thread() │
│ └── ... (DOCLING_SERVE_ENG_LOC_NUM_WORKERS) │
└──────────────────────────────────────────────────────────┘

Layer 1 — Uvicorn HTTP Workers#

UVICORN_WORKERS (env prefix UVICORN_, default None = 1 process) controls how many OS-level processes uvicorn forks 1. These are passed directly to uvicorn.run(workers=...) 2. When UVICORN_WORKERS > 1 or --reload is active, the server runs as a subprocess-based multi-process setup 3; CLI args are ignored in that mode and must be set via env vars 4.

Each uvicorn process gets its own independent LocalOrchestrator instance (via @lru_cache on get_async_orchestrator()) 5, its own asyncio event loop, and its own pool of processing workers. There is no shared state between uvicorn workers.

Layer 2 — Orchestrator Processing Workers (Local Backend)#

Inside each uvicorn process, LocalOrchestrator runs an asyncio.Queue and spawns N coroutine-based workers 6:

• DOCLING_SERVE_ENG_LOC_NUM_WORKERS (default: 2) — number of concurrent AsyncLocalWorker instances 7
• Workers are launched as asyncio.create_task(w.loop()) in the FastAPI lifespan at startup 8
• Each worker loops indefinitely: await self.orchestrator.task_queue.get() → process → notify 9

The asyncio-to-thread bridge: ML inference is CPU/GPU-bound. Each worker offloads the conversion call to a thread via asyncio.to_thread(run_task), which runs in Python's default ThreadPoolExecutor. This keeps the asyncio event loop (and HTTP handling) unblocked during heavy inference.

Startup and Health Supervision#

The FastAPI lifespan context manager:

1. Calls orchestrator.warm_up_caches() if DOCLING_SERVE_LOAD_MODELS_AT_BOOT=true to eagerly load ML models 10
2. Launches process_queue() as a supervised background task 11
3. If the queue processor crashes with an unhandled exception, _supervise_queue_processor sets a failure flag 12, causing /livez to return 503 so the platform restarts the pod

/ready returns 503 until _models_ready is set after warm-up 13.

Model Sharing Across Workers#

The key memory tradeoff in the local backend is DOCLING_SERVE_ENG_LOC_SHARE_MODELS (default: false) 14:

This flag is forwarded via LocalOrchestratorConfig(shared_models=...) and checked in AsyncLocalWorker.loop(). When use_shared_manager=True, the worker reuses self.orchestrator.cm; when False, it creates a new DoclingConverterManager(self.orchestrator.cm.config) 15.

RQ and Ray Backends#

With ENG_KIND=rq, the uvicorn API pod itself does no ML inference — it only pushes task IDs to a Redis queue. Separate docling-serve rq-worker processes pull from the queue, each loading models independently 17. The API process still runs process_queue() but as a pub/sub listener for result notifications, not as a converter.

With ENG_KIND=ray, conversion is handled by Ray Serve actors. The converter actor pool autoscales between DOCLING_SERVE_ENG_RAY_MIN_ACTORS (default: 1) and DOCLING_SERVE_ENG_RAY_MAX_ACTORS (default: 10) 18, with target_requests_per_replica controlling the autoscaling trigger 19. Per-tenant concurrency is capped by DOCLING_SERVE_ENG_RAY_MAX_CONCURRENT_TASKS (default: 5) 20. See GPU Memory Management for Ray actor memory configuration.

Key Settings Reference#

Key source files:

• docling_serve/settings.py — all env var definitions
• docling_serve/orchestrator_factory.py — backend selection and config wiring
• docling_jobkit/orchestrators/local/orchestrator.py — LocalOrchestrator and process_queue()
• docling_jobkit/orchestrators/local/worker.py — AsyncLocalWorker loop and asyncio.to_thread bridge
• docling_serve/app.py (lines 160–224) — lifespan startup, queue supervision, health probes