GPU Memory Management in docling-serve#

GPU and system memory management in docling-serve is handled across three layers: model sharing in the local orchestrator, per-actor resource allocation in the Ray orchestrator, and runtime monitoring via management endpoints. The approach varies significantly by orchestration backend (local, rq, or ray), selected via DOCLING_SERVE_ENG_KIND.

Local Orchestrator: Model Sharing#

The local backend runs workers as forked subprocesses on a single machine. The key memory setting is eng_loc_share_models (DOCLING_SERVE_ENG_LOC_SHARE_MODELS, default: false).

• When false (default), each worker loads its own copy of all ML models into memory (and GPU VRAM). With eng_loc_num_workers=2 (default), this doubles GPU memory consumption.
• When true, models are shared across workers, reducing per-worker VRAM overhead. This is appropriate when GPU memory is constrained and workers are co-located on the same node.

This setting is passed directly to LocalOrchestratorConfig as the shared_models parameter; the actual sharing mechanism is implemented in docling-jobkit.

Ray Orchestrator: Distributed Resource Allocation#

The Ray backend is the primary production-grade orchestrator and exposes the most extensive resource controls. All settings use the DOCLING_SERVE_ENG_RAY_* prefix and are forwarded to the Ray orchestrator in orchestrator_factory.py.

Actor Memory Requests#

Ray actors (converter, coordinator, dispatcher) each have configurable memory budgets:

None means no explicit memory cap — Ray's default scheduler decides placement. The former alias eng_ray_memory_limit_per_actor is deprecated in favor of eng_ray_converter_actor_memory_request 5.

CPU Allocation per Actor#

Replica Placement and Node Packing#

Two settings cap how many Ray Serve replicas of each actor type land on a single node, preventing one node from being saturated (added in PR #629):

• eng_ray_converter_max_replicas_per_node (default: None = no cap) 9
• eng_ray_coordinator_max_replicas_per_node (default: None = no cap) 10

Setting eng_ray_coordinator_max_replicas_per_node=1 spreads coordinators one-per-node; leaving converter cap unset allows Ray to densely pack converters.

Autoscaling Bounds#

The converter actor pool autoscales between eng_ray_min_actors (default: 1) and eng_ray_max_actors (default: 10), controlled by eng_ray_target_requests_per_replica (default: 1.0). Scale-up and scale-down delays are configured via eng_ray_upscale_delay_s (default: 30 s) and eng_ray_downscale_delay_s (default: 600 s).

OOM Protection#

Ray deployments include built-in OOM protection, enabled by default 11:

• eng_ray_enable_oom_protection (default: true) — activates memory pressure monitoring in the Ray orchestrator.
• eng_ray_memory_warning_threshold (default: 0.9) — triggers warnings at 90% memory utilization.

These values are passed directly to the jobkit RayOrchestratorConfig 12. The actual enforcement logic (e.g., whether tasks are deferred or actors restarted under pressure) lives in docling-jobkit.

In multi-tenant environments, per-tenant concurrency caps (eng_ray_max_concurrent_tasks, default: 5) and optional queue limits (eng_ray_max_queued_tasks) limit memory pressure from any single tenant.

Memory Monitoring Endpoints#

When enable_management_endpoints is set to true (DOCLING_SERVE_ENABLE_MANAGEMENT_ENDPOINTS, default: false), two diagnostic routes become available (added in PR #513):

GET /v1/memory/stats 13
Returns process RSS (via psutil) plus cgroup memory breakdown from /sys/fs/cgroup/memory.current and /sys/fs/cgroup/memory.stat:

{ "rss": ..., "anon": ..., "file": ..., "slab": ..., "cgroup_total": ... }

Values are in MB. cgroup_total is what Linux uses for OOM kill decisions.

GET /v1/memory/counts 14
Forces a Python GC cycle then returns GC generation counts/thresholds, total tracked object count, asyncio task counts, and a ranked list of the top 20 object types by instance count. Useful for diagnosing Python-side memory leaks.

Both endpoints return 403 Forbidden if management endpoints are disabled 15.

Container-Level Memory Optimization#

All official docling-serve container images preload mimalloc (v3.2.8, compiled from source) via LD_PRELOAD to reduce heap fragmentation and lower overall resident memory (PR #512). This applies system-wide to the service process and is transparent to application code.

Additionally, OMP_NUM_THREADS=4 is set by default in the container, limiting OpenMP parallelism and its associated memory overhead 16.

Key Source References#