Remote Inference Services#

Docling supports offloading model inference to remote servers via two distinct protocols and backend types:

1. KServe v2-compatible servers — for OCR, layout detection (object detection), and image classification tasks. Communicates via HTTP REST or gRPC using the KServe Predict Protocol v2. Typical backend: NVIDIA Triton Inference Server.
2. OpenAI-compatible VLM endpoints — for vision-language model tasks (code/formula extraction, VLM-based document understanding). Communicates over HTTP using the OpenAI Chat Completions API. Compatible with Ollama, LM Studio, vLLM, and OpenAI.

All remote engines are opt-in: you must set pipeline_options.enable_remote_services = True or the engine constructor raises OperationNotAllowed 1 2.

KServe v2 Engines (OCR, Object Detection, Image Classification)#

Transports#

Both the HTTP and gRPC transports implement the same KserveV2Client interface — infer() accepts numpy arrays and returns numpy arrays — so engine code is transport-agnostic.

gRPC is the default transport 3. The gRPC client requires the remote-serving extras (pip install 'docling[remote-serving]'); the HTTP client only requires requests 4.

Both transports support binary tensor payloads (use_binary_data=True by default), which is more efficient than JSON encoding for large pixel arrays 3. Set use_binary_data=False for servers that do not support this extension.

The grpc_max_message_bytes default is 64 MB 3. For dns:///host:port URLs, the gRPC client auto-injects round_robin load-balancing unless a policy is already specified 5.

Engine classes#

On initialize(), each engine:

1. Downloads the HuggingFace preprocessor and label config locally (no model weights needed).
2. Opens the transport client.
3. Calls get_model_metadata() to discover input/output tensor names from the server 6 7.

Factory pattern#

The object-detection and image-classification engines are constructed via factory functions:

• create_object_detection_engine() — selects ApiKserveV2ObjectDetectionEngine when the engine type is API_KSERVE_V2.
• create_image_classification_engine() — selects ApiKserveV2ImageClassificationEngine when the engine type is API_KSERVE_V2.

Both factories receive and forward enable_remote_services to the engine constructors. The resolve_kserve_transport_base_url() utility in kserve_transport_utils.py normalizes the URL before the client is opened (prepending http:// for HTTP URLs lacking a scheme).

Configuration options (shared across all KServe v2 engines)#

3

OCR-specific options#

KserveV2OcrOptions (set via pipeline_options.ocr_options) adds:

• lang (default ["english", "chinese"]) — passed to the server
• scale (default 2.0) — image scale multiplier (e.g., 72 DPI → 144 DPI)
• model_name (default "ocr")

3

Example:

from docling.datamodel.pipeline_options import PdfPipelineOptions, KserveV2OcrOptions

pipeline_options = PdfPipelineOptions()
pipeline_options.enable_remote_services = True
pipeline_options.do_ocr = True
pipeline_options.ocr_options = KserveV2OcrOptions(
    url="localhost:8001", # gRPC endpoint
    model_name="ocr",
    transport="grpc",
    lang=["english"],
    scale=2.0,
)

3

OpenAI-Compatible VLM Endpoints#

For VLM tasks (SmolDocling-style pipeline, code/formula enrichment), use ApiVlmEngineOptions:

from docling.datamodel.vlm_engine_options import ApiVlmEngineOptions
from docling.datamodel.pipeline_options import VlmPipelineOptions, VlmConvertOptions

options = VlmPipelineOptions(
    vlm_options=VlmConvertOptions.from_preset(
        "smoldocling",
        engine_options=ApiVlmEngineOptions(url="http://localhost:11434"),
    )
)

8

Key ApiVlmEngineOptions fields 9:

The API VLM engine is constructed via create_vlm_engine() in docling/models/inference_engines/vlm/factory.py. It is one of several VlmEngineType variants; see VLM Inference Engine for the full local/remote backend matrix.

Key Source Files#