Achieving Optimal GPU Performance#

This guide describes how to maximize GPU performance for Docling pipelines, with dedicated sections for Linux and Windows. It covers device selection, differences between the standard and VLM pipelines, and provides example snippets for configuring batch size and concurrency in the VLM pipeline.

Linux#

Standard Pipeline#

For orchestrated environments (e.g., Kubernetes), enable GPU acceleration by setting the accelerator type and limit in your pipeline task:

converter.set_accelerator_type("nvidia.com/gpu")
converter.set_accelerator_limit("1")

You can further target specific GPU nodes using node selectors and tolerations:

from kfp import kubernetes

kubernetes.add_node_selector(
    task=converter,
    label_key="nvidia.com/gpu.product",
    label_value="NVIDIA-A10",
)

kubernetes.add_toleration(
    task=converter,
    key="gpu_compute",
    operator="Equal",
    value="true",
    effect="NoSchedule",
)

Reference

Direct device selection outside orchestrated environments is not explicitly exposed.

VLM Pipeline#

For best GPU utilization, use a local inference server. On Linux, vllm, LM Studio, and Ollama are supported. Example endpoints:

• vllm: http://localhost:8000/v1/chat/completions
• LM Studio: http://localhost:1234/v1/chat/completions
• Ollama: http://localhost:11434/v1/chat/completions

Configure the VLM pipeline to use a local server:

from docling_jobkit.datamodel.convert import ApiVlmOptions

vlm_options = ApiVlmOptions(
    url="http://localhost:8000/v1/chat/completions", # vllm
    params={
        "model": "ibm-granite/granite-docling-258M",
        "max_tokens": 4096,
    },
    concurrency=4, # Number of concurrent requests
    prompt="Convert this page to docling.",
    timeout=90,
)

Set batch size in your batch processing function:

def compute_batches(source, target, batch_size=20):
    # ...
    batch_keys = generate_batch_keys(filtered_source_keys, batch_size=batch_size)

In Kubernetes pipeline YAML, set concurrency with parallelismLimit:

parallelismLimit: 20

Reference

Windows#

Standard Pipeline#

GPU device selection is not explicitly exposed outside orchestrated environments. If running in Kubernetes, use the same configuration as Linux.

VLM Pipeline#

On Windows, LM Studio and Ollama are supported as local inference servers. Example endpoints:

• LM Studio: http://localhost:1234/v1/chat/completions
• Ollama: http://localhost:11434/v1/chat/completions

Configure the VLM pipeline:

vlm_options = ApiVlmOptions(
    url="http://localhost:1234/v1/chat/completions", # LM Studio
    params={
        "model": "granite3.2-vision:2b",
        "max_tokens": 4096,
    },
    concurrency=2,
    prompt="Convert this page to docling.",
    timeout=90,
)

Set batch size in your batch processing function:

def compute_batches(source, target, batch_size=10):
    # ...
    batch_keys = generate_batch_keys(filtered_source_keys, batch_size=batch_size)

In Kubernetes pipeline YAML, set concurrency with parallelismLimit:

parallelismLimit: 10

Reference

Pipeline Differences#

The standard pipeline relies on orchestrator-level GPU resource configuration. The VLM pipeline achieves best performance by offloading model inference to a local server, with explicit control over batch size and concurrency. VLM pipeline configuration is mutually exclusive between local and API models—only one can be set at a time.

Example: Full VLM Pipeline Configuration#

from docling_jobkit.datamodel.convert import ApiVlmOptions

vlm_options = ApiVlmOptions(
    url="http://localhost:1234/v1/chat/completions", # LM Studio or Ollama
    params={
        "model": "granite3.2-vision:2b",
        "max_tokens": 4096,
    },
    concurrency=2,
    prompt="Convert this page to docling.",
    timeout=90,
)

Additional Notes#

No project-specific caveats for GPU performance on Linux vs Windows were found beyond device and server support. Always ensure your local inference server is configured to use the GPU and that drivers are up to date.