Pipeline Batch Size Configuration#

The StandardPdfPipeline (threaded mode) processes document pages through five concurrent stages — preprocess → ocr → layout → table → assemble — connected by bounded queues. For the three computationally intensive middle stages (OCR, layout, table), pages are collected into batches before calling the underlying model. Batch size controls the classic memory-vs-throughput trade-off: larger batches saturate the GPU/CPU more efficiently but hold more page images in memory simultaneously.

Batch-size parameters live on PdfPipelineOptions (and its subclass ThreadedPdfPipelineOptions). These defaults apply whenever the pipeline is run directly via DocumentConverter:

The preprocess and assemble stages are fixed at batch_size=1 regardless of these settings 1.

How the Pipeline Reads These Values#

StandardPdfPipeline._create_run_ctx() reads the options and passes them directly when constructing each ThreadedPipelineStage:

• ocr_batch_size → ocr stage 2
• layout_batch_size → layout stage 3
• table_batch_size → table stage 4
• batch_polling_interval_seconds → all stages as batch_timeout
• queue_max_size → all inter-stage ThreadedQueue instances

docling-serve Overrides#

When running via docling-serve, operator-level overrides are set as environment variables. The relevant fields on DoclingServeSettings all default to None:

DOCLING_SERVE_OCR_BATCH_SIZE
DOCLING_SERVE_LAYOUT_BATCH_SIZE
DOCLING_SERVE_TABLE_BATCH_SIZE
DOCLING_SERVE_QUEUE_MAX_SIZE
DOCLING_SERVE_BATCH_POLLING_INTERVAL_SECONDS

These are wired into DoclingConverterManagerConfig in orchestrator_factory.py, and into the RQ worker path in __main__.py 5. Inside DoclingConverterManagerConfig, the method _parse_standard_pdf_opts() conditionally writes each non-None value onto PdfPipelineOptions — the docling defaults remain untouched when the serve-level setting is None 6.

This was introduced in docling-serve PR #428, which replaced the older DOCLING_NUM_THREADS variable with the stage-granular batch-size settings.

Memory vs. Throughput Trade-off#

• Increase batch sizes to improve GPU utilization on multi-page documents (more pages processed per model call).
• Decrease batch sizes (down to 1) when memory is constrained or documents are mostly single-page.
• Reduce batch_polling_interval_seconds to decrease per-document latency at the cost of less efficient batching.
• Reduce queue_max_size to bound peak memory; stages will apply back-pressure rather than buffering unbounded pages.

Higher batch sizes do not affect the preprocess or assemble stages, which are always processed one page at a time 7.

Key Source References#