GPU and Accelerator Support#

RAGFlow's inference pipeline for document understanding (OCR, layout recognition, table structure recognition) supports three execution modes: CPU (default), NVIDIA GPU via CUDA, and Huawei Ascend NPU. The primary abstraction is ONNX Runtime's pluggable execution provider system, with Ascend using a parallel native inference stack.

ONNX Runtime Execution Providers (CUDA / CPU)#

All vision models — text detection (det.onnx), text recognition (rec.onnx), and layout/table models — are loaded through load_model() in deepdoc/vision/ocr.py. On startup, this function:

1. Calls cuda_is_available(), which imports torch and checks torch.cuda.is_available() for the target device_id 1.
2. If CUDA is available, creates an ort.InferenceSession with CUDAExecutionProvider and configurable GPU options 2.
3. Otherwise, falls back to CPUExecutionProvider with CPU memory arena shrinkage enabled 3.

Key CUDA tuning env vars (all optional):

4

Multi-GPU Parallelism#

Setting PARALLEL_DEVICES=N (where N > 0) causes the OCR class to instantiate one TextDetector and TextRecognizer per GPU device (device IDs 0 through N−1), enabling concurrent document processing across multiple GPUs 5. Each detector/recognizer is bound to its assigned device_id.

Dependency Selection#

pyproject.toml auto-selects the right ONNX Runtime package:

• onnxruntime-gpu==1.23.2 on Linux/Windows x86_64 (includes CUDA EP)
• onnxruntime==1.23.2 on macOS or non-x86_64 platforms (CPU-only)

6

Huawei Ascend NPU Backend#

Ascend support is a parallel inference stack using Huawei's ais_bench library and compiled .om model files instead of ONNX. It applies to layout recognition and table structure recognition only (OCR remains ONNX-based).

Backend Selection#

Two env vars switch the backend at runtime:

In pdf_parser.py, setting LAYOUT_RECOGNIZER_TYPE=ascend instantiates AscendLayoutRecognizer instead of the ONNX-based LayoutRecognizer. In table_structure_recognizer.py, setting TABLE_STRUCTURE_RECOGNIZER_TYPE=ascend routes to _run_ascend_tsr().

AscendLayoutRecognizer#

Defined in deepdoc/vision/layout_recognizer.py:

• Loads a compiled .om model from rag/res/deepdoc/{domain}.om via ais_bench.infer.interface.InferSession.
• Device selection via ASCEND_LAYOUT_RECOGNIZER_DEVICE_ID (default 0).
• Supports the same layout label set as the ONNX backend (title, Text, Table, Figure, Equation, etc.).

Model Files#

Ascend requires Huawei-compiled .om files; standard ONNX .onnx files are not compatible. Models must be placed in rag/res/deepdoc/:

• Layout: layout.om, layout.table.om, etc. (domain-specific)
• Table structure: tsr.om

7 8

Docker Deployment#

GPU (NVIDIA) Container#

The ragflow-gpu Docker Compose profile passes all NVIDIA GPUs into the container via the NVIDIA runtime 9. The tei-gpu profile does the same for the optional Text Embeddings Inference sidecar 10.

CPU-Only / Deepdoc OSS Container#

The standalone deepdoc Docker service (deepdoc_oss image, profile deepdoc) runs ONNX inference only. The docker_stubs.py script generates lightweight stub packages at build time so that the vision modules load without pulling in torch, pdfplumber, or Ascend (ais_bench) dependencies. On CPU-only images, cuda_is_available() silently returns False and ONNX Runtime uses CPUExecutionProvider 11.

Key Source Files#