OCR Backend and Model Loading#

RAGFlow's document understanding pipeline (deepdoc/vision/) runs three categories of vision models — text detection (det.onnx), text recognition (rec.onnx), and layout/table-structure recognition (layout*.onnx, tsr.onnx) — across three hardware backends: CPU, NVIDIA GPU (CUDA), and Huawei Ascend NPU. The primary abstraction for CPU/GPU is ONNX Runtime; Ascend uses a parallel native stack via ais_bench.

Centralized Model Loading and Caching#

All ONNX models — including layout and table-structure recognizers — are loaded through the single load_model(model_dir, nm, device_id) function in deepdoc/vision/ocr.py.

Caching: A module-level loaded_models dict 1 stores every loaded (InferenceSession, RunOptions) pair. The cache key is the model file path plus device_id 2, so the same model on two different GPUs is stored independently. On cache hit, the function returns immediately without reloading.

CUDA/CPU provider selection: load_model calls an inner cuda_is_available() 3 that imports torch and checks torch.cuda.is_available() for the target device ID. On success, an ort.InferenceSession is created with CUDAExecutionProvider; otherwise it falls back to CPUExecutionProvider with CPU memory arena shrinkage enabled 4.

ONNX session options are shared across both providers 5:

• execution_mode = ORT_SEQUENTIAL
• enable_cpu_mem_arena = False
• Thread counts from OCR_INTRA_OP_NUM_THREADS / OCR_INTER_OP_NUM_THREADS (default 2)

The Recognizer base class (deepdoc/vision/recognizer.py) delegates directly to load_model and stores the returned session as self.ort_sess. Both LayoutRecognizer 6 and TableStructureRecognizer 7 extend Recognizer and inherit this path. All ONNX model files reside in rag/res/deepdoc/ and are auto-downloaded from InfiniFlow/deepdoc on HuggingFace if absent.

CUDA Environment Variables#

8

These variables apply to all ONNX-based models: det, rec, layout, and tsr.

Multi-GPU (OCR text pipeline)#

The OCR class 9 checks settings.PARALLEL_DEVICES (set dynamically to torch.cuda.device_count() in common/settings.py 10). When PARALLEL_DEVICES > 0, it instantiates one TextDetector and TextRecognizer per GPU device ID (0 through N−1), enabling concurrent per-document processing. Layout and table recognizers do not currently use multi-GPU sharding via this mechanism.

Huawei Ascend NPU Backend#

The Ascend path is a parallel stack that bypasses ONNX Runtime entirely. It applies only to layout and table-structure recognition — OCR text detection and recognition remain ONNX-based.

Backend selection is controlled by two env vars:

In deepdoc/parser/pdf_parser.py 11, setting LAYOUT_RECOGNIZER_TYPE=ascend instantiates AscendLayoutRecognizer instead of LayoutRecognizer. For TSR, TABLE_STRUCTURE_RECOGNIZER_TYPE=ascend routes to _run_ascend_tsr().

AscendLayoutRecognizer 12:

• Loads a compiled .om file from rag/res/deepdoc/{domain}.om via ais_bench.infer.interface.InferSession
• Device selected by ASCEND_LAYOUT_RECOGNIZER_DEVICE_ID (default 0)
• Does not use the global loaded_models cache; session is created per instance

_run_ascend_tsr 13:

• Loads rag/res/deepdoc/tsr.om and creates an InferSession per invocation
• Uses the same ASCEND_LAYOUT_RECOGNIZER_DEVICE_ID env var

Ascend requires Huawei-compiled .om files; standard .onnx files are incompatible.

Key Source Files#