Picture Chunker Media Processing#

The Picture chunker (rag/app/picture.py) is RAGFlow's entry point for indexing standalone image and video files. Its public API is the chunk() function, which routes files into one of two processing paths based on file extension, then tokenizes the resulting text into a single document chunk.

File Routing: Images vs. Videos#

Detection is purely extension-based. A top-level constant VIDEO_EXTS lists the supported video formats:

.mp4, .mov, .avi, .flv, .mpeg, .mpg, .webm, .wmv, .3gp, .3gpp, .mkv

Everything else is treated as an image 1. Both paths set doc_type_kwd accordingly ("video" or "image") and tokenize results into a single doc dict returned as a one-element list.

Video Path#

For video files 2:

1. Sets doc_type_kwd = "video".
2. Resolves the tenant's default IMAGE2TEXT model via get_tenant_default_model_by_type and wraps it in an LLMBundle.
3. Calls cv_mdl.async_chat(..., video_bytes=binary, filename=filename, video_prompt=video_prompt) — an async call run with asyncio.run().
4. Tokenizes the returned summary text via tokenize(doc, ans, eng) and returns [doc].

An optional video_prompt can be passed through parser_config to steer the LLM's summarization 3.

Supported CV backends for video:
Only models that implement async_chat with a video_bytes parameter are compatible. Currently, GeminiCV (Google Gemini) and QWenCV (Alibaba Qwen/DashScope) are implemented in rag/llm/cv_model.py 4. Gemini handles videos ≤ 20 MB via inline blob; larger videos use the Google Files API. The video support was introduced by PR #10671.

Image Path#

For image files 5:

1. Opens the binary as a PIL Image, converts to RGB.
2. OCR stage: first attempts PaddleOCR if configured; falls back to the local DeepDoc OCR engine (deepdoc.vision.OCR) 6.
3. OCR-sufficient check: if the extracted text exceeds 32 tokens/characters, it skips the vision LLM entirely and tokenizes the OCR result directly 7.
4. Vision LLM fallback: if OCR text is sparse, calls cv_mdl.describe(img_binary.read()) on the tenant's IMAGE2TEXT model to generate a natural-language description, then concatenates it with the OCR text before tokenizing 8.
5. Wraps the final chunk list through attach_media_context to optionally prepend/append surrounding text tokens (controlled by image_context_size from parser_config).

PaddleOCR Path#

The _try_paddleocr_image() helper is invoked when parser_config["layout_recognize"] resolves to "PaddleOCR". It:

• Resolves the PaddleOCR model via get_first_provider_model_name or the env-based fallback ensure_paddleocr_from_env 9.
• Writes the image to a temp file and calls pdf_parser.parse_image(filepath=..., binary=...) 10.
• Returns the text string on success, or empty string on failure/missing model — the caller then falls back silently to the local OCR 11.

vision_llm_chunk() Utility#

vision_llm_chunk(binary, vision_model, prompt, callback) is a lower-level helper that converts a PIL image to markdown text via a VLM using vision_model.describe_with_prompt(). It:

• Skips images smaller than 11×11 pixels to avoid provider size-limit errors 12.
• Saves as JPEG, falling back to PNG on encode error 13.
• Cleans the output with clean_markdown_block before returning 14.

This function is not called by chunk() directly. It is imported by other document parsers:

• deepdoc/parser/pdf_parser.py — for images embedded in PDFs 15
• deepdoc/parser/figure_parser.py 16
• deepdoc/parser/mineru_parser.py 17

Processing Flow Summary#

chunk(filename, binary, ...)
    │
    ├─ filename ends in VIDEO_EXTS?
    │ └─ Yes → async_chat(video_bytes=...) → tokenize → [doc]
    │
    └─ No (image)
            ├─ PaddleOCR (if layout_recognize="PaddleOCR")
            ├─ DeepDoc local OCR (fallback)
            ├─ OCR text long enough? → tokenize → attach_media_context → [doc]
            └─ OCR text short → CV LLM describe() + OCR → tokenize → attach_media_context → [doc]

Key Configuration Parameters (parser_config)#