Document Figure Classification#

Overview#

DocumentPictureClassifier is the enrichment model responsible for classifying every PictureItem in a DoclingDocument into a fine-grained visual category. It is distinct from layout detection (which labels regions as PICTURE vs. TABLE, etc.) and from VLM-based description generation. Enabling it sets PictureItem.meta.classification on each processed picture 1.

Classification is triggered by setting do_picture_classification=True in PdfPipelineOptions (or ConvertPipelineOptions).

Two-Tier Storage Model#

Results are stored in two parallel locations:

PictureMeta (which holds classification) extends FloatingMeta and also holds molecule, tabular_chart, and code 2.

Engine Abstraction#

The classifier delegates inference to a pluggable BaseImageClassificationEngine 3. Three concrete implementations are registered by engine type 4:

Each engine must implement three abstract methods 5:

• initialize() — load model/allocate resources
• predict_batch(input_batch) → List[ImageClassificationEngineOutput]
• get_label_mapping() → Dict[int, str]

Engine selection and model coordinates are configured via DocumentPictureClassifierOptions, which wraps an ImageClassificationModelSpec defaulting to the IMAGE_CLASSIFICATION_DOCUMENT_FIGURE preset (pointing to docling-project/DocumentFigureClassifier-v2.5).

DocumentPictureClassifier Pipeline Integration#

The model class is DocumentPictureClassifier, which:

1. Accepts only PictureItem nodes (is_processable) 6.
2. Renders images at 2× scale (images_scale = 2) 7.
3. Converts images to RGB and batches them into ImageClassificationEngineInput objects 8.
4. Calls engine.predict_batch(), maps label IDs → names via get_label_mapping() 9.
5. Writes PictureClassificationMetaField into item.meta.classification 10.

The 26-Class Taxonomy (DocumentFigureClassifier-v2.5)#

The model is an EfficientNet-B0 fine-tuned on a subset of HuggingFace/finepdfs, trained on a dataset 10× larger than v2.0 11. It accepts 224×224 RGB images and achieves 90.7% accuracy and 0.689 balanced accuracy on a held-out test set 11.

The 26 active labels, grouped by category, are defined in PictureClassificationLabel:

Note: The model's ONNX label list 11 does not include other_chart; the enum also retains legacy labels (cad_drawing, heatmap, molecular_structure, etc.) for backward compatibility with older serialized documents.

Inference backends are available via both Transformers and ONNX Runtime as documented on the model card.

Key Source Files#