Zoom on Layer 1 (L1): Layout Detection & Routing in a Modular IDP Pipeline#

Problem Statement: Intelligent Routing for Mixed-Content Medical Documents#

Dental and medical forms present a unique document analysis challenge: a single page typically contains multiple content types superimposed on one another, each requiring fundamentally different processing techniques. A typical form may include:

• Printed (typeset) text: form labels, instructions, and pre-printed field names
• Handwritten annotations: patient data, clinician notes, and responses filled in by hand
• Stamps and seals: official rubber or ink stamps indicating approval or certification
• Tables: structured grids with rows and columns for organizing information
• Signatures: handwritten in designated boxes for authorization

This heterogeneity creates a fundamental dispatching problem: no single processing engine can correctly handle all these region types. Standard optical character recognition (OCR) works well on clean printed text but degrades heavily on handwriting. Handwritten text recognition (HTR) tools are tuned specifically for handwriting but are unnecessary—and wasteful—for clean print. Stamps and seals require specialized figure classification. Tables demand structure-aware parsing. Running the wrong tool on a region type yields poor results and wastes computational resources.

The Three-Layer Progressive Ambiguity Reduction Architecture#

To address this dispatching challenge, we adopt a strict progressive ambiguity reduction philosophy across three layers:

• L0: Physical denoising — Addresses scan quality issues through deskewing, binarization, and noise removal. The output is a clean per-page image ready for analysis.

• L1: Spatial layout detection — Identifies WHAT content exists WHERE on the page, classifies each region by type (printed vs. handwritten text, table, figure, etc.), and routes it to the appropriate specialized tool. This is the dispatching layer.

• L2: Specialized tools per region type — Each region is processed by the tool best suited to its content: OCR for printed text, HTR for handwriting, table parsers for grids, figure classifiers for stamps and seals.

This architecture ensures that ambiguity is reduced incrementally: L0 handles physical distortions, L1 performs spatial and semantic classification, and L2 applies targeted extraction. Each layer operates on the output of the previous one, and no layer attempts to solve problems outside its scope.

Desire to Use Docling's Egret Layout Model#

For the L1 spatial layout detection task, Docling's Egret layout model is an attractive choice. Egret is a state-of-the-art document layout detector based on the RT-DETR object detection architecture, trained on a large corpus of document images. It identifies and localizes structural elements on a page with high accuracy.

The Critical Gap: No "Handwritten" Label#

However, Egret's label taxonomy presents a significant limitation for mixed-content documents. The Egret and Heron models output exactly 17 structural labels 1:

• Caption
• Footnote
• Formula
• List-item
• Page-footer
• Page-header
• Picture
• Section-header
• Table
• Text
• Title
• Document Index
• Code
• Checkbox-Selected
• Checkbox-Unselected
• Form
• Key-Value Region

Notably absent from this taxonomy is any label for "Handwritten" content. The Text label covers ALL text blocks—both printed and handwritten—without distinction. This means that Egret cannot, on its own, distinguish between a printed paragraph and a handwritten annotation in the same spatial location. For a medical form where handwritten patient data is filled into printed fields, Egret will detect text regions but cannot classify them by writing modality.

The LayoutPredictor.predict() method returns bounding box dictionaries with keys label, confidence, l (left), t (top), r (right), and b (bottom) 2. Coordinates are in pixel space with origin at the top-left corner of the image. While this provides precise spatial localization and structural classification, it does not provide the printed-vs-handwritten distinction required for intelligent routing in L1.

This gap is the core motivation for the architectural choices that follow: we must augment Egret's structural detection with an additional classification step to enable true content-aware routing before specialized L2 tools are invoked.

Options Considered#

This section evaluates three architectural approaches for bridging the gap between Egret's structural label taxonomy (which lacks a "handwritten" class) and the requirement to route handwritten regions to specialized tools. Each option represents a different point in the Docling architecture where the handwriting classification decision can be made.

Option 1: Custom Layout Plugin (Rejected)#

This approach replaces Egret with a custom layout model that extends the standard taxonomy to include a "Handwritten" label alongside the existing 17 structural labels.

Docling's plugin system provides the necessary infrastructure. Custom layout models implement the BaseLayoutModel abstract interface 3, which requires two methods: get_options_type() to declare configuration schema and predict_layout(conv_res, pages) to produce layout predictions. Custom models register via setuptools entry points and load when allow_external_plugins=True is set in PipelineOptions 4.

The plugin factory uses a PluginManager to discover registered models. By default, only plugins whose module names start with "docling." are loaded; external plugins are filtered unless explicitly permitted 5. This security mechanism ensures untrusted code does not execute by default.

Limitations:

1. Single-model constraint: The factory pattern instantiates exactly one layout model per pipeline 6. No ensemble or parallel execution of multiple layout models is supported. Choosing the custom model means forfeiting Egret entirely—you cannot run both.

2. Training burden: Creating a model that matches Egret's accuracy on structural detection while adding handwriting classification requires substantial labeled data, compute resources, and ML engineering effort. The custom model must achieve parity with Egret's proven performance on the 17 existing labels.

3. Maintenance divergence: Maintaining a custom model separates your system from upstream Docling improvements. As Egret evolves with better structural detection, the custom model must be retrained to incorporate those advances.

Why rejected: The high cost of training, the single-model architectural constraint that eliminates Egret's proven accuracy, and the ongoing maintenance burden of keeping a custom model current with upstream improvements make this option impractical for most use cases.

Option 2: Enrichment Model — Post-Layout Classification (Viable Alternative)#

This approach keeps Egret as the layout model and adds a secondary classifier that runs as an enrichment step after the full Docling pipeline completes.

The enrichment phase executes after document assembly 7. Enrichment models follow the BaseItemAndImageEnrichmentModel pattern 8, which provides two key capabilities:

• prepare_element(conv_res, element): Crops region images from page images based on bounding boxes. For elements with provenance, it extracts the cropped region from the source page at the configured scale, optionally expanding the bounding box by an expansion_factor 9.
• __call__(doc, element_batch): Processes a batch of prepared elements (images + items) and returns enriched items.

The DocumentPictureClassifier demonstrates this pattern in production 10. It filters processable elements (PictureItem instances), batches their cropped images, runs inference via an image classification engine, and stores predictions in item.meta.classification 11.

A handwriting enrichment model would follow the same pattern: filter "Text" regions from the assembled document, batch their cropped images, classify as printed or handwritten, and attach an is_handwritten flag to each region's metadata. Routing logic would then execute outside the Docling pipeline, consuming the enriched document and dispatching regions to specialized tools based on classification results.

Pros:

• Stays fully within Docling's architecture using production-tested enrichment infrastructure
• Leverages Docling's coordinate management and image cropping utilities
• Clean separation of concerns: Egret handles structure, enrichment adds handwriting classification
• Reuses the proven enrichment batching and element preparation patterns

Cons:

• The FULL pipeline (layout → OCR → table parsing → assembly → enrichment) must complete before routing decisions can be made
• If a region is handwritten, Docling already spent time running OCR on it unnecessarily
• Dispatching to specialized HTR tools happens outside and after the full pipeline execution
• All regions pass through the OCR stage even if they will ultimately be rerouted to an HTR tool

When it's appropriate: This option suits scenarios where you want classification metadata attached to a complete DoclingDocument for downstream systems. It works well when integration with other systems expects enriched Docling documents, or when the overhead of running OCR on handwritten regions (even though results will be discarded) is acceptable relative to the convenience of staying within Docling's architecture.

Option 3: External Orchestrator with Egret Standalone (Chosen Solution)#

This approach extracts Egret's LayoutPredictor from docling-ibm-models and uses it as a standalone component outside the full Docling pipeline. Routing decisions happen immediately after layout detection, before invoking any OCR, table parsing, or pipeline overhead.

The LayoutPredictor is explicitly designed for standalone use. Its constructor requires only an artifact path, device selection, number of threads, confidence threshold, and optional class blacklist 12. It accepts PIL Images or numpy arrays as input and returns bounding boxes with labels and confidence scores:

from docling_ibm_models.layoutmodel.layout_predictor import LayoutPredictor
from PIL import Image

egret = LayoutPredictor(
    artifact_path="path/to/egret-xlarge",
    device="cuda",
    num_threads=4
)
page_image = Image.open("page_001.png")
regions = list(egret.predict(page_image))
# Returns: [{"label": "Text", "confidence": 0.95, "l": 50, "t": 100, "r": 400, "b": 200}, ...]

The predict() method yields dictionaries with keys "label", "confidence", "l" (left), "t" (top), "r" (right), and "b" (bottom). Coordinates are in pixel space with origin at the top-left corner 13. For batch processing, predict_batch() accepts a list of images and returns a list of prediction lists, providing better efficiency than sequential predict() calls 14.

The demo script confirms this standalone usage pattern 15: instantiate the predictor, load page images, call predict(), and process results—no pipeline or conversion infrastructure required.

With regions detected, an external handwriting classifier examines each "Text" region's cropped image and adds an is_handwritten flag. Routing logic then dispatches immediately:

• Printed text → Docling OCR pipeline or standard OCR
• Handwritten text → HTR tool
• Tables → Docling TableFormer or VLM-based table extraction
• Pictures → Stamp/signature detectors or visual classifiers

Pros:

• Routing happens immediately after detection, before any OCR or pipeline stages run
• No unnecessary OCR on handwritten regions—each region type routes to its appropriate tool from the start
• Full orchestration control: compose any external HTR, stamp detector, or table tool
• No Docling pipeline overhead for regions that don't need it
• Directly reuses Egret at its most fundamental abstraction level

Cons:

• You manage orchestration, coordinate tracking, and result assembly
• Less "batteries included" than the full Docling pipeline
• Responsibility for coordinate alignment and combining results from disparate tools falls on the orchestrator

Why chosen: This approach implements a true L1 dispatcher pattern where routing decisions happen before specialized processing, maximizing efficiency and flexibility. It uses Egret for what it does best (structural detection) without fighting its limitations (no handwriting labels), and avoids wasting computation on OCR for regions that will ultimately require HTR tools.

Trade-off Analysis#

The three architectural options differ fundamentally in where routing decisions occur within the processing flow, leading to distinct implications for performance, flexibility, and integration complexity.

Comparison Table#

Key Architectural Difference: Routing Timing#

The critical distinction lies in when the routing decision is made:

Options 1 and 2 make routing decisions at or after the end of the Docling pipeline. In Option 2, the enrichment model runs during the document enrichment phase 7, which occurs after layout detection, OCR, and table parsing have already completed. This means handwritten text regions pass through Docling's OCR before being flagged as handwritten—processing that should have been routed to HTR instead.

Option 3 makes the routing decision immediately after L1 structural detection. Only regions confirmed as printed text proceed to Docling's OCR pipeline. Handwritten regions bypass OCR entirely and route directly to HTR, never touching tools designed for printed text. This represents the true L1 dispatcher pattern where routing precedes specialized processing.

Performance Implications#

For documents containing mixed content, routing timing significantly impacts computational efficiency:

• Option 2 waste: A document with 30% handwritten content wastes approximately 30% of OCR compute on regions that should not have been processed by print-oriented OCR.
• Option 3 parallelization: After routing, specialized L2 tools can run concurrently—print OCR, HTR, and table parsing can all execute in parallel across different region types.
• GPU memory: Option 3 only loads LayoutPredictor at L1; specialized models (OCR, HTR, TableFormer) load on demand at L2 based on actual routing decisions.

Flexibility and Integration#

Option 1 faces a fundamental constraint: Docling's plugin system allows only one layout model to run at a time 4. The plugin system uses entry points for discovery 5, but no parallel or ensemble execution is supported. Training a custom layout model that includes handwritten labels requires substantial labeled data and computational resources.

Option 2 stays fully within Docling's architecture, following the pattern established by DocumentPictureClassifier 10 which inherits from BaseItemAndImageEnrichmentModel 8. The enrichment model receives cropped images and document items, classifies them, and enriches the metadata. Downstream consumers receive a unified DoclingDocument with consistent metadata fields.

Option 3 provides maximum flexibility to integrate any external tool without modifying Docling's codebase. This is particularly important in regulated environments (medical, legal) where tool selection may be mandated by compliance requirements. The orchestrator controls the entire flow, composing arbitrary tools at each routing decision.

When to Choose Each Option#

Choose Option 2 when:

• You need a complete, enriched DoclingDocument output with consistent metadata
• Downstream consumers expect a unified document format with printed/handwritten classification
• Pipeline simplicity is more important than routing efficiency
• The handwritten percentage is low and OCR overhead is acceptable

Choose Option 3 when:

• True pre-processing routing is required—dispatching before specialized tools run
• Different SLAs are needed for different region types (e.g., 2-second latency for printed, 10-second for handwritten)
• You are building a microservices or queue-based IDP architecture
• OCR cost or latency on handwritten regions is unacceptable
• Integration with external mandated tools is required

Option 1 is generally not recommended due to high training costs and the single-model constraint that prevents leveraging Egret's proven accuracy for structural detection while separately addressing the handwriting classification gap.

Chosen Solution: Detailed Architecture (Option 3)#

Option 3 uses an external orchestrator with Docling's Egret layout model deployed as a standalone component. This architecture implements a true L1 dispatcher that routes document regions to specialized tools immediately after detection, before any pipeline processing begins.

Visual Flow Diagram#

[Input Document (PDF / Scan)]
          ↓
  [L0: Preprocessing]
  · deskew · binarize · denoise
          ↓
    [Clean Page Images]
  (PIL Image per page)
          ↓
[L1: Egret LayoutPredictor]
  (docling-ibm-models standalone)
          ↓
  Regions with structural labels
  {label, confidence, l, t, r, b}
          ↓
[L1.5: Handwriting Classifier]
  (external, applied to "Text" regions)
          ↓
  Enriched regions:
  {label, is_handwritten, ...}
          ↓
  [L2 Routing Dispatcher]
     ┌────────────┬────────────┬────────────┬────────────┐
     ↓ ↓ ↓ ↓ ↓
[Printed Text] [Handwritten] [Tables] [Pictures] [Checkboxes]
[Docling OCR] [HTR Tool] [Table- [Figure [Checkbox
               [/ pipeline] Former] Classifier] Detector]
     └────────────┴────────────┴────────────┴────────────┘
          ↓
[L3: Assembly & Coordinate Integration]
          ↓
 [Final Structured Document]

Step 1 — L0 Preprocessing#

The first layer accepts input documents (PDF or scanned images) and prepares clean page images for layout detection:

• Apply deskew to correct rotation from scanning
• Apply binarization to convert to clean black/white
• Apply denoising to remove scan artifacts
• Output clean PIL Image per page at known DPI

For standalone L0 preprocessing, tools like OpenCV or Pillow handle basic operations. Docling itself offers some preprocessing capabilities, but this architecture separates preprocessing as an explicit layer. The key requirement is producing a clean PIL Image object per page that can be fed directly to the layout predictor.

Step 2 — L1 Structural Detection (Egret Standalone)#

The core of this architecture uses LayoutPredictor from docling-ibm-models as a standalone component, independent of Docling's full pipeline 16.

from docling_ibm_models.layoutmodel.layout_predictor import LayoutPredictor
from huggingface_hub import snapshot_download
from PIL import Image

# Download the model once
artifact_path = snapshot_download(repo_id="docling-project/docling-layout-egret-xlarge")

# Initialize the LayoutPredictor (standalone, no Docling pipeline)
egret = LayoutPredictor(
    artifact_path=artifact_path,
    device="cuda", # Use "cpu" or "mps" as appropriate
    num_threads=4, # Only relevant when device="cpu"
    base_threshold=0.3, # Confidence threshold for predictions
)

# Predict on a single page image
page_image = Image.open("page_001.png")
regions = list(egret.predict(page_image))

# Output format:
# [
# {"label": "Text", "confidence": 0.95, "l": 50, "t": 100, "r": 400, "b": 200},
# {"label": "Table", "confidence": 0.91, "l": 50, "t": 250, "r": 600, "b": 500},
# {"label": "Picture", "confidence": 0.88, "l": 450, "t": 50, "r": 620, "b": 190},
# ]

The predict() method accepts either PIL Image or numpy array input 17. It returns a generator of bounding box dictionaries with pixel-space coordinates where the origin (0, 0) is the top-left corner 18. Each prediction includes the structural label (one of the 17 standard labels from LayoutLabels), confidence score, and bounding box coordinates: l (left), t (top), r (right), b (bottom) 19.

The 17 structural labels defined in the LayoutLabels class include: Caption, Footnote, Formula, List-item, Page-footer, Page-header, Picture, Section-header, Table, Text, Title, Document Index, Code, Checkbox-Selected, Checkbox-Unselected, Form, and Key-Value Region 1.

For processing multiple pages efficiently, use predict_batch() instead of calling predict() repeatedly 14. The demo script shows the standalone usage pattern 20.

Step 3 — L1.5 Handwriting Classification#

Since Egret's structural labels do not distinguish printed from handwritten text, we add an external classifier to enrich "Text" regions with handwriting detection:

from PIL import Image
from typing import Any

def classify_regions(page_image: Image.Image, regions: list[dict], hwc_model: Any) -> list[dict]:
    """
    Enrich 'Text' regions with a handwriting classification label.
    hwc_model: any classifier with a .predict(image) -> {'label': str, 'confidence': float} interface
    """
    enriched = []
    for region in regions:
        if region["label"] == "Text":
            # Crop the region from the page image
            crop = page_image.crop((region["l"], region["t"], region["r"], region["b"]))
            result = hwc_model.predict(crop)
            region["is_handwritten"] = (result["label"] == "handwritten")
            region["hw_confidence"] = result["confidence"]
        else:
            region["is_handwritten"] = False
        enriched.append(region)
    return enriched

The handwriting classifier is external to Docling—any lightweight image classifier fine-tuned to distinguish printed versus handwritten text will work. Models like a fine-tuned Vision Transformer (ViT) or EfficientNet are common choices. This step operates only on "Text" regions, leaving other structural labels unchanged. Docling does not currently provide this component, so it must be supplied separately.

Step 4 — L2 Routing#

With enriched region metadata, the routing dispatcher directs each region to its specialized processing tool:

from enum import Enum

class RegionRoute(Enum):
    PRINTED_OCR = "printed_ocr"
    HANDWRITTEN_HTR = "handwritten_htr"
    TABLE_PARSER = "table_parser"
    FIGURE_CLASSIFIER = "figure_classifier"
    CHECKBOX_DETECTOR = "checkbox_detector"
    SKIP = "skip"

ROUTE_MAP = {
    "Text": None, # determined by is_handwritten flag
    "Table": RegionRoute.TABLE_PARSER,
    "Picture": RegionRoute.FIGURE_CLASSIFIER,
    "Checkbox-Selected": RegionRoute.CHECKBOX_DETECTOR,
    "Checkbox-Unselected": RegionRoute.CHECKBOX_DETECTOR,
    "Form": RegionRoute.PRINTED_OCR,
    "Key-Value Region": RegionRoute.PRINTED_OCR,
    "Title": RegionRoute.PRINTED_OCR,
    "Section-header": RegionRoute.PRINTED_OCR,
    "Caption": RegionRoute.PRINTED_OCR,
    "List-item": RegionRoute.PRINTED_OCR,
    "Footnote": RegionRoute.PRINTED_OCR,
    "Page-header": RegionRoute.PRINTED_OCR,
    "Page-footer": RegionRoute.PRINTED_OCR,
    "Formula": RegionRoute.SKIP, # or specialized formula parser
    "Code": RegionRoute.PRINTED_OCR,
    "Document Index": RegionRoute.PRINTED_OCR,
}

def dispatch_region(region: dict) -> RegionRoute:
    label = region["label"]
    if label == "Text":
        if region.get("is_handwritten", False):
            return RegionRoute.HANDWRITTEN_HTR
        else:
            return RegionRoute.PRINTED_OCR
    return ROUTE_MAP.get(label, RegionRoute.SKIP)

def route_all(regions: list[dict]) -> dict[RegionRoute, list[dict]]:
    routes: dict[RegionRoute, list[dict]] = {}
    for region in regions:
        route = dispatch_region(region)
        routes.setdefault(route, []).append(region)
    return routes

The routing logic is straightforward: "Text" regions route based on the is_handwritten flag, while other structural labels follow fixed mappings. Handwritten text goes to an HTR (Handwritten Text Recognition) tool. Printed text, titles, sections, and similar content route to standard OCR or Docling's pipeline. Tables route to TableFormer or Granite-Vision. Pictures route to figure classification (for detecting stamps, signatures, logos). Checkboxes route to a checkbox detector. Formulas can either be skipped or sent to a specialized formula parser.

This dispatcher implements true L1 routing: the decision happens immediately after detection, before any processing tools run. This avoids wasting computation on inappropriate tools (e.g., running OCR on handwritten text or applying HTR to printed text).

Step 5 — Assembly#

After each L2 tool processes its assigned regions, the assembly step merges results into a unified structured document:

def assemble_document(routed_results: dict, page_image_size: tuple) -> list[dict]:
    """
    Merge results from all L2 tools into a reading-order list of content blocks.
    """
    all_blocks = []
    for route, results in routed_results.items():
        all_blocks.extend(results)
    # Sort by top-to-bottom, left-to-right reading order
    all_blocks.sort(key=lambda b: (b["t"], b["l"]))
    return all_blocks

Each L2 tool returns text or structured data with coordinates in the same pixel-space coordinate system used by Egret. The assembly step:

1. Collects results from all specialized tools
2. Sorts assembled regions by reading order (top-to-bottom, left-to-right by default using the t and l coordinates)
3. Maps to the target output format (DoclingDocument, JSON, plain text, or other structured format)

Because all coordinates remain in pixel-space relative to the page image, coordinate integration is straightforward. More sophisticated reading order algorithms can be applied if needed, but simple top-left sorting works well for most document layouts.

Why This Is Compatible with Docling's Philosophy#

Using LayoutPredictor standalone (Option 3) is not a workaround or deviation from Docling's design—it is a legitimate, fully supported usage pattern explicitly enabled by Docling's architecture.

LayoutPredictor as a Standalone Component#

The LayoutPredictor class lives in docling-ibm-models 21, a separate package from the main docling pipeline orchestration library. This separation is intentional: docling-ibm-models is a pure models library containing reusable ML model implementations that can be invoked independently of any pipeline 22. The LayoutPredictor.__init__() method accepts only model-intrinsic parameters: artifact_path, device, num_threads, base_threshold, and blacklist_classes 12. It requires no ConversionResult, no pipeline state, and no Docling-specific data structures. Input is a simple PIL Image or numpy array; output is a Python generator yielding plain dictionaries with keys "label", "confidence", "l", "t", "r", "b" 13. This design makes LayoutPredictor a self-contained, reusable component.

The official demo script demo_layout_predictor.py 15 demonstrates this exact standalone usage pattern: it instantiates LayoutPredictor directly, passes PIL images from disk, and processes the output without ever constructing a Docling pipeline 23. This is not a hack—it is the documented, intended way to use the model as a standalone component.

Architectural Separation: Models vs. Orchestration#

Docling's architecture deliberately separates model implementations from pipeline orchestration. The docling-ibm-models package contains pure model code (layout detection, table structure models, etc.), while the docling package provides the DocumentConverter and pipeline machinery that orchestrates these models. The BaseLayoutModel interface in the main docling package 3 is an adapter that wraps standalone models for use in the pipeline—it is not the only way to use those models. By using LayoutPredictor directly, Option 3 leverages Docling's structural detection capabilities at their most reusable abstraction layer, without requiring the full pipeline overhead.

Leveraging Strengths, Acknowledging Gaps#

Option 3 uses Docling for what it does best: state-of-the-art structural layout detection via the Egret model family, which outputs 17 precise labels for document elements like tables, section headers, figures, and text 1. These strengths are fully preserved in standalone usage. The absence of a "handwritten" label is a known model limitation, not a limitation of the standalone approach—this gap would exist regardless of whether LayoutPredictor is used inside or outside the pipeline. Option 3 addresses this gap externally with a lightweight handwriting classifier rather than forcing the pipeline to support multiple layout models simultaneously.

Not Fighting Constraints#

Docling's pipeline enforces a single-layout-model-at-a-time constraint 24, which simplifies orchestration and avoids conflicts between competing model outputs. Rather than attempting to bend the pipeline to support multiple models or custom ensemble logic, Option 3 steps outside the pipeline entirely when that constraint becomes limiting. The blacklist_classes parameter 25 allows filtering irrelevant labels at initialization time, demonstrating that the model is designed with flexible deployment scenarios in mind.

Modular Model Family#

The three Egret variants (medium, large, xlarge) 26 all share the same output interface—the same 17 labels 1 and the same dictionary structure 19. The LayoutModelConfig dataclass 27 and LayoutModelType enum 28 formalize these models as first-class configurations, not experimental alternatives. You can swap between model sizes in standalone usage without changing routing logic, demonstrating the modularity Docling's design enables.

Available Egret Models for L1 Detection#

Docling provides three Egret model variants for layout detection, each offering a different balance between speed and accuracy 26. All three variants output the same 17 structural labels and share a common API, making it trivial to switch between them during development or deployment 1.

Model Variants#

DOCLING_LAYOUT_EGRET_MEDIUM 29

• Repository: docling-project/docling-layout-egret-medium
• Profile: Balanced speed and accuracy
• Best for: High-throughput batch processing, development iteration, and CPU deployments where inference speed is critical

DOCLING_LAYOUT_EGRET_LARGE 30

• Repository: docling-project/docling-layout-egret-large
• Profile: Higher accuracy with moderate inference speed
• Best for: Production deployments with GPU resources and balanced workloads requiring good accuracy without sacrificing too much speed

DOCLING_LAYOUT_EGRET_XLARGE 31

• Repository: docling-project/docling-layout-egret-xlarge
• Profile: Highest accuracy at the cost of slower inference
• Best for: Quality-critical applications, small batch sizes, and scenarios where false negatives are expensive (e.g., medical document processing)

Shared Label Set#

All three Egret variants detect the same 17 structural categories 1:

Caption, Footnote, Formula, List-item, Page-footer, Page-header, Picture, Section-header, Table, Text, Title, Document Index, Code, Checkbox-Selected, Checkbox-Unselected, Form, Key-Value Region

Critical limitation: None of these labels distinguish handwritten from printed text. Any region classified as "Text" may contain either printed or handwritten content, requiring an external classifier to bridge this gap.

Usage Example#

Download and instantiate a model using huggingface_hub and LayoutPredictor 12:

from huggingface_hub import snapshot_download
from docling_ibm_models.layoutmodel.layout_predictor import LayoutPredictor

# Choose your variant
repo_id = "docling-project/docling-layout-egret-xlarge" # or -large or -medium
artifact_path = snapshot_download(repo_id=repo_id)

predictor = LayoutPredictor(
    artifact_path=artifact_path,
    device="cuda",
    base_threshold=0.3,
)

Selection Guidance#

Practical advice:

• Start development with Medium for fast iteration cycles
• Benchmark Large vs. XLarge on your specific document corpus before committing to production
• All three variants are API-compatible — swapping models requires only changing repo_id or artifact_path

Note: For use within the full Docling pipeline, specify the model variant via LayoutOptions(model_spec=DOCLING_LAYOUT_EGRET_XLARGE) 32.

Implementation Guidelines#

This section provides practical guidance for teams building the external orchestrator architecture with Egret as the standalone L1 dispatcher.

1. Environment Setup#

Install the required packages:

pip install docling-ibm-models huggingface-hub Pillow

For GPU acceleration, ensure CUDA is available and PyTorch is CUDA-enabled. Verify with:

import torch
print(torch.cuda.is_available()) # Should return True

Download the Egret model once and cache locally using snapshot_download() 33:

from huggingface_hub import snapshot_download
artifact_path = snapshot_download(repo_id="docling-project/docling-layout-egret-xlarge")

Store the artifact path for reuse across multiple documents.

2. Configuration and Tuning#

The LayoutPredictor accepts several tuning parameters 12:

base_threshold (default 0.3): Controls the confidence threshold for detections. Lower values increase recall (catch more regions) but may introduce false positives. Raise the threshold to reduce spurious detections. For medical forms with dense content, consider values between 0.25 and 0.35 34.

blacklist_classes: Filters out labels you don't need. For medical forms, you might blacklist "Document Index" or "Code" if those categories are not relevant 25.

device: Set to "cuda" for GPU acceleration or "cpu" for CPU inference 35.

num_threads: When using CPU, increase this value (e.g., 8 or more) on multi-core servers to improve throughput 36.

The LayoutPredictor uses a global threading lock during initialization to ensure thread-safe model loading 37.

3. Batch Processing for Performance#

The predict_batch() method is significantly more efficient than calling predict() in a loop for multi-page documents 14:

from docling_ibm_models.layoutmodel.layout_predictor import LayoutPredictor
from PIL import Image

# Initialize predictor once
egret = LayoutPredictor(
    artifact_path="path/to/egret-xlarge",
    device="cuda",
    num_threads=4
)

# Efficient batch processing for a multi-page document
page_images = [Image.open(f"page_{i:03d}.png") for i in range(num_pages)]
all_page_regions = egret.predict_batch(page_images)
# Returns List[List[dict]] — one list per page
for page_idx, page_regions in enumerate(all_page_regions):
    print(f"Page {page_idx}: {len(page_regions)} regions detected")

Batch processing amortizes model overhead across multiple images and is essential for production throughput.

4. Error Handling and Edge Cases#

Overlapping regions: Egret may produce overlapping bounding boxes. Implement Non-Maximum Suppression (NMS) post-processing if your application requires non-overlapping regions.

Empty pages: The predict() method returns an empty list for blank pages 17. Handle this gracefully to avoid downstream errors.

Very small regions: Filter out regions below a minimum area threshold (e.g., (r - l) * (b - t) < min_area) to eliminate noise.

Low-confidence regions: Consider a secondary threshold for "uncertain" regions (e.g., 0.3 < confidence < 0.5) that can be sent to a fallback tool or flagged for manual review.

Mixed-language documents: Egret is layout-agnostic; language-specific processing is handled by downstream L2 OCR or HTR tools.

5. Coordinate System#

All coordinates from LayoutPredictor are in pixel space relative to the input image, with origin at the top-left corner 18. Bounding boxes are returned as [left, top, right, bottom].

If you resize the image before passing it to predict(), scale coordinates back accordingly:

scale_x = original_width / resized_width
scale_y = original_height / resized_height
original_bbox = {
    "l": region["l"] * scale_x,
    "t": region["t"] * scale_y,
    "r": region["r"] * scale_x,
    "b": region["b"] * scale_y
}

For multi-DPI documents, normalize to a consistent DPI (e.g., 150 or 300 DPI) before processing to ensure stable layout detection.

6. Integration Testing Strategy#

Unit test: Verify that LayoutPredictor output matches the expected dictionary structure with keys "label", "confidence", "l", "t", "r", "b" 38.

Integration test: Test routing logic with synthetic regions by mocking predict() output to ensure correct dispatching to L2 tools.

Regression test: Maintain a golden dataset of medical forms with annotated region types. Track detection accuracy across model versions.

Threshold calibration: Run on a holdout validation set and plot precision/recall curves to find the optimal base_threshold for your use case.

7. Performance Considerations#

GPU vs CPU: GPU inference (device="cuda") is dramatically faster and essential for production workloads. CPU inference is viable for low-volume scenarios but expect 5-10× slower performance 35.

CPU threading: Use num_threads=8 or higher on multi-core servers when GPU is unavailable 36.

Memory: The XLarge model requires more GPU VRAM. Ensure sufficient memory or use the Medium or Large variants for constrained environments.

Parallelization: L2 tools for different region types can run concurrently—process printed text OCR and handwritten text HTR in parallel to reduce total latency.

8. Thread Safety#

The LayoutPredictor uses a global lock during initialization to prevent threading issues during model loading 37. After initialization, inference runs under @torch.inference_mode() and is safe for single-threaded use 39.

For multi-threaded servers, instantiate one LayoutPredictor per thread or use a pool pattern to avoid contention. Do not share a single predictor instance across threads without external synchronization.

Conclusion#

The challenge of routing mixed-content dental and medical documents — with their interplay of printed fields, handwritten annotations, stamps, tables, and signatures — is not solvable by any single document processing tool. It demands a deliberate architectural layer that sees the page as a whole, understands its spatial composition, and dispatches each region to the tool best suited for it.

Option 3 — an external orchestrator backed by Egret LayoutPredictor as a standalone component — is the right choice for a true L1 dispatcher. It answers the core design question decisively: routing decisions happen before specialized tools run, not after. This means:

• Handwritten regions are never fed to a print-optimized OCR engine.
• Table regions go directly to a table parser, skipping irrelevant steps.
• Stamp and signature regions reach a figure classifier without incurring OCR overhead.

The architectural layers work together cleanly:

This approach makes principled use of Docling's strengths — the Egret model family's high-accuracy structural detection — without asking it to do something it was not designed for (handwriting detection). The gap is bridged by a lightweight external classifier operating at the crop level, which is a tractable and well-studied problem.

Next Steps for Implementation#

1. Install dependencies: pip install docling-ibm-models huggingface-hub Pillow
2. Download your chosen Egret variant using snapshot_download("docling-project/docling-layout-egret-xlarge") 31
3. Prototype L1 using LayoutPredictor.predict() on a representative sample of your document corpus 2
4. Audit the region label distribution on your documents to understand the mix of structural types
5. Select or fine-tune a handwriting classifier on samples from your specific document domain
6. Build the routing dispatcher starting from the dispatch_region() pattern presented in Section 4
7. Integrate L2 tools incrementally — start with printed OCR, add HTR next, then table and figure handling
8. Benchmark and calibrate base_threshold on a held-out evaluation set to balance recall and precision for your use case

The modular nature of this pipeline means each layer can be developed, tested, and improved independently. Teams can iterate on the handwriting classifier without touching the routing logic, or swap the HTR tool without modifying L1. This is the hallmark of a well-designed IDP architecture.