Text Element Merging and Dehyphenation#

After spatial reading order is established, ReadingOrderModel detects and joins text elements that were split across layout boundaries — a common occurrence in typeset PDFs where words break across columns or pages. The merging pipeline handles both cross-element text joining and dehyphenation (stripping hyphens from split words), and also propagates hyperlinks across merged items.

Entry point: ReadingOrderModel.__call__() runs four sequential sub-steps: predict_reading_order, predict_to_captions, predict_to_footnotes, and predict_merges. The merge mapping from predict_merges is passed into _readingorder_elements_to_docling_doc(), where matched elements are consumed and their text is joined onto the primary item's text/orig fields and prov list.

Merge Detection: predict_merges#

predict_merges() (in docling-ibm-models) identifies which consecutive elements should be merged. Its logic:

1. Label filter — skips PAGE_HEADER, PAGE_FOOTER, TABLE, PICTURE, CAPTION, FOOTNOTE. Only TEXT elements are candidates by default; LIST_ITEM was added recently (see below).

2. Spatial requirement — the current element must be strictly left of the next, or on a different page.

3. Regex pattern matching — both patterns must match for a merge to occur:

   • m1: r".+([a-z,\-\u00AD])(\s*)" — the current element's text ends with a lowercase letter, comma, hard hyphen (-), or soft hyphen (U+00AD).
   • m2: r"(\s*[a-zA-Z\u00C0-\u024F])(.+)" — the next element's text starts with any ASCII letter or Latin Extended-A character (U+00C0–U+024F), supporting accented characters.

4. Chaining — merges propagate: if A→B qualifies and B→C qualifies, all three are merged into A.

PR #155 extended m1 to include soft hyphens (\u00AD) and m2 to accept uppercase and Unicode letters. Previously, only lowercase ASCII endings/starts were recognized, missing soft-hyphen splits and German nouns (or any word beginning with an uppercase letter after a column break).

Text Joining: _merge_elements#

_merge_elements() in readingorder_model.py applies the actual join. The joining strategy depends on how the primary element ends:

PR #3232 introduced the soft-hyphen stripping (new_item.text = new_item.text[:-1] + merged_elem.text). PR #3888 added the hard-hyphen path for lowercase continuations (e.g., algo- + rithms → algorithms), while hard hyphens before uppercase (e.g., algo- + Rithms) are preserved with a space.

Note: Intra-cluster dehyphenation (joining lines within a single text cluster) is a separate step handled by sanitize_text() in PageAssembleModel at the page assembly stage.

LIST_ITEM Merging#

predict_merges historically only considered DocItemLabel.TEXT, causing hyphenated words split across list-item boundaries to produce spurious extra list items (e.g., algo- becoming item 2, and rithms; becoming item 3) 1.

Two coordinated fixes addressed this:

• PR #171 in docling-ibm-models 2: extends the merge candidate condition to if elem.label in [DocItemLabel.TEXT, DocItemLabel.LIST_ITEM]. For LIST_ITEM, a simpler heuristic is used — text.rstrip().endswith(("-", "\u00ad")) and next_text.lstrip()[0].islower() — rather than the full regex, to avoid false positives.
• PR #3888 in docling 3: updates _merge_elements to strip hard hyphens from list-item text when the continuation is lowercase.

Hyperlink Propagation Across Merged Elements#

PR #3131 introduced end-to-end hyperlink tracking. TextElement.hyperlink is resolved during page assembly via spatial matching and then threaded through the reading order model in _handle_text_element() and _add_caption_or_footnote().

During merges, if the primary and merged elements carry different hyperlink targets, the hyperlink is dropped (new_item.hyperlink = None). This is a conservative policy: a cross-element merge spanning two distinct link targets is treated as ambiguous.

Key Source Files#