PDF Text Extraction Filtering#
During PDF text extraction, MinerU must distinguish valid body text characters from structural noise — primarily diagonal watermarks — before filling characters into text spans. The filtering logic lives in mineru/utils/span_pre_proc.py and was significantly enhanced in PR #5216 (merged 2026-07-03).
Quick orientation: the full filter runs in
_get_chars_for_span_fill(); the rotation check itself is_is_supported_rotation().
The Core Problem#
PDFs frequently contain two superficially similar phenomena that must be handled differently:
| Phenomenon | Rotation | Line context | Correct treatment |
|---|---|---|---|
| Diagonal watermark | Non-standard (e.g. 45°) | Entire line is rotated | Exclude from span fill |
| Pseudo-italic / emphasis | Non-standard | Individual chars inside a standard-direction line | Include in span fill |
Prior to PR #5216, the pipeline filtered characters with a single check — _is_supported_rotation(char['rotation']) — dropping all non-standard-rotation characters. This correctly removed whole-line watermarks but also discarded locally-rotated emphasis characters within body text.
Filtering Pipeline#
The entry point is _get_chars_for_span_fill(page_chars), called from txt_spans_extract() . It replaces the previous one-liner filter and runs the following stages:
-
Base allowlist — All characters with a supported rotation (0°, 90°, 180°, 270°, within ±0.1°) are added to a set keyed by
_get_char_fill_key. -
Early exit — If no non-standard-rotation characters exist on the page, the base allowlist is returned immediately (fast path for clean documents).
-
Line-context analysis — For each line returned by
get_lines_from_chars():- Lines whose own rotation is non-standard are skipped entirely — these are whole-line diagonal watermarks.
- Lines that contain no visible standard-rotation character (checked by
_is_visible_standard_rotation_char) are also skipped — this guards against lines consisting only of whitespace or control characters. - Any locally-rotated character inside a qualifying line is added to the allowlist, preserving pseudo-italic emphasis.
-
Final list — Characters are returned in original order, filtered to only those whose key is in the allowlist.
Supporting Helpers#
| Function | Purpose | Source |
|---|---|---|
_is_supported_rotation(rotation) | Returns True for 0°/90°/180°/270° ±0.1° | |
_get_char_fill_key(char) | Stable identity key; prefers char_idx, falls back to id(char) for older pdftext | |
_iter_line_chars(line) | Flattens line → spans → chars, tolerating missing chars fields | |
_is_visible_standard_rotation_char(char) | Guards against whitespace, \r/\n, missing bbox, or non-positive bbox dimensions |
Where Filtered Characters Are Used#
After _get_chars_for_span_fill returns the allowlisted characters, fill_char_in_spans() spatially maps each character to its containing span using a grid index. Spans that end up empty or contain only private-use Unicode characters are flagged for OCR fallback.
Related Files#
mineru/utils/pdf_text_tool.py— Upstream character extraction (get_page_chars) and deduplication (_deduplicate_near_identical_chars), which feeds data intospan_pre_proc.py.mineru/utils/char_utils.py— Character normalization (ligature handling, full-/half-width conversion) applied after span filling.