Document Token and Heading Schema#

This article covers two closely related concerns in docling-core:

1. Special token registration — how DocTags and DocLang token vocabularies are defined and validated.
2. Heading item schemas — the SectionHeaderItem and FieldHeadingItem types, their shared LevelNumber constraint, and the level-numbering inconsistency between the DocTags token registry and the data model.

Token Vocabularies#

DocTags Tokens (DocumentToken)#

DocumentToken (defined in docling_core/types/doc/tokens.py) is a str, Enum that enumerates all structural tags used in the DocTags XML-like format produced by vision models such as SmolDocling. It covers semantic tags (text, title, caption, picture, table, …), OTSL table structure, list and inline grouping, and special markers (page_break, otsl, smiles).

get_special_tokens() returns the complete LLM vocabulary: paired <tag> / </tag> for every enum member, then <section_header_level_N> / </section_header_level_N> for N in range(6) (i.e., 0–5), picture classification tokens, code language tokens, table OTSL tokens, and location tokens <loc_N> 1.

create_token_name_from_doc_item_label(label, level=1) maps a DocItemLabel to its string token name. For SECTION_HEADER it produces section_header_level_{level}, passing through the caller-supplied level directly 2.

DocLang Tokens (DocLangToken / DocLangVocabulary)#

The DocLang serialization format uses a richer vocabulary defined in _doclang_utils.py. Token metadata is centralized in DocLangVocabulary:

• ALLOWED_ATTRIBUTES — enumerates which DocLangAttributeKey values are valid on each token. Both DocLangToken.HEADING and DocLangToken.FIELD_HEADING accept only the level attribute 3.
• ALLOWED_ATTRIBUTE_RANGE — defines numeric bounds. HEADING and FIELD_HEADING both have level constrained to [1, 6] 4.
• IS_SELFCLOSING — tracks self-closing tokens such as <page_break/>, <location …/>, and all OTSL structural markers 5.

DocLangVocabulary.get_special_tokens() emits the full token list for LLM tokenizer registration. For HEADING and FIELD_HEADING it emits a bare <heading> (level 1, no attribute) plus <heading level="N"> for N in 2–6; closing tags are always bare </heading> 6.

_create_heading_token(level) / _create_field_heading_token(level) create opening tags and validate the level via _create_level_open_token, which raises ValueError for out-of-range values 7.

Heading Item Schemas#

SectionHeaderItem#

Defined in docling_core/types/doc/items/text.py:

class SectionHeaderItem(TextItem):
    label: Literal[DocItemLabel.SECTION_HEADER] = DocItemLabel.SECTION_HEADER
    level: LevelNumber = 1

Represents document section headings. Used by DoclingDocument.add_heading(text, level, ...) 8.

FieldHeadingItem#

Defined in docling_core/types/doc/items/form.py:

class FieldHeadingItem(TextItem):
    label: Literal[DocItemLabel.FIELD_HEADING] = DocItemLabel.FIELD_HEADING
    level: LevelNumber = 1

Used for headings inside form field regions (FieldRegionItem). In the DocLang serializer, context is detected via _has_field_region_ancestor() to decide whether to wrap output in <field_region> 9.

Shared LevelNumber Constraint#

Both classes use the same type alias 10:

LevelNumber = Annotated[int, Field(ge=1, le=100)]

Level range: 1–100, validated by Pydantic at construction time.

Level Numbering Inconsistencies#

There are two inconsistencies to be aware of:

Practical implication: A SectionHeaderItem with level=7 is valid at the Pydantic layer but will raise a ValueError during DocLang serialization. DocTags token registration includes section_header_level_0 (a level that no model actually outputs) due to the range(6) loop starting at 0, while the documented usage starts at level 1.

Key Files#