Tokenizer Backends#

Tokenizer backends provide the token-counting mechanism used by HybridChunker to enforce per-chunk token budgets. All backends implement the BaseTokenizer abstract interface, a Pydantic BaseModel + ABC that requires three methods:

Key invariant: the tokenizer passed to HybridChunker must match the embedding model's tokenizer — token counts are only meaningful relative to that model's context window. 1

HuggingFaceTokenizer#

File: docling_core/transforms/chunker/tokenizer/huggingface.py
Install: pip install 'docling-core[chunking]' (pulls in transformers + huggingface-hub) 2

Wraps any PreTrainedTokenizerBase from 🤗 Transformers. 3

• Token counting calls tokenizer.tokenize(text=text) and returns len(...).
• max_tokens is optional at construction. When omitted, the validator attempts to auto-detect it by downloading sentence_bert_config.json from the HuggingFace Hub for the model and reading max_seq_length. If that file is absent, construction raises RuntimeError. 4
• Convenience constructor HuggingFaceTokenizer.from_pretrained(model_name, max_tokens=None) wraps AutoTokenizer.from_pretrained and accepts an optional explicit max_tokens.
• The module exposes get_default_tokenizer(), which returns a HuggingFaceTokenizer backed by sentence-transformers/all-MiniLM-L6-v2 — this is also the default used by HybridChunker when no tokenizer is supplied. 5

⚠️ HuggingFace Transformers may emit "Token indices sequence length is longer than the specified maximum sequence length" during HybridChunker initialization. This is a false alarm and can be safely ignored. 1

OpenAITokenizer#

File: docling_core/transforms/chunker/tokenizer/openai.py
Install: pip install 'docling-core[chunking-openai]' (pulls in tiktoken) 6

Wraps a tiktoken.Encoding object. 7

• Both tokenizer (a tiktoken.Encoding) and max_tokens (an int) are required at construction — there is no auto-detection. 8
• Token counting calls tokenizer.encode(text=text) and returns len(...).
• A model_validator at construction time checks that tiktoken is importable and raises ImportError with the install hint if not. 9

Typical usage:

import tiktoken
from docling_core.transforms.chunker.tokenizer.openai import OpenAITokenizer

tokenizer = OpenAITokenizer(
    tokenizer=tiktoken.encoding_for_model("gpt-4o"),
    max_tokens=128 * 1024,
)

10

Comparison at a Glance#

No official benchmarks comparing the two implementations have been published.

Adding a Custom Backend#

Subclass BaseTokenizer (a Pydantic BaseModel) and implement the three abstract methods. Pass the instance directly to HybridChunker(tokenizer=...).