Extraction Pipeline Prompting#

The VLM extraction pipeline supports two distinct prompting strategies, selected via the ExtractionPromptStyle enum (NUEXTRACT or GRANITE_VISION). The strategy is set in VlmExtractionPipelineOptions.extraction_prompt_style and flows through to TransformersExtractionModel, which dispatches to the appropriate input-builder at runtime 1.

NUEXTRACT: Schema-Injected Tokenizer Chat Template#

build_nuextract_inputs() constructs inputs by passing the JSON schema template directly into the tokenizer's chat template via processor.tokenizer.apply_chat_template(..., template=<schema>). Key details:

• The raw schema string is passed as the template argument — the schema is embedded inside the tokenizer's native chat template rather than being inserted as plain text in a user message.
• Vision content is handled by qwen-vl-utils (process_vision_info, fetch_image), which is a hard runtime dependency for this style. An explicit ImportError is raised if the package is absent 2.
• At model init, processor.tokenizer.padding_side is set to "left" and a GenerationConfig is loaded from disk — both specific to NUEXTRACT 3.
• When Python < 3.14, the model is compiled with torch.compile() at startup 4.
• The target model is NuExtract-2.0-2B (HuggingFace: numind/NuExtract-2.0-2B).

GRANITE_VISION: Fixed Natural-Language Instruction Prompt#

build_granite_vision_inputs() constructs a standard HuggingFace chat conversation where each message contains:

1. An {"type": "image"} content item.
2. A {"type": "text", "text": <extraction_prompt>} content item.

The extraction prompt is built by _build_extraction_prompt(template), which wraps the schema in a fixed natural-language instruction:

"Extract structured data from this document image. Return a JSON object matching this schema: … Return null for fields you cannot find. Return ONLY valid JSON, no other text."

processor.apply_chat_template() (top-level processor, not the tokenizer sub-object) is used to render the conversation — no qwen-vl-utils required. The target model is granite-vision-4.1-4b (HuggingFace: ibm-granite/granite-vision-4.1-4b, requires trust_remote_code=True).

Dispatch and Configuration#

The style must match the chosen model — mixing them produces incorrect output 5.

Key Files#