Parser-Chunk Contract#

The Parser-Chunk Contract is the inter-component data handoff between the Parser pipeline stage and downstream consumers (chunkers and the Compiler). It is enforced by two Pydantic models and the output_format routing signal.

Pipeline Position#

The standard ingestion chain in rag/flow/ is:

File → Parser → TokenChunker / TitleChunker → Compiler → Indexer

The Parser writes its result into a set of named output slots. The next stage reads those slots via the TokenChunkerFromUpstream schema. A mismatch between what the Parser writes and what output_format declares causes silent zero-chunk failures — the chunker reads the wrong (empty) field and returns {"chunks": []} with no error. 1

The output_format Signal#

Every parser method calls self.set_output("output_format", ...) first, then writes exactly one of the four format-specific output slots 2:

The _image() method unconditionally hardcodes output_format = "json" regardless of DSL config 3 — a deliberate fix for a prior bug where format mismatch produced zero chunks 1.

Schema Definitions#

Parser output → Chunker input: TokenChunkerFromUpstream#

Defined in rag/flow/chunker/schema.py 4:

The model uses populate_by_name=True, extra="forbid" 5, so upstreams can write json, markdown, etc. as keys directly, and unknown keys raise an error.

Parser upstream input: ParserFromUpstream#

Defined in rag/flow/parser/schema.py 6. Contains only name (required), file (optional dict), and two boolean flags abstract and author. It does not carry output_format — that is set by the Parser during processing.

How Chunkers Consume the Contract#

TokenChunker._invoke() pattern 7:

1. String formats (markdown, text, html): reads getattr(from_upstream, f"{output_format}_result") and processes the string payload.
2. json format: reads from_upstream.json_result as a list[dict]. Each dict carries a doc_type_kwd field ("text", "table", "image") that determines how the chunk is handled 8.
3. chunks format: reads from_upstream.chunks directly — used by the Compiler stage 9.

The json format: block dict structure#

When output_format="json", parsers emit a list[dict]. The canonical fields on each dict are:

PDF parsers normalize all blocks through a shared bbox-normalization loop before writing output 10. After normalization, normalize_pdf_items_metadata() is called before the json slot is written 11.

Parser-Specific Section Tuple (pre-normalization)#

The low-level parsers (MinerU, DeepDoc, Docling, etc.) work in a distinct intermediate representation before the Parser flow component converts it into the contract above. They return section tuples from parse_pdf():

• pipeline/manual parse_method: (text: str, layout_type: str, position_tag: str) — a 3-tuple 12
• paper parse_method: (text_with_tag: str, layout_type: str) — 2-tuple 13

position_tag is a string of the form @@{page}-{page}\t{x0}\t{x1}\t{top}\t{bottom}## 14. The Parser._pdf() method iterates these tuples, builds bbox dicts, and converts them into the final contract format 15.

MinerU maps its native content_list.json content types (text, table, image, equation, code, list) to section tuples via _transfer_to_sections(), discarding header, footer, page_number, and discarded blocks.

Allowed output_format Values per File Type#

Enforced by ParserParam.check() 16:

Key Files#