Parser Output Lifecycle#

The ingestion pipeline distinguishes three states for parser output: missing (parser returned None or a falsey value), present-but-empty (parser returned an empty list [] or a dict with an empty payload key), and non-empty (valid chunks for indexing). Getting this distinction wrong causes either silent zero-chunk failures (empty treated as non-empty → no-op downstream) or false error states (empty treated as missing → spurious error progress). PR #14220 "Fix: accept empty value as 0 chunk" hardened the full pipeline to handle all three cases correctly.

Relevant files:

• rag/app/picture.py — picture parser producing empty output on vision model failure
• rag/svr/task_executor.py — run_dataflow and do_handle_task controlling progress/success determination
• rag/flow/tokenizer/schema.py / tokenizer.py — presence-vs-truthiness guards (changed in PR #14220)
• rag/flow/chunker/schema.py — TokenChunkerFromUpstream contract 1

The Three Output States#

How Each State Arises in picture.py#

The picture parser (rag/app/picture.py) produces all three states:

• Non-empty (OCR path): If OCR text exceeds 32 tokens, the parser tokenizes and returns attach_media_context([doc], ...) directly, bypassing the vision LLM entirely 2.
• Non-empty (vision LLM path): If OCR text is sparse, the vision model is called; on success the result is concatenated with OCR text and returned as [doc] 3.
• Empty (vision LLM failure): If the vision model raises (missing model, quota error, network), the parser calls callback(prog=-1, msg=str(e)) to mark the task as errored, then return [] 4. The final return [] at the end of the function covers any other code path that falls through without returning 5.
• PaddleOCR fallback: _try_paddleocr_image() returns an empty string on any failure 6; the caller falls back silently to the local DeepDoc OCR engine 7.

The key behavioral difference: a missing vision model causes the parser to emit prog=-1 (error) and return []. The empty list signals "nothing to index", but the task is already flagged as failed at the progress level.

Tokenizer Schema: Presence vs. Truthiness#

Before PR #14220, the tokenizer schema's _check_payloads method used truthiness checks (not self.chunks, not self.json_result, etc.), which treated an empty list [] and None identically — both as "missing". This caused silent failures when a parser legitimately produced zero chunks.

The fix 8 in rag/flow/tokenizer/schema.py:

• output_format == "chunks": validation now checks self.chunks is not None instead of bool(self.chunks). An empty [] is a valid payload.
• markdown / text / html / json: switched from not <field> to <field> is None. An empty string or empty list is considered present.

The downstream TokenChunkerFromUpstream schema in rag/flow/chunker/schema.py uses populate_by_name=True, extra="forbid" 9, so unknown keys still raise errors but missing optional fields are allowed. The output_format signal routes to the correct result field; a mismatch between what the parser writes and what output_format declares causes zero-chunk output with no error 10.

Task Executor: Presence Checks and Early Exits#

run_dataflow in task_executor.py has two distinct early-exit points that handle missing and empty outputs separately.

Exit 1: Missing output (pipeline returned falsey)#

if not chunks:
    PipelineOperationLogService.create(...)
    return

Lines 797–802: if pipeline.run() returns None or any falsey value, the executor logs a PARSE operation and returns immediately without setting any error progress. The document remains at whatever progress the pipeline left it.

Exit 2: Empty normalized output (present-but-empty key)#

After format normalization 11, a second check fires:

if not chunks:
    PipelineOperationLogService.create(...)
    return

Lines 829–832: even if the pipeline returned a dict with a key present (e.g., {"chunks": []}), normalization yields an empty list, and the executor exits cleanly before embedding or indexing.

Presence vs. Truthiness in Normalization#

The normalization block 11 uses presence checks, not truthiness:

if "chunks" in chunks: # NOT chunks.get("chunks")
    ...
elif "json" in chunks: # NOT chunks.get("json")
    ...

This is the core fix from PR #14220: "chunks" in chunks is True even when chunks["chunks"] == [], so the correct code path is taken and the empty list is preserved through normalization rather than falling through to an unrelated branch.

Standard (non-dataflow) path#

In do_handle_task 12, the standard chunking path treats empty output as success, not error:

if not chunks:
    progress_callback(1.0, msg=f"No chunk built from {task_document_name}")
    return

Empty output → progress = 1.0. This is intentional: a valid document with no extractable content (e.g., a blank image) should not be marked as failed.

Progress Reporting and Success/Failure Determination#

Progress values in set_progress 13 follow this convention:

Picture parser error path#

When the vision LLM is missing or fails, picture.py calls callback(prog=-1, msg=str(e)) 14 before returning []. This sets the task document to the error state before the empty list propagates upstream. The task executor's empty-output early exit then fires, but the document is already marked failed at the DB level.

Empty output as success#

For the standard chunking path (do_handle_task), an empty chunk list is reported as progress_callback(1.0, ...) 15. DocumentService.increment_chunk_num is not called (it's only called after successful insertion at line 1655), so the document's chunk count stays at 0 while progress reaches 1.0.

Dataflow path#

In run_dataflow, empty outputs (both exit points) do not set any progress explicitly. PipelineOperationLogService.create is called to record the parse attempt . On the success path, set_progress(task_id, prog=1.0, ...) fires at line 939 and DocumentService.increment_chunk_num is called at line 935 17.

Operation log#

PipelineOperationLogService.create is called on all run_dataflow outcomes — empty, errored, and successful — ensuring every parse attempt is traceable . For non-dataflow tasks, PipelineOperationLogService.record_pipeline_operation is called in the finally block of handle_task 18.