LLM Document Processing#

ScrapeGraphAI handles documents that exceed a model's context window through a four-stage pipeline: token budget resolution, token-budget-aware chunking in ParseNode, parallel per-chunk inference via LangChain's RunnableParallel in GenerateAnswerNode, and merge-step synthesis where a final LLM call consolidates chunk results into a single coherent answer.

This pattern is used by all scraping graphs (e.g., SmartScraperGraph) and sidesteps truncation errors without requiring retrieval-augmented generation (RAG) for cases where the full document must be scanned.

Stage 1: Token Budget — models_tokens + AbstractGraph#

Every graph initializes an LLM via AbstractGraph._create_llm(). During initialization, the model's maximum token count (model_token) is resolved from the models_tokens registry — a nested dict keyed by provider and model name (e.g., openai/gpt-4o → 128000) — and stored as self.model_token 1. If a model is not in the registry, it defaults to 8192 tokens 2. This value is then passed directly to ParseNode as chunk_size 3.

Stage 2: Chunking — ParseNode + split_text_into_chunks#

ParseNode.execute() splits the fetched document into token-bounded chunks. The effective chunk size is reduced to leave room for prompt overhead:

• HTML mode (default): chunk_size - 250 4
• Non-HTML mode: min(chunk_size - 500, chunk_size * 0.8) 5

Chunking is delegated to split_text_into_chunks(), which defaults to the semchunk library for semantically coherent splits. It applies a further 10% reduction (chunk_size * 0.9) before calling semchunk.chunk() 6. Token counting is standardized across all LLM providers using tiktoken with the gpt-4o encoding, regardless of the actual model in use 7.

The resulting List[str] chunks are written to state["parsed_doc"] 8.

Stage 3: Parallel Inference — GenerateAnswerNode#

GenerateAnswerNode.execute() branches on chunk count:

• Single chunk (len(doc) == 1): Uses TEMPLATE_NO_CHUNKS / TEMPLATE_NO_CHUNKS_MD — a single prompt with the full content and user question 9.
• Multiple chunks: Builds a chains_dict where each key is chunk1, chunk2, … and each value is a PromptTemplate | llm chain using TEMPLATE_CHUNKS / TEMPLATE_CHUNKS_MD — the chunk's content is embedded as a partial variable, so the only runtime input is {question} 10. All chains are wrapped in a LangChain RunnableParallel, which dispatches all chunk LLM calls concurrently and returns a dict of results 11.

Stage 4: Merge#

After parallel chunk processing, batch_results (the dict of per-chunk answers) is fed into a final merge chain using TEMPLATE_MERGE / TEMPLATE_MERGE_MD 12. The merge prompt explicitly instructs the LLM to:

• Combine all chunk answers into one response without repetitions 13
• Respect any maximum item count from the user's instructions 14
• Return valid JSON conforming to the output schema 15

All chains (chunk and merge) run with a configurable timeout, defaulting to 480 seconds 16. Timeout and JSON parse errors return structured error dicts rather than raising 17.

Key Files#