State-Based Pipeline Architecture#

ScrapeGraphAI pipelines are implemented as directed acyclic graphs (with optional conditional branches) where nodes communicate exclusively through a shared mutable state dictionary. No node holds a reference to another; they only read from and write to a single state: dict that is threaded through every execution step. This pattern is defined by two core classes:

• BaseNode — abstract base for every node; declares input expression and output key list
• BaseGraph — orchestrates execution order and state passing

How State Flows Through the Graph#

At execution time, a caller creates an initial_state dict and passes it to BaseGraph.execute(). The graph traverses nodes in topological order starting from the entry_point, calling each node's execute(state) method, which returns the updated state. The same dict is passed forward to every subsequent node 1.

In SmartScraperGraph, the initial state is seeded with two keys — user_prompt and either url or local_dir — then progressively enriched by each node 2:

{ "user_prompt": ..., "url": ... }
  → FetchNode adds "doc"
  → ParseNode adds "parsed_doc"
  → GenerateAnswerNode adds "answer"

Input/Output Expressions (Boolean Logic)#

Every node declares:

• input — a boolean expression (string) over state key names
• output — a list of state key names the node will write

The input expression is evaluated by _parse_input_keys(state, expression), which supports:

The parser evaluates OR branches left-to-right and returns the first branch where all keys exist in the current state. This allows nodes to declare flexible requirements without knowing which upstream path produced the data 3.

Real examples from SmartScraperGraph._create_graph() 4:

• FetchNode: input="url | local_dir" — accepts either a URL or a local directory
• ParseNode: input="doc" — strictly requires doc
• GenerateAnswerNode: input="user_prompt & (relevant_chunks | parsed_doc | doc)" — requires the prompt plus whichever document representation is available

Conditional Branching#

A special node type, conditional_node, returns a node name string (rather than an updated state) to dynamically select the next node 5. A ConditionalNode must have exactly two outgoing edges 6. SmartScraperGraph uses this for the optional reattempt feature: if GenerateAnswerNode produces "NA", the graph loops to a regen node 7.

Graph Construction Pattern#

BaseGraph requires three things at construction time 8:

1. nodes — ordered list of node instances
2. edges — list of (from_node, to_node) tuples
3. entry_point — the first node to execute

SmartScraperGraph._create_graph() selects among eight pre-defined graph variations based on the boolean flags (html_mode, reasoning, reattempt) 9. Each variation is a different combination of the same node pool wired differently.

Key Source Files#