MarkdownNodeParser#
Overview#
MarkdownNodeParser splits a Markdown document into TextNode objects based on ATX-style (#) headings. Each node spans from one heading to the next, and carries a header_path metadata field recording the ancestor-heading breadcrumb (e.g. /Main Header/Sub-header/). Content before the first heading is emitted as its own node.
Source: llama-index-core/llama_index/core/node_parser/file/markdown.py
Tests: llama-index-core/tests/node_parser/test_markdown.py
Heading Parsing#
get_nodes_from_node iterates over lines and applies re.match(r"^(#+)\s(.*)", line) to detect headings . This pattern follows CommonMark §4.6 ATX headings: one or more # characters followed by a space and the heading text. Consequences:
#Not a header(no space) is not treated as a heading .# indented(leading spaces) is not detected because the regex anchors at^— indented lines pass through as body content.- All six heading levels (
#through######) are recognised; there is no upper bound enforced. The level count is simplylen(header_match.group(1)).
Header stack and header_path#
A header_stack list of (level, text) tuples is maintained . On each new heading, any stack entries with a level ≥ the incoming level are popped before the new entry is pushed . The header_path written to node metadata is built from all entries except the last (the current heading) , e.g.:
| Heading sequence | header_path on its node |
|---|---|
# A | / |
## B under A | /A/ |
### C under B | /A/B/ |
# D (resets to root) | / |
Level jumps (e.g. # A → ### C) are accepted; the intermediate level is simply absent from the stack .
The separator character defaults to / and is configurable via header_path_separator .
Fenced Code Block Handling#
Headings inside fenced code blocks must be suppressed. The parser tracks a boolean code_block flag and toggles it whenever a line (after lstrip()) starts with ````` (cite:code_file:0fdca43d:60-63). While code_block is True, the heading regex is skipped.
Known Bugs (as of ddeb22219)#
The shipped code at this commit has two open correctness issues, each with a pending fix:
1. Tilde fences not recognised (PR #22123)
CommonMark §4.5 allows fenced code blocks opened with ~~~ (three or more tildes) in addition to ````` backticks. The current parser only checks for backtick fences (cite:code_file:0fdca43d:60), so a # line inside a ~~~ block is incorrectly treated as a heading, producing spurious node splits and a corrupted header_path hierarchy.
Fix approach (PR #22123): replace the lstrip().startswith("```") check with a regex matching ^\s*({3,}|~{3,})`, and record which fence character opened the block; close only on a matching character.
2. Opening fence length not respected (PR #23283)
The toggle logic does not track the opening fence length. A four-backtick opening fence (``````) can be accidentally closed by three backticks (`````) appearing inside it, causing subsequent # content in the block to be parsed as a heading.
Fix approach (PR #23283): replace the boolean with a code_fence_length integer; close the block only when the closing run of backticks is ≥ the opening length.
Configuration#
| Parameter | Default | Effect |
|---|---|---|
include_metadata | True | Adds header_path to each node's metadata |
include_prev_next_rel | True | Wires PREVIOUS/NEXT relationships between nodes |
header_path_separator | "/" | Separator used inside header_path values |
Instantiate via MarkdownNodeParser() or MarkdownNodeParser.from_defaults(...) .
Quick Reference#
- Entry point:
get_nodes_from_node— single-document splitting logic - Node builder:
_build_node_from_split— attachesheader_pathmetadata - Batch entry:
_parse_nodes— iterates over a sequence of input nodes - Open PRs fixing fence edge cases: #22123 (tilde fences), #23283 (fence length)