Markdown Preprocessing and Repair#

The react-streamdown package processes LLM-generated text through two sequential layers before rendering: a text normalization layer (preprocess.ts) and an incomplete-markdown repair layer (remend.ts). Both layers run on the full accumulated text each time the stream flushes.

Key files:

• packages/react-streamdown/src/preprocess.ts — math delimiter normalization, currency escaping
• packages/react-streamdown/src/remend.ts — tailBoundedRemend, findRemendWindowStart, scanBlocks
• packages/react-streamdown/src/primitives/StreamdownText.tsx — wires both layers into the streaming pipeline

Quick-reference table of public exports:

preprocess.ts — Text Normalization#

preprocess.ts normalizes text before the Markdown parser receives it, converting alternative math delimiters that language models emit to forms remark-math recognizes and escaping currency amounts so single-dollar math does not consume prices. All rewrites skip verbatim regions — HTML blocks, code spans, and fences — so delimiters shown as code remain unchanged 1.

rewriteOutsideCode#

The internal walker rewriteOutsideCode applies a rewrite function only to text outside HTML blocks, code spans, and fences. It steps over \x escapes so an escaped backtick does not open a span 2. Backtick regions use backtickEnd, which delegates to either fenceEnd or codeSpanEnd depending on whether opensBacktickFence accepts the run.

Fence Detection#

A backtick run opens a fence when it contains ≥3 backticks, only blockquote or list markers precede it on the line, and the info string contains no backtick 3. Indentation is not capped at the three columns CommonMark allows because the cap is container-relative and the walker does not track containers 4. This trade-off means a run indented four columns at the root, where CommonMark reads an indented code block, opens a fence here 5.

A tilde run opens a fence when it contains ≥3 tildes and carries the same line prefix (blockquote or list markers) 6. Tilde runs never open code spans, so no info-string check is needed 7.

fenceEnd finds the closing run — at least as long as the opener and the only content on its line — accounting for blockquote depth 8. A quoted fence ends where its blockquote does if no closer appears 9.

codeSpanEnd closes on an exact-length backtick run; a blank line ends the search because spans cannot cross paragraph boundaries 10.

Math Delimiter Normalization#

Three functions convert alternative math delimiters to the $...$ / $$...$$ form remark-math parses:

• rewriteLatexBracketDelimiters converts \(...\) → $...$ and [...] → $$...$$ 11. A single or double leading backslash is accepted, since models emit both depending on escaping 12.

• rewriteCustomMathTags converts [/math]...[/math] → $$...$$ and [/inline]...[/inline] → $...$ 13.

• normalizeMathDelimiters composes both 14.

emitDisplayMath is the helper for multiline display math. It emits $$body$$ on one line for single-line bodies and the fenced form for bodies spanning lines, with $$ markers on their own lines 15. Markers written at the root column would end the list item or blockquote the math was written inside, so they carry that container's prefix and the body is aligned to it 16.

escapeCurrencyDollars#

escapeCurrencyDollars escapes $ preceding a digit so remark-math does not consume prices as math delimiters 17. The decision is made on the delimiter pair: the text up to the next $ is inspected to determine if it's plausible math via isMathBody 18. This keeps wrong guesses local — an accepted span contains no $, so an inserted escape can never fall between a delimiter pair and shift every delimiter that follows 19.

The function uses rewriteOutsideHtml (not rewriteOutsideCode) because it has its own verbatim-run logic via endOfVerbatimRun.

remend.ts — Tail-Bounded Incomplete-Markdown Repair#

remend.ts wraps the remend library with a segmented strategy: it identifies protected regions (closed code fences, HTML blocks, and math blocks) and any construct still open at the end, then applies full remend repair only to the final unsettled block while running escape-only passes on earlier prose segments 20.

scanBlocks#

The scanBlocks function performs a single-pass line-by-line scan that returns four values:

• boundary: character index after the most recent blank line outside any open fence, HTML block, or $$ math block — the last top-level block boundary 21
• protectedRanges: flat array of start/end pairs marking closed fences, HTML blocks, $$ blocks, and line-start inline math; these regions are copied raw without remend escaping 22
• openStart: start index of a fence, HTML, or math block still open at the end, or -1 if none 23
• katexCloses: whether remend's katex completion (a bare $$ line) can close an open $$ block — true only when the block was opened with exactly two dollars, outside any blockquote, and outside the list item of its opening line 2425

Protected ranges start at line boundaries because remend drops a trailing space from its input; a cut mid-line would lose characters 26.

Fence Tracking#

A fence opens at any indentation 27, since a marker indented four or more columns is either a fence nested in a list item or an indented code block 28. The opener's indentation is stored in fenceIndent 29. A fence closes only on a marker of the same character (backtick or tilde), at the same blockquote depth, indented at most three columns past the opener, with a run at least as long as the opener, followed only by whitespace 30.

Math Block Tracking#

Dollar-sign math follows remark-math conventions 31. A run of two or more $ characters at the start of a line's content (after list and blockquote markers) opens a $$ math block only if no other dollar appears on that line 32. The block closes on a line holding only a dollar run at least as long, at the correct blockquote depth and indentation 33. Math blocks are raw: fence markers inside them open nothing.

Inline math (two or more dollars not at block start, or mid-line) is protected only when it opens at line start and closes on the same line with a run of exactly the same length 34.

HTML Block Tracking#

HTML blocks follow all seven CommonMark kinds 35. An HTML block opens when a line's content (indented fewer than four columns past blockquote markers themselves indented fewer than four) starts one of the seven patterns micromark recognizes 36. The body is raw: fence or math markers inside it open nothing 37.

Blank-Line Boundary Tracking#

A blank line outside any open fence, HTML block, or math block commits the boundary 38. The boundary marks the end of the last settled block. A bare > line is blank inside a blockquote but opens a new block after a blank line 39.

List and Blockquote Context#

The scan uses skipListMarkers to track list and blockquote nesting. List markers and blockquote markers affect where fences, math blocks, and HTML blocks can open and close. A fence or math block opened after list markers ends with that list item at the first line indented fewer columns than the item's content 40.

findRemendWindowStart#

findRemendWindowStart is a one-line wrapper that returns scanBlocks(text).boundary. It preserves the same public signature as the previous implementation 41.

tailBoundedRemend#

The tailBoundedRemend function is the main export. It segments the message and applies different remend configurations to each segment 20:

1. Promote comparisonOperators to handler: If comparisonOperators !== false, the function promotes it to a prose-only handler so comparison-operator escapes run only on prose segments, not on raw protected blocks 42.

2. Fast path: If boundary ≤ 0, no protected ranges exist, and no block is open (openStart === -1), the function calls remend(text, options) directly 43.

3. Segmented processing: Otherwise, it walks through protected ranges and prose segments:

   • For each protected range [from, to]: runs remend(text.slice(cursor, from), prefixOptions) on prose before it, then copies text.slice(from, to) raw 44
   • If a block is still open (openStart !== -1): runs remend on prose before the open block with COMPLETION_OFF, then either copies the open block raw (when katex cannot close it) or runs remend with katex-only options (when katexCloses is true) 45
   • Otherwise: runs remend on prose before the boundary with COMPLETION_OFF, then applies full remend to the tail starting at the boundary 46

COMPLETION_OFF#

The COMPLETION_OFF object disables all remend completion options. It is enforced with a satisfies constraint at compile time: a new remend completion option will fail the build instead of silently reaching settled text 47.

PrefixSafeOption#

The PrefixSafeOption type lists options safe for settled prefix text: singleTilde, comparisonOperators, handlers, and linkMode. These apply escapes rather than completions 48.

escapeComparisonOperators#

The escapeComparisonOperators function escapes list-item-like comparison operators (e.g., - > 25) so they are not misread as list items 49. When comparisonOperators !== false, tailBoundedRemend promotes it to a handler that runs as a prose-only handler rather than on raw text 42.

Integration in StreamdownTextPrimitive#

StreamdownTextPrimitive enables tailBoundedRemend when mode === "streaming", repair is not disabled, and no custom parseMarkdownIntoBlocksFn is provided. It sets parseIncompleteMarkdown={false} when forwarding to Streamdown to prevent double repair 50.

Design Decisions and Known Limitations#

The pipeline reflects tradeoffs between correctness, maintainability, and the constraints of line-based scanning without full container tracking.

Fence detection: preprocess.ts vs remend.ts#

Two separate fence detectors exist and intentionally differ in their indentation rules:

• preprocess.ts opensBacktickFence (packages/react-streamdown/src/preprocess.ts) omits the CommonMark three-column indentation cap because the cap is container-relative and the walker does not track containers. A run indented four or more columns at root opens a fence here, where CommonMark would read it as an indented code block. This is a deliberate tradeoff, documented in the function comment: "Indentation is not capped at the three columns CommonMark allows, because the cap is relative to the enclosing container and this walker does not track containers" 51.

• remend.ts scanBlocks fence opener opens a fence at any indentation: "a marker indented four or more columns is either a fence nested in a list item or an indented code block, code either way" 26. The closer must sit at most three columns past the opener 52, matching how fenceEnd in preprocess reads closers 53.

Both arrive at the same practical outcome—protecting code from rewriting—but for different reasons. The preprocess walker exists to normalize math delimiters outside code, while the remend scanner exists to exclude settled blocks from incomplete-markdown repair.

Indented code blocks not protected#

Root-level indented code blocks (e.g., four-space indented lines at root) still receive tilde and comparison-operator escapes. Tracking indented code blocks requires list container state, which the line scanner does not maintain 26. This is a known limitation: Intro\n\n lm(y~x) becomes y\~x escaped 54. The issue is tracked in open issues #6945 and #7449, referenced in 54 and 55.

Mid-line $$ not treated as display-math opener#

remark-math opens display math only where a line's content starts with $$ (the blockStart position in the scan) 56. The scan now opens a $$ block only at blockStart, not at any mid-line position 56. This fixed a bug where Price: $$$ tier plus a later $$ block caused the block to lose protection, as the mid-line $$ kept the scanner inside math for the rest of the message 56.

Custom handlers receive prose segments separately#

Since tailBoundedRemend now segments the message, custom remend.handlers receive each run of prose as a separate call, not the full final block as one string 54. This is a behavioral change from the initial tail-bounded implementation.

Maintaining upstream remend compatibility#

The background note flags: "react-streamdown's tailBoundedRemend code-span scanner must preserve remend's lookup and rendering behavior as upstream streamdown/remend packages change" 55. The scanBlocks leaf rules are designed to match remend's own parsing: info-string check, exact run length, paragraph-bounded spans, and backslash parity for escapes 55. The remend.ts file notes: "upstream, vercel/streamdown#571 (open) moves remend to a region scanner that recognizes ~~~ fences at any indentation; its tilde escape still ignores math regions, so the $$ half is not covered there" 55.

PR History#