Model Thinking and Reasoning#

RAGFlow supports reasoning/thinking models (e.g., DeepSeek-R1, Qwen3, Claude, OpenAI o-series, Kimi) that emit chain-of-thought content before their final answer. This content flows through the system in a standardized <think>...</think> wrapper and is handled consistently across generation config, streaming, API responses, and document exports.

1. Generation Configuration#

LLMParam.gen_conf() in agent/component/llm.py builds the parameter dict sent to the model. The thinking attribute is forwarded only when it is set and not "default" 1:

if hasattr(self, "thinking") and self.thinking and self.thinking != "default":
    conf["thinking"] = self.thinking

This is the entry point for UI-configured thinking control (e.g., enabling/disabling thinking on an LLM component). A prior bug (issue #16321, fixed in PR #16640) prevented thinking from being forwarded, so think tags were never properly controlled.

Provider-Specific Normalization#

_apply_model_family_policies() in rag/llm/chat_model.py translates the generic thinking field into each provider's native format before the API call:

The LITELLM_ALLOWED_GEN_CONF_KEYS set explicitly permits thinking, enable_thinking, and reasoning_effort to reach the LiteLLM completion layer 2. The base ALLOWED_GEN_CONF_KEYS (used by direct OpenAI-compatible clients) does not include these keys, so model-family policies must translate them into extra_body or strip them before the call 3.

Qwen3 defaults to enable_thinking=False unless explicitly overridden 4. Google Gemini models default thinking_budget=0 5.

2. Streaming: Think Tag Normalization#

Providers expose reasoning in different fields (reasoning_content, reasoning). The streaming layer in _async_chat_streamly() and async_chat_streamly() normalizes these into inline <think>…</think> text chunks 6:

_reasoning = getattr(delta, "reasoning_content", None) or getattr(delta, "reasoning", None)
if _reasoning:
    if not reasoning_start:
        ans = "<think>"
    ans += _reasoning + "</think>"

This wrapping happens for both the Base (direct OpenAI) and LiteLLMBase streaming paths 7. The with_reasoning=False kwarg can suppress wrapping when callers want raw text only (e.g., when QwQ is used as a non-streaming chat model) 8.

_stream_with_think_delta State Machine#

_stream_with_think_delta() in api/db/services/dialog_service.py is an async generator that sits between the raw LLM stream and consumers. It:

1. Tracks an in_think flag to distinguish reasoning from answer content.
2. Yields ("marker", "<think>", state) and ("marker", "</think>", state) as distinct events, separate from ("text", chunk, state) events.
3. Buffers answer text until at least min_tokens tokens accumulate to reduce event chattering 9.

The _ThinkStreamState class 10 holds the incremental parse state across chunks, including close_pending to handle cases where </think> appears at the end of a chunk without following answer text.

Both the dialog service (async_chat) 11 and the LLM agent component (_stream_output_async, _generate_streamly) consume this generator.

3. Canvas Event Protocol (start_to_think / end_to_think)#

The Canvas.run() loop in agent/canvas.py processes each Message component's streaming output. When it receives <think> or </think> markers from _stream_with_think_delta, it emits distinct SSE events instead of text content 13:

if m == "<think>":
    return decorate("message", {"content": "", "start_to_think": True})
elif m == "</think>":
    return decorate("message", {"content": "", "end_to_think": True})

The agent API layer in api/apps/restful_apis/agent_api.py reconstructs the full content string (with tags) when accumulating full_content from these events:

if ans.get("data", {}).get("start_to_think", False):
    full_content += "<think>"
elif ans.get("data", {}).get("end_to_think", False):
    full_content += "</think>"

Streaming clients receive start_to_think/end_to_think flags in individual SSE chunks and are responsible for visually separating or discarding think content. The web UI renders this as a collapsible "Thinking..." section.

4. Filtering Think Content from API Responses and Exports#

Non-Streaming API Responses#

_extract_visible_answer() in dialog_service.py 14 strips reasoning from the final accumulated text: if </think> is present, it splits at the last </think> and returns the trailing answer portion (preserving the think block only if thought content was non-empty). This is used to produce the clean answer for non-streaming chat API responses.

Known issue (now fixed): In earlier versions (≤0.25.6/0.26.1), the tag-stripping regex re.sub(r"</?think>", "", text) removed only tags, leaving reasoning text in place. _remove_reasoning_content() downstream couldn't find the now-tagless thinking text and returned it verbatim. Fixed in PR #16640 by ensuring thinking is properly forwarded through gen_conf so models called with enable_thinking=false never emit think tags at all. See issue #16321 for the full discussion.

Document Export#

Message._strip_thinking() in agent/component/message.py is applied before any Word/PDF/Excel conversion to prevent reasoning blocks from leaking into exported documents:

# Remove complete think blocks
cleaned = re.sub(r"<think>.*?</think>", "", content, flags=re.DOTALL)
# Remove dangling unclosed <think> + trailing content
cleaned = re.sub(r"<think>.*$", "", cleaned, flags=re.DOTALL)
# Remove leftover standalone tags
cleaned = re.sub(r"</?think>", "", cleaned)

LLM Component (Non-Streaming)#

The _invoke_async method in agent/component/llm.py applies a post-processing regex to strip </think> and everything before it from structured-output answers 15:

ans = re.sub(r"^.*</think>", "", ans, flags=re.DOTALL)

5. Practical Notes for API Consumers#

• Non-streaming (stream=false): Think content should be stripped automatically. If it leaks, verify your LLM component's thinking configuration is set to "disabled" (requires the PR #16640 fix to be applied).
• Streaming (stream=true): Watch for start_to_think: true and end_to_think: true flags in SSE chunks. Content emitted between those flags is reasoning text. To strip it client-side: re.sub(r'<think>[\s\S]*?</think>', '', raw_answer).
• Agent (/api/v1/agents/chat/completions): In non-streaming mode, thinking tags are reconstructed in full_content and should be stripped by _extract_visible_answer(). In streaming mode, watch for start_to_think/end_to_think flags. See issue #16321 for a detailed discussion of edge cases and workarounds.