Agent Prompt Processing#

The agent prompt processing pipeline is responsible for preparing, fitting, and dispatching messages to the LLM in canvas-based agent workflows. It is distinct from the simpler chat-assistant path and involves several transformation steps before any text reaches the model.

Pipeline Overview#

The pipeline has four main stages, all in Python:

_prepare_prompt_variables()
  → _sys_prompt_and_msg() # merge history + format prompts
  → [file/image injection] # attach sys.files content
  → _extract_prompts() # strip special XML tags from sys_prompt
↓
fit_messages() / message_fit_in() # token-budget trimming
↓
validate_fitted_messages() # guard against empty user message
↓
_generate_async() / async_chat() # LLM call

Key entry points:

• LLM._invoke_async() — non-streaming agent LLM path
• LLM._stream_output_async() — streaming path (used when a Message component is downstream)
• AgentWithTools._invoke_async() — agent-with-tools path; delegates to LLM._invoke_async when no tools are configured

Stage 1 – Prompt Variable Preparation#

_prepare_prompt_variables() collects all canvas input variables, formats image/file attachments, then calls _sys_prompt_and_msg() to produce the initial message list.

_sys_prompt_and_msg() key logic:

• Iterates self._param.prompts (the configured user/assistant turns)
• Formats each turn's content via string_format(content, args) using resolved input variables
• Trailing same-role replacement: if msg[-1]["role"] equals the incoming prompt's role, the last message is replaced (not appended), preventing duplicate user turns. This was a bug vector (see Known Issues below).

After _sys_prompt_and_msg(), system-file text attachments are merged into the last user message, and _extract_prompts() strips special XML tags (TASK_ANALYSIS, PLAN_GENERATION, REFLECTION, CONTEXT_SUMMARY, CONTEXT_RANKING, CITATION_GUIDELINES) from sys_prompt into a structured dict returned as user_defined_prompt.

Stage 2 – User Prompt Fallback Injection (AgentWithTools)#

In AgentWithTools._invoke_async(), if a runtime user_prompt kwarg is present, it is built into a structured message with optional REASONING, CONTEXT, and QUERY sections and injected directly into self._param.prompts 1. This overrides any canvas-configured prompts. If user_prompt is absent and prompts is empty/default, the component falls back to the query from canvas state. This mechanism was broken prior to PR #16570 when template resolution and the URL-construction for the model were not applied to runtime prompts.

Stage 3 – Message Fitting#

LLM.fit_messages() wraps message_fit_in() from rag/prompts/generator.py:

1. Prepends the system prompt as {"role": "system", ...} and deep-copies msg
2. Passes the combined list to message_fit_in(msg, budget) where budget = int(effective_context_length * 0.97) 2
3. effective_context_length normalizes max_tokens=0 → 8192 3

message_fit_in() trimming strategy (rag/prompts/generator.py:69-136):

• If total tokens fit, returns immediately
• Otherwise, discards all non-system messages except the last one
• If still over budget, trims content: if the system prompt represents > 80% of tokens, it trims the user message first; otherwise, it trims the system prompt first
• Non-positive max_length is clamped to 8192 rather than causing a zero-context wipe 4

Context budget helper:

effective_context_length = max_tokens or 8192
context_fit_budget = int(effective_context_length * 0.97)

Stage 4 – Post-Fit Validation#

validate_fitted_messages() checks that the fitted list has at least two messages and that the final message is a non-empty user turn. If validation fails, a **ERROR** string is returned instead of reaching the LLM. On the non-streaming path, the error is written to the _ERROR output slot 5; on the streaming path, it is emitted as output content or _ERROR depending on whether an exception default value is configured 6.

Known Issues and Fixes#

Affected versions: v0.26.1 and nightly builds before 2026-07-01 7. Both fixes are merged as of PR #16413 (merged 2026-07-01) and PR #16570 (merged 2026-07-03).

Note: This bug only affects canvas-based agents; the chat-assistant code path does not share this pipeline 7.

Key Source Files#