Streaming Tool Execution#

Streaming tool execution describes how LangBot's LocalAgentRunner interleaves streamed LLM token delivery with synchronous tool/function call invocation. The key behavioral fact: tool execution fully blocks the streaming output — no new LLM tokens are delivered to the user while a tool is running. Each tool-call round is atomic from the user's perspective, with the next stream resuming only after all tools in that round have returned.

Entry point: LocalAgentRunner.run() in src/langbot/pkg/provider/runners/localagent.py.

Streaming vs. Non-Streaming Path#

Whether streaming is used is determined per-query by calling query.adapter.is_stream_output_supported(). If the adapter does not support streaming (or raises AttributeError), the runner falls back to _invoke_with_fallback(), which returns a single Message object 1.

When streaming is active, _invoke_stream_with_fallback() 2 is called instead. It eagerly consumes the first chunk from each candidate model to verify the stream is healthy before committing; fallback to the next model is only possible before any chunks have been yielded. Once streaming starts the model is locked in for the entire conversation turn, including subsequent tool-call rounds 3.

_StreamAccumulator: Throttled Chunk Delivery#

Raw LLM streams emit many small chunks. _StreamAccumulator buffers them and emits a consolidated MessageChunk every 8 raw chunks or on the final chunk 4. This throttling reduces downstream pressure without significantly increasing latency.

Tool-call argument fragments (OpenAI-style delta JSON) are merged across chunks into tool_calls_map. The assembled ToolCall objects are only included on the final accumulated chunk — they are never exposed mid-stream . This means downstream stages never see a partial tool call.

The remove_think option strips <think>…</think> blocks from the accumulated content before emission 6.

On each tool-call round a fresh _StreamAccumulator is created 7, deliberately not seeded with the prior round's text. Re-seeding would cause the platform adapter to receive (and forward) duplicate opening lines.

The Tool-Call Round Loop and Streaming Pause#

After the initial LLM stream finishes, pending_tool_calls = final_msg.tool_calls seeds the while loop 8. The loop:

1. Executes each tool call serially via tool_mgr.execute_func_call() 9 — this await is blocking: the generator pauses here and yields nothing to the user.
2. Yields tool result messages as MessageChunk(role='tool', ...) 10. These are passed through the pipeline but are not rendered as user-facing chat messages.
3. Starts the next LLM stream with the full message history including tool results, then loops back if the new response also contains tool calls 11.

A hard cap of MAX_TOOL_CALL_ROUNDS = 128 12 prevents runaway loops from a looping or adversarial model. If the cap is hit, the runner logs a warning and stops the loop without raising an error 13.

What Users See During Tool Execution#

The ResponseWrapper pipeline stage controls what is delivered to the user for each message in resp_messages.

During tool-call rounds: when result.tool_calls is non-empty, a "Call FuncName..." placeholder is appended to the response chain 14. This placeholder is only actually sent to the user if output.misc.track-function-calls is true in the pipeline config 15. Without that flag, tool-call rounds are silent from the user's perspective.

Final answer: sandbox outbound attachments (files/images the agent produced) are collected and appended to the message chain only on _is_final_assistant_message() — a message that is role='assistant', has no tool_calls, and is either a non-chunk or a chunk with is_final=True. This guard ensures attachments are not appended to intermediate streaming chunks or tool-call confirmation messages 16.

Tool Dispatch and MCP Timeouts#

ToolManager.execute_func_call() dispatches to four loaders in priority order: native → plugin → MCP → skill. The first loader that claims the tool name wins; ToolNotFoundError is raised if none match.

Every tool invocation is wrapped by _invoke_tool_with_monitoring(), which records start time, duration, success/error status, and routes results to the monitoring service.

MCP tools are the most likely source of long tool-execution pauses. RuntimeMCPSession.invoke_mcp_tool() applies a per-server tool_call_timeout_sec (default 300 seconds) . A timeout raises MCPToolCallTimeoutError . If the server's session has expired mid-call, the session triggers an async reconnect and retries the call once before failing 19.

Key Source Files#