Tool Call Validation (Dangling tool_use / tool_result Pairs)#
Overview#
Phoenix does not validate that every tool_use block in an assistant message has a matching tool_result block in the following user message — and vice versa — before forwarding the message list to an LLM provider. This gap affects two code paths: Playground span replay and agent session replay. The Anthropic Messages API enforces this constraint strictly and returns a 400 if it is violated; OpenAI is more lenient but structurally also requires paired tool messages.
The Problem: Dangling tool_use/tool_result Pairs#
When a span that contains a mid-conversation tool_use response is opened in the Phoenix Playground via LLM Span Replay, the replay pipeline reconstitutes the captured assistant message — including its tool_use content block — but does not carry over the matching tool_result user message from the trace.
The result is a message list where the assistant's tool_use has no paired tool_result, which causes Anthropic's API to return:
400 - messages.1: `tool_use` ids were found without `tool_result` blocks
immediately after: toolu_01PK6mZ2d237bSRpSooKrL6L
The bug is not specific to the memory_20250818 tool where it was first observed — it affects any Anthropic tool call captured mid-conversation, including bash, text_editor, and computer.
The Three-Layer Replay Pipeline#
Span replay passes through three layers, none of which sanitize unpaired tool blocks:
1. Backend Extraction (dataset_helpers.py)#
get_dataset_example_output in src/phoenix/server/api/helpers/dataset_helpers.py extracts output messages from a span. It faithfully reconstructs tool call data (IDs, function names, arguments) from OpenInference span attributes but performs no cross-validation against corresponding tool_result messages in adjacent spans.
2. Frontend Hydration (playgroundUtils.ts)#
processAttributeMessagesToChatMessage and getOutputFromAttributes convert raw span attributes into ChatMessage objects for the playground UI. processAttributeToolCalls maps the captured tool call blocks into provider-specific formats (e.g., tool_use for Anthropic). These functions correctly reconstruct the assistant message with its tool_use block but do not check whether a corresponding tool_result message exists in the reconstituted message list.
3. Server-Side Message Building (playground_clients.py)#
_build_anthropic_messages in AnthropicStreamingClient converts the playground's internal message list into Anthropic MessageParam objects. It maps TOOL role messages to tool_result blocks , but silently defaults tool_use_id to an empty string when the ID is missing rather than raising an error — unlike the OpenAI client, which explicitly raises a ValueError for missing tool_call_id . No check is performed to validate that each tool_use in an AI message has a paired tool_result in the next user message before the API call is made.
Asymmetric Error Handling Across Providers#
| Client | Missing tool result | Missing tool ID |
|---|---|---|
OpenAI (to_openai_chat_completion_param) | Passes through (OpenAI may accept) | raise ValueError |
Anthropic (_build_anthropic_messages) | Silent — provider returns 400 | Defaults to "" string |
The OpenAI client raises early on a missing tool_call_id; the Anthropic client does not, surfacing the error only as a raw 400 from the remote API.
Where a Fix Should Go#
The issue thread identifies three acceptable mitigation strategies:
- Carry over the original
tool_resultblock from the captured trace into the next message. - Drop the dangling
tool_useblock so the replayed request is well-formed. - Warn the user before submission rather than surfacing a raw
400.
A fix would most naturally live in one or both of:
processAttributeMessagesToChatMessage/getOutputFromAttributesinplaygroundUtils.ts— detect and handle danglingtool_useblocks during frontend hydration._build_anthropic_messagesinplayground_clients.py— validate pairing before the API call and raise a descriptive error or strip the dangling block.