LLM Tool Calling#

Overview#

Sure integrates with Anthropic and OpenAI for chat, streaming, and function/tool calling — the mechanism by which the AI assistant can invoke application functions (e.g., create_category, get_transactions) and feed results back into the conversation. The architecture is provider-agnostic: shared data structures in Provider::LlmConcept allow both Anthropic and OpenAI backends to flow through the same downstream execution and persistence logic.

Provider Selection#

Provider::Registry resolves which LLM backend to use. preferred_llm_provider 1 checks Setting.llm_provider and tries providers in order — defaulting to Anthropic-first if configured, OpenAI-first otherwise — falling back to whichever has valid credentials. Returns nil if neither is configured; callers must guard on that.

Credentials are read from env vars (ANTHROPIC_ACCESS_TOKEN / ANTHROPIC_API_KEY, OPENAI_ACCESS_TOKEN) or Setting DB values 2.

Core Data Structures (Provider::LlmConcept)#

Provider::LlmConcept defines the provider-agnostic value objects used throughout the tool-calling pipeline 3:

The chat_response interface 4 is what every provider implements. Key parameters: functions (tool definitions), function_results (prior round results), streamer (callback proc for streaming), previous_response_id (for stateful OpenAI Responses API turns).

Tool/Function Definition (Assistant::Function)#

Assistant::Function is the abstract base class every callable tool subclasses. Subclasses must implement:

• self.name / self.description — used in the function definition sent to the LLM 5
• call(params) — executed when the LLM requests the function 6
• params_schema (via build_schema) — JSON Schema object describing parameters; strict_mode? defaults to true, requiring all properties in required 7

to_definition serializes the function into { name:, description:, params_schema:, strict: } 8, which is forwarded to providers by FunctionToolCaller#function_definitions.

Concrete implementations live in app/models/assistant/function/ (e.g., Assistant::Function::CreateCategory).

Request Building: Anthropic (ChatConfig)#

Provider::Anthropic::ChatConfig assembles the Anthropic API request:

• System prompt is wrapped with cache_control: { type: "ephemeral" } for prompt caching — ~10× cheaper on cache hits 9.
• Tool definitions are built from functions definitions; the last tool block also gets cache_control to cache tool definitions 10.
• The strict key (OpenAI-only) is stripped from schemas before forwarding to Anthropic 11.

Response Parsing: Anthropic (ChatParser)#

Provider::Anthropic::ChatParser maps raw Anthropic SDK message objects to ChatResponse:

• Text blocks (type == :text) → ChatMessage with joined output text 12
• Tool-use blocks (type == :tool_use) → ChatFunctionRequest; input is serialized to JSON if not already a String 13
• Handles both object-style (.respond_to?(:key)) and hash-style blocks for SDK version flexibility 14

Tool Execution (FunctionToolCaller)#

Assistant::FunctionToolCaller receives the list of ChatFunctionRequest objects from the parsed response and executes them:

1. Looks up the matching Assistant::Function instance by name 15
2. JSON.parses function_args and calls fn.call(fn_args) 16
3. Wraps the result in a ToolCall::Function record via from_function_request 17
4. Any exception is re-raised as FunctionExecutionError with function name + args in the message 18

function_definitions 19 serializes all registered functions via to_definition for inclusion in the next API request.

Persistence (ToolCall::Function)#

ToolCall::Function is the ActiveRecord model persisting each tool invocation:

• from_function_request(function_request, result) — creates the record from the LLM's request + Ruby execution result 20
• to_result — formats the result for injection into the next LLM turn 21
• to_tool_call — formats the call in the OpenAI tool_calls message format for conversation history replay 22

Orchestration: Multi-Turn Tool Calling (Assistant::Responder)#

Assistant::Responder drives the full round-trip:

1. Sends the initial request to llm.chat_response(...) with a streamer proc 23
2. The streamer emits :output_text events (text chunks) and :response events (final message) to registered listeners
3. If the response contains function_requests, calls handle_follow_up_response 24:
   • Executes all requested functions via FunctionToolCaller#fulfill_requests
   • Emits :response with function_tool_calls for persistence
   • Issues a follow-up chat_response with function_results injected
4. Recursion is intentionally blocked: follow-up responses ignore further function_requests to prevent runaway LLM spend 25

For conversation history, Anthropic uses raw Message records (chat_message_records); OpenAI uses a pre-built role/tool_call_id payload (openai_messages_payload) 26.

Key File Index#