LLM Provider Configuration#

Sure's AI assistant uses an OpenAI-compatible API as its single unified interface, which means any provider that speaks the OpenAI chat-completions protocol can be used — including OpenAI, Groq, OpenRouter, LM Studio, Ollama, and other self-hosted inference servers. 1

Prerequisite: Only gpt-4.1 was supported prior to v0.6.5-alpha. Multi-provider support was introduced in that release. 2

Two Independent AI Pipelines#

Sure runs two AI subsystems that are configured and operate independently 3:

A second "external" assistant mode exists (ASSISTANT_TYPE=external) for delegating chat entirely to a remote agent, but that system is separate from provider configuration; see docs/hosting/ai.md for details.

Provider Resolution#

Provider::Registry resolves which backend to instantiate. For LLM calls the registry exposes two named providers — openai and anthropic 4.

preferred_llm_provider (used by batch/background flows) checks Setting.llm_provider and tries providers in order, preferring Anthropic-first when configured, otherwise OpenAI-first, and falling back to whichever has credentials 5. It returns nil if neither provider has credentials — callers must guard on that.

OpenAI provider instantiation#

Registry#openai reads credentials in this priority order:

1. ENV["OPENAI_ACCESS_TOKEN"] → Setting.openai_access_token
2. ENV["OPENAI_URI_BASE"] → Setting.openai_uri_base (optional — omit for OpenAI.com)
3. ENV["OPENAI_MODEL"] → Setting.openai_model (required when a custom URI base is set)

If uri_base is present but model is blank, the registry logs an error and returns nil — the provider is treated as unconfigured 6. The same constraint is enforced at the model level: Provider::Openai#initialize raises Provider::Openai::Error immediately when custom_provider? is true but no model is supplied.

The default model when no custom base URL is set is gpt-4.1.

Anthropic provider instantiation#

Registry#anthropic reads:

1. ENV["ANTHROPIC_ACCESS_TOKEN"] or ENV["ANTHROPIC_API_KEY"] → Setting.anthropic_access_token
2. ENV["ANTHROPIC_BASE_URL"] → Setting.anthropic_base_url (optional)
3. ENV["ANTHROPIC_MODEL"] → Setting.anthropic_model (optional)

The LLM_PROVIDER env var (default: "openai") controls which provider preferred_llm_provider tries first 7.

Configuration Reference#

All settings can be configured via environment variables or the Settings UI (Settings → Self-Hosting → AI Provider). Environment variables take precedence; when set, the corresponding UI field is disabled. 8

API tokens stored in the DB are encrypted at rest using ActiveSupport::MessageEncryptor keyed from SECRET_KEY_BASE. Affected fields: openai_access_token, anthropic_access_token, external_assistant_token. 9

Key environment variables#

10 11

Tested provider configurations#

OpenAI (no extra config needed):

OPENAI_ACCESS_TOKEN=sk-proj-...

Groq / OpenRouter / other OpenAI-compatible services:

OPENAI_ACCESS_TOKEN=your-api-key
OPENAI_URI_BASE=https://openrouter.ai/api/v1 # or https://api.groq.com/openai/v1
OPENAI_MODEL=google/gemini-2.5-flash # provider/model slug

Ollama (local):

OPENAI_ACCESS_TOKEN=ollama-local # any non-empty string
OPENAI_URI_BASE=http://localhost:11434/v1
OPENAI_MODEL=llama3.1:13b
LLM_CONTEXT_WINDOW=8192
OPENAI_REQUEST_TIMEOUT=180

12

Extra HTTP Headers#

Some OpenAI-compatible gateways require custom headers on requests (e.g. OpenRouter's HTTP-Referer). Set OPENAI_EXTRA_HEADERS to a JSON object mapping header names to values:

# Single-quoted so the shell does not interpret braces or quotes.
# A value containing the literal {session_id} is replaced with the chat's UUID
# on each chat request, identifying the conversation rather than the install.
OPENAI_EXTRA_HEADERS='{"x-opencode-session":"{session_id}"}'

# Static value instead — sent on every OpenAI-provider request, including
# batch jobs (auto-categorize, merchant detection, PDF processing):
OPENAI_EXTRA_HEADERS='{"x-opencode-session":"b3f1c2d4-0000-0000-0000-000000000000"}'

Behavior:

• A value containing the literal {session_id} is substituted with the chat's UUID on each chat request, so requests are attributable per conversation. Batch flows only receive static (non-{session_id}) headers, because a session only exists for a chat. If your gateway requires the header on every endpoint, use a static value.
• Extra headers are attached to chat requests made by the OpenAI-compatible provider. Batch flows (auto-categorize, merchant detection, PDF processing) only receive static headers.
• Values are stringified: nested JSON objects/arrays become Ruby inspect-style strings, not valid JSON. Header values must be plain strings.
• User headers merge over the client's managed headers — setting Authorization here would override the access token.
• Unset, blank, malformed, or non-object JSON is ignored with an error in the logs; chat keeps working. The raw value is never logged.
• These headers are NOT sent to embedding, vector-store, or AI-health-probe calls — those build separate clients. A gateway requiring the header on those endpoints is not supported.
• ENV-only: there is no settings-page equivalent. The value is re-read every time a provider client is built (nothing is cached), but updating the environment requires restarting the app.

Function/Tool Calling Requirement#

The builtin chat assistant and auto-categorization both rely on OpenAI-style function/tool calling (structured JSON schemas sent with the request). Not all models and not all OpenAI-compatible endpoints support tool calling. 13

When a model or provider does not support tools, Provider::Openai detects the failure and raises Provider::Openai::FunctionCallingNotSupportedError — a specialized subclass of Provider::Error — with an actionable message including the model name and provider URL. 14 Detection logic checks for:

• HTTP 404 responses (common when the endpoint doesn't recognize the tools parameter)
• Error messages containing substrings like "tools", "tool_choice", "function", "does not support", or "invalid/unknown parameter" 14

The chat UI surfaces this error message directly (rather than a generic "Failed to generate response") so the user can identify the misconfigured model.

Model capability notes:

• Cloud: gpt-4.1, gpt-5, OpenRouter's anthropic/claude-* models have reliable tool support
• Local: qwen3-30b, qwen3-8b, llama3.1-8b support tool calling; many smaller or older models do not
• Auto-categorization does not require function calling — a plain completion model works there 15

Known Limitation: Single Tool Round Per Turn#

The builtin responder currently caps tool execution at one round per turn — if the model requests additional tool calls in its follow-up response, those calls are silently dropped and the assistant message stays pending, leaving the UI hanging on "Thinking…" 16

This is an intentional guard against runaway spend on paid APIs but is problematic for self-hosted deployments where local model calls are free. Issue #2241 and PR #2253 track adding a configurable iteration cap and proper error surfacing when the limit is hit.

Workaround: choose models/prompts that resolve queries in a single tool round, or backport PR #2253. 16

Model Change Auto-Clears AI Cache#

When Setting.openai_model is changed (via UI or API), a ClearAiCacheJob is automatically enqueued for every family to force re-categorization with the new model. This can incur API costs proportional to the number of transactions. 17

Key Files#