Model Provider Integration#

Agenta routes LLM calls by matching a model string to stored vault credentials and building a LiteLLM-compatible kwargs dict at runtime. Two credential kinds exist: standard (a plain API key for hosted providers like OpenAI or Anthropic) and custom (structured settings for providers that need more than a key — AWS Bedrock, Azure, Vertex AI, or any OpenAI-compatible endpoint). The full resolution pipeline runs inside the SDK on every inference call, after the backend injects the project's vault secrets into the request context.

Data Model#

Provider credentials are defined in api/oss/src/core/secrets/dtos.py and validated by enums in api/oss/src/core/secrets/enums.py.

SecretKind 1: provider_key or custom_provider.

Standard secrets (SecretKind.provider_key) use StandardProviderDTO and require only a key string. Recognized providers are enumerated in StandardProviderKind: OpenAI, Anthropic, Gemini, Groq, Cohere, Mistral AI, Together AI, OpenRouter, and others 2.

Custom secrets (SecretKind.custom_provider) use CustomProviderDTO, which adds url, version, optional key, and a freeform extras dict, plus a list of CustomModelSettingsDTO entries. CustomProviderKind extends the standard set with custom, azure, bedrock, sagemaker, and vertex 3.

provider_slug is automatically set from the connection's header name during creation 4. On read, SecretResponseDTO.build_up_model_keys constructs composite model key strings of the form {provider_slug}/{kind}/{model_slug} — e.g., mybedrock/bedrock/anthropic.claude-3-5-sonnet. These composite keys are the model identifiers passed at inference time.

SDK Credential Resolution#

SecretsManager.get_provider_settings(model) is the SDK entry point. It runs four steps:

1. Pull secrets from route context — reads the secrets list injected by the backend into routing_context.
2. Parse into standard vs. custom lists — _parse_secrets splits by kind, and for custom secrets merges api_base and api_version from the provider's url/version fields into extras 5.
3. Resolve provider name — looks up the model in model_to_provider_mapping, a dict built from supported_llm_models in sdk/agenta/sdk/assets.py. If absent, falls back to _custom_providers_get to find a matching custom provider by scanning models lists 6.
4. Build kwargs — for standard secrets, copies api_key 7; for custom secrets, spreads extras (AWS creds, api_base, etc.) into the kwargs dict 8.

Custom model string conversion. Before the kwargs are built, _get_compatible_model strips the provider_slug/ prefix so LiteLLM receives a valid model string. OpenAI-compatible endpoints (kind custom) are rewritten from {slug}/custom/{model} to openai/{model}, since LiteLLM routes them via the OpenAI adapter.

The standard provider list in assets.py covers Anthropic, Cohere, DeepInfra, Gemini, Groq, Mistral AI, OpenAI, OpenRouter, Perplexity AI, Together AI, and Vertex AI 9.

Frontend: Connect-Model Gate & Provider Drawer#

PR #5096 redesigned the model-connection UI. Key behaviors:

• Connect-a-model gate now fires only when all three conditions are true: vault credential count is zero, the agent is not self_managed, and a browser-local localStorage flag (agenta:provider-key-setup-done, backed by Jotai atomWithStorage in agenta-entities/src/secret/state/atoms.ts) is unset 10. Previously it fired on every load.
• Drawer layout is Harness → Model → Provider credentials. Selecting a model auto-highlights its provider in the credentials rail. The rail is model-scoped: it shows only the selected model's provider, any custom providers whose models list includes that model, and relevant "Add …" rows (Azure, Bedrock, Vertex AI, custom) 10.
• Connection slug threading. Model selection now passes the connection slug explicitly through the UI stack instead of guessing it from the model ID string, eliminating a class of stale-slug runtime failures 10.
• Provider family normalization (e.g., converting together_ai to a canonical form) was extracted into a shared @agenta/shared helper to remove per-call-site duplication 10.
• CustomProviderForm was extracted from ConfigureProviderDrawerContent into @agenta/entity-ui/secretProvider so the drawer and the inline pane share one implementation 10.

Known Issues#

Custom provider shadows standard key (open, issue #5117)#

When a custom-provider connection includes a model name that a standard provider key also serves (e.g., gpt-4o-mini), the custom connection resolves first because the candidate pool ranks model-name matches above provider-family matches for slug-less default connections. The run uses the custom provider's credentials (which may be wrong) instead of the standard key 11.

Ambiguous connection silently drops credentials (open, issue #5117)#

When connection resolution encounters two equally valid connections and raises an AmbiguousConnectionError (a subclass of ConnectionResolutionError), the service-layer catch block degrades to credential_mode="runtime_provided" with an empty env — the same path taken for a genuinely missing connection. The run proceeds with no credentials, and the user sees a generic "add your key" error with no indication that the real problem is a conflict between two existing connections 11.

Bedrock concurrent credential leakage (partially fixed, issue #4244 / PR #4797)#

Concurrent Bedrock calls in the same worker could cross-contaminate AWS credentials because per-request keys were installed into os.environ around an await point. The fix removes the os.environ mutation entirely and passes AWS credentials as request-scoped kwargs to LiteLLM, eliminating the leak for Bedrock, Bedrock Converse, and Sagemaker paths 12.

Old-format prompt model path (issue #4933)#

Old prompts stored the model at prompt.model instead of prompt.llm_config.model. The evaluator LLM loop in handlers.py did not call get_provider_settings_from_workflow(), so custom provider models fell through with no api_base or api_key. Creating a new prompt with the correct llm_config.model path resolves the issue; the legacy path requires a migration/normalization step in the frontend 13.