Custom Provider Validation#

Overview#

Phoenix validates custom LLM provider configurations at multiple layers — GraphQL input, client instantiation, and database persistence — converting internal ValueError exceptions into user-facing BadRequest (HTTP 400) responses with actionable error messages. This ensures misconfigured providers fail fast with clear diagnostics rather than propagating silently.

Custom providers are modeled as GenerativeModelKind.CUSTOM in the GraphQL schema 1, distinct from built-in providers.

Validation Layers#

1. GraphQL Input Layer#

Annotation / Evaluator output configs enforce "exactly one type" logic at the GraphQL input level. AnnotationConfigInput.__post_init__ raises BadRequest("Exactly one of categorical, continuous, or freeform must be set") if the discriminated union constraint is violated 2.

Model config input (GenerativeModelInput) marks provider_key and name as required non-optional fields; optional provider-specific fields (base_url, endpoint, api_version, region) are validated downstream during client instantiation 3.

2. Provider Registry & Client Dispatch#

The PlaygroundClientRegistry singleton resolves a concrete streaming client class from (provider_key, model_name). If no class is found and no PROVIDER_DEFAULT fallback exists, get_client() returns None 4.

In chat_mutations.py, a None result immediately raises BadRequest(f"Unknown LLM provider: '{provider_key.value}'") before any network call is attempted 5.

Any exception raised during client instantiation that isn't already a CustomGraphQLError is caught and re-raised as BadRequest with a formatted message 6:

"Failed to connect to LLM API for {provider} {model}: {error}"

3. Per-Provider Client Validation#

Each streaming client __init__ validates its required fields and raises BadRequest with provider-specific messages:

The shared helpers _get_credential_value() and _require_credential() are the standard pattern for reading provider credentials from the user-supplied list before falling back to environment variables 11.

4. Annotation Config Pydantic Validation#

Pydantic validators in annotation_configs.py enforce field-level rules:

• Non-empty labels — _categorical_value_label_is_non_empty_string 12
• Non-empty values list — _categorical_values_are_non_empty_list 13
• Unique labels — _categorical_values_have_unique_labels 14
• Bound ordering — ContinuousAnnotationConfig.check_bounds() enforces lower_bound < upper_bound 15

Mutation helpers (_to_pydantic_categorical_annotation_config, etc.) wrap Pydantic instantiation in try/except ValueError and re-raise as BadRequest so errors surface through the GraphQL API 16.

Recent Changes#

• PR #10766 ("feat: add custom providers to model menu", Dec 2025) — Added end-to-end support for selecting custom providers in the playground model menu, including GenerativeModelKind.CUSTOM classification and proper DB-record-based provider resolution to prevent request/DB mismatches. Introduced a Private Use Area (PUA) delimiter (\uE000) for safe model-name key encoding in the frontend 17.

• PR #11726 ("fix: make name non-nullable for output configs", Feb 2026) — Tightened evaluator output config validation by introducing explicitly typed output config models with non-nullable name fields, database-enforced uniqueness checking, and converting ValueError to BadRequest in GenerativeModelCustomerProviderConfigInput for better HTTP semantics 18.

Key Files#