Provider Model Discovery#

RAGFlow supports two ways to populate a provider's model list: a static catalog from conf/llm_factories.json and a live remote fetch via provider-specific classes in rag/llm/model_meta.py. The two lists are merged at request time, with remote results winning on name conflicts 1.

This system was introduced in PR #15711 ("Feat: get model list from remote").

The ModelMeta Registry#

rag/llm/model_meta.py defines a Base abstract class and a set of provider-specific subclasses. Each subclass carries a _FACTORY_NAME string and implements get_model_list(), which hits the provider's live API and returns a list of {name, model_types, features, max_tokens} dicts.

The ModelMeta dict is populated at import time in rag/llm/__init__.py: the auto-discovery loop inspects all members of the model_meta module, finds every subclass of Base that has _FACTORY_NAME, and registers it 2. A _FACTORY_NAME can be a list to register a single class under multiple keys 3.

Providers registered in ModelMeta (as of the current codebase):

OpenAIAPICompatible._infer_model_types() classifies names by substring hints (embed, bge, rerank, whisper, vl, gpt-4o, etc.) 13. This approach is fragile for arbitrary model names — a known limitation acknowledged in the design.

How list_provider_models Uses ModelMeta#

provider_api_service.list_provider_models() is the merge point:

1. Build static_llms from FACTORY_LLM_INFOS (the JSON catalog).
2. If the provider name is in ModelMeta, call await ModelMeta[provider_name](api_key, base_url).get_model_list() to get remote_models 14.
3. Merge into a dict keyed by model name, remote overrides static, sort by name, return 15.

The HTTP endpoint GET /providers/<provider>/models proxies this, accepting optional api_key and base_url query parameters to support pre-save discovery (i.e., before an instance is saved).

Two-Phase Provider Connection Verification#

When a provider instance is created or updated, verify_api_key() runs a live probe in two phases:

Phase 1 — Model list: Uses the model_info list passed by the caller (or falls back to the static catalog from llm_factories.json) to determine which model types to probe 16.

Phase 2 — Live probe per type: Iterates through model types and fires one probe per capability, stopping as soon as any type passes 17:

Each probe runs inside asyncio.wait_for(..., timeout=timeout_seconds) where the timeout defaults to 10 s and is overridden by LLM_TIMEOUT_SECONDS 18. Results are persisted as ModelVerifyStatusEnum.SUCCESS or FAIL in each model's extra.verify field and surfaced by list_instance_models().

The standalone POST /providers/<provider>/connection endpoint 19 can run verification independently (e.g., before creating an instance), and if instance_id is supplied, it writes the per-model verify results back to the DB.

GPUStack's Integration Gap#

GPUStack has full chat-model support via LiteLLMBase with the openai/ prefix 20, a Go-layer driver added in PR #15024, and is enumerated in SupportedLiteLLMProvider. However, GPUStack has no ModelMeta class in model_meta.py 21.

Practical consequence: calling GET /providers/GPUStack/models with credentials returns only the static entries from llm_factories.json — which for a self-hosted provider with user-supplied models is typically empty. Dynamic model discovery (fetching ${base_url}/v1/models) does not fire for GPUStack.

Since GPUStack's API is OpenAI-compatible, adding a ModelMeta class is straightforward: subclass OpenAIAPICompatible with _FACTORY_NAME = "GPUStack". GPUStack's Go driver already implements ListModels via GET ${base}/v1/models 22, confirming the endpoint is available.

Key Source Files#