LLM Provider Integration#

RAGFlow supports a large catalog of LLM providers through a unified abstraction layer. Chat models (and most other model types) are dispatched through LiteLLM via the LiteLLMBase class in rag/llm/chat_model.py, while providers with custom auth or endpoint conventions get provider-specific branches inside that same class.

Three dictionaries in rag/llm/__init__.py define the provider catalog:

• SupportedLiteLLMProvider — the authoritative enum of all LiteLLM-routed providers, including Ollama, GPUStack, OpenAI, Azure-OpenAI, and ~40 others.
• LITELLM_PROVIDER_PREFIX — maps each provider to the LiteLLM model prefix (e.g., "ollama_chat/" for Ollama, "openai/" for GPUStack, "azure/" for Azure OpenAI).
• FACTORY_DEFAULT_BASE_URL — maps providers to their default API base URL. Providers without a managed endpoint (e.g., Ollama) have an empty string and require a user-supplied base URL.

At import time, rag/llm/__init__.py auto-discovers all model class submodules (chat_model, embedding_model, etc.) and populates dictionaries like ChatModel, EmbeddingModel, RerankModel, etc., keyed by _FACTORY_NAME.

Provider Instance Configuration and Credential Storage#

Since v0.27.0, providers are stored in a three-tier hierarchy: TenantModelProvider → TenantModelInstance → TenantModel 1. Each TenantModelInstance row holds the api_key, optional base_url, and a JSON extra blob for per-instance overrides.

The high-level CRUD and verification logic lives in api/apps/services/provider_api_service.py. Key functions:

Two normalization helpers run before storage and verification:

• _normalize_provider_base_url — For VLLM only: strips trailing / and appends /v1 if missing.
• _normalize_provider_api_key — For VLLM with no key: injects the placeholder "x" so the empty-key code path is never hit.

GPUStack uses a different URL joining strategy: urljoin(self.base_url, "v1") is applied inside _construct_completion_args at request time 2, rather than at storage time. A general ensure_v1 utility in rag/utils/url_utils.py handles the same normalization for other paths 3.

API Key Verification Flow#

verify_api_key runs automatically on every create_provider_instance and (by default) on update_provider_instance. It iterates through the provider's model list and fires a live probe for each model type — stopping as soon as one type succeeds 4:

Each probe is wrapped in asyncio.wait_for(..., timeout=timeout_seconds) 5. The timeout defaults to 10 seconds and is overridden by the LLM_TIMEOUT_SECONDS environment variable 5. Each model's result is recorded as ModelVerifyStatusEnum.SUCCESS or FAIL and persisted in the model's extra.verify field.

Per-model verify status is surfaced via list_instance_models, which returns "verify" alongside name, model_type, max_tokens, and features.

Ollama-Specific Behavior#

Ollama requires user-supplied base URLs and has no API key requirement. Several version-specific changes affect behavior:

v0.26.2 bug — mandatory API key: Adding an Ollama instance always failed with "Fail to access model(Ollama/…) using this api key" because the REST API validation required api_key for all non-VLLM providers 6.

v0.26.3 fix — optional API key (PR #16519): The create_provider_instance endpoint was changed to require only instance_name, making api_key optional. The hardcoded default model entry in llm_factories.json was also removed to enable dynamic model discovery 7.

Post-v0.26.3 timeout issue: Even after the fix, adding Ollama models could still fail if the verify_api_key chat probe (which sends a real streaming completion) took longer than the default 10-second timeout. This is particularly acute for qwen-series models with thinking enabled 7. Workaround: Set LLM_TIMEOUT_SECONDS=60 in docker/.env.

Docker networking: When RAGFlow runs in Docker, 127.0.0.1 resolves to the container, not the host. Use http://host.docker.internal:<port> as the base URL for a host-local Ollama instance 7.

Reverse-proxy Bearer auth: If Ollama sits behind a reverse proxy enforcing Bearer authentication, LiteLLMBase._construct_completion_args transparently injects Authorization: Bearer <api_key> in the request headers when an API key is present 8.

Key Source Files#

Related articles:

• Model Provider Architecture — TenantModelProvider → TenantModelInstance → TenantModel three-tier hierarchy and bit-flag model_type encoding
• LLM Configuration and Selection — runtime llm_id resolution, resolve_model_config, per-request override flow
• Database Migrations — v0.26.0 / v0.27.0 migration stages that created the three-tier tables