LiteLLM Integration#

Overview#

LitellmProvider is a Keep AI provider (category ["AI"]) that routes LLM calls through a LiteLLM proxy server via its OpenAI-compatible /chat/completions endpoint. Because LiteLLM acts as a unified gateway for 100+ LLMs, this provider lets Keep workflows target any supported model (OpenAI, Anthropic, local Ollama, etc.) without swapping providers. The implementation lives in keep/providers/litellm_provider/litellm_provider.py.

Authentication & Configuration#

Auth is defined in LitellmProviderAuthConfig:

When api_key is set, it is sent as Authorization: Bearer <key> 1.

_query() — Core Request/Response Flow#

_query() is the single entry point for all LLM calls. Key parameters:

• prompt — user message content
• model — defaults to "gpt-3.5-turbo" 2
• temperature — defaults to 0.7 3
• max_tokens — defaults to 1024 4
• structured_output_format — optional JSON schema dict; triggers special handling (see below)

The method POSTs to {api_url}/chat/completions with a 60-second timeout 5 and extracts the reply from result["choices"][0]["message"]["content"] 6. Any requests.exceptions.RequestException is re-raised as ProviderException 7.

Structured Output#

When structured_output_format is provided, the provider injects a system message before the user prompt instructing the model to return JSON conforming to the schema 8. After extraction it calls json.loads() on the reply; on failure it raises ProviderException with the raw text and model response for debugging 9.

Example workflow usage — see enrich_using_structured_output_from_openai.yaml, where the step uses structured_output_format to make the model return environment and impacted_customer_name fields; the same YAML explicitly notes this can also use LiteLLM 10.

Response Validation & Error Recovery (PR #6671)#

PR #6671 (fix(litellm): fail loudly on null content and accept fenced JSON) addresses two failure modes introduced by reasoning models and instruction-following quirks:

1. Null Content Detection#

Reasoning models (e.g. o1, DeepSeek-R1) can exhaust the entire max_tokens budget on internal reasoning tokens, returning content: null. Previously this produced a silent {"response": None} success. The fix raises ProviderException with the finish_reason and guidance to raise max_tokens or use a non-reasoning model 11.

2. Markdown Code Fence Stripping#

Some models wrap JSON replies in json … fences even when told not to. The new _strip_code_fence() static method:

1. Strips the opening `````
2. Optionally removes a leading json language tag
3. Strips the closing `````
4. Returns the input unchanged if no fence is detected (safe no-op)

_strip_code_fence() is applied only when structured_output_format is set, preserving backward compatibility for plain-text responses 11.

Workflow Integration#

In a Keep workflow YAML, the provider is used as a step 12:

steps:
  - name: Query litellm
    provider: litellm
    config: "{{ provider.my_provider_name }}"
    with:
      prompt: {value}
      model: {value}
      temperature: {value}
      max_tokens: {value}
      structured_output_format: {value}

Step results are accessed at steps.<step-name>.results.response. When structured_output_format is used, results.response is a parsed Python dict/object 13.

Key Files & References#