OpenAI-Compatible Server Integration#

Dify supports OpenAI-compatible inference servers — including vLLM, SGLang, Ollama, and OpenRouter — through two integration layers:

1. Model plugins in dify-official-plugins handle raw LLM calls (/v1/chat/completions) with SSE streaming response parsing.
2. Agent V2 (dify-agent microservice) handles tool-calling agent loops via pydantic-ai, and imposes stricter requirements on provider behavior. 1

Known compatibility issues cluster around three areas: SSE format validation, function calling requirements for Agent V2, and response parsing quirks in workflow and agent execution contexts. The plain Python OpenAI SDK is more tolerant of provider quirks than Dify's parsers — providers that pass the SDK may still fail in Dify. 2

SSE Format Validation and "Non-JSON encountered." Error#

Where it originates#

The OpenRouter and Ollama plugin LLM implementations share nearly identical SSE streaming parsers. Each line is processed as 3:

decoded_chunk = chunk.strip().removeprefix("data:").lstrip()

The .lstrip() after removeprefix("data:") handles both data:{JSON} (no space) and data: {JSON} (standard space) formats. When the decoded content fails JSON validation, the plugin yields a final chunk with finish_reason="Non-JSON encountered." and terminates the stream — it does not raise an exception.

How it surfaces in workflows#

In a workflow LLM block this appears as: 2

{
  "text": "",
  "finish_reason": "Non-JSON encountered.",
  "usage": { "prompt_tokens": 0, "completion_tokens": 0, ... },
  "latency": 1.0
}

Zero tokens consumed and sub-second latency indicate the error occurs before the model generates any output — the LLM call itself never succeeds.

Common root causes#

The stream_mode_delimiter credential field (default "\n\n") can be adjusted for providers that use alternative delimiters, such as "\n". 4

Agent V2 Function Calling Requirements#

Agent V2 (Dify ≥ 1.16.0) uses pydantic-ai as its execution engine and requires native OpenAI-style function calling — the tools parameter in requests and tool_calls field in responses. There is no built-in CoT/ReAct fallback in Agent V2. 5 1

vLLM configuration#

For vLLM-hosted models, function calling is not enabled by default. Required flags:

--enable-auto-tool-choice
--tool-call-parser <parser> # e.g., "mistral", "llama3_json", "hermes"

Not all models served by vLLM support function calling — it depends on the specific model and chat template. Models like Llama-3, Qwen2, and Mistral generally support it. 5

Finish reason normalization#

The adapter normalizes "tool_calls", "function_call", and "function_calls" finish reasons to a single "tool_call" value internally. 1

Fallback#

If the model does not support function calling, use the classic agent mode (ReAct/CoT), which is available in all Dify versions and does not require tool_calls support. 5

Deployment note#

Agent V2 is experimental in 1.15.0–1.16.0-rc1 and the dify-agent microservice is not included in the default docker-compose.yaml. 5

vLLM + Agent V2 Compatibility Bugs (Fixed in 1.16.x)#

Several specific incompatibilities with vLLM and similar OpenAI-compatible providers were fixed in the 1.16.0 timeframe:

Error messages swallowed (PR #38584)#

Agent V2 was returning a generic "Internal Server Error" instead of the real provider error (authentication failure, rate limit, bad request, etc.). Structured errors are now passed through with correct HTTP status codes. 6

Text history sent as part lists (PR #38464)#

Text-only assistant message history was serialized as part objects (e.g., [{"type": "text", "text": "..."}]) instead of plain strings. Providers like vLLM that don't handle multipart structures for text-only content would reject or misparse these. The fix collapses text-only parts into a single concatenated string. 7

Tool call ID normalization (PR #38592)#

vLLM may omit or send placeholder tool call IDs ("none", "null", empty string, whitespace). These caused distinct tool calls to collapse into the same thought record. _normalize_tool_call_id() converts such values to None, and stable fallback vendor_id values are generated from chunk sequence position. 8

Qwen/vLLM system message ordering (PR #39136, open)#

Qwen served via vLLM requires the system message to be the first element in the messages array. Previously, instruction messages could be inserted mid-array. The fix in _map_messages_to_prompt_messages() merges all system messages into one and places it at index 0. 9 10

Key Source Files and References#

Related Issues#

• #15120 — "Non-JSON encountered." with OpenAI-compatible models in workflows 2
• #38882 — Agent V2 incompatibility with vLLM-deployed models 5