LLM-as-a-Judge Evaluation#

LLM-as-a-Judge is Langfuse's automated evaluation system that uses a configured LLM to score traces, dataset runs, and other observability events against a defined rubric. The judge LLM receives a prompt template (with interpolated trace variables) and must return structured JSON — there is no fallback for plain-text responses.

Supported LLM Providers#

Six adapters are natively supported 1:

OpenAI-compatible third-party providers (e.g., BigModel/GLM, Qwen) can be configured as a custom openai adapter with a custom baseURL, but structured output support depends entirely on the upstream provider 2.

Structured Output: The Hard Requirement#

Every eval call goes through fetchLLMCompletion with a structuredOutputSchema — a Zod schema derived at runtime from the evaluator's output definition. The LangChain withStructuredOutput() method is used to enforce schema compliance at the provider level. There is no fallback parsing of plain-text responses 2.

Score Types and Schemas#

The RawEvalOutputResult type allows score: number | boolean | string | string[] 3. buildResultSchemaForResolvedOutputDefinition builds the correct Zod schema at runtime:

• Numeric → z.number()
• Boolean → z.boolean()
• Categorical (single) → z.enum([...categories])
• Categorical (multi-match) → z.array(...).min(1) with uniqueness enforcement

Thinking/Reasoning Model Quirks#

For adapters that emit "thinking" blocks (VertexAI, GoogleAIStudio), fetchLLMCompletion forces method: "functionCalling" when calling withStructuredOutput 4. This prevents reasoning blocks from corrupting JSON schema parsing. For OpenAI reasoning models (o1, o3, o4-mini, gpt-5 series), isOpenAIReasoningModel is used to switch from maxTokens to maxCompletionTokens 5.

Default Evaluator Model Validation#

Before a model can be used for evaluations, it passes a mandatory live test call via DefaultEvalModelService.upsertDefaultModel, which calls testModelCall. The test sends this prompt to the model:

"Extract a score (1-5) and reasoning from this text: 'This is a test. It worked perfectly because it matched all passing criteria.'"

The default schema used during the test is 6:

z.object({ score: z.string(), reasoning: z.string() })

Known issue: This test schema uses z.string() for score, while actual evaluation schemas use z.number() for numeric evaluators. A model that returns {"score": 5} will fail the setup test even though it would work at runtime. This was partially fixed in PR #12540 (v3.161.0) but the testModelCall default schema still uses z.string() 2. A caller may pass a custom structuredOutputSchema to testModelCall to override this.

If the test call fails, upsertDefaultModel throws ForbiddenError: "Model configuration not valid for evaluation. <error message>" 7. This error surfaces in the UI and prevents saving the configuration.

Evaluator Blocking#

Once an evaluator is saved and running, failures during execution can automatically block it. The full blocking system is documented in the Evaluator Configuration and Status Management knowledge article. Key blocking reasons relevant to structured output / model config:

Key Files#