Eval Output Schema#

There are two distinct eval output schema systems in Langfuse, both of which converge on the same score event format at write-time:

1. LLM-as-a-Judge output definition — a persisted, user-configured schema that controls what the LLM must return. Defined in packages/shared/src/features/evals/outputDefinition.ts.
2. Code evaluator dispatcher contract — the runtime schema that validates what a user-written Python or TypeScript evaluate function returns. Defined in packages/shared/src/server/evals/codeEvalDispatcherTypes.ts.

LLM-as-a-Judge Output Definition Schema#

Source: outputDefinition.ts

Persisted schema (PersistedEvalOutputDefinition): a union of a legacy string-field format and the current v2 structured format 1. V2 schemas are versioned objects with a version: 2 discriminant 2:

Each V2 schema has a score field (description + optional categories/shouldAllowMultipleMatches for categorical) and a reasoning field, both with a description string used to populate the LLM prompt 2.

Runtime flow:

1. resolvePersistedEvalOutputDefinition normalizes legacy and v2 rows into ResolvedEvalOutputDefinition — a flat shape with dataType, reasoningDescription, scoreDescription, and (for categorical) categories + shouldAllowMultipleMatches.
2. buildEvalOutputResultSchema / compilePersistedEvalOutputDefinition produce a Zod schema passed to the LLM via withStructuredOutput(). The schema enforces:
   • Numeric → z.number()
   • Boolean → z.boolean()
   • Categorical (single) → z.enum([...categories])
   • Categorical (multi-match) → z.array(...).min(1) with uniqueness check 3
3. validateEvalOutputResult runs safeParse against the compiled schema and normalizes the raw { score, reasoning } object into the typed EvalOutputResult union 4:
   • { dataType: "NUMERIC", score: number, reasoning: string }
   • { dataType: "BOOLEAN", score: boolean, reasoning: string }
   • { dataType: "CATEGORICAL", matches: string[], reasoning: string }

Code Evaluator Score Contract (CodeEvalScore)#

Source: codeEvalDispatcherTypes.ts

The CodeEvalScoreSchema validates each score object returned by user-written code. It is a union over four dataType variants, plus a no-dataType fallback:

All fields are camelCase in this schema. Shared optional fields: name (required), value (required), dataType (required), comment, configId, metadata.

The full result envelope is { scores: CodeEvalScore[] } (min 1 score), validated by parseDispatchResult, which throws a CodeEvalDispatcherError with code INVALID_RESULT on failure.

Python vs TypeScript Case Conventions#

The user-facing function contract uses different field names depending on the runtime language 5:

The server-side CodeEvalScoreSchema accepts only camelCase. The case translation from Python snake_case to camelCase happens inside the dispatcher runtime (AWS Lambda handler for Python, or the insecure-local dispatcher for TypeScript). By the time scores reach parseDispatchResult, all field names must be camelCase. A Python evaluator returning { "data_type": "BOOLEAN" } instead of { "dataType": "BOOLEAN" } will fail this validation with an INVALID_RESULT error 6.

Score Event Write Path#

After dispatch validation, the buildEvalScoreWritePayloads function in evalScoreEvent.ts converts EvalOutputResult or CodeEvalScoreWithName objects into ScoreEventType envelopes. All fields remain camelCase (dataType, traceId, observationId, executionTraceId). These events are consumed by the standard ingestion pipeline where validateAndInflateScore handles configId lookups and final score hydration.

Key Files#

Public docs: Code evaluators — Score fields reference