Eval Job Execution#

Overview#

Eval job execution in Langfuse is a two-phase pipeline: job creation (deciding which traces/observations to evaluate and persisting JobExecution records) followed by job execution (fetching data, calling the LLM judge, and writing score events back through the ingestion queue). Both phases are handled in the worker service under worker/src/features/evaluation/.

The primary source files are:

• evalService.ts — job creation and LLM-as-a-judge execution logic
• evalScoreEvent.ts — score event construction
• evalExecutionDeps.ts — dependency injection for LLM calls, S3 uploads, queue enqueue
• evalRuntime.ts — prompt compilation and message building utilities

Phase 1: Job Creation (createEvalJobs)#

createEvalJobs is triggered by three event types:

Steps:

1. Fetch active job configs from Postgres (jobType=EVAL, status=ACTIVE, blockedAt=null) filtered by project and optionally by configId or timeScope 1.
2. Anti-loop guard: skip job creation for traces whose environment starts with "langfuse-" when the source is trace-upsert, preventing eval → eval trace → eval infinite loops 2.
3. Cache optimization: with multiple configs, pre-fetch the trace once (excluding input/output, including metadata for in-memory filter evaluation) and pre-fetch dataset item IDs from ClickHouse 3.
4. Dedup existing jobs: batch-query existing JobExecution rows matched by (jobConfigurationId, jobInputTraceId, jobInputDatasetItemId, jobInputObservationId) 4.
5. Per-config loop: for each active config, evaluate trace filters (in-memory if possible, DB fallback otherwise), validate observation existence if an observationId is set, apply sampling (config.sampling probability), and deduplicate 5.
6. Persist JobExecution with the following identifier fields 6:
   • jobInputTraceId — the trace being evaluated
   • jobInputTraceTimestamp — used for ClickHouse time-range queries during execution
   • jobInputDatasetItemId / jobInputDatasetItemValidFrom — dataset item context (if applicable)
   • jobInputObservationId — observation-level targeting (if applicable)
7. Enqueue to EvalExecutionQueue with a sharding key {projectId}-{jobExecutionId} and an optional delay (in ms) 7.

If a second trace event "deselects" an already-queued job (filter no longer matches), the existing job is set to CANCELLED 8.

Phase 2: Job Execution (evaluate / executeLLMAsJudgeEvaluation)#

The BullMQ worker picks up the queued event and calls evaluate, which:

1. Loads the JobExecution, JobConfiguration, and EvalTemplate from Postgres.
2. Checks isJobConfigExecutable — cancels the job if the config is blocked or inactive.
3. Calls extractVariablesFromTracingData to resolve template variables:
   • dataset_item → Postgres lookup by jobInputDatasetItemId + validFrom
   • trace → ClickHouse lookup by jobInputTraceId + jobInputTraceTimestamp
   • observation types → ClickHouse lookup by trace ID + observation name (mapping.objectName)
   • Results are internally cached per function call to avoid redundant lookups.
4. Calls executeLLMAsJudgeEvaluation.

Inside executeLLMAsJudgeEvaluation#

This is the shared core for both trace-level and observation-level evaluations 9:

1. Compile prompt: compileEvalPrompt interpolates extracted variables into the template string; buildEvalMessages wraps the result into a single ChatMessageType.User message.
2. Validate output definition: compilePersistedEvalOutputDefinition normalizes the template's outputDefinition and produces a Zod schema for structured output.
3. Fetch model config: via deps.fetchModelConfig (backed by DefaultEvalModelService). On invalid config, blocks the evaluator config with reason INVALID_MODEL_CONFIG.
4. Generate executionTraceId: a W3C-format trace ID derived from jobExecutionId, used to link the internal LLM call trace back to the eval job 10.
5. Build execution metadata: buildEvalExecutionMetadata produces a snake_case key/value record containing job_execution_id, job_configuration_id, target_trace_id, target_observation_id, and target_dataset_item_id.
6. Call LLM: deps.callLLM passes the structured output schema to fetchLLMCompletion (via withStructuredOutput). The call is traced under LangfuseInternalTraceEnvironment.LLMJudge to avoid triggering further eval jobs 11.
7. Validate LLM output: validateEvalOutputResult runs safeParse against the compiled schema.
8. Build and persist score events: buildEvalScoreWritePayloads → upload to S3 → enqueue to IngestionQueue.
9. Mark job complete: updates status=COMPLETED, records jobOutputScoreId and executionTraceId 12.

Score Event Construction (buildEvalScoreWritePayloads)#

buildEvalScoreWritePayloads converts an EvalOutputResult into one or more EvalScoreWritePayload objects:

Each ScoreEventType body includes 13:

• traceId / observationId — identifiers from the JobExecution record
• executionTraceId — links the score back to the internal LLM judge trace
• metadata — the snake_case execution metadata record
• source: ScoreSourceEnum.EVAL
• comment — populated with the LLM's reasoning output

Score events are written to S3 via deps.uploadScore and enqueued to the IngestionQueue via deps.enqueueScoreIngestion 14, then processed by the standard ingestion pipeline where validateAndInflateScore handles configId resolution.

Key Identifiers Flowing Through the Pipeline#

Related Articles#

• LLM-as-a-Judge Evaluation — supported providers, structured output, model validation
• Eval Output Schema — output definition types, schema compilation, score write path
• Evaluator Configuration and Status Management — blocking, deactivation, config guards
• BullMQ Worker Lifecycle — queue registration, stall handling, metrics