Evaluator Configuration and Status Management#

Langfuse evaluators (LLM-as-a-Judge jobs) have two distinct status mechanisms:

1. Blocking — automatic pausing when LLM/model configuration is missing or broken 1
2. Deactivation — manual ACTIVE/INACTIVE toggle with server-side guards that prevent certain changes once an evaluator has run on existing traces

Config Blocking#

Evaluators are automatically blocked when the worker detects a configuration problem during execution. The isJobConfigExecutable function gates all eval runs — a config must have status === ACTIVE and blockedAt === null to be executable. If blocked, the job execution is cancelled immediately.

Blocking is triggered from four sources 2:

Block Reasons & Messages#

Block reason, short label, and user-facing message are defined in evalConfigBlocking.ts:

Database Fields#

Three nullable columns on JobConfiguration track blocked state 1:

• blockedAt — timestamp when the block was set
• blockReason — EvaluatorBlockReason enum value
• blockMessage — human-readable remediation text

Notifications & Side Effects#

When blockEvaluatorConfigs is called, it:

1. Updates the DB record with blockedAt, blockReason, and blockMessage
2. Sends an email to project admins via notifyBlockedEvaluatorConfigs
3. Invalidates project eval config caches so workers stop picking up the config immediately

Frontend: Blocked State UI#

The EvaluatorPausedCallout component renders a yellow alert banner when evalConfig.blockedAt is set. It surfaces:

• The blockMetadata.shortLabel and elapsed time since blocked
• The full blockMessage from getEvaluatorBlockMetadata
• A context-aware "fix it" link via getEvaluatorBlockResolutionPath: LLM auth/missing issues resolve to /settings/llm-connections; other issues link to the evaluator template or general evals page
• A Reactivate button (calls updateEvalJob with status: ACTIVE), with onSuccess and onError toast notifications 3

Reactivation & Block Field Reset#

When updateEvalJob receives a status update, the router automatically clears block fields by merging resetEvalConfigBlockFields into the update payload 4. Before activating a previously-blocked config, the router calls validateEvalTemplateCanRun to confirm the underlying issue is resolved 5.

Server-Side Guards on updateEvalJob#

Beyond blocking, updateEvalJob enforces two immutability rules for evaluators that have already run on existing traces 6:

1. timeScope immutability — Once EXISTING is in timeScope, it cannot be removed. Throws BAD_REQUEST: "The evaluator ran on existing traces already. This cannot be changed anymore." 7

2. EXISTING-only deactivation guard — For legacy TRACE/DATASET targets (not EVENT), if timeScope is ["EXISTING"] only, the evaluator cannot be set to INACTIVE. Throws BAD_REQUEST: "The evaluator is running on existing traces only and cannot be deactivated." 8

The DeactivateEvalConfig switch pre-disables itself when timeScope === ["EXISTING"] 9, preventing the second guard from being triggered via the toggle UI.

Frontend Error Handling: Silent Failure & Fix#

Bug (issue #13315): DeactivateEvalConfig had no onError handler on the updateEvalJob mutation. When the server rejected a request (e.g., EXISTING-scope guard), the error was silently swallowed — the form reverted with no toast or inline message 10.

Fix (PR #14064): Added showErrorToast in the onError callback, consistent with the pattern in evaluator-paused-callout.tsx and inner-evaluator-form.tsx 11.

Known residual issue: The onClick handler still calls mutateAsync without await or .catch() 12, leaving a dangling unhandled promise rejection on failure. The recommended fix is switching to mutate (fire-and-forget), as evaluator-paused-callout.tsx does 13.

Key Files#