Observation Eval Scheduling#

Observation-level evaluators (job configs with targetObject = EVENT or EXPERIMENT) are scheduled through a separate pipeline from trace-level evaluators, and are only triggered by the OTel ingestion path. Understanding this split is critical for debugging missing eval scores.

Core architectural split:

Both paths ultimately call the same executeLLMAsJudgeEvaluation() core 1, but differ in trigger mechanism, config fetch, and variable extraction source.

Ingestion Paths#

OTel Ingestion — the only path that auto-triggers observation-level evals#

The otelIngestionQueueProcessor handles this end-to-end:

1. Downloads raw ResourceSpans from S3, converts to events, and concurrently writes traces (via processEventBatch) and observations (via IngestionService.mergeAndWrite) to ClickHouse 2.
2. After writes complete, calls processor.processToEvent(parsedSpans) to produce enriched event records carrying trace-level attributes (userId, sessionId, tags, release) 3.
3. Calls fetchObservationEvalConfigs for the project. If configs exist, runs scheduleObservationEvals() per observation.

This enrichment step — provided by OtelIngestionProcessor.processToEvent() — is what makes observation-level scheduling possible and is absent from the legacy ingestion path.

Legacy SDK Ingestion — trace-level evals only#

The IngestionService (used by langfuse SDK v3 client.generation() / client.trace() calls) writes observations to ClickHouse and publishes to TraceUpsertQueue 4. The queue consumer calls createEvalJobs() with sourceEventType: "trace-upsert", which only processes TRACE/DATASET-targeted configs. scheduleObservationEvals() is never called.

This is confirmed intentional behavior by Langfuse maintainers — the legacy flow will not be extended 5. Observation-level evaluators require migrating to OTel-based SDKs:

• Python SDK v4+
• JS/TS SDK v5+
• Or OTel-native ingestion version 4 (x-langfuse-ingestion-version: 4 header)

The log message "No active evaluation jobs found" / "Cached no traceBased eval configs" appearing after legacy-SDK ingestion is expected — it is emitted by the trace-level path, which is unaware of observation eval configs 5.

Batch Actions (ui-create-eval)#

Uses createEvalJobs() with sourceEventType: "ui-create-eval" — the trace-level path. Observation-level EVENT/EXPERIMENT evaluators are not triggered via this path 6.

Experiments (dataset-run-item-upsert)#

When a dataset run item is ingested, createEvalJobs() is called with sourceEventType: "dataset-run-item-upsert". Critical guard: if datasetItem.observationId is set, the function explicitly skips creating a trace-level eval job 7 — doing so would prematurely attach a score at the trace level. Observation-level scoring for experiment items is expected to flow through the OTel path's EXPERIMENT target object handling in scheduleObservationEvals.

scheduleObservationEvals Logic#

scheduleObservationEvals receives a pre-fetched list of observation eval configs and a single observation record:

1. Filter + sample: for each config, evaluates filter conditions via InMemoryFilterService and applies the config's sampling rate 8. Skips S3 upload if nothing matches.
2. S3 upload (once per observation, not per config): uploads the full observation payload for later variable extraction by the executor 9.
3. Per-config job creation 10:
   • Generates a deterministic jobExecutionId from configId:observationId
   • Upserts a JobExecution row with jobInputTraceId + jobInputObservationId
   • Enqueues on LLMAsJudgeExecutionQueue with delay: 0

Experiment configs additionally require the observation to be the experiment root span (observation.span_id === observation.experiment_item_root_span_id) to fire 11.

Key Files#

Debugging#

• Evals not triggering at all: check whether ingestion goes through the legacy SDK (will never trigger observation evals) vs. OTel SDK.
• Check if jobs are being created: filter traces in the UI by environment = langfuse-llm-as-a-judge 5.
• Observation-not-found retries: retryObservationNotFound uses exponential backoff (30s → 1m → 2m → 4m, max 5 retries) when an eval job requires a specific observation that hasn't arrived yet 12.
• "No active evaluation jobs found" log: this is emitted by the trace-level path and is expected when using observation-targeted evaluators with legacy ingestion 5.