DatasetEvaluator#

DatasetEvaluator is a first-class GraphQL node in Phoenix representing the scoped connection between a Dataset and an Evaluator. It controls how an evaluator (LLM-based, code-based, or built-in) is configured and run against a specific dataset, storing the input mapping and lifecycle metadata alongside the link.

The table below summarizes the three evaluator kinds and their deletion behavior:

GraphQL Schema & Evolution#

The DatasetEvaluator node was introduced in PR #10511 by promoting the datasets_evaluators join table into a proper GraphQL type. The promoted node gained fields displayName, updatedAt, dataset, evaluator, and inputMapping; mutations were updated to return DatasetEvaluatorMutationPayload; and new dataset-scoped queries datasetEvaluator / datasetEvaluators were added.

Subsequent schema changes:

• Name field (PR #11052): display_name was renamed to name with an Identifier type and the unique constraint tightened to (dataset_id, name).
• Mutation removals: createCodeEvaluator was removed in PR #11127; the unused and buggy deleteEvaluators mutation was removed in PR #13104.
• Built-in kind (PR #11094): BUILTIN was added to the EvaluatorKind enum after introduction of the persistent builtin_evaluators table.

Built-in Evaluator Table & Sync#

PR #11094 introduced a builtin_evaluators database table backed by an in-memory evaluator registry. On startup, sync_builtin_evaluators() (in builtin_evaluator_sync.py) upserts registry-defined entries by stable key values and removes stale rows. This replaced the previous builtin_evaluator_id foreign key: all evaluator kinds now route through a single evaluator_id on dataset_evaluators.

PR #11250 standardized all built-in evaluator identifiers to snake_case ("ExactMatch" → "exact_match", "LevenshteinDistance" → "levenshtein_distance", etc.) and removed the CamelCase-to-snake_case conversion helper from the sync path — the server now persists evaluator_cls.name directly.

Data Persistence Bugs During Creation#

Two distinct creation-time bugs were fixed:

1. Missing input_mapping (PR #10885): create_dataset_llm_evaluator failed to persist input_mapping as a plain dict, causing the evaluator to crash on first run. Fix: store input.input_mapping.to_dict() or default to {"literal_mapping": {}, "path_mapping": {}}. The mutation response was also updated to include the PlaygroundDatasetSection_evaluator fragment so the UI receives all required fields immediately after creation.

2. Double instantiation (PR #11127): create_dataset_builtin_evaluator instantiated the ORM object twice — once outside and once inside the DB session — silently dropping output_config_override, description, and user_id. Fix: single instantiation inside the session. The same PR also added a token_hex(12) suffix to evaluator project names to prevent unique-constraint violations when recreating evaluators.

Cascade Deletion Semantics#

PR #13104 fixed a critical bug (issue #13090) where deleting a dataset's link to a BUILTIN evaluator would cascade-delete the shared BuiltinEvaluator row, silently breaking all other datasets referencing the same built-in.

Current deleteDatasetEvaluators behavior:

1. Always deletes the DatasetEvaluators link row.
2. BUILTIN evaluators: the underlying evaluator definition is never touched.
3. LLM/CODE evaluators: the Evaluator row is deleted only when no other DatasetEvaluators row references it (enforced via a NOT EXISTS predicate).
4. Prompts: optionally deleted if requested and unreferenced.

Database-specific implementation: PostgreSQL uses a data-modifying CTE 1 to fold the "find orphaned evaluators" and "delete links" steps into one round-trip; SQLite uses a portable SELECT-then-DELETE pattern.

Key Source Files#