Evaluator Management UI#

The evaluator management UI is centered on the EvaluatorsModal — a three-step wizard modal that handles browsing, creating, and configuring evaluator configurations in the auto-evaluation flow. It is surfaced via the SelectEvaluatorSection component 1 inside the New Evaluation page.

The modal manages all state internally (evaluators, configs, testsets, selected variant, view mode) and dispatches between three steps driven by a current index 2:

Step 2 is conditionally appended to the steps array only when a template is selected 3. The modal width adapts dynamically — 600px at step 2 without debug panel, 90vw otherwise, with a min-w-[800px] and max-w-[1800px] clamp 4.

Component Architecture#

All components live under web/oss/src/components/pages/evaluations/autoEvaluation/EvaluatorsModal/.

Step 0 — Evaluators 5: Displays existing evaluator configurations in either a list table (EvaluatorList) or card grid (EvaluatorCard), toggled by a Radio.Group icon-button pair. View preference is persisted to localStorage under the key "evaluator_view" 6. Each config supports edit, clone, and delete actions; deletion is guarded by a DeleteModal that validates whether the resource is still in use.

Step 1 — NewEvaluator 7: A template browser displaying all registered evaluator types. Renders only the list view (NewEvaluatorList); a card variant (NewEvaluatorCard) also exists. Selecting a template sets selectedEvaluator and advances to step 2 8.

Step 2 — ConfigureEvaluator 9: A dynamic form for the chosen template. Uses DynamicFormField (supports Monaco editor for JSON/code inputs), AdvancedSettings for collapsible advanced options, and Messages for chat-based role configuration (system/user/assistant). Supports create, edit, and clone modes via editMode and cloneConfig flags 10.

After a successful save, if openedFromNewEvaluation is true the modal closes; otherwise it returns to step 0 11.

Tag-Based Filtering#

Both the Evaluators step and the NewEvaluator step use getEvaluatorTags() from web/oss/src/lib/helpers/evaluate.ts to populate their filter tab bars.

The four base categories are Classifiers, Similarity, AI / LLM, and Functional. In demo mode, a RAG category is prepended 12.

Each step renders a Radio.Group of tab buttons. A "View all" default shows every item; selecting a tag filters by item.tags.includes(selectedEvaluatorCategory) 13. An Input.Search further narrows results by name 14. The two filters are composed via useMemo.

UI Pattern Evolution#

The evaluator selector UI has gone through several iterations:

PR #3256 — Dropdown → Popover (Dec 2025)
Fixed layout issues with the evaluator template selector by converting EvaluatorTemplateDropdown from an AntD Dropdown to a Popover, giving finer control over dimensions. Also introduced responsive drawer sizing with CSS clamp() and a togglable test panel.

PR #3257 — Inline creation drawers (Dec 2025)
The biggest UX inflection point. Users previously had to navigate away to create evaluators during the new-evaluation flow. This PR introduced CreateEvaluatorDrawer and CreateHumanEvaluatorDrawer for inline creation without context-switching, shared filtering utilities (evaluatorFiltering.ts with an ENABLED_EVALUATORS whitelist), and category tabs on the EvaluatorTemplateDropdown.

PR #4151 — Adapter-driven tabs (Apr 2026)
Extended the selection adapter system with a buildTabs?: (items: T[]) => TabDefinition[] hook for dynamic tab generation. Added controlled open/onOpenChange props to EvaluatorTemplateDropdown so parent components can manage open state. Expanded PopoverCascaderVariant significantly for this capability.

PR #4630 — Cascade selector polish (Jun 2026)
Introduced rich metadata in the cascading selector: version count and date subtitles via evaluatorWorkflowMetaDescription.ts, parent/child checkbox multi-select, bulk clear actions, and auto-opening of the child panel on popover open. These enhancements were localized to the playground evaluator selector.

Key Files Reference#