Experiments#

Experiments is a first-class Langfuse feature for evaluating prompts and model configurations against a dataset, entirely within the UI. It sits on top of the existing DatasetRuns Postgres table but layers a dedicated event-driven data model (getExperimentsFromEvents, getExperimentItemsFromEvents, etc.) that provides faster loading and richer filtering than the legacy Dataset Runs view.

The feature lives at:

• Entry page: web/src/pages/project/[projectId]/experiments.tsx
• Results page: web/src/pages/project/[projectId]/experiments/results.tsx
• Feature directory: web/src/features/experiments/

RBAC scopes:

• promptExperiments:CUD — create/trigger experiments 1
• promptExperiments:read — view experiments/results 2

Data Architecture: Experiments vs. Dataset Runs#

An Experiment is a DatasetRun record in Postgres, but with structured metadata and a dedicated processing pipeline:

On creation, createExperiment in the router 3 does two things:

1. Creates a DatasetRuns row with experiment metadata (prompt_id, provider, model, model_params, experiment_name, experiment_run_name) stored in the metadata field.
2. Enqueues a job on ExperimentCreateQueue → QueueName.ExperimentCreate 4. A background worker then iterates dataset items, calls the LLM for each, and writes traces/observations back via the ingestion pipeline.

URL translation between the legacy compare?runs=A&B format and the new experiments/results?baseline=A&c=B format is handled by experimentUrlTranslation.ts.

Creation Workflow#

The entry point is CreateExperimentsForm, which presents two modes:

1. Via UI — renders MultiStepExperimentForm, a 5-step breadcrumb wizard.
2. Via SDK/API — shows docs links and an optional webhook/remote-trigger configuration (RemoteExperimentUpsertForm / RemoteExperimentTriggerModal).

The 5-step wizard 5 steps are:

Form state is managed by react-hook-form with CreateExperimentData Zod schema 7. The experiment name and runName are auto-generated from the selected prompt/version/dataset 8. Submission calls api.experiments.createExperiment.useMutation 9.

CreateExperimentsForm is also embedded in the Prompt detail page (prompt-detail.tsx) so an experiment can be launched directly from a prompt version 10.

Experiments List & Filtering#

ExperimentsTable at /project/[projectId]/experiments shows all experiments for a project with:

• Columns: name, description, item count, error count, start time, linked dataset (badge), linked prompts (badges), avg latency, total cost, and score sub-columns 11.
• Score sub-columns come in three flavors rendered via useScoreColumns: trace-level item scores, observation-level item scores, and experiment-level scores 12.
• Dataset filter: the experimentDatasetId filter option lets users scope the table to a specific dataset. Filter options (including available dataset IDs and score names) are fetched via experiments.filterOptions 13.
• Compare action: selecting multiple rows shows a "Compare (N)" button that navigates to /experiments/results?baseline=<first>&c=<others> 14.
• Row click navigates to the results page with that experiment as baseline 15.

The page is guarded by a beta toggle (useExperimentAccess + ExperimentsBetaSwitch). Users without the beta enabled see a feature-opt-in screen; disabling the toggle redirects back to /datasets 16.

Results & Comparison View#

The results page (results.tsx) centers on ExperimentItemsTable, which renders one row per dataset item. Each row stacks the output, scores, latency, and cost from the baseline experiment and any number of comparison experiments side-by-side. Up to 5 color-coded slots are supported (baseline + 4 comparisons).

The router's items procedure fetches per-item data across all selected experiment IDs, supporting:

• baseExperimentId / compExperimentIds for multi-experiment comparison 17.
• filterByExperiment — per-experiment filter arrays, enabling dataset-item-level filtering scoped to a specific run 18.
• itemVisibility toggle (baseline-only vs all) to show only items present in the baseline or all items across all selected experiments 19.

An ExperimentOverviewPanel sidebar shows aggregate metrics for each selected experiment.

Key files quick reference: