Dataset Run Deletion#

Dataset run deletion is an async, two-phase process: the Postgres record is deleted synchronously, then ClickHouse run items (dataset_run_items_rmt) are cleaned up asynchronously via a BullMQ queue. This design was introduced in PR #8074 to handle ClickHouse writes correctly when deleting dataset-related entities.

Two deletion scopes are supported, differentiated by a deletionType discriminated union 1:

API Layer#

Both deletion paths live in dataset-router.ts and require the datasets:CUD RBAC scope:

• deleteDataset 2: Deletes the dataset row from Postgres, then calls addToDeleteDatasetQueue with deletionType: "dataset".
• deleteDatasetRuns 3: Accepts an array of datasetRunIds, deletes the DatasetRuns rows from Postgres, then calls addToDeleteDatasetQueue with deletionType: "dataset-runs" and the run IDs. The datasetId input is marked optional for backward-compat; if omitted, it falls back to the first run's datasetId.

Both mutations emit an auditLog entry for each deleted resource 4 5.

Queue Infrastructure#

Queue definition — DatasetDeleteQueue is a BullMQ singleton backed by a dedicated Redis connection 6. Job options: 2 attempts, 30 s exponential backoff, removeOnComplete: true, removeOnFail: 100_000.

Producer — addToDeleteDatasetQueue enqueues a QueueJobs.DatasetDelete job ("dataset-delete-job" on "dataset-delete-queue"). Payload schema is DatasetQueueEventSchema — a Zod discriminated union on deletionType 1.

Worker / processor — processClickhouseDatasetDelete switches on deletionType and calls the appropriate ClickHouse repository function:

deletionType: "dataset" → deleteDatasetRunItemsByDatasetId
deletionType: "dataset-runs" → deleteDatasetRunItemsByDatasetRunIds

ClickHouse Repository#

All three delete functions in dataset-run-items.ts issue DELETE FROM dataset_run_items_rmt parameterized queries with a configurable timeout (LANGFUSE_CLICKHOUSE_DELETION_TIMEOUT_MS):

• deleteDatasetRunItemsByProjectId — project-wide deletion (used during project deletion, not dataset-run deletion).
• deleteDatasetRunItemsByDatasetId — filters on project_id + dataset_id.
• deleteDatasetRunItemsByDatasetRunIds — filters on project_id + dataset_id + dataset_run_id IN (...).

UI Components#

Single run — DeleteDatasetRunButton is a reusable component rendered in the row-actions dropdown of DatasetRunsTable. It checks datasets:CUD access via useHasProjectAccess 7: if the user lacks the scope, the button renders disabled but visible (never hidden). If the user has access, the button opens a confirmation dialog before calling api.datasets.deleteDatasetRuns.useMutation with a single-element datasetRunIds array 8. On success, it either navigates to a redirectUrl or invalidates the dataset query cache 9.

Bulk delete — DatasetRunTableMultiSelectAction in DatasetRunsTable.tsx surfaces a dropdown "Actions" button when rows are selected. It uses the same deleteDatasetRuns mutation with all selected IDs, shows a confirmation dialog with a count-aware message, and clears row selection on success.

Both UI flows share the single datasets.deleteDatasetRuns tRPC mutation — there is no separate mutation for bulk vs. single deletion.

Key Files#