Dataset Deletion#
Deleting a dataset in Dify is a multi-step async process that must clean up: vector store collections/tables, database rows (documents, segments, metadata, app joins), and file storage objects. The logic is intentionally distributed across an event signal, a Celery task, and the index processor abstraction layer to handle failures in each layer independently.
Event-Driven Trigger#
Deletion is initiated in DatasetService.delete_dataset, which:
- Emits the
dataset_was_deletedBlinker signal — before deleting the dataset row — so handlers can still read dataset properties. - Deletes the
Datasetrow and commits.
The signal is defined in api/events/dataset_event.py, and the handler in clean_when_dataset_deleted.py immediately enqueues a Celery task via .delay() .
Guard: If
dataset.doc_formordataset.indexing_techniqueis missing, the handler returns early without enqueuing — this prevents the task from running on datasets that were never indexed .
clean_dataset_task (Celery, dataset queue)#
clean_dataset_task is the core cleanup routine. Key behaviours:
doc_formfallback: Ifdoc_formisNoneor blank, it falls back toPARAGRAPH_INDEXso the vector DB cleanup can still run .- Vector DB cleanup is isolated:
index_processor.clean()is wrapped in its owntry/except; failure logs an error but does not abort the rest of the cleanup . This resilience was added after billing API failures were found to crash the entire task . - Unconditional segment deletion:
DocumentSegmentrows are deleted using adataset_id-scoped query outside the documents-existence check, so orphans left by a concurrentclean_document_taskare always cleaned up . This was a deliberate fix for a race condition . - Full DB cleanup: Removes
DatasetProcessRule,DatasetQuery,AppDatasetJoin,DatasetMetadata,DatasetMetadataBinding,Pipeline/Workflow(if apipeline_idexists), and sourceUploadFilerecords . - Rollback on error: The outer
exceptblock rolls back the session before logging, preventing dirty session state .
Known Bugs & Race Conditions#
| Issue | Root Cause | Fix |
|---|---|---|
| Orphaned segments | clean_document_task deletes document rows before clean_dataset_task runs; the latter then skipped segment cleanup because documents was empty | Moved DocumentSegment bulk-delete outside the if documents block |
| Vector DB skipped for empty datasets | index_processor.clean() was inside the if documents conditional, skipping all 33 vector DB backends when no documents existed | Moved vector DB cleanup before the if documents block |
| Segment-index race condition | Async index-delete task queried child-chunk IDs after DB rows were already deleted | Pre-compute ChildChunk.index_node_id values before deletion; pass them directly to the async task |
| Billing API crash | Vector(dataset) construction inside index_processor.clean() triggered BillingService network calls; any failure propagated uncaught | Wrap index_processor.clean() in try/except; introduce _LazyEmbeddings proxy to defer billing calls |
| Infinite loop / disk exhaustion | In _delete_records (used by remove_app_and_related_data_task), a break inside the inner for loop on exception only exits the inner loop, not the outer while True; a poison-pill record triggers endless retries generating 95 GB+ of logs | Replace break with continue; add a success_count circuit-breaker to exit the outer loop when an entire batch makes no progress |
Key Source Files#
| File | Purpose |
|---|---|
api/events/dataset_event.py | Blinker signal definition |
api/events/event_handlers/clean_when_dataset_deleted.py | Signal handler → enqueues Celery task |
api/tasks/clean_dataset_task.py | Celery task: vector DB + DB row + file storage cleanup |
api/services/dataset_service.py (line ~1346) | delete_dataset — emits signal, deletes row |
api/tasks/remove_app_and_related_data_task.py (line ~696) | _delete_records — batched deletion utility with known infinite-loop bug |
