File Storage Synchronization#

Dify stores file metadata (path key, tenant, type, size) as UploadFile rows in PostgreSQL while the actual bytes live in the configured storage backend (local filesystem via OpenDAL, S3, MinIO, etc.). These two stores must stay in sync: a database row with no corresponding physical file causes a FileNotFoundError in opendal_storage.load_stream, which surfaces as a 502 upstream error and a 404 on /file-preview endpoints 1.

The inverse problem — physical files with no DB row — wastes disk space but does not cause runtime errors.

Normal deletion paths keep the stores in sync:

• Single-file deletion — FileService.delete_file wraps storage.delete() and session.delete() in a single session.begin() transaction. Both succeed or both roll back.
• Document cleanup — clean_document_task first collects all file keys, commits DB deletions in explicit transactions, then deletes the physical files with per-file try/except blocks so partial storage failures don't leave orphaned DB rows.
• Dataset cleanup — clean_dataset_task follows the same pattern with isolated vector DB cleanup 2.

Sync breaks when operations bypass this transaction management.

Common Causes#

The most commonly reported cause is an out-of-band cron script that deletes files from ./volumes/app/storage/upload_files/ directly 1. After a major version upgrade, the missing-file errors become visible because older versions may have tolerated the gap differently.

Diagnosing a Sync Gap#

1. Check which DB-referenced files are missing from disk:

docker compose exec db psql -U postgres -d dify \
  -c "SELECT key FROM upload_files;" -t > /tmp/db_files.txt

while read -r key; do
  [ ! -f "./volumes/app/storage/$key" ] && echo "MISSING: $key"
done < /tmp/db_files.txt

2. Verify the container can see storage at all:

docker compose exec api ls /app/api/storage/upload_files/<tenant-id>/

If this fails, the volume mount (./volumes/app/storage:/app/api/storage) is broken — not a sync issue.

3. Check OPENDAL_FS_ROOT matches the mount path (default: storage) 1. A misconfigured root causes every file lookup to fail regardless of whether the file exists.

Remediation#

Dify ships two CLI commands for reconciliation 4:

# Fix: DB records pointing to missing files
docker compose exec api flask clear-orphaned-file-records

# Fix: storage files with no DB record
docker compose exec api flask remove-orphaned-files-on-storage

If files are genuinely lost (not misrouted), re-upload them through the Dify UI after running clear-orphaned-file-records.

Prevention#

Do not delete files directly from the storage directory. Use Dify's UI/API delete flows, which trigger the transactional cleanup tasks above.

Dify provides built-in scheduled cleanup tasks (all disabled by default) as alternatives to custom cron scripts 5:

These tasks clean both DB records and storage files together, respecting the PLAN_SANDBOX_CLEAN_DAY_SETTING retention period (default: 30 days) 6.

For upgrades: before bringing up the new container, verify files are accessible inside the container with docker compose exec api ls /app/api/storage/upload_files/ and confirm the volume mount in docker-compose.yml points to the correct path from the previous deployment.