File Upload Processing#

Dify's backend file upload pipeline handles incoming files through FileService.upload_file in api/services/file_service.py. This is the primary entry point for all uploads — knowledge base documents, workflow inputs, and batch CSV runs. The service enforces validation, sanitizes filenames, stores the file bytes in the configured storage backend (local filesystem via OpenDAL, S3, MinIO, etc.), and writes an UploadFile metadata row to PostgreSQL. Keeping these two stores in sync is a recurring concern; see File Storage Synchronization for details.

Filename Validation and Sanitization#

upload_file applies these checks in order:

Before PR #31412 3, filenames with other special characters like @ and | caused the upload to fail. The fix replaced rejection with automatic sanitization — invalid characters are replaced with underscores — so the upload succeeds with a cleaned filename.

For ZIP archive entries, the separate _sanitize_zip_entry_name method strips directory components (os.path.basename) and replaces any residual / or \\ with underscores. This prevents path traversal in exported or imported ZIP payloads.

The storage key itself is always UUID-based, so the original (or sanitized) filename is only metadata; it never influences where the bytes land on disk.

Encoding Detection#

Dify migrated from chardet to charset_normalizer (PR #29022, 4) for more accurate multi-language encoding detection. The library is now used in three places:

1. RAG file extraction — detect_file_encodings in api/core/rag/extractor/helpers.py calls charset_normalizer.from_path(...).best() inside a ThreadPoolExecutor with a 5-second timeout. The FileEncoding namedtuple carries encoding, confidence, and language.

2. CSV and text extractors — The CSV extractor 5 and text extractor 6 both attempt to open files with the specified encoding first; on UnicodeDecodeError with autodetect_encoding=True, they call detect_file_encodings() and retry through the returned candidates in confidence order.

3. Web content reader — web_reader_tool.py uses charset_normalizer.from_bytes(...).best() on HTTP response bodies, falling back to utf-8 or response.text if detection fails.

Bug history:

• PR #29022 introduced the charset_normalizer switch but the initial implementation returned raw results as dict-like objects. PR #29595 7 fixed a resulting TypeError: tuple indices must be integers or slices, not str in detect_file_encodings by switching from dict-key access (encoding["encoding"]) to attribute access (encoding.encoding).
• PR #31204 8 added frontend encoding detection for batch CSV uploads, using jschardet in the browser to decode GBK / GB18030 / BIG5 files before sending them to the backend, since the RAG backend path alone did not cover the batch-run CSV input flow.

Database–Storage Synchronization#

Every uploaded file has two representations that must stay in sync:

• PostgreSQL — UploadFile row with path key, tenant, MIME type, and size
• Storage backend — actual bytes addressed by a UUID-based key

A DB row with no physical file causes a FileNotFoundError in opendal_storage.load_stream 9, which surfaces as 502 errors and 404 responses on /file-preview endpoints.

Normal deletion paths maintain sync:

• FileService.delete_file wraps storage.delete() and session.delete() in a single transaction.
• clean_document_task collects file keys, commits DB deletions, then removes physical files with per-file try/except blocks.

Common causes of desync: 9

• Out-of-band cron/script deleting files from ./volumes/app/storage/upload_files/ directly
• Incomplete volume copy during a version upgrade
• storage.save() succeeds but the subsequent db.commit() fails (orphaned file, no row)

Remediation CLI commands in api/commands/storage.py:

# Remove DB records whose files are missing from storage
docker compose exec api flask clear-orphaned-file-records

# Remove storage files that have no DB record
docker compose exec api flask remove-orphaned-files-on-storage

Key Source Files#