File Download Architecture#

Dify uses three distinct download strategies depending on whether content is sourced from an external datasource plugin, a single uploaded document, or a batch of documents. Each strategy has a different data path, I/O model, and truncation risk profile.

1. Azure Blob / Large Binary Blob Assembly (Datasource Streaming)#

Entry point: DatasourceManager.stream_online_results → stream_node_events

When a workflow node invokes an ONLINE_DRIVE datasource (e.g., Azure Blob), the plugin yields a generator of DatasourceMessage objects. AzureBlobDataSource._download_file selects one of three sub-strategies based on auth method and file size:

Blob chunk assembly — merge_blob_chunks: The raw generator is wrapped by merge_blob_chunks, which accumulates BLOB_CHUNK messages keyed by chunk ID until the final chunk arrives, then emits one complete BLOB message. The maximum assembled size is controlled by dify_config.PLUGIN_MAX_FILE_SIZE (default 50 MB) 1. Exceeding this limit raises a ValueError and drops the accumulated buffer — this is the primary truncation risk for large datasource files. PR #34662 extended this merging to the datasource path (previously only the tool path in api/core/plugin/impl/tool.py had it; the datasource path in api/core/plugin/impl/datasource.py was unprotected).

Downstream transformation: After merging, DatasourceFileMessageTransformer.transform_datasource_invoke_messages saves the assembled blob to storage via ToolFileManager.create_file_by_raw and returns a BINARY_LINK or IMAGE_LINK message with a URL of the form /files/datasources/{tool_file_id}.

2. Single-Document Signed URL Download#

Entry point: GET /datasets/<dataset_id>/documents/<document_id>/download → DocumentDownloadApi.get

This path returns a signed URL directly from storage — no bytes are proxied through the API server.

DocumentService.get_document_download_url(document, session)
  └── file_helpers.get_signed_file_url(upload_file_id=..., as_attachment=True)

2

The URL is generated by FileService.get_file_presigned_url, which calls storage.generate_presigned_url(file_key, expires_in=dify_config.FILES_ACCESS_TIMEOUT, ...). The client browser downloads directly from the storage backend (S3, Azure Blob, local, etc.).

Truncation risk: Near-zero under normal conditions. The file is already at rest in storage; the download is a direct pre-signed fetch with no intermediate assembly. Failure modes are storage unavailability and URL expiry (FILES_ACCESS_TIMEOUT).

3. Batch ZIP File Download#

Entry point: POST /datasets/<dataset_id>/documents/download-zip → DocumentBatchDownloadZipApi.post

For downloading multiple upload-file documents at once, Dify builds and streams a single ZIP archive to avoid browser multi-download limits:

1. DocumentService.prepare_document_batch_download_zip resolves UploadFile rows for the requested document IDs after checking dataset permissions.
2. FileService.build_upload_files_zip_tempfile writes the ZIP into a NamedTemporaryFile:
   • Entry names are sanitized against path traversal (_sanitize_zip_entry_name) and deduplicated (_dedupe_zip_entry_name).
   • File bytes are streamed chunk-by-chunk from storage (storage.load(..., stream=True)) using ZIP_DEFLATED compression.
   • The temp file path is yielded to the caller; the file stays on disk until the response finishes.
3. Flask's send_file(zip_path, as_attachment=True) serves the ZIP. response.call_on_close(cleanup.close) deletes the tempfile after streaming completes 3.

Truncation risk: Moderate. ZIP assembly is synchronous in the API process. Large batches increase disk pressure on the temp directory. There is no total-batch size cap at the service layer — only per-file upload limits enforced at ingest time.

Truncation Risk Summary#

Key Files#