File Download Architecture#

Dify uses five distinct download strategies depending on whether content is sourced from an external datasource plugin, a single uploaded document, a batch of documents, an app package export, or an app package import from files or URLs. Each strategy has a different data path, I/O model, and truncation risk profile.

1. Azure Blob / Large Binary Blob Assembly (Datasource Streaming)#

Entry point: DatasourceManager.stream_online_results → stream_node_events

When a workflow node invokes an ONLINE_DRIVE datasource (e.g., Azure Blob), the plugin yields a generator of DatasourceMessage objects. AzureBlobDataSource._download_file selects one of three sub-strategies based on auth method and file size:

Blob chunk assembly — merge_blob_chunks: The raw generator is wrapped by merge_blob_chunks, which accumulates BLOB_CHUNK messages keyed by chunk ID until the final chunk arrives, then emits one complete BLOB message. The maximum assembled size is controlled by dify_config.PLUGIN_MAX_FILE_SIZE (default 50 MB) 1. Exceeding this limit raises a ValueError and drops the accumulated buffer — this is the primary truncation risk for large datasource files. PR #34662 extended this merging to the datasource path (previously only the tool path in api/core/plugin/impl/tool.py had it; the datasource path in api/core/plugin/impl/datasource.py was unprotected).

Downstream transformation: After merging, DatasourceFileMessageTransformer.transform_datasource_invoke_messages saves the assembled blob to storage via ToolFileManager.create_file_by_raw and returns a BINARY_LINK or IMAGE_LINK message with a URL of the form /files/datasources/{tool_file_id}.

2. Single-Document Signed URL Download#

Entry point: GET /datasets/<dataset_id>/documents/<document_id>/download → DocumentDownloadApi.get

This path returns a signed URL directly from storage — no bytes are proxied through the API server.

DocumentService.get_document_download_url(document, session)
  └── file_helpers.get_signed_file_url(upload_file_id=..., as_attachment=True)

2

The URL is generated by FileService.get_file_presigned_url, which calls storage.generate_presigned_url(file_key, expires_in=dify_config.FILES_ACCESS_TIMEOUT, ...). The client browser downloads directly from the storage backend (S3, Azure Blob, local, etc.).

Truncation risk: Near-zero under normal conditions. The file is already at rest in storage; the download is a direct pre-signed fetch with no intermediate assembly. Failure modes are storage unavailability and URL expiry (FILES_ACCESS_TIMEOUT).

3. Batch ZIP File Download#

Entry point: POST /datasets/<dataset_id>/documents/download-zip → DocumentBatchDownloadZipApi.post

For downloading multiple upload-file documents at once, Dify builds and streams a single ZIP archive to avoid browser multi-download limits:

1. DocumentService.prepare_document_batch_download_zip resolves UploadFile rows for the requested document IDs after checking dataset permissions.
2. FileService.build_upload_files_zip_tempfile writes the ZIP into a NamedTemporaryFile:
   • Entry names are sanitized against path traversal (_sanitize_zip_entry_name) and deduplicated (_dedupe_zip_entry_name).
   • File bytes are streamed chunk-by-chunk from storage (storage.load(..., stream=True)) using ZIP_DEFLATED compression.
   • The temp file path is yielded to the caller; the file stays on disk until the response finishes.
3. Flask's send_file(zip_path, as_attachment=True) serves the ZIP. response.call_on_close(cleanup.close) deletes the tempfile after streaming completes 3.

Truncation risk: Moderate. ZIP assembly is synchronous in the API process. Large batches increase disk pressure on the temp directory. There is no total-batch size cap at the service layer — only per-file upload limits enforced at ingest time.

4. App Package Export (.ifpkg)#

Entry point: GET /console/api/apps/{app_id}/export → AppExportApi.get → RosterAgentPackageExporter.export

The App export route accepts optional query parameters:

• format: yaml or ifpkg. All app types (Workflow, Chatflow, Agent) default to .ifpkg format; YAML export remains available as an explicit option (format=yaml).
• version_id: Optional published Agent version UUID. When specified, the export uses the selected version's configuration and workspace skill bindings instead of the active configuration.

The export implementation in AppExportApi.get streams .ifpkg archives with Content-Type: application/zip using Flask's send_file, while YAML exports use AppDslService and return JSON-wrapped DSL.

Version selection behavior:

• When version_id is not specified, the export uses the editable shared draft. If no draft exists, the export falls back to the active published snapshot.
• When version_id is specified:
  • The export uses the selected published version's configuration snapshot and workspace skill bindings.
  • Requires a paid plan on Cloud (sandbox users must upgrade).
  • The version must be owned by the current workspace.
  • Historical version visibility must be enabled.
  • Validation is performed by AgentRosterService.get_visible_agent_version_snapshot.
  • Raises AgentVersionNotFoundError if the version is unavailable or not visible in history.

For exporting applications as portable .ifpkg archives:

1. Agent Apps: RosterAgentPackageExporter.export collects the Roster Agent configuration (based on version_id selection or active draft/snapshot fallback), resolves all referenced skills and files from ToolFile and UploadFile tables, and gathers workspace skills via SkillManagementService.list_runtime_agent_skill_archives. RosterAgentPackageExporter._build_archive constructs a versioned ZIP archive with:
   • manifest.yaml: Package metadata and resource index (format: "dify.roster-agent") (RosterAgentPackageManifest)
   • app.yaml: Agent application configuration with portable references (AgentAppDsl)
   • Skill packages (s_NNNNNN.zip): Agent-scoped and workspace skills
   • Config files (f_NNNNNN.*): Agent configuration files with preserved extensions
   • Image icons (i_NNNNNN.*): App and Agent image icons with preserved extensions
2. Workflow and Chatflow Apps: AppPackageService.export calls AppDslService.export_data to serialize the workflow DSL, collecting inline Agent resources from node-bound snapshots via AgentPackageResourceExporter. The ZIP archive includes:
   • manifest.yaml: Package metadata (format: "dify.app") with Agent resource index
   • app.yaml: Workflow/Chatflow DSL with portable Agent package references (capped at the same DSL_MAX_SIZE limit as YAML imports: 10 MiB)
   • Skill packages (s_NNNNNN.zip): Agent-scoped and workspace skills from each node's bound snapshot
   • Config files (f_NNNNNN.*): Agent configuration files with preserved extensions
   • Image icons (i_NNNNNN.*): App and Agent image icons with preserved extensions
3. Package validation and assembly:

   • Agent packages validate each member against configurable limits defined in FileUploadConfig:

     Limit	Config key	Default
     Maximum single skill package size	UPLOAD_SKILL_FILE_SIZE_LIMIT	50 MB
     Maximum total package size	AGENT_PACKAGE_MAX_BYTES	512 MB
     Maximum YAML document size (manifest/app)	AGENT_PACKAGE_MAX_MANIFEST_BYTES	5 MB
     Maximum number of archive entries	AGENT_PACKAGE_MAX_ENTRIES	5000

     Archive assembly is streamed into a SpooledTemporaryFile (spooled at 16 MB). Each embedded skill package is inspected via SkillPackageService.inspect to enforce nested uncompressed size limits (SKILL_PACKAGE_MAX_UNCOMPRESSED_BYTES, default 200 MB). The exporter computes SHA-256 digests and sizes for each member, populating the manifest resource index. The completed archive is yielded as a RosterAgentPackageExport with a sanitized filename (<agent-name>.ifpkg).
     Image icons are validated against UPLOAD_IMAGE_FILE_SIZE_LIMIT and deduplicated in the manifest (shared icons stored once). Icon file references are converted to portable resource IDs (i_NNNNNN) in the exported metadata.

   • Workflow/Chatflow packages enforce the DSL_MAX_SIZE limit (10 MiB) on the serialized DSL during export. The archive is built into a SpooledTemporaryFile (spooled at 16 MB). When inline Agents are present, the resource exporter collects each Agent's configuration snapshot, skill packages, and config files from the bound WorkflowAgentNodeBinding and materializes them using the same validation and assembly path as Roster Agent packages. The completed archive is validated by re-reading through the same import validation path and yielded with a sanitized filename (<app-name>.ifpkg).

4. Errors are surfaced through dedicated exceptions in api/services/agent/errors.py: RosterAgentPackageTooLargeError, RosterAgentPackageExportFailedError, InvalidRosterAgentPackageError.

Truncation risk: Low to moderate. Configurable size and entry-count limits prevent unbounded resource consumption. Skill package inspection adds validation overhead but shares the same boundary framework as skill uploads, reducing inconsistency. Disk pressure depends on the 16 MB spool threshold and total package size.

5. App Package Import (File and URL)#

Entry points:

• File upload: POST /console/api/apps/imports with mode=package-file and multipart file → AppImportApi._import_package
• URL import: POST /console/api/apps/imports with mode=yaml-url and yaml_url → AppImportApi._import_url → download_app_import_source

When the /apps/imports endpoint receives a package file or a yaml_url parameter pointing to an app package rather than a YAML/DSL file, the import flow attempts format auto-detection before importing:

1. URL import only: download_app_import_source performs a single bounded HTTP download:
   • Supports extensionless download URLs and GitHub URLs (automatically normalized from github.com/org/repo/blob/... to raw.githubusercontent.com/org/repo/...)
   • Uses the existing remote_fetcher infrastructure (bounded downloads, SSRF protection, timeout enforcement)
   • Rejects compressed HTTP responses to prevent memory exhaustion from decompression
   • Enforces a size limit of max(AGENT_PACKAGE_MAX_BYTES, DSL_MAX_SIZE) (default 512 MB for packages, 10 MiB for DSL)
   • Downloads into a SpooledTemporaryFile (16 MB spool threshold)
   • Single download attempt — no retries on failure
2. AppPackageService.read_package attempts format auto-detection:
   • Inspects the package as a ZIP archive and reads the manifest.yaml to determine the package format
   • If format == "dify.roster-agent", returns None to signal the stream should be rewound and dispatched to _import_agent_package for Roster Agent import via RosterAgentPackageImporter
   • If format == "dify.app", validates the package structure, extracts app.yaml, and returns a PreparedAppPackage containing the DSL and validated inline Agent resources
   • Validates archive integrity, member path safety, Agent resource ownership, and uncompressed size against the shared AGENT_PACKAGE_MAX_BYTES limit (512 MB)
   • Enforces the DSL_MAX_SIZE limit (10 MiB) on the extracted DSL
   • Rewinds the source stream before returning, ensuring the caller can re-read for Agent package import if needed
3. Ordinary app packages (Workflow/Chatflow): The prepared package is passed to AppDslService.import_app, which materializes bundled Agent resources via AgentPackageResourceImporter before database writes:
   • Resource materialization reads skill archives and config files from the prepared package, uploads them to the target workspace storage, and creates ToolFile and UploadFile records. The updated DSL is validated and passed to the existing app creation/overwrite flow.
   • Icon files are restored as destination-owned uploads with file references updated to the new workspace's file IDs.
   • When resources cannot be restored (e.g., damaged skill archives), the importer records missing-resource warnings and marks the references as is_missing=True in the imported Agent configuration
     • Legacy Agent DSL asset validation: When importing a legacy Agent DSL package, skill and file references are validated within the target tenant. Unavailable or invalid references (IDs that don't exist or aren't accessible in the target tenant) are cleared from the imported configuration and marked as missing. These missing resources generate import warnings that are returned to the user. Valid resources that have been restored from archives remain usable. This validation is performed by a private helper (_mark_missing_package_assets in AgentDslService) to ensure references resolve correctly in the target environment.
   • Overwrite validation (_validate_workflow_overwrite) ensures compatibility between the target app mode and the imported DSL (both must be Workflow or Advanced Chat)
   • Additional node-type validation prevents importing workflows with incompatible node types (e.g., Answer nodes in Advanced Chat apps)
   • For each inline Agent node, the import creates a new Agent row, a WorkflowAgentNodeBinding linking the node to the materialized snapshot, and an AgentConfigSnapshot with remapped resource IDs
4. Agent packages: Dispatched to RosterAgentPackageImporter for skill installation and Agent app creation (overwrite not supported for Agent packages)
   • Icon files are restored as destination-owned uploads with file references updated to the new workspace's file IDs. Packages without icons remain compatible.
5. Errors are surfaced through InvalidRosterAgentPackageError and RosterAgentPackageTooLargeError

Truncation risk: Low to moderate. Bounded downloads enforce package-specific quotas through the existing remote-file fetcher. The 16 MB spool threshold matches package export behavior. Single-attempt download (no retries) reduces resource consumption on transient failures. Format detection prevents double downloads. Ordinary app packages share the 10 MiB DSL import limit with YAML imports.

Truncation Risk Summary#

Key Files#