OpenDAL Storage Backend#

Overview#

OpenDALStorage is the class that backs both the opendal and local storage types in Dify. It wraps the Apache OpenDAL Python library to provide a single, uniform storage interface across multiple backend schemes (filesystem, S3-compatible, etc.). For local development the fs scheme is used, making all file I/O operate on the host filesystem.

Key files:

api/extensions/storage/opendal_storage.py — OpenDALStorage implementation
api/extensions/ext_storage.py — Storage facade + app wiring
api/extensions/storage/base_storage.py — BaseStorage abstract interface
api/extensions/storage/storage_type.py — StorageType enum

How It Fits Into the Storage Layer#

All storage access in the API goes through the singleton storage object (an instance of the Storage facade). On startup, Storage.init_app() calls get_storage_factory() which matches on STORAGE_TYPE:

`STORAGE_TYPE` value	What is instantiated
`opendal`	`OpenDALStorage(dify_config.OPENDAL_SCHEME)` — scheme from config
`local` (deprecated)	`OpenDALStorage(scheme="fs", root=dify_config.STORAGE_LOCAL_PATH)`
`s3`, `azure-blob`, …	Dedicated storage classes (not OpenDAL)

Both opendal and local storage types ultimately create an OpenDALStorage instance — they differ only in how the fs root is resolved.

The Storage facade delegates all method calls (save, load_once, load_stream, download, exists, delete, scan) to self.storage_runner, which is the concrete OpenDALStorage instance .

OpenDALStorage Internals#

Initialization#

OpenDALStorage.__init__ accepts a scheme string and optional **kwargs. If no kwargs are provided, _get_opendal_kwargs() reads scheme-specific config from environment variables and .env, looking for keys prefixed with OPENDAL_<SCHEME>_ (e.g., OPENDAL_FS_ROOT for the fs scheme).

For the fs scheme, the root directory defaults to "storage" and is created if it doesn't exist .

An opendal.layers.RetryLayer (max 3 attempts, factor 2.0 with jitter) is applied to every operator at construction time .

Operations#

Method	Behavior
`save`	Synchronous `op.write(path, bs)` — writes entire `bytes` payload at once
`load_once`	Synchronous `op.read(path)` — returns entire file as `bytes`
`load_stream`	Opens file in `"rb"` mode with a 4096-byte chunk size, yields chunks via a `Generator`
`download`	Reads full file and writes to a local path using `Path.write_bytes`
`exists`	Delegates to `op.exists(path)`
`delete`	Checks existence first; no-ops silently if file is absent
`scan`	Uses OpenDAL 0.46.0+ `op.list(path, recursive=True)`; filters by `entry.metadata.is_dir`

4096-Byte Chunk Streaming#

load_stream passes chunck=4096 (note the typo in the source — it mirrors OpenDAL's parameter name) to op.open() and reads 4096 bytes per iteration . This is a fixed constant, not configurable at call time.

Configuration#

Default configuration for local development :

STORAGE_TYPE=opendal
OPENDAL_SCHEME=fs
OPENDAL_FS_ROOT=storage

OPENDAL_SCHEME defaults to "fs" in OpenDALStorageConfig . STORAGE_TYPE defaults to "opendal" in the middleware config . STORAGE_LOCAL_PATH (used when STORAGE_TYPE=local) defaults to "storage" and is marked deprecated .

For non-fs schemes (e.g., S3-compatible via OpenDAL), set OPENDAL_SCHEME and supply scheme-specific env vars using the OPENDAL_<SCHEME>_<KEY> naming convention parsed by _get_opendal_kwargs.

Notes for Developers#

local storage type is deprecated. Prefer STORAGE_TYPE=opendal with OPENDAL_SCHEME=fs. The local path still resolves to OpenDALStorage but may be removed in a future release.
Retry is always on. Every operator gets a retry layer automatically — callers do not need to handle transient failures themselves.
load_stream vs load_once. Use storage.load(filename, stream=True) for large files to avoid loading the full payload into memory; stream=False (default) returns bytes directly .
scan requires OpenDAL ≥ 0.46.0 for the recursive=True list API .