Cloud Service Authentication in OpenDAL#

OpenDAL authenticates cloud storage requests by delegating to the reqsign crate. Each backend holds a signer and one or more credential loaders that implement provider-specific signing algorithms. Before every outbound request, the backend calls its sign() or sign_query() method which (1) loads a credential, (2) mutates the request headers or query string, and (3) forwards the signed request.

The pattern is uniform across providers: build credentials → sign request → strip any headers the provider rejects (e.g., the HOST header on GCS 1).

Provider Overview#

Google Cloud Storage (GCS)#

GcsCore carries two separate loaders because GCS uses different credentials for different operations 2:

• GoogleTokenLoader — fetches OAuth2 access tokens for standard JSON-API requests (async, with exponential-backoff retry).
• GoogleCredentialLoader — loads raw service-account credentials for presigned URL generation via the XML API (sync).

load_token(): If a static token string was configured, wraps it directly as a GoogleToken. Otherwise calls token_loader.load() with retry. Returns None only when allow_anonymous is set; otherwise returns a ConfigInvalid error.

sign(): Calls load_token() → signer.sign(req, &token). Used for all normal API calls.

sign_query(): Calls load_credential() → signer.sign_query(req, duration, &cred). Used only for presigned (query-signed) URLs via the XML API.

Configuration Options#

Configured through GcsConfig / GcsBuilder:

The builder assembles a GoogleCredentialLoader from content/path first, then wraps it in a GoogleTokenLoader with scope and service-account settings 3.

AWS S3#

S3Core stores a boxed AwsCredentialLoad trait object, enabling any custom credential loader to be injected 4. An AtomicBool (credential_loaded) tracks whether credentials were successfully fetched at least once; subsequent failures are treated as errors rather than silent anonymous fallbacks 5.

The builder wires up the loader chain in backend.rs 6:

1. Custom loader — any type implementing AwsCredentialLoad, injected via customized_credential_load().
2. Assume-role — AwsAssumeRoleLoader with optional external ID, backed by STS.
3. Default loader — AwsDefaultLoader, sourcing credentials from environment variables, ~/.aws/credentials, or EC2 instance metadata (which can be disabled).

Signing uses AWS Signature Version 4 via AwsV4Signer 7.

Azure Blob Storage#

AzblobCore holds AzureStorageLoader and AzureStorageSigner 8. Credentials are mandatory — load_credential() returns an error if nothing is found 9.

Supported credential types, configured via AzureStorageConfig 10:

• Account key — HMAC-SHA256 shared-key authentication.
• SAS token — pre-signed token passed as a query parameter.
• Environment variables are the primary source; the loader is created with AzureStorageLoader::new(config_loader) 11.

sign() prepends the required x-ms-version: 2022-11-02 header before delegating to the signer 12.

Key Entry Points for Deeper Investigation#