Storage Options and Credential Vending#

LanceDB supports dynamic cloud storage credential management through a mechanism called credential vending — where a namespace server supplies short-lived credentials that are automatically refreshed before they expire. This is critical for long-running operations on remote storage (S3, GCS, etc.) where static credentials would time out mid-operation.

Architecture Overview#

Storage options (credentials, endpoint configuration, region, etc.) flow from two possible sources:

• Static — passed once at connect time via storage_options on the connection request. These are stored in ListingDatabaseOptions.storage_options 1 and inherited by all tables opened through the database.
• Dynamic (vended) — supplied by a StorageOptionsProvider trait (imported from lance_io::object_store) that LanceDB calls to retrieve fresh credentials on demand. The provider is wired into an ObjectStoreParams via the private helper set_storage_options_provider, which wraps both the static initial options and the provider in a StorageOptionsAccessor.

On OpenTableBuilder, the public entry point is storage_options_provider(), which sets the provider for read operations. A parallel method exists for write operations on CreateTableBuilder.

vend_input_storage_options#

vend_input_storage_options is a namespace client property (not a function) set on DirectoryNamespace to instruct the namespace server to return storage credentials alongside table metadata. Three related properties control the behavior 2:

When enabled, the namespace adds an expires_at_millis timestamp field to returned storage options. LanceDB uses this timestamp to decide when to re-call the provider 4.

Credential Expiration and Refresh Flow#

1. Table open — describe_table is called on the namespace, returning storage options with expires_at_millis.
2. Read / write — The StorageOptionsAccessor checks whether the timestamp has passed before each object store operation.
3. Refresh — When expired, the accessor calls the provider, which in turn calls describe_table again on the namespace. describe_table call count is a measurable signal for credential refresh 5.
4. Caching — Credentials are cached per operation to avoid redundant refreshes 6.

Integration tests test_namespace_credential_refresh_on_read and test_namespace_credential_refresh_on_write verify end-to-end refresh by setting a 3-second expiry and asserting that describe_table is called exactly once after expiry.

A unit test test_manifest_enabled_vend_input_storage_options confirms the expires_at_millis field is present in the options returned by latest_storage_options().

Inspecting Storage Options (Python API)#

The Python Table class exposes two methods 7:

• initial_storage_options() — returns the static options passed at connection/table setup.
• latest_storage_options() — calls the provider if one is configured, returning the most recent credentials. Use this in credential-vending scenarios.

Both are marked as internal APIs and their return values are subject to change.

Remote URI Scheme Routing#

URI routing happens in ListingDatabase::connect_with_options(). The flow:

1. Parse — url::Url::parse(uri) extracts the scheme 8.
2. Windows path guard — single-character schemes (drive letters like C:) are routed to Self::open_path() 9.
3. Query extraction — engine and mirrored_store query params are stripped before passing the URI downstream; remaining params are preserved 10.
4. Commit engine rewrite — If an engine param is present, the scheme is rewritten as {scheme}+{engine} (e.g., s3+ddb://) to engage an alternative commit store 11.
5. Dispatch — ObjectStore::from_uri_and_params() (from lance) receives the cleaned URI and routes s3://, gs://, az://, hf://, and other schemes to their respective backend implementations 12.

Static storage options are passed into this call via ObjectStoreParams.storage_options_accessor 13. There is no LanceDB-level special casing for hf://; it is handled entirely by the Lance ObjectStore layer.

Key Files#