Namespace Manifest Management#

Overview#

LanceDB supports two complementary modes for table discovery and metadata management in local/native connections:

1. Directory-listing mode (default): Tables are discovered by listing .lance directories under the connection root. Table names are derived from directory names.
2. Manifest-enabled mode: A namespace-backed manifest is the source of truth for table metadata. This enables richer operations (versioned declares, managed versioning, hierarchical namespaces) and automatic migration from directory listing.

The central control point is the manifest_enabled boolean on ConnectRequest.

Directory-Listing Mode (Default)#

ListingDatabase is the default database backend. It discovers tables with a read_dir call and filters for paths ending in .lance 1. Table URIs are constructed as <base_uri>/<table_name>.lance 2.

ListingDatabase always carries a backing LanceNamespaceDatabase instance (namespace_database) that handles operations involving an explicit namespace_path — table operations at the root namespace bypass it and use object-store directory listing directly 3.

Key option constants (listing.rs):

• new_table_data_storage_version — storage format version for new tables
• new_table_enable_v2_manifest_paths — enable V2 manifest paths (more efficient, requires newer clients)
• new_table_enable_stable_row_ids — enables stable row IDs across compaction/delete operations

manifest_enabled Configuration#

Setting manifest_enabled: true on a connection causes ConnectBuilder::execute() to call connect_manifest_enabled_namespace_database() instead of the standard connect_with_options() path 4. The returned connection is a LanceNamespaceDatabase directly — not a ListingDatabase.

Manifest-enabled properties injected automatically 5:

• manifest_enabled = "true"
• dir_listing_to_manifest_migration_enabled = "true" — existing directory-listed tables are migrated into the manifest on access.

Restrictions for manifest-enabled connections 6: URI schemes with + (e.g., s3+ddb://), engine query parameters, and mirroredStore query parameters are rejected — these commit strategies are incompatible with manifest-based metadata.

Python API#

import lancedb
db = lancedb.connect("/path/to/db", manifest_enabled=True)

The Python connect() function exposes manifest_enabled: bool = False 7. When True, it forwards to LanceDBConnection with the flag set 8.

Rust API#

lancedb::connect("/path/to/db")
    .manifest_enabled(true)
    .execute()
    .await?;

ConnectBuilder::manifest_enabled() sets the flag on ConnectRequest.

LanceNamespaceDatabase and Manifest-Based Table Management#

LanceNamespaceDatabase is the implementation used when manifest-enabled (and also when connecting via connect_namespace). It delegates all table operations to a LanceNamespace trait object.

Table lifecycle in manifest mode:

1. Create: Calls declare_table() on the namespace to reserve a location, then writes data to that location 9. A DeclareTableRequest conflict is resolved by inspecting whether the table is fully realized (has version + schema) before surfacing TableAlreadyExists 10.
2. Open: Delegates to NativeTable::open_from_namespace() 11.
3. List: Calls namespace.list_tables() 12 instead of read_dir.
4. Drop: Calls namespace.drop_table() 13.

ExternalManifestStore and Managed Versioning#

When a namespace response includes managed_versioning: Some(true), table creation installs an ExternalManifestCommitHandler as the Lance commit handler 14:

let external_store = LanceNamespaceExternalManifestStore::for_table_uri(
    self.namespace.clone(), table_id, &location
)?;
params.commit_handler = Some(Arc::new(ExternalManifestCommitHandler {
    external_manifest_store: Arc::new(external_store),
}));

• LanceNamespaceExternalManifestStore — from lance::io::commit::namespace_manifest; stores manifest version metadata in the namespace server rather than in object storage files 15.
• ExternalManifestCommitHandler — from lance_table::io::commit::external_manifest; implements CommitHandler to route commits through the external store 16.

This enables the namespace server to serve as the version-of-record, which allows features like atomic cross-table version coordination and server-side table discovery without object-store listing.

The managed_versioning flag is activated on the namespace side. For example, setting the table_version_tracking_enabled namespace client property via namespace_client_property("table_version_tracking_enabled", "true") causes the namespace to return managed_versioning: Some(true) in describe_table responses 17.

Choosing a Mode#

Key Source Files#