Table Listing API#

LanceDB exposes two methods for listing tables in a database: the deprecated table_names() and its replacement list_tables(). The migration adds namespace support, proper cursor-based pagination, and a scalable object-store-backed listing path.

Methods at a Glance#

Both methods are declared on the Database trait in rust/lancedb/src/database.rs.

table_names (Deprecated)#

TableNamesRequest carries three fields:

• namespace_path: Vec<String> — empty = root namespace
• start_after: Option<String> — resume after this name (lexicographic)
• limit: Option<u32> — max results to return

In ListingDatabase, the implementation reads all .lance directories via read_dir, sorts them, then slices from the first entry whose name is strictly greater than start_after. This means enumerating the entire database before applying pagination — on S3 with 100,000 tables, this took 11.2 seconds just to return the first 10 results 3.

list_tables (Current)#

list_tables in ListingDatabase uses the same read_dir-then-slice approach for the root namespace but improves pagination semantics. When a page_token is provided, it finds the first entry strictly greater than the token; when a limit is set, it peeks one entry beyond the page to determine whether a next page exists, returning the last name of the current page as next_page_token 4.

For non-root namespaces, list_tables delegates to LanceNamespaceDatabase, which routes to the underlying lance_namespace layer.

The Python binding for list_tables 5 accepts namespace_path, page_token, and limit, and returns a dict with tables and page_token keys.

Namespace Routing#

ListingDatabase dispatches listing based on the id / namespace_path field:

• Empty / root namespace: handled directly via ObjectStore::read_dir on the base path 6
• Non-empty namespace: forwarded to self.namespace_database(), which wraps a LanceNamespaceDatabase 7

A prior bug (PR #2842) incorrectly converted an empty namespace vector to None in LanceNamespaceDatabase::table_names, causing root-namespace listing to fail. The fix ensures id: Some(request.namespace) is always set, treating an empty Vec as the root namespace sentinel.

Pagination Off-By-One Bug (Fixed)#

The original list_tables had an off-by-one error: the page_token returned was the first name of the next page, so when callers used it to resume, the comparison name > page_token skipped that boundary table. Paging through tables [a, b, c, d, e] with limit=2 would return [a, b] then [d, e], silently dropping c 3.

The current implementation (PR #3777) corrects this by peeking one entry beyond the requested page. If f.len() > limit, the token is f[limit] (the first item of the next page), and the slice is truncated to limit. On the next call, the comparison name > page_token correctly starts strictly after that boundary item 4.

Performance Note#

PR #3777 introduced streaming listing (ObjectStore::read_dir_stream) to push limit and page_token into the object store's list API, making cost proportional to page size rather than total database size. On 100K tables in S3, listing the first 10 dropped from 11.2 s to 0.09 s. The table_names approach cannot be made efficient this way — it requires sorted output, but store-streamed listing returns items in key order, not sorted order — which is also why table_names is deprecated 3.

Key Source References#