Table Optimization#
Overview#
LanceDB table optimization is modeled after PostgreSQL's VACUUM. It bundles three sub-operations — Compact, Prune, and Index — into a single optimize() call. Because LanceDB uses a copy-on-write, append-only filesystem model, every write (add, delete, update) creates new files; optimization is the mechanism for reclaiming space and maintaining query performance over time.
The primary entry point is Table.optimize(). The legacy compact_files() and cleanup_old_versions() methods on LanceTable were deprecated in v0.21.0 .
Call Stack#
Python (sync) LanceTable.optimize() table.py:3770-3821
└─ LOOP.run(AsyncTable.optimize())
Python (async) AsyncTable.optimize() table.py:5923-5985
└─ self._inner.optimize(...) (PyO3 binding)
PyO3 Wrapper PyTable::optimize() python/src/table.rs:1151-1208
└─ OptimizeAction::Compact
└─ OptimizeAction::Prune
└─ OptimizeAction::Index
Rust Core execute_optimize() optimize.rs:163-213
├─ compact_files_impl() optimize.rs:148-158
├─ cleanup_old_versions() optimize.rs:129-140
└─ optimize_indices() optimize.rs:105-112
Lance Backend lance::dataset::optimize::compact_files()
lance::dataset::Dataset::cleanup_old_versions()
dataset.optimize_indices()
Three Optimization Actions#
The Rust core dispatches via the OptimizeAction enum:
1. Compact (OptimizeAction::Compact)#
Merges small fragment files into larger ones . Implemented in compact_files_impl(), which delegates to lance::dataset::optimize::compact_files. Accepts CompactionOptions and optional IndexRemapperOptions. Run frequently if writes are frequent; not needed for read-only workloads .
2. Prune (OptimizeAction::Prune)#
Removes old dataset versions from disk . Implemented in cleanup_old_versions(), which calls lance::dataset::Dataset::cleanup_old_versions. Default retention when called via OptimizeAction::All is 7 days . The delete_unverified flag (default False) controls whether files <7 days old that may belong to in-progress transactions are removed — setting it to True risks corruption if other writers are active .
3. Index (OptimizeAction::Index)#
Folds unindexed rows (added since the last index build) into existing indices without full retraining . Implemented in optimize_indices(). Faster than retraining but does not update cluster centroids for IVF indices — periodic full retraining is still advisable after large data loads .
PyO3 Wrapper (python/src/table.rs)#
PyTable::optimize() accepts cleanup_since_ms (milliseconds, converted from timedelta upstream) and delete_unverified, then issues three sequential async Rust calls — Compact, Prune, Index — and returns an OptimizeStats containing CompactionStats and RemovalStats.
Python API (python/python/lancedb/table.py)#
| Method | Class | Notes |
|---|---|---|
LanceTable.optimize() | LanceTable | Sync; wraps async via LOOP.run(). Params: cleanup_older_than (timedelta, default 7 days), delete_unverified. |
AsyncTable.optimize() | AsyncTable | Async; converts timedelta to ms, passes to PyO3 binding. retrain param deprecated. |
LanceTable.compact_files() | LanceTable | Deprecated v0.21.0. Delegates to lance.dataset.DatasetOptimizer.compact_files. |
LanceTable.cleanup_old_versions() | LanceTable | Delegates directly to lance.LanceDataset. |
A rule of thumb from the docstring: run optimize() after adding/modifying ≥100,000 records or after >20 data modification operations .
Key Source Files#
| File | Purpose |
|---|---|
rust/lancedb/src/table/optimize.rs | Rust core: OptimizeAction enum, execute_optimize(), compact_files_impl(), cleanup_old_versions(), optimize_indices() |
python/src/table.rs | PyO3 binding: PyTable::optimize(), stats structs |
python/python/lancedb/table.py | Python API: LanceTable.optimize(), AsyncTable.optimize(), deprecated helpers |