Vector Search Result Ordering#

LanceDB stores data in the Lance columnar format across multiple fragments (file-backed data chunks). Vector search result ordering covers how distance-ranked results are gathered from those fragments, merged, and returned to the caller. There are two distinct pipelines: a plain/ANN vector search path that delegates sorting to the Lance Scanner, and a hybrid (vector + FTS) path that explicitly concatenates, normalizes, and reranks results in LanceDB itself. A third concern — silent data corruption at write boundaries — can cause vectors to be silently misaligned before they ever reach the query layer.

ANN Query Pipeline#

The entry point for query plan construction is create_plan() in rust/lancedb/src/table/query.rs.

Top-k overshoot for pagination. For vector queries, LanceDB calls scanner.nearest(&column, query_vector, top_k) where top_k = limit + offset 1. Fetching extra results is necessary so the scanner can return the right window after the offset is applied, regardless of which fragments contribute to the result set.

Fragment merging is Lance-internal. The Scanner handles cross-fragment distance-sorted merging internally. LanceDB then applies scanner.limit(limit, offset) on top 2. The _distance column is populated by the Lance engine and contains the distance metric value for each returned row.

Result ordering is not guaranteed deterministic across calls — the docs explicitly note that batch sizes and row ordering within results are non-deterministic 3.

Fast search (scanner.fast_search()) restricts the search to indexed fragments only, trading consistency for speed 4. Use Table.optimize() to pull newly-added rows into the index.

Refine factor. When refine_factor is set, LanceDB fetches limit × refine_factor ANN candidates using the quantized index, then fetches the full uncompressed vectors for those candidates and re-ranks by exact distance. This corrects ordering errors introduced by quantization 5. Without it, _distance values are approximate quantized distances.

Multi-vector queries. When multiple query vectors are provided, LanceDB creates one plan per vector and unions them 6, adding a query_index column to distinguish results.

Hybrid Search Merge (Vector + FTS)#

Hybrid search is handled by execute_hybrid() in rust/lancedb/src/query.rs:

1. Vector and FTS sub-queries run in parallel.
2. All result batches from each arm are collected and concatenated with concat_batches() 7.
3. _distance and _score columns are normalized (rank-based or score-based) 8.
4. A reranker merges and sorts the combined result set. The default is the RRF (Reciprocal Rank Fusion) reranker, which deduplicates by row ID and scores each document by 1.0 / (rank + k), then sorts descending.

Known reranker bugs (now fixed):

• LinearCombinationReranker had inverted scoring (1 - combined_score) and a wrong penalty for documents missing FTS matches — fixed in PR #3437 (merged 2026-05-26).
• MRRReranker incorrectly averaged reciprocal ranks over only the systems where a document appeared, rather than all systems — fixed in PR #3599 (merged 2026-07-01).

Silent Corruption Risks at Write Boundaries#

large_list → fixed_size_list child-array offset bug (Issue #3602)#

Polars and pyarrow export list columns as large_list. LanceDB casts these to fixed_size_list during ingestion. When the input batch is a zero-copy Arrow slice (e.g., a batch that is the tail of a multi-row-group parquet chunk), the FixedSizeListArray values child has a non-zero offset into a shared values buffer.

The writer reads from buffer position 0, ignoring the child offset. This silently shifts vectors by a constant number of rows — each affected row receives a vector belonging to a different row. Scalar columns remain correctly aligned, making the corruption invisible without cross-checking vectors against the source data.

Conditions that must all be true to trigger it 9:

1. Vector column is supplied as large_list or list (not fixed_size_list directly).
2. Input arrives as a zero-copy slice with a non-zero offset into a shared values buffer.
3. Rows are wide enough to cross the internal buffer-splitting threshold.

Workarounds:

• Cast vector columns to fixed_size_list before passing data to LanceDB.
• Call .combine_chunks() on the batch to materialize the slice (sets child offset to 0).

Stale index entries after partial-schema merge_insert (lance#6514)#

A partial-schema merge_insert (not all columns updated) can cause touched fragments to be dropped from the index bitmap while stale entries remain in underlying indexes (B-tree, vector index). This produces two symptoms in LanceDB:

• Issue #3280 10: A second merge_insert after create_scalar_index threw Ambiguous merge inserts are prohibited — because stale index entries matched the same target row multiple times. Fixed in lance ≥ 4.x via the invalidated_fragment_bitmap field in IndexMetadata.
• Issue #3515 11: Scalar index on the merge column causes merge_insert(...).when_matched_update_all() to silently report 0 updated rows. Fix in PR #3612.

Mitigation: Drop and recreate the index, or upgrade to a lance version that includes the invalidated_fragment_bitmap fix.

Key Query Parameters#

See VectorQueryRequest for the full field list and defaults, and QueryBase for shared filter/select/ordering options.