Write Path Architecture#

Overview#

LanceDB's write path is implemented in Rust under rust/lancedb/src/table/ and exposes three main data ingestion operations: add, update, and merge_insert. Each path preprocesses data differently before delegating to lance-core. All paths ultimately produce a new dataset version committed to the lance storage layer.

add — Append or Overwrite#

Entry point: add_data.rs → AddDataBuilder

AddDataBuilder::into_plan builds a DataFusion execution plan that applies the following preprocessing steps in order 1:

1. Schema validation — rejects columns not present in the table schema; missing table columns are allowed (filled with nulls). Skipped entirely in Overwrite mode 2.
2. Embedding computation — if the table has embedded columns, scannable_with_embeddings fills them 3.
3. Schema casting — cast_to_table_schema (see Schema Casting) projects and casts input fields to match the table schema. Not applied in Overwrite mode — the input schema replaces the table schema entirely 4.
4. NaN guard — reject_nan_vectors wraps the plan to error on any NaN values in vector columns (configurable via NaNVectorBehavior::Keep) 5.

The resulting plan is passed to InsertExec 6, which runs it across partitions, accumulates lance uncommitted transactions, and finally commits via CommitBuilder::execute() 7.

Key behavior differences between Append and Overwrite:

• Append: schema validation + casting applied; input must be a sub-schema.
• Overwrite: both validation and casting are skipped; input schema replaces the table schema. Raw binary is not coerced into blob struct layout on overwrite 8.

update — SQL-Expression Column Updates#

Entry point: update.rs → execute_update

The update path is the simplest of the three. No DataFusion preprocessing pipeline is built. Instead 9:

1. Snapshots the current dataset.
2. Creates a LanceUpdateBuilder directly from lance-core.
3. Configures update_where() (WHERE clause) and one or more set() expressions (SET clause).
4. Calls builder.build()?.execute() — lance-core handles the rest.
5. Updates the cached dataset wrapper with the new version.

Columns to update are specified as (column_name, sql_expression) pairs 10; the expression is evaluated against each row's current values. No embedding re-computation or schema coercion occurs on this path.

merge_insert — Upsert / Conditional Insert-or-Update#

Entry point: merge.rs → execute_merge_insert

merge_insert is the most complex path and has two sub-routes:

Standard Path#

When no LsmWriteSpec is installed (or when use_lsm(false) is set), the standard path 11:

1. Creates LanceMergeInsertBuilder and configures three behaviors:
   • when_matched — UpdateAll, DoNothing, or conditional update.
   • when_not_matched — InsertAll or DoNothing.
   • when_not_matched_by_source — Delete, conditional delete, or Keep.
2. Calls builder.try_build()?.execute_reader(new_data) to delegate entirely to lance-core.
3. Returns MergeResult with insert/update/delete counts 12.

Input on this path goes directly to lance-core as a RecordBatchReader without the DataFusion preprocessing pipeline used by add. Indexes can be leveraged for the join key via use_index (default: true) 13.

LSM / MemWAL Path#

When a LsmWriteSpec is installed and the merge shape is an upsert (when_matched_update_all + when_not_matched_insert_all), lsm_dispatch_decision routes to execute_lsm_merge_insert 14:

1. Collects all input batches and aligns them to the target schema.
2. Validates that all rows route to a single shard.
3. Writes to a ShardWriter (MemWAL) via writer.put(batches) — no commit is issued immediately.

In this mode, MergeResult only populates num_rows; insert/update breakdown is not known until compaction 15.

Schema Casting (cast_to_table_schema)#

Source: datafusion/cast.rs → cast_to_table_schema / build_field_exprs

Applied on add (Append mode only) via a DataFusion ProjectionExec. Per-field rules in build_field_exprs 16:

Column matching is by name, not position, so schema evolution (adding columns after embedding columns) is handled correctly 18.

Blob V2 Descriptor Coercion#

Source: datafusion/blob_coerce.rs → coerce_blob_expr

Applied when the table field is marked as blob v2 and the input field differs. Three accepted input forms 19:

1. Raw binary (Binary, LargeBinary, BinaryView) — data is placed in the data child field; all other struct fields become typed nulls.
2. Pre-built struct with data or uri child present — fields are extracted via get_field(), cast to declared types, and any missing declared fields are filled with typed nulls.
3. Null — written as a null descriptor row.

The result is a DataFusion named_struct physical expression producing a properly shaped blob v2 descriptor. Users supply raw bytes; they never need to construct the descriptor manually.

Key Source Files#