Blob Storage#

Lance blob v2 columns store large binary payloads out of line — queries return small descriptors, not bytes. Callers read actual blob data through dedicated fetch APIs after obtaining stable row IDs from a query. 1

Blob columns require:

• Lance file format ≥ v2.2 (auto-enforced at table creation) 2
• Stable row IDs enabled on the table 3

Blob APIs are local-table only; remote (LanceDB Cloud) tables return NotImplementedError. 4

Schema and Column Type#

A blob v2 column is an Arrow extension type (lance.blob.v2) backed by a 4-field struct 5:

Python — declare with lancedb.schema.blob(name, nullable=True), which returns a pa.Field of type BlobType.

Rust — declare with lancedb::blob(name, nullable), which wraps lance::blob::blob_field.

The extension marker ARROW:extension:name = "lance.blob.v2" on the field metadata distinguishes v2 blobs from legacy v1 blobs (lance-encoding:blob). 6

File Format Requirements (data_storage_version)#

ensure_blob_storage_version() is called during table creation and automatically bumps data_storage_version to at least LanceFileVersion::V2_2 whenever the schema contains any blob v2 column. If the caller specifies a higher version (e.g., V2_3), that is preserved. 7

The tests in blob.rs document this behavior explicitly:

• Default params → bumped to V2_2 8
• Explicit V2_0 → overridden to V2_2 9
• Explicit V2_3 → kept at V2_3 7
• No blob columns → no-op 10

The descriptor struct layout (Struct<data, uri> with the lance.blob.v2 extension marker) cannot be safely stored in format versions below V2_2. 3

Null vs. Empty Value Semantics#

All three fetch APIs preserve the following invariant 11 12:

• Null blob → null slot in the result (None in Python, null in Arrow)
• Valid empty blob (zero bytes, non-null) → b"" in the result

This is enforced in the Rust layer via take_blobs_aligned and take_blob_ranges_aligned, which match on payload.data: Some(data) appends the value; None appends a null slot.

In take_blob_files_aligned, null rows propagate as None entries in the returned Vec<Option<BlobFile>>. 13 Lance v10.0.0-beta.3 fixed a potential panic here by switching from .unwrap() to .flatten() when advancing the handle iterator over nullable results. 14

Read APIs#

Three materialization strategies are available. All require row IDs obtained via with_row_id(True) on a query. Results preserve input length and order. 15

BlobFile is a RawIOBase subclass with seek(), tell(), size(), read_range(offset, length), and async aread(). 16

fetch_blob_ranges is the right choice when you have an index of clip offsets inside videos or byte windows into large files — all ranges are submitted as a single planned Lance operation. 17

Rust entry points are take_blobs_aligned, take_blob_ranges_aligned, and take_blob_files_aligned in rust/lancedb/src/blob.rs.

Write Path#

On write, LanceDB automatically coerces Binary / LargeBinary Arrow inputs into the blob struct layout. Three input forms are accepted 18:

1. Raw binary — lands in the data child field
2. Pre-built struct with data or uri child already populated
3. Null — written as a null descriptor row

The coercion logic lives in rust/lancedb/src/table/datafusion/blob_coerce.rs. Users do not need to construct the descriptor struct manually.

Nested Blobs and Dotted Paths#

Blob columns can be nested inside structs or lists. The fetch APIs address them with dotted paths (e.g., "info.blob"). 19

Schema traversal is recursive and handles Struct, List, LargeList, and FixedSizeList. 20 21

Example from tests: a schema with info.blob (blob nested inside struct field info) — blob_column_names() returns ["info.blob"]. 22

Limitations and Error Handling#

• Local-only: All blob fetch APIs are unsupported on remote (LanceDB Cloud) tables. 23
• Legacy v1 columns rejected: Columns with lance-encoding:blob metadata return InvalidInput with a migration hint pointing to lance.blob.v2. 24
• Invalid row IDs: If fetch_blobs / fetch_blob_ranges receives row IDs not present in the table, it returns InvalidInput. 25
• Non-blob column name: Passing a non-blob column name to any fetch API returns InvalidInput. 24

Key source files:

• rust/lancedb/src/blob.rs — Rust blob v2 definitions, ensure_blob_storage_version, aligned fetch helpers 26
• python/python/lancedb/_blob.py — Python BlobFile and projection helpers 27
• python/python/lancedb/schema.py — BlobType, blob(), schema traversal 28
• python/python/lancedb/table.py — Abstract fetch_blobs, fetch_blob_ranges, fetch_blob_files interfaces 15
• rust/lancedb/src/table/datafusion/blob_coerce.rs — Write-path coercion 29