Dosu LogoDosu Logo
Ask
Join our Discord
LanceDB's SpacePublic
LanceDB
DocumentsLanceDB's Space
Null Type Handling in Lance Writer
Null Type Handling in Lance Writer
Type
Topic
Status
Published
Created
Aug 1, 2026
Updated
Aug 1, 2026
Created by
Dosu Bot
Updated by
Dosu Bot

Null Type Handling in Lance Writer#

Overview#

When PyArrow infers the type of an all-null column, it produces DataType::Null rather than a concrete type (e.g. binary, large_binary). This is expected PyArrow behavior:

pa.array([None]).type # -> null
pa.array([b"x", None]).type # -> binary

The Lance writer's write path does not handle DataType::Null input for two extension column types — blob v2 and lance.json — causing Table.add() to fail whenever every row in a batch has None for those columns. Plain Arrow types (large_binary, binary, string, int64, etc.) accept all-null batches without issue .

Affected Paths#

Blob v2 (lancedb.blob())#

The blob coercion logic lives in blob_coerce.rs. Its entry point, coerce_blob_expr, matches on the input field's data type and accepts only Binary | LargeBinary | BinaryView (for raw bytes) or a Struct with a data/uri child (for pre-built descriptors). There is no arm for DataType::Null, so an all-null column falls through to the catch-all error :

ValueError: Invalid input, cannot coerce column 'val' with type Null into a blob
v2 struct. expected Binary, LargeBinary, BinaryView, or a Struct with a 'data' or 'uri' child

Arrow can cast Null → LargeBinary to produce an all-null array, and the existing raw-binary branch already fills non-data children with typed nulls via typed_null. Adding DataType::Null to the first match arm is the minimal fix.

lance.json (pa.json_())#

When a table is created with pa.json_(), the column is stored internally as lance.json (a LargeBinary field carrying ARROW:extension:name = lance.json metadata). If the input batch is all-null, PyArrow infers the column as DataType::Null.

build_field_exprs in cast.rs has a special case that bypasses the DataFusion cast when the input is arrow.json and the table field is lance.json , allowing lance-core to handle the conversion itself. However, this guard does not fire for DataType::Null input because is_arrow_json_field only recognizes fields carrying arrow.json extension metadata — a Null-typed field has none. The call then falls through to the general cast path, which tries to reconcile Null with LargeBinary in a way that produces a schema mismatch:

RuntimeError: lance error: Append with different schema: `val` should have
type json but type was large_binary

Write Path Context#

Table.add() calls AddDataBuilder::into_plan, which invokes cast_to_table_schema for non-overwrite writes . That function calls build_field_exprs , which dispatches to coerce_blob_expr for blob v2 columns and applies the arrow.json pass-through guard for json columns . Both gaps surface at this point in the pipeline.

Table.add()
  └─ AddDataBuilder::into_plan (add_data.rs)
       └─ cast_to_table_schema (cast.rs)
            ├─ coerce_blob_expr (blob_coerce.rs) ← no Null arm [blob bug]
            └─ arrow.json guard (cast.rs) ← guard misses Null input [json bug]

Impact and Workaround#

The bug is especially disruptive for row-at-a-time inserts where extension columns are optional: a single-row batch with None for the extension column is trivially all-null, so the majority of such writes fail .

Workaround: Explicitly declare the column as large_binary in the input schema so PyArrow skips inference. The existing coercion then handles all-null, all-populated, and mixed batches identically :

insert_schema = pa.schema([
    pa.field(f.name, pa.large_binary(), nullable=True)
    if f.type == lancedb.blob("_").type else f
    for f in schema
])
await table.add(pa.Table.from_pylist(rows, schema=insert_schema))

Tracking#

  • Issue #3759 — bug(python): add() rejects an all-null batch for Lance extension columns (json, blob v2) (open as of 2026-08-01)
  • PR #3429 — fix: allow appending arrow.json data into lance.json tables — merged 2026-05-27; introduced the arrow.json pass-through guard but did not cover DataType::Null input

Key Source Files#

FilePurpose
rust/lancedb/src/table/datafusion/blob_coerce.rsBlob v2 coercion logic; missing Null match arm
rust/lancedb/src/table/datafusion/cast.rsSchema-cast pipeline; arrow.json pass-through guard
rust/lancedb/src/table/add_data.rsAddDataBuilder::into_plan; orchestrates casting
Documents
Apache Arrow Type System
Blob Storage
Cross-Platform Build Support
Deadlock Prevention
Debugger-Safe Object Representation
Hybrid Query Builder
Namespace Layer Architecture
Namespace Manifest Management
Null Type Handling in Lance Writer
PyO3 Async Runtime Bridge
SDK and Multi-Language Architecture
Table Listing API
Table Optimization
Vector Search Result Ordering
Vendored Dependency Version Skew
Write Path Architecture
Monitoring with OpenTelemetry Metrics