Null Type Handling in Lance Writer#

Overview#

When PyArrow infers the type of an all-null column, it produces DataType::Null rather than a concrete type (e.g. binary, large_binary). This is expected PyArrow behavior:

pa.array([None]).type # -> null
pa.array([b"x", None]).type # -> binary

The Lance writer's write path does not handle DataType::Null input for two extension column types — blob v2 and lance.json — causing Table.add() to fail whenever every row in a batch has None for those columns. Plain Arrow types (large_binary, binary, string, int64, etc.) accept all-null batches without issue 1.

Affected Paths#

Blob v2 (lancedb.blob())#

The blob coercion logic lives in blob_coerce.rs. Its entry point, coerce_blob_expr, matches on the input field's data type and accepts only Binary | LargeBinary | BinaryView (for raw bytes) or a Struct with a data/uri child (for pre-built descriptors). There is no arm for DataType::Null, so an all-null column falls through to the catch-all error 2:

ValueError: Invalid input, cannot coerce column 'val' with type Null into a blob
v2 struct. expected Binary, LargeBinary, BinaryView, or a Struct with a 'data' or 'uri' child

Arrow can cast Null → LargeBinary to produce an all-null array, and the existing raw-binary branch already fills non-data children with typed nulls via typed_null. Adding DataType::Null to the first match arm is the minimal fix.

lance.json (pa.json_())#

When a table is created with pa.json_(), the column is stored internally as lance.json (a LargeBinary field carrying ARROW:extension:name = lance.json metadata). If the input batch is all-null, PyArrow infers the column as DataType::Null.

build_field_exprs in cast.rs has a special case that bypasses the DataFusion cast when the input is arrow.json and the table field is lance.json 3, allowing lance-core to handle the conversion itself. However, this guard does not fire for DataType::Null input because is_arrow_json_field only recognizes fields carrying arrow.json extension metadata — a Null-typed field has none. The call then falls through to the general cast path, which tries to reconcile Null with LargeBinary in a way that produces a schema mismatch:

RuntimeError: lance error: Append with different schema: `val` should have
type json but type was large_binary

Write Path Context#

Table.add() calls AddDataBuilder::into_plan, which invokes cast_to_table_schema for non-overwrite writes 4. That function calls build_field_exprs 5, which dispatches to coerce_blob_expr for blob v2 columns 6 and applies the arrow.json pass-through guard for json columns 3. Both gaps surface at this point in the pipeline.

Table.add()
  └─ AddDataBuilder::into_plan (add_data.rs)
       └─ cast_to_table_schema (cast.rs)
            ├─ coerce_blob_expr (blob_coerce.rs) ← no Null arm [blob bug]
            └─ arrow.json guard (cast.rs) ← guard misses Null input [json bug]

Impact and Workaround#

The bug is especially disruptive for row-at-a-time inserts where extension columns are optional: a single-row batch with None for the extension column is trivially all-null, so the majority of such writes fail 1.

Workaround: Explicitly declare the column as large_binary in the input schema so PyArrow skips inference. The existing coercion then handles all-null, all-populated, and mixed batches identically 1:

insert_schema = pa.schema([
    pa.field(f.name, pa.large_binary(), nullable=True)
    if f.type == lancedb.blob("_").type else f
    for f in schema
])
await table.add(pa.Table.from_pylist(rows, schema=insert_schema))

Tracking#

• Issue #3759 — bug(python): add() rejects an all-null batch for Lance extension columns (json, blob v2) (open as of 2026-08-01)
• PR #3429 — fix: allow appending arrow.json data into lance.json tables — merged 2026-05-27; introduced the arrow.json pass-through guard but did not cover DataType::Null input

Key Source Files#