Apache Arrow Type System in the Node.js SDK#

The Node.js SDK's Arrow type handling lives primarily in two files:

• nodejs/lancedb/arrow.ts — schema inference, table construction, and IPC serialization.
• nodejs/lancedb/sanitize.ts — type normalization across multiple Arrow library instances.

Everything in arrow.ts is re-exported from apache-arrow, so consumers can use LanceDB's Arrow instance directly via import { ... } from "@lancedb/lancedb/arrow" 1.

Schema Inference from JavaScript Records#

makeArrowTable is the primary entry point for converting Record<string, unknown>[] (row-major JS objects) into a columnar Arrow Table. Schema inference is handled by inferSchema.

Inference Rules#

When no schema is provided, JS values are mapped to Arrow types by inferType:

2

Vector Column Heuristics#

Arrays are a special case. Before falling back to List, inferType checks two conditions 3:

1. Explicit vectorColumns option — if the column name appears in MakeArrowTableOptions.vectorColumns, the array is encoded as a FixedSizeList with the configured float type (default Float32).
2. Name heuristic — if the field name contains "vector" or "embedding" (case-insensitive), the array is automatically promoted to a FixedSizeList<Float32> (or FixedSizeList<Uint8> for integer data). See nameSuggestsVectorColumn.

By default, a column named "vector" is treated as Float32 vector without any explicit configuration 4.

Typed Array → FixedSizeList#

TypedArray column values (e.g. Float32Array, Uint8Array) bypass the name heuristic and are always inferred as FixedSizeList via typedArrayToArrowType. This handles Float32Array, Float64Array, Uint8Array, Uint16Array, Uint32Array, Int8Array, Int16Array, and Int32Array.

Nested Structs#

inferSchema uses a PathTree to traverse nested object fields. Plain objects (not Date, Buffer, Array, Set, Map, or typed arrays) are recursed into and emitted as Struct columns 5. Nested paths are represented as dotted strings and unified under the Struct Arrow type.

Schema Subset Behavior#

If a schema is provided but some schema fields are absent from the data, makeArrowTable returns a subset of the schema matching only the fields present in the records 6. This is intentional to support partial-write patterns.

Known Bug: Type Equality in inferSchema#

inferSchema is supposed to detect type mismatches across records (e.g., a column that is Float64 in row 0 but Utf8 in row 1). However, the comparison uses object identity (!==) rather than semantic equality 7. Because each inferType call creates a new DataType instance, logically identical types are always considered different, and the mismatch code path is never reached. Compounding this, the error is created but never thrown, so inconsistent types are silently ignored.

This is tracked in issue #3781. The proposed fix is to compare semantic type representations (e.g. via dataTypeToJson) and actually throw on mismatch.

Semantic Type Comparison: dataTypeToJson#

LanceDB's approach to type equality is serialization-based. dataTypeToJson converts any DataType to a plain JsonDataType object using a switch on typeId. The JSON representation matches the format used by the Rust lance crate 8. Comparing the JSON output of two types is the intended way to check semantic equivalence — and this is the approach proposed in issue #3781 to fix the inferred-type mismatch bug.

Key mappings in dataTypeToJson 9:

• Float → "halffloat" / "float" / "double" based on precision
• Timestamp → "timestamp:{unit}:{timezone}"
• Decimal → "decimal:{bitWidth}:{precision}:{scale}"
• FixedSizeList → { type: "fixed_size_list", fields: [...], length: N }
• Dictionary → "dict:{valueType}:{indexType}:false"

Multi-Version Sanitization#

Node.js allows multiple Arrow library versions to coexist. Because apache-arrow uses instanceof checks that are version-specific, types from a different Arrow instance fail those checks. sanitize.ts solves this by reconstructing types from their structural properties rather than comparing identity.

The core function is sanitizeType: it reads typeId from any object (function or number form) and dispatches to per-type constructors from LanceDB's own Arrow instance. Cascading sanitizers handle nested types:

• sanitizeField → calls sanitizeType on the field's type 10
• sanitizeSchema → maps sanitizeField over all fields 11
• sanitizeTable → sanitizes schema and all record batches

All public entry points (makeArrowTable, fromTableToBuffer, etc.) call sanitizeSchema before processing user-supplied schemas, ensuring cross-version safety 12.

Related Bug: Schema Metadata Validation#

sanitizeMetadata validates that schema metadata is a Map<string, string> 13, but sanitizeSchema short-circuits on instanceof Schema and skips metadata validation when the schema already passes the instance check 14. This means malformed metadata (e.g. Map<string, number>) is silently accepted when the schema is already the correct Arrow instance. Tracked in issue #3729.

Key Entry Points#