Monitoring with OpenTelemetry Metrics#

Overview#

LanceDB exposes internal performance metrics through OpenTelemetry, giving you visibility into object store behavior — request counts, latency, bytes transferred, errors, and throttles — directly in your existing observability stack.

The integration is built on a pull-based adapter in the LanceDB Rust core 1. Lance publishes metrics through the metrics crate facade; the adapter aggregates them into lock-free cumulative storage and exposes a snapshot API that language bindings call whenever your OpenTelemetry MetricReader fires. The bridge is generic: every metric Lance describes is surfaced automatically with no per-metric Python or JavaScript code needed.

What is exposed#

Currently, the following object store metrics are available:

All metrics carry operation (e.g., read, write, head) and base (the storage scheme, e.g., file, s3, gs, az) labels so you can filter and aggregate at any granularity.

Architecture#

The integration follows OpenTelemetry's separation of concerns:

• LanceDB acts as a library: it wires up observable instruments and registers callbacks that snapshot the internal state on demand.
• Your application acts as the host: it configures the MeterProvider, chooses a MetricReader, and points an exporter at whatever backend you use (Prometheus, Grafana, Datadog, OTLP endpoint, etc.).

This design means upgrading your observability backend requires no changes to LanceDB code.

Language and version support#

Requires Lance ≥ v9.0.0-beta.19, which ships the object-store metrics APIs 1.

Installation#

Python#

The OpenTelemetry integration requires the otel extra, which installs the OpenTelemetry API package:

pip install "lancedb[otel]"

This pulls in opentelemetry-api only — the thin, zero-dependency API surface that LanceDB uses to register instruments 1. To actually export metrics you need an OpenTelemetry SDK. Install the standard SDK with:

pip install opentelemetry-sdk

For production use, you will typically also install an exporter such as opentelemetry-exporter-otlp or opentelemetry-exporter-prometheus.

Design note: LanceDB follows the OpenTelemetry recommendation that libraries depend only on the API, while the application chooses and configures the SDK. This keeps LanceDB's dependency footprint minimal.

Node.js#

@opentelemetry/api is a direct runtime dependency of @lancedb/lancedb, so no extra installation is needed for the API 1. To export metrics, add the SDK:

npm install @opentelemetry/sdk-metrics
# or
pnpm add @opentelemetry/sdk-metrics

For production pipelines you may also want @opentelemetry/exporter-metrics-otlp-http or a Prometheus bridge.

Python Integration#

API#

from lancedb.otel import instrument_lancedb_metrics

result: bool = instrument_lancedb_metrics(meter_provider=None)

Parameters

Returns bool — True if the recorder was successfully installed and instruments were registered. False if a different metrics recorder is already installed in the process; in that case a UserWarning is emitted and no instruments are created 1.

Calling instrument_lancedb_metrics() more than once is safe — instruments are created only on the first successful call, and subsequent calls return True immediately.

If opentelemetry-api is not installed, an ImportError is raised with a message pointing to the correct install command.

Basic example (in-memory, for debugging)#

import lancedb
import pyarrow as pa
from lancedb.otel import instrument_lancedb_metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import ConsoleMetricExporter, PeriodicExportingMetricReader

# Configure a MeterProvider that prints metrics to stdout every 10 seconds.
exporter = ConsoleMetricExporter()
reader = PeriodicExportingMetricReader(exporter, export_interval_millis=10_000)
provider = MeterProvider(metric_readers=[reader])

# Register LanceDB instruments on the provider. Call this once at startup.
instrument_lancedb_metrics(provider)

# Use LanceDB normally — metrics are collected automatically.
db = lancedb.connect("/tmp/mydb")
table = db.create_table("items", pa.table({"id": pa.array(range(1000))}))
table.search([1, 2, 3]).limit(10).to_arrow()

provider.shutdown()

Verifying with InMemoryMetricReader#

During testing or verification, use InMemoryMetricReader to inspect collected metrics without a network exporter — the same pattern used in LanceDB's own test suite 1:

import lancedb
import pyarrow as pa
from lancedb.otel import instrument_lancedb_metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import InMemoryMetricReader

reader = InMemoryMetricReader()
provider = MeterProvider(metric_readers=[reader])
instrument_lancedb_metrics(provider)

db = lancedb.connect("/tmp/testdb")
db.create_table("t", pa.table({"id": pa.array(range(256))}))

# Inspect collected metrics.
data = reader.get_metrics_data()
for rm in data.resource_metrics:
    for sm in rm.scope_metrics:
        for metric in sm.metrics:
            print(metric.name, [p.value for p in metric.data.data_points])

Production example (OTLP export)#

from lancedb.otel import instrument_lancedb_metrics
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter

exporter = OTLPMetricExporter(endpoint="http://otel-collector:4317", insecure=True)
reader = PeriodicExportingMetricReader(exporter, export_interval_millis=30_000)
provider = MeterProvider(metric_readers=[reader])

# Must be called before LanceDB begins serving traffic.
instrument_lancedb_metrics(provider)

Replace the OTLPMetricExporter endpoint with your collector, Grafana Agent, or vendor ingest URL. The metrics flow into any backend that accepts OTLP (Prometheus, Grafana, Datadog, Honeycomb, etc.).

Node.js Integration#

API#

import { instrumentLanceDbMetrics } from "@lancedb/lancedb";

const result: boolean = instrumentLanceDbMetrics(meterProvider?);

Parameters

Returns boolean — true if the recorder was installed and instruments were registered. false if a different metrics recorder is already installed in the process; in that case a warning is logged via console.warn and no instruments are created 1.

Calling instrumentLanceDbMetrics() more than once is safe — instruments are created only on the first successful call.

Basic example (pull-based reader, for testing/debugging)#

The following pattern is used in LanceDB's own test suite 1:

import {
  MeterProvider,
  MetricReader,
} from "@opentelemetry/sdk-metrics";
import { connect, instrumentLanceDbMetrics } from "@lancedb/lancedb";

// A minimal pull-based reader. Call collect() to trigger observable callbacks.
class ManualMetricReader extends MetricReader {
  protected async onForceFlush(): Promise<void> {}
  protected async onShutdown(): Promise<void> {}
}

const reader = new ManualMetricReader();
const provider = new MeterProvider({ readers: [reader] });

// Register LanceDB instruments. Call this once, before any DB operations.
instrumentLanceDbMetrics(provider);

// Use LanceDB normally.
const db = await connect("/tmp/mydb");
const table = await db.createTable("items", Array.from({ length: 256 }, (_, i) => ({ id: i })));
await table.countRows();

// Collect metrics on demand.
const collected = await reader.collect();
for (const scope of collected.resourceMetrics.scopeMetrics) {
  for (const metric of scope.metrics) {
    console.log(metric.descriptor.name, metric.dataPoints);
  }
}

await provider.shutdown();

Production example (OTLP export)#

import { MeterProvider, PeriodicExportingMetricReader } from "@opentelemetry/sdk-metrics";
import { OTLPMetricExporter } from "@opentelemetry/exporter-metrics-otlp-http";
import { instrumentLanceDbMetrics } from "@lancedb/lancedb";

const exporter = new OTLPMetricExporter({
  url: "http://otel-collector:4318/v1/metrics",
});

const reader = new PeriodicExportingMetricReader({
  exporter,
  exportIntervalMillis: 30_000,
});

const provider = new MeterProvider({ readers: [reader] });

// Must be called before LanceDB begins serving traffic.
instrumentLanceDbMetrics(provider);

Install the exporter package separately:

npm install @opentelemetry/exporter-metrics-otlp-http

Metric Types#

LanceDB uses three metric kinds, each mapped to an OpenTelemetry observable instrument type 1.

Counters#

Counters map directly to ObservableCounter instruments. They report cumulative totals that only increase over time. Example: lance_object_store_requests_total.

Gauges#

Gauges map directly to ObservableGauge instruments. They report an instantaneous value that can go up or down. Example: lance_object_store_in_flight_requests.

Histograms#

OpenTelemetry has no asynchronous histogram instrument, so each histogram is decomposed into three separate observable counters following Prometheus conventions 1:

For example, lance_object_store_request_duration_seconds produces:

• lance_object_store_request_duration_seconds_bucket (unitless, with le attribute)
• lance_object_store_request_duration_seconds_count (unitless)
• lance_object_store_request_duration_seconds_sum (unit: s)

Why is _bucket unitless? The bucket and count instruments count observations (dimensionless integers), not the measured quantity itself. Only _sum accumulates the measured values and therefore retains the physical unit. This matches the Prometheus exposition format.

Available Metrics#

All currently available metrics come from the Lance object store layer 1. All metrics carry two labels:

• operation — the storage operation being performed (e.g., get, put, delete, list, head, copy).
• base — the storage scheme / backend (e.g., file, s3, gs, az, memory).

lance_object_store_requests_total#

Kind: Counter
Unit: (unitless — count of requests)

Total number of completed object store requests. Increment once per request, regardless of success or failure.

Use this metric to understand request throughput and to compare request rates across different operations and backends.

lance_object_store_request_duration_seconds#

Kind: Histogram → exported as _bucket, _count, _sum
Unit: s (seconds, on _sum only)

Latency of each completed object store request. The histogram buckets let you compute percentiles (p50, p95, p99) in your observability backend.

lance_object_store_in_flight_requests#

Kind: Gauge
Unit: (unitless — count of concurrent requests)

Instantaneous count of in-flight (started but not yet completed) object store requests. Useful for diagnosing concurrency bottlenecks or unexpected request pile-ups.

lance_object_store_retryable_responses_total#

Kind: Counter
Unit: (unitless — count of responses)

Total number of throttle or otherwise retryable responses received from the object store. A sustained non-zero value indicates the backend is rate-limiting your workload.

Note: The metric catalog is populated at startup when the recorder is installed. New metrics introduced in future Lance versions are surfaced automatically — no SDK upgrade required.

Important Notes#

The metrics recorder is process-global#

The underlying metrics crate permits exactly one global recorder per process. Once the LanceDB recorder is installed, no other metrics-compatible recorder can be installed in the same process 1.

• If you call instrument_lancedb_metrics() / instrumentLanceDbMetrics() a second time after a successful first call, the function is a no-op and returns True/true — the recorder and instruments already exist.
• If a different recorder was installed before you called the LanceDB function, the function emits a warning and returns False/false. No LanceDB instruments are created in that case.

Python warning text:

UserWarning: Could not install the LanceDB metrics recorder: another `metrics`
recorder is already installed in this process. LanceDB metrics will
not be exported via OpenTelemetry.

Node.js warning text (console.warn):

Could not install the LanceDB metrics recorder: another `metrics` recorder
is already installed in this process. LanceDB metrics will not be exported
via OpenTelemetry.

GIL is released during snapshot (Python)#

When the OpenTelemetry MetricReader collects metrics, the Python binding releases the GIL while walking the metric registry 1. This means periodic metric collection does not stall other Python threads.

Feature flags (Rust crate only)#

If you are using LanceDB directly as a Rust crate, the metrics integration is disabled by default. Enable it with the appropriate Cargo feature 1:

# Cargo.toml
[dependencies]
lancedb = { version = "...", features = ["metrics-otel"] }

Both Python and Node.js builds already enable metrics-otel by default, so there is nothing extra to configure in those environments.

Troubleshooting#

Metrics don't appear in my backend#

Check your SDK configuration. instrument_lancedb_metrics() / instrumentLanceDbMetrics() only registers OpenTelemetry observable instruments and their callbacks. It does not configure how or where metrics are exported. You must set up a MetricReader and an exporter on your MeterProvider separately.

Common mistakes:

• Passing no exporter to MeterProvider — without a reader, the SDK collects nothing.
• Using the default global MeterProvider (a no-op provider) — always pass an explicitly configured provider, or configure the global provider before calling the instrument function.
• Calling instrument_lancedb_metrics() before your MeterProvider is configured — instruments are bound to the provider at call time.

A quick sanity check using Python's InMemoryMetricReader:

from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import InMemoryMetricReader
from lancedb.otel import instrument_lancedb_metrics
import lancedb, pyarrow as pa

reader = InMemoryMetricReader()
provider = MeterProvider(metric_readers=[reader])
instrument_lancedb_metrics(provider)

db = lancedb.connect("/tmp/check")
db.create_table("check", pa.table({"x": pa.array([1, 2, 3])}))

data = reader.get_metrics_data()
names = [m.name for rm in data.resource_metrics for sm in rm.scope_metrics for m in sm.metrics]
print(names) # Should include lance_object_store_requests_total, etc.

Warning: "another metrics recorder is already installed"#

This means some other library or framework in your process already claimed the metrics global slot before LanceDB. The metrics crate allows only one recorder per process.

Options:

1. Move LanceDB instrumentation earlier — call instrument_lancedb_metrics() before any other library initializes its recorder.
2. Check for conflicting libraries — search your dependency tree for other crates or packages that install a metrics recorder.
3. Accept the limitation — if you cannot resolve the conflict, LanceDB metrics will not be exported in that process.

ImportError on instrument_lancedb_metrics (Python)#

If you see:

ImportError: instrument_lancedb_metrics requires the OpenTelemetry API/SDK.
Install it with `pip install lancedb[otel]` or `pip install opentelemetry-sdk`.

Install the required package:

pip install "lancedb[otel]"
# Or, to also get the full SDK:
pip install opentelemetry-sdk

Version requirement#

The OpenTelemetry metrics integration requires Lance ≥ v9.0.0-beta.19, which ships the object-store metrics APIs. Make sure your LanceDB installation is up to date 1.