ClickHouse Query Execution#

Overview#

Langfuse executes ClickHouse queries through a thin abstraction layer in packages/shared/src/server/repositories/clickhouse.ts and packages/shared/src/server/clickhouse/client.ts. These two files define all query primitives and the client lifecycle.

Execution Primitives#

Four main functions handle query execution 1:

All SELECT functions guard against reads from the legacy events table at the call site via assertNoLegacyEventsRead.

queryClickhouse includes retry logic for transient network errors 2. Retryable patterns include socket hang up, connection reset, ECONNRESET, ETIMEDOUT, etc., with up to LANGFUSE_CLICKHOUSE_QUERY_MAX_ATTEMPTS attempts at a fixed 100 ms delay.

Client Management#

ClickHouseClientManager is a singleton that pools clients by configuration hash. A new @clickhouse/client instance is created only when the config key changes 3. Key behaviors:

• Service routing: Three service tiers — ReadWrite, ReadOnly, EventsReadOnly — resolved from env vars CLICKHOUSE_URL, CLICKHOUSE_READ_ONLY_URL, CLICKHOUSE_EVENTS_READ_ONLY_URL 4.
• Async inserts: All clients enable async_insert: 1 and wait_for_async_insert: 1 by default 5.
• Progress headers: When request_timeout > 30 s, the client enables send_progress_in_http_headers with a 10-second interval 6.
• OTel propagation: Active span context is injected into HTTP headers so ClickHouse queries appear in traces 7.

The clickhouseClient(opts?, preferredService?) helper is the standard entry point 8.

Server-Side Timeout Enforcement (PR #14730)#

PR #14730 added automatic derivation of max_execution_time from the client request_timeout. When a client is created with a request_timeout, the ClickHouse server setting is set to ⌈request_timeout_ms / 1000⌉ + 5 seconds via a getRequestTimeoutClickHouseSettings helper 9.

This means ClickHouse will kill the query ~5 seconds after the client gives up, preventing orphaned queries from running indefinitely. Key details:

• When no request_timeout is configured, the default 30 s timeout is used, resulting in max_execution_time: 35 9.
• timeout_before_checking_execution_speed is set to 0 so the limit is checked immediately.
• Callers can override the derived value by passing explicit clickhouse_settings — those are merged after the derived settings 9.
• request_timeout is part of the client config hash, so different timeout values get separate cached client instances.

Fire-and-Poll Pattern#

For long-running queries (e.g., background migrations, bulk exports), Langfuse uses a fire-and-poll pattern:

1. Fire: Call commandClickhouse() with an AbortController signal and a pre-generated queryId passed in tags 10. The query is dispatched but the HTTP connection will be severed intentionally.
2. Confirm start: After 5 seconds, call pollQueryStatus(queryId) to verify the query registered on the server 11.
3. Abort the HTTP connection: Once confirmed running, call abortController.abort(). The ClickHouse query continues server-side 12.
4. Poll for completion: A ConcurrentQueryManager periodically calls pollQueryStatus for all active queryIds (default 30-second interval, up to 4 concurrent queries) 13.

pollQueryStatus internals#

pollQueryStatus performs a two-stage lookup with 60 s timeouts each:

1. SELECT query_id FROM clusterAllReplicas('default', 'system.processes') — returns "running" if found 14.
2. SELECT type, exception_code FROM clusterAllReplicas('default', 'system.query_log') — maps QueryFinish → "completed", ExceptionWhileProcessing → "failed", empty → "not_found" 15.

getQueryError(queryId) retrieves the exception message from system.query_log for failed queries.

Both functions use skip_unavailable_shards: 1 to tolerate partial shard availability.

The generateQueryId helper 16 produces IDs in the format backfill-{chunkId}-{8-char uuid}, making them human-readable in ClickHouse system tables.

Error Handling#

ClickHouseResourceError is a typed wrapper for server-side resource failures. It detects three classes by substring-matching the error message 17:

• MEMORY_LIMIT — "memory limit exceeded"
• OVERCOMMIT — "OvercommitTracker"
• TIMEOUT — "Timeout", "timeout", "timed out"

All query functions call wrapIfResourceError in their catch blocks, so callers can instanceof ClickHouseResourceError to distinguish resource failures from other errors.

ClickHouse also has a quirk where exceptions mid-stream are emitted as rows with a single exception key rather than as HTTP errors. handleExceptionRow detects and re-throws these.

Key Source Files#