Background Migration Timeout Configuration#

Langfuse uses two layered timeout mechanisms to protect ClickHouse from runaway queries in long-running background migrations and blob-storage export jobs: a client-side request timeout (enforced by the Node.js HTTP layer) and a server-side execution cap (max_execution_time) derived automatically from that timeout. Both are tunable via environment variables defined in packages/shared/src/env.ts.

Key Environment Variables#

Both variables are declared and validated via Zod in the shared env schema 1.

History: LANGFUSE_CLICKHOUSE_DATA_EXPORT_REQUEST_TIMEOUT_MS was originally introduced with a 10-minute default. PR #14753 raised this to 60 minutes after production logs showed TIMEOUT_EXCEEDED errors at ~606 s during full-history backfills for large projects.

How Client Timeout Becomes a Server Timeout#

PR #14730 added logic in packages/shared/src/server/clickhouse/client.ts that derives ClickHouse server settings from the client's request_timeout:

max_execution_time = ceil(request_timeout_ms / 1000) + 5
timeout_before_checking_execution_speed = 0

The 5-second grace window ensures ClickHouse kills orphaned server-side queries just after the client HTTP connection gives up, preventing runaway queries that continue to consume resources after the client has moved on. 2

Setting override precedence inside ClickHouseClientManager.getClient():

1. Env-var async-insert overrides
2. Cloud-region options
3. Service-level ClickHouse settings
4. Derived timeout settings (max_execution_time, timeout_before_checking_execution_speed)
5. Caller-supplied clickhouse_settings — these win and can override the derived values

How It's Used in Background Migrations#

BackfillEventsHistoric (v4 Historic Backfill)#

The backfillEventsHistoric worker fires long INSERT … SELECT queries from observations_pid_tid_sorting into events_full. Each query is fired with an AbortController: the HTTP connection is intentionally dropped after the query is confirmed running on the server, and completion is tracked via polling system.query_log 3.

Because these queries run detached from the client HTTP lifecycle, they are not capped by LANGFUSE_CLICKHOUSE_DATA_EXPORT_REQUEST_TIMEOUT_MS. The only per-query timeout in the recovery path is a hardcoded 60-second request_timeout used when checking system.processes during worker restarts 4.

If you need to impose a server-side cap on backfill chunk queries, you can pass clickhouseSettings: { max_execution_time: <seconds> } in the commandClickhouse call inside fireQuery() 5, since caller-supplied settings take highest precedence.

Blob Storage Export / Analytics Integration Export#

All blob-storage and analytics-integration export jobs read LANGFUSE_CLICKHOUSE_DATA_EXPORT_REQUEST_TIMEOUT_MS and pass it as request_timeout to the ClickHouse client. The derived max_execution_time is automatically set to ceil(3600) + 5 = 3605 s under the default 60-minute setting. 6

Relevant Files#