ClickHouse Full-Text Search#

Overview#

Langfuse implements full-text search (FTS) in ClickHouse using TEXT skip indexes (splitByNonAlpha tokenizer) on the events_full table. This is a v4-era feature requiring ClickHouse ≥ 25.12 and applies to input, output, metadata_names, and metadata_values columns. Legacy traces and observations tables use bloom-filter skip indexes for equality/exact lookups, not FTS.

The core FTS query helpers live in packages/shared/src/server/queries/clickhouse-sql/fts.ts 1.

Schema: TEXT Index Definition#

Migration 0039_create_events_full.up.sql 2 defines the events_full table with:

INDEX idx_input input TYPE TEXT(splitByNonAlpha) GRANULARITY 1,
INDEX idx_output output TYPE TEXT(splitByNonAlpha) GRANULARITY 1,
INDEX idx_metadata_names metadata_names TYPE TEXT(splitByNonAlpha) GRANULARITY 1,
INDEX idx_metadata_values metadata_values TYPE TEXT(splitByNonAlpha) GRANULARITY 1

A follow-on migration 0043_add_events_metadata_ngram_index.up.sql adds an ngram index on metadata columns 2. The enable_full_text_index = 1 table setting must be set to activate TEXT index usage 3.

Client Setting: enable_full_text_index#

The ClickHouse client for the events read path injects enable_full_text_index: 1 via ServiceClickhouseSettings 3. This setting is gated to the EventsReadOnly connection type and is not applied globally. Without it, ClickHouse ignores the TEXT index even when it exists. See packages/shared/src/server/clickhouse/client.ts.

The matches Operator#

What it does#

matches is a public filter operator on the v2 observations API 4. It routes queries through the TEXT index for input, output, and metadata columns on the events_* tables. It is not available on v1 endpoints.

The operator is defined as FTS_MATCH_OPERATOR = "matches" in packages/shared/src/interfaces/filters.ts 4.

Generated SQL#

The final SQL condition for a matches filter on a text field 5:

position(lower(field), lower(value)) > 0
AND hasAllTokens(lower(field), arraySlice(arrayDistinct(tokens(lower(value))), 1, 64))

• position(lower(field), lower(value)) > 0 — enforces ordered, contiguous literal-substring match (case-insensitive).
• hasAllTokens(...) — serves as a ClickHouse TEXT index prefilter only; it prunes granules but the position() check enforces actual semantics 6.

For metadata array matching, ftsMetadataArrayIndexedSubstringCondition is used instead — case-sensitive (no lower()): hasKey AND hasAllTokens(valuesColumn, ...) AND position(valueAccessor, value) > 0.

Validation#

assertValidFtsMatchFilter throws InvalidRequestError if:

• The value contains no alphanumeric tokens (empty query).
• The filter target is not input, output, or metadata on an events table.

validateIndexedInputOutputFilters in events.ts rejects v2 requests that include an IO filter but no = or matches predicate, preventing accidental full-table IO scans 4.

hasAllTokens Token Limit (≤ 64 tokens)#

ClickHouse's hasAllTokens function supports a maximum of 64 token arguments. Queries with search strings tokenizing to >64 tokens would fail at runtime 7.

Fix (PR #14114): Token arrays are wrapped in arraySlice(arrayDistinct(tokens(...)), 1, 64) before being passed to hasAllTokens. The constant FTS_HAS_ALL_TOKENS_MAX_SEARCH_TOKENS = 64 is defined in fts.ts 8. Correctness is unaffected because position() or exact-equality conditions enforce full semantic matching.

Semantic Evolution#

Version Requirements#

• ClickHouse ≥ 25.12 required for TEXT indexes and enable_block_number_column 2.
• ClickHouse 24.3 support was dropped when v4 events tables were promoted to migrations.
• For general ClickHouse version compatibility issues (analyzer bugs, lazy materialization), see the ClickHouse Version Compatibility knowledge base article.

Key Files#