ClickHouse Full-Text Search#
Overview#
Langfuse implements full-text search (FTS) in ClickHouse using TEXT skip indexes (splitByNonAlpha tokenizer) on the events_full table. This is a v4-era feature requiring ClickHouse ≥ 25.12 and applies to input, output, metadata_names, and metadata_values columns. Legacy traces and observations tables use bloom-filter skip indexes for equality/exact lookups, not FTS.
The core FTS query helpers live in packages/shared/src/server/queries/clickhouse-sql/fts.ts .
Schema: TEXT Index Definition#
Migration 0039_create_events_full.up.sql defines the events_full table with:
INDEX idx_input input TYPE TEXT(splitByNonAlpha) GRANULARITY 1,
INDEX idx_output output TYPE TEXT(splitByNonAlpha) GRANULARITY 1,
INDEX idx_metadata_names metadata_names TYPE TEXT(splitByNonAlpha) GRANULARITY 1,
INDEX idx_metadata_values metadata_values TYPE TEXT(splitByNonAlpha) GRANULARITY 1
A follow-on migration 0043_add_events_metadata_ngram_index.up.sql adds an ngram index on metadata columns . The enable_full_text_index = 1 table setting must be set to activate TEXT index usage .
Client Setting: enable_full_text_index#
The ClickHouse client for the events read path injects enable_full_text_index: 1 via ServiceClickhouseSettings . This setting is gated to the EventsReadOnly connection type and is not applied globally. Without it, ClickHouse ignores the TEXT index even when it exists. See packages/shared/src/server/clickhouse/client.ts.
The matches Operator#
What it does#
matches is a public filter operator on the v2 observations API . It routes queries through the TEXT index for input, output, and metadata columns on the events_* tables. It is not available on v1 endpoints.
The operator is defined as FTS_MATCH_OPERATOR = "matches" in packages/shared/src/interfaces/filters.ts .
Generated SQL#
The final SQL condition for a matches filter on a text field :
position(lower(field), lower(value)) > 0
AND hasAllTokens(lower(field), arraySlice(arrayDistinct(tokens(lower(value))), 1, 64))
position(lower(field), lower(value)) > 0— enforces ordered, contiguous literal-substring match (case-insensitive).hasAllTokens(...)— serves as a ClickHouse TEXT index prefilter only; it prunes granules but theposition()check enforces actual semantics .
For metadata array matching, ftsMetadataArrayIndexedSubstringCondition is used instead — case-sensitive (no lower()): hasKey AND hasAllTokens(valuesColumn, ...) AND position(valueAccessor, value) > 0.
Validation#
assertValidFtsMatchFilter throws InvalidRequestError if:
- The value contains no alphanumeric tokens (empty query).
- The filter target is not
input,output, ormetadataon an events table.
validateIndexedInputOutputFilters in events.ts rejects v2 requests that include an IO filter but no = or matches predicate, preventing accidental full-table IO scans .
hasAllTokens Token Limit (≤ 64 tokens)#
ClickHouse's hasAllTokens function supports a maximum of 64 token arguments. Queries with search strings tokenizing to >64 tokens would fail at runtime .
Fix (PR #14114): Token arrays are wrapped in arraySlice(arrayDistinct(tokens(...)), 1, 64) before being passed to hasAllTokens. The constant FTS_HAS_ALL_TOKENS_MAX_SEARCH_TOKENS = 64 is defined in fts.ts . Correctness is unaffected because position() or exact-equality conditions enforce full semantic matching.
Semantic Evolution#
| PR | Change |
|---|---|
| #13782 | Initial FTS: TEXT indexes on events_full/events_core; enable_full_text_index: 1; hasAllTokens conjunct added implicitly to ILIKE content search |
| #13863 | Explicit matches operator on v2 API; FTS SQL centralized in FTS_OPERATOR_DESCRIPTORS; v2 IO filters must include indexed predicate |
| #13928 | matches semantics tightened from pure token conjunction to position() AND hasAllTokens() — enforces ordered literal-substring matching, not bag-of-words |
| #14114 | Cap token array to 64 via arraySlice; prevents ClickHouse runtime error on long queries |
| #14812 | Events tables promoted to formal CH migrations (0038–0043); v4 defaults flipped to events_only; ClickHouse 24.3 dropped |
Version Requirements#
- ClickHouse ≥ 25.12 required for TEXT indexes and
enable_block_number_column. - ClickHouse 24.3 support was dropped when v4 events tables were promoted to migrations.
- For general ClickHouse version compatibility issues (analyzer bugs, lazy materialization), see the ClickHouse Version Compatibility knowledge base article.
Key Files#
| File | Purpose |
|---|---|
packages/shared/src/server/queries/clickhouse-sql/fts.ts | All FTS helpers: predicates, descriptors, validation |
packages/shared/src/server/queries/clickhouse-sql/clickhouse-filter.ts | StringFilter/StringObjectFilter apply matches via FTS descriptors |
packages/shared/src/interfaces/filters.ts | FTS_MATCH_OPERATOR, FtsMatchOperator type |
packages/shared/clickhouse/migrations/*/0039_create_events_full.up.sql | TEXT index DDL for events_full |
packages/shared/src/server/clickhouse/client.ts | enable_full_text_index: 1 in EventsReadOnly settings |
packages/shared/src/server/repositories/events.ts | validateIndexedInputOutputFilters |