CJK Input and Unicode Handling#

Langfuse has multiple layers of Unicode support to handle Chinese, Japanese, and Korean (CJK) text correctly at ingestion, search, display, and export. The three main concerns are: (1) how the Python SDK stores non-ASCII content, (2) how ClickHouse queries handle UTF-8 strings, and (3) how frontend search inputs and display components deal with Unicode content.

The Core Problem: Python SDK Unicode Escaping#

The Python SDK's EventSerializer calls json.dumps(..., ensure_ascii=True), which encodes every code point ≥ U+0080 as a \uXXXX escape sequence. This means CJK text like 你好 is stored verbatim in ClickHouse as the 12-byte ASCII string \u4f60\u597d. The OpenTelemetry ingestion path follows the same convention. 1

This has downstream implications for search, display, and export — all three need to be aware of this encoding.

Full-Text Search: Dual-Form ILIKE#

The clickhouseSearchCondition function in packages/shared/src/server/queries/clickhouse-sql/search.ts builds ILIKE-based WHERE clauses for trace/observation search. For input and output columns, a naive ILIKE '%你好%' would never match stored \u4f60\u597d content.

Fix (PR #13644): When the search query contains non-ASCII characters, the function also generates an escaped variant via a toJsonUnicodeEscaped helper that replicates Python's ensure_ascii=True behavior — including UTF-16 surrogate pairs for astral-plane characters (emoji, supplementary CJK). 1

The resulting condition is:

col ILIKE {searchString: String} OR col ILIKE {searchStringEscaped: String}

ASCII-only queries skip the second clause entirely (the escaped form is identical to the input), preserving the existing query plan for the common case. The dual-form logic applies only to input/output columns; identifier columns (id, user_id, name) are unaffected. 1

UTF-8 Safe Truncation in ClickHouse#

ClickHouse's left() function counts bytes, not characters, and will split multi-byte UTF-8 sequences mid-character. All repositories use leftUTF8() instead, which counts Unicode code points. 2

Key locations:

• event-query-builder.ts — conditional leftUTF8(input, charLimit) / leftUTF8(output, charLimit) when truncation is enabled
• events.ts — leftUTF8(t.input, ${env.LANGFUSE_SERVER_SIDE_IO_CHAR_LIMIT}) for trace I/O
• observations.ts and traces.ts — same pattern for the legacy tables
• experiments.ts — applied to experiment_item_expected_output

The truncation limit is controlled by LANGFUSE_SERVER_SIDE_IO_CHAR_LIMIT. For the events pipeline, truncation is either pre-applied by the events_core materialized view or applied at query time via leftUTF8() depending on which table is read. 3

Unicode Display and Export: decodeUnicodeEscapesOnly#

The packages/shared/src/utils/unicode.ts module exports decodeUnicodeEscapesOnly, a single-pass custom decoder that converts \uXXXX sequences back to the original characters. It handles surrogate pairs (\uD83D\uDE00 → 😀) and is robust to truncated/invalid escapes. 4

It runs in two modes 5:

• Non-greedy (default): respects backslash parity, skips \\u-prefixed patterns
• Greedy (true): decodes all \uXXXX patterns regardless of preceding backslashes — used for doubly-escaped content from the Python SDK

Usage sites:

• UI rendering: IOTableCell component and PrettyJsonView use greedy mode so stored \uXXXX sequences display as readable characters 6
• Batch exports (CSV/JSON/JSONL): the stringify / stringifyForCsv helpers in packages/shared/src/server/utils/transforms/stringify.ts apply decoding to all string values 7
• Trace JSON download: the "Download trace as JSON" API routes through the same stringify helper 8

Unicode in Chat/Message Search: NFKD Normalization#

In-page search over chat message content uses CodeMirror. A dependency bump to CodeMirror 6.7.0 (PR #13577) added NFKD normalization support, enabling compatibility matching of fullwidth/halfwidth and composed CJK characters (e.g., searching セン inside ㌢). A prior fix 9 corrected mismatches between the editor's highlight count and the controller's result count caused by fullwidth vs. halfwidth text — the controller was updated to use SearchQuery.getCursor() instead of manual toLocaleLowerCase() + indexOf().

Unicode in Prompt Variables and Identifiers#

Prompt variable validation migrated from ASCII-only and hardcoded JAPANESE_CHAR_RANGE regexes to Unicode property escapes (\p{L} for letters, \p{N} for digits), enabling CJK, Cyrillic, Arabic, and all other scripts in variable names. 10

Signup name validation similarly uses Unicode property escapes with NFC normalization and smart-quote handling. 11

What Is NOT Implemented#

• IME composition event handling (compositionstart / compositionend): No frontend component currently suppresses search/filter queries during an active IME composition. This means a filter that fires on each keystroke will run against the intermediate pinyin/zhuyin buffer before the user commits a character. 12
• NFKC normalization on search queries or filter values: The ClickHouse filter layer (StringFilter, StringObjectFilter, etc. in clickhouse-filter.ts) does not apply any Unicode normalization. Fullwidth/halfwidth variant reconciliation (e.g., ｇｐｔ－４ vs. gpt-4) would need normalization at either the client or query layer. 12