V4 Data Pipeline Migration#

Langfuse V4 replaces the separate traces + observations ClickHouse tables with a unified events-based model: every span, observation, and trace root is a row in a single wide table with trace-level fields denormalized onto each row 1. The migration is additive — new tables are added alongside the old ones — and uses a dual-write pattern so deployments can pace their own cutover 2.

            Old model New model
  ┌──────────┐ ┌──────────────┐ ┌──────────────────────────────┐
  │ traces │ │ observations │ → │ events_full │
  └──────────┘ └──────────────┘ │ (every span, denormalized) │
   joined at read time └──────────────────────────────┘
                                               ↑ MV
                                      ┌────────────────┐
                                      │ events_core │
                                      │ (truncated I/O)│
                                      └────────────────┘

New ClickHouse tables (DDL in dev-tables.sh):

Traces have no separate entity in V4. Each trace is represented as a virtual root span with span_id = t-{traceId}, making the unified table queryable for both spans and trace-level aggregations 1.

Breaking changes after cutover to events_only:

• Legacy batch ingestion (POST /api/public/ingestion, traces/spans/generations/events) → OTLP (POST /api/public/otel/v1/traces) 1
• Read APIs (GET /api/public/traces, /observations, /sessions, etc.) → Observations API v2
• Python SDK ≤ v2 and JS/TS SDK ≤ v3 are rejected 1
• Trace-level LLM-as-a-Judge evaluators stop running 1

All of these keep working in legacy or dual write mode, so the cutover is controlled by the operator.

Write Mode Transitions#

The LANGFUSE_MIGRATION_V4_WRITE_MODE env var controls which tables ingestion targets. All migration modes are temporary — legacy and dual will be removed in a future major version 1.

Important sequencing rules 1:

• Upgrade ClickHouse to ≥ 25.12 before the server upgrade (also requires PostgreSQL ≥ 15, Redis ≥ 7.0).
• Keep LANGFUSE_BACKGROUND_MIGRATION_V4_ENABLE_HISTORIC_BACKFILL=false while on legacy — the backfill runs once, and data written between its cutoff and the start of dual write would be permanently missing from the new tables.
• LANGFUSE_MIGRATION_V4_NATIVE_OTEL_BEHAVIOUR: set to dual_write to retain server-side attribute propagation for OTel producers that haven't adopted client-side propagation; defaults to direct (V4 behavior).

Full configuration reference 1:

Dual write mode sample config 1:

LANGFUSE_MIGRATION_V4_WRITE_MODE=dual
LANGFUSE_MIGRATION_V4_NATIVE_OTEL_BEHAVIOUR=dual_write
LANGFUSE_BACKGROUND_MIGRATION_V4_ENABLE_HISTORIC_BACKFILL=false
LANGFUSE_MIGRATION_V4_ALLOW_PREVIEW_OPT_IN=false

ClickHouse schema migrations (new events tables, materialized views) are applied automatically on server startup via golang-migrate 3. No manual schema work is required.

Dual-Write Pipeline and EventPropagation Worker#

In dual mode, IngestionService writes to both the legacy observations table and observations_batch_staging. This is gated by the writeToStagingTables flag / LANGFUSE_EXPERIMENT_INSERT_INTO_EVENTS_TABLE env var 2.

Two staging record types 2:

• Observation records — written verbatim, stamped with a partition-aware timestamp
• Trace records — converted via convertTraceToStagingObservation to a pseudo-span with span_id = trace_id

Partition-aware timestamp — getPartitionAwareTimestamp(): if createdAtTimestamp is within the last 2 minutes, use it as-is; otherwise, use Date.now(). This prevents churning already-processed 3-minute partitions 2.

Who writes directly vs. through staging 1:

• Python SDK ≥ 4.7.0, JS/TS SDK ≥ 5.4.0, or OTel with x-langfuse-ingestion-version: 4 → writes events_full directly, no delay
• Older SDKs → staging pipeline → ~10 minute delay

EventPropagation Job#

handleEventPropagationJob runs every minute via BullMQ and propagates completed staging partitions into events_full 2:

Redis cursor → select oldest completed partition → JOIN with traces → INSERT INTO events_full → update cursor

1. Cursor — reads langfuse:event-propagation:last-processed-partition from Redis 2
2. Partition selection — queries system.parts for the oldest partition older than LANGFUSE_EXPERIMENT_EVENT_PROPAGATION_PARTITION_DELAY_MINUTES (default 10 min) and after the cursor 2
3. JOIN + INSERT — joins rows against traces (±1-day window, 7-day max), denormalizing name, user_id, session_id, tags, release, metadata onto each observation row 2
4. Parent span resolution — parent_span_id = COALESCE(parent_observation_id, concat('t-', trace_id)), ensuring the tree is always connected even if a parent span never arrived 2
5. Cursor update — writes the processed partition timestamp back to Redis 2

Concurrency: global concurrency = 1, ensuring strictly sequential partition processing in timestamp order (eventPropagationQueue.ts) 2.

Required env vars to enable the worker 2:

QUEUE_CONSUMER_EVENT_PROPAGATION_QUEUE_IS_ENABLED=true
LANGFUSE_EXPERIMENT_INSERT_INTO_EVENTS_TABLE=true

Health monitoring: GET /api/health?failIfEventPropagationStuck=true on worker port 3030 returns 503 when the job hasn't progressed within LANGFUSE_EVENT_PROPAGATION_STUCK_THRESHOLD_MINUTES (default 15). Safe to keep configured after cutover — it passes when the dual write is not running 1.

Staging partitions are retained for 48 hours, giving a multi-day grace window for recovery after an incident 1.

Historic Backfill (M1–M5 Chain)#

Data ingested before the dual write was active exists only in the old tables. Two options exist 1:

• Automated backfill — rewrites all historic data into events_full; requires ~3× ClickHouse disk headroom. Enable with LANGFUSE_BACKGROUND_MIGRATION_V4_ENABLE_HISTORIC_BACKFILL=true only after dual write is active and confirmed healthy.
• Retention-based rollover — skip the backfill; keep dual write active for one full data retention window. Zero extra disk required.

The Five-Step Chain#

The backfill runs as an ordered chain; each step only starts after its predecessor finishes successfully. Built on ChunkedClickhouseBackfillMigration 4:

Fire-and-poll pattern for long-running ClickHouse INSERT … SELECT queries 4: dispatch with AbortController, confirm running via system.processes, abort the HTTP connection, then poll system.query_log for completion. Sets max_execution_time: 0 to bypass client-side timeouts.

State persistence: phase (init → loading_chunks → backfill → completed), chunk todos, and active query IDs are serialized to background_migrations.state in Postgres on every transition — making the chain fully resumable after worker restarts 4.

Cluster awareness: detectTableEngine() branches between SharedMergeTree (ClickHouse Cloud), ReplicatedMergeTree (self-hosted HA), and MergeTree (single-node), adapting DDL and SYSTEM commands accordingly 4.

Track backfill progress on the background migrations page in the Langfuse UI, or via the background migrations docs.

OTel Ingestion Migration#

V4 replaces the Langfuse-proprietary batch ingestion API with standard OTLP. All spans must be sent to /api/public/otel/v1/traces with x-langfuse-ingestion-version: 4 5.

Legacy API → OTLP Mapping#

Client-side attribute propagation is required in V4. In the old model, trace attributes were joined server-side at read time. Now they must be present on every observation row. Use OTel Baggage with a span processor to copy selected entries to attributes on every span 5. See Propagating Trace Attributes to All Spans.

LANGFUSE_MIGRATION_V4_NATIVE_OTEL_BEHAVIOUR=dual_write retains server-side propagation for OTel producers that haven't completed client-side propagation yet — spans without x-langfuse-ingestion-version: 4 go through the staging pipeline (~10 min delay) 1.

SDK Compatibility Matrix#

Full ingestion checklist and validation steps: Migrate custom ingestion to Langfuse v4 5.

Cutover, Cleanup, and Key Source Files#

Cutover#

Remove migration overrides (or explicitly set events_only) once 1:

• All producers run compatible SDKs, and
• Historic data is covered (backfill complete, or one full retention window dual-written)

After cutover: legacy endpoints return 404, older SDKs are rejected, trace-level evaluators stop running. This is the point of commitment — rolling back to a v3 read path would miss data written since the switch.

Optional Cleanup#

1. Drop scratch table: set LANGFUSE_BACKGROUND_MIGRATION_V4_DROP_PID_TID_SORTING_TABLES=true after confirming backfill health 1

2. Truncate old tables (irreversible — only after cutover is confirmed):

   ⧉

   TRUNCATE TABLE traces;
   TRUNCATE TABLE observations;
   -- On clustered deployments:
   TRUNCATE TABLE traces ON CLUSTER default;
   TRUNCATE TABLE observations ON CLUSTER default;
   

   Use TRUNCATE, not DROP — Langfuse expects the tables to exist 1.

Rollback#

V4 schema migrations are additive (new tables only). While on legacy or dual, the old tables receive all data — rolling back to the latest v3 release is safe. After events_only cutover, new data lands only in the new tables; rollback is not safe 1.

Key Source Files#

Official docs:

• Upgrade v3 → v4 (self-hosted)
• Migrate custom ingestion to v4
• Background Migrations
• v4 GitHub Discussion