OTel Ingestion and Trace Hierarchy#

Overview#

Langfuse's OTel ingestion pipeline converts standard OpenTelemetry ResourceSpan payloads into Langfuse traces and observations. The flow is:

1. HTTP → S3: The API handler uploads raw resourceSpans to blob storage and enqueues a job on the OtelIngestionQueue 1
2. Worker → OtelIngestionProcessor: The otelIngestionQueueProcessor downloads the file and passes it to OtelIngestionProcessor.processToIngestionEvents(), which produces a flat list of IngestionEventType events 2
3. Split by type: Events are split into traces (processEventBatch) and observations (IngestionService.mergeAndWrite), processed concurrently 3
4. ClickHouse write: IngestionService merges events with any existing ClickHouse record and writes the final record 4

For each OTel span, the processor emits up to two events:

• A trace-create event (full, partial, or shallow — see below)
• An observation event (span-create, generation-create, etc.)

Primary source files:

• OtelIngestionProcessor.ts — core conversion logic
• otelIngestionQueue.ts — worker job handler
• IngestionService/index.ts — merge-and-write logic
• attributes.ts — all LangfuseOtelSpanAttributes enum values

Root Span Detection and trace-create Strategy#

Root Span Logic#

A span is classified as a root span if it has no parentSpanId or if it carries the langfuse.internal.as_root = "true" attribute 5:

isRootSpan = !parentObservationId || String(attributes[LangfuseOtelSpanAttributes.AS_ROOT]) === "true"

AS_ROOT is defined as "langfuse.internal.as_root" in the attributes enum. Setting it to "true" forces a span to behave as a trace root regardless of OTel parentage — useful when a Langfuse-instrumented service runs inside a broader OTel trace where the real root is not observable.

Three Grades of trace-create Events#

The processor emits one of three trace shapes per unique traceId 6:

hasTraceUpdates is true when any of the Langfuse trace attributes (e.g. langfuse.trace.name, user.id, session.id, langfuse.trace.tags) or framework-specific equivalents (Vercel AI SDK ai.telemetry.metadata.*, LlamaIndex tag.tags) are present 7.

Trace Deduplication#

To avoid emitting redundant trace-create events for already-seen traces, the processor checks a Redis key (langfuse:project:{projectId}:trace:{traceId}:seen) with a 10-minute TTL 8. A trace-create is only emitted when the span is a root, has trace-level updates, or the traceId has not been seen in the last 10 minutes. Within a single batch, shallow trace-create events are suppressed if a full or partial one exists for the same traceId 9.

Parent-Child Observation Relationships#

Each OTel span maps to a Langfuse observation where:

• observation.id = hex-encoded OTel spanId
• observation.parentObservationId = hex-encoded OTel parentSpanId (null for root spans) 10

Both Python and JS SDKs send IDs in different formats; parseId() handles both hex strings and integer arrays 11.

The parentObservationId is written directly to the ClickHouse observations table as parent_observation_id 12. There is no blocking wait for the parent to exist. If a child span arrives before its parent (common in async/concurrent traces), the child is written with an orphaned reference. This is resolved later by the event propagation layer.

Immutable Fields#

Once written to ClickHouse, id, project_id, trace_id, start_time, created_at, and environment are immutable — subsequent events for the same observation ID cannot overwrite these. Mutable fields (name, metadata, input, output, model, usage, etc.) are merged by taking the most recent non-null value from time-sorted events.

Eventual Consistency for Orphaned Parent References#

Langfuse does not retry or block when a child observation references a parent that hasn't been written yet. Instead, it relies on two layers of eventual consistency:

1. Dual-Write to Staging Table#

After ClickHouse write, each observation is also written to ObservationsBatchStaging with a partition-aware timestamp 13. The getPartitionAwareTimestamp() method ensures events within the last 2 minutes use their original createdAt, while older events get the current timestamp — preventing stale partition updates after the 4-minute lock window 14.

2. EventPropagation COALESCE Fallback#

The handleEventPropagationJob worker processes batches from ObservationsBatchStaging and writes them to events_full. During this step, the parent reference is resolved:

CASE
  WHEN obs.id = concat('t-', obs.trace_id) THEN ''
  ELSE coalesce(obs.parent_observation_id, concat('t-', obs.trace_id))
END AS parent_span_id

15

If parent_observation_id is null (i.e., the parent never arrived), the trace's root pseudo-span (t-{traceId}) becomes the fallback parent. This ensures the events_full tree is always connected.

3. Eval-Specific Retry#

For evaluation jobs that require a specific parent observation to exist, a separate retryObservationNotFound mechanism provides exponential backoff (30s → 1m → 2m → 4m) with a maximum of 5 retries. This is not used in the general ingestion path.

Key Attributes Reference#

All Langfuse-specific OTel span attributes are defined in the LangfuseOtelSpanAttributes enum:

The processor supports input/output extraction for many third-party instrumentation libraries (Vercel AI SDK, Genkit, MLFlow, TraceLoop, Pydantic AI, LiveKit, LlamaIndex, etc.) via framework-specific attribute paths 16.