Observability and Tracing#

Dify's observability infrastructure runs on two complementary layers 1:

1. OTel instrumentation — native span-level tracing wired into Flask, Celery, and key execution paths via decorators and auto-instrumentation (api/extensions/otel/).
2. Ops trace pipeline — an event-driven system that ships structured run data to external providers (Langfuse, LangSmith, Phoenix, Aliyun, etc.) via a batched Celery queue.

The unified tracing framework (opt-in via OPS_TRACE_UNIFIED_ENABLED) bridges both layers with a provider-neutral canonical span model. The telemetry gateway is the single emit() entry point: SignalType.TRACE events go to TraceQueueManager → Celery; metric/log events go to enterprise telemetry (EE only); large payloads (>1 MB) are offloaded to object storage before enqueuing 1.

App invocation
  → TraceQueueManager.add_trace_task()
      → threading.Timer (batched flush every N seconds)
          → Celery: process_trace_tasks
              → provider adapter (Langfuse / LangSmith / Phoenix / Aliyun …)

OTel Instrumentation Layer#

Root: api/extensions/otel/

@trace_span() decorator#

@trace_span() is the primary way to create OTel spans around functions. It resolves a singleton SpanHandler and records exceptions on failure. Two custom handlers attach richer context 1:

• AppGenerateHandler — attaches app_id, tenant_id, user_type.
• WorkflowAppRunnerHandler — attaches workflow-level attributes.

Enabled by ENABLE_OTEL or ENABLE_OTEL_FOR_INSTRUMENT config flags.

Context propagation#

propagate_context() captures OTel and contextvars state at decoration time and re-attaches it on execution — essential for async tasks and Celery workers. setup_context_propagation() establishes a CompositePropagator combining W3C TraceContext and B3 Multi-Format for interoperability with Jaeger, Zipkin, etc. 1

Auto-instrumentation#

init_instruments(app) wires up automatic OTel instrumentation for Flask, SQLAlchemy (with SQL commenter), Redis, HTTPX, and Celery 1.

Semantic conventions#

Node-type span parsers#

api/extensions/otel/parser/ enriches spans with semantic attributes per node type 1:

GenAI Span Conventions (v1.17+)#

PR #39339 established GenAI span conventions across providers and extended the Aliyun trace provider 1:

• TTFT: gen_ai.response.time_to_first_token (nanoseconds) on LLM spans for both workflow and message traces.
• Agent spans: AGENT span kind (replacing TASK) for agent-strategy nodes with gen_ai.operation.name=invoke_agent, agent name, and token usage.
• ReAct decomposition: Per-round STEP spans with child LLM and TOOL spans, anchored to monotonic node start time. Agent log entries are classified: Thought → LLM, CALL … → TOOL.
• Tool attributes: gen_ai.span.kind=TOOL, gen_ai.operation.name=execute_tool, plus gen_ai.tool.name/type/description/call.id/call.arguments/call.result across message tool traces, workflow tool nodes, and ReAct CALL spans.
• Failed LLM fallback: When process_data is empty on invocation failure, spans fall back to node inputs/error so gen_ai.request.model, gen_ai.provider.name, and gen_ai.prompt are still populated.
• gen_ai.response.model added to LLM spans in follow-on PR #40667 1.

Unified Tracing Framework#

Root: api/core/ops/unified_trace/

Opt-in via OPS_TRACE_UNIFIED_ENABLED=true. Builds a single canonical span tree in Core once, then delegates only translation and transport to thin provider adapters 1.

Canonical entities (entities.py):

• CanonicalSpanKind: CHAIN, LLM, RETRIEVER, TOOL, AGENT
• CanonicalSpan: one operation with explicit parent_id, timing, status, metadata
• CanonicalTrace: validated parent-before-child span tree with integrity checks enforced at construction

Provider routing: OpsTraceManager._get_dispatch_entry() routes per-provider at construction time. Currently registered in the unified registry: Phoenix and LangSmith (registry.py). Other providers (including Langfuse and Aliyun) use legacy provider-specific adapters 1.

Cross-node parent context: parent_context.py stores a ProviderParentContext envelope (trace_id, parent_id, provider-specific restoration state) in Redis across async Celery dispatch boundaries. TTL is OPS_TRACE_PARENT_CONTEXT_TTL_SECONDS (default 3900 s), sized to cover the max retry window of 780 retries × 5 s. If context is unavailable, the Celery task raises RetryableTraceDispatchError and retries 1.

Agent Observability Service#

api/services/agent/observability_service.py powers the agent monitoring dashboard. AgentObservabilityService aggregates daily token, cost, and latency stats for both webapp agents (querying Message) and workflow-embedded agent nodes (joining workflow_runs + workflow_node_executions via workflow_agent_node_bindings) 1. Key behaviors:

• Latency is merged with a weighted average to avoid double-counting across sources 1.
• serialize_workflow_node_message() reads latency from execution_metadata.agent_log.agent_backend.usage.latency, falling back to node_execution.elapsed_time 1.

Agent Trajectory Tracing#

Agent trajectory tracing is an independent, opt-in export pipeline specifically for agent execution traces, separate from system tracing. It uses the DIFY_AGENT_TRAJECTORY_* environment variables to export agent runs to a separate OTLP endpoint, typically for centralized monitoring in self-hosted deployments.

Key differences from system tracing:

• System tracing: General observability instrumentation across Flask, Celery, Redis, HTTPX, and workflow execution paths. Configured via standard OTEL_* environment variables and exported through the OTel SDK.
• Agent trajectory tracing: Dedicated pipeline for agent run spans only. Configured via DIFY_AGENT_TRAJECTORY_* variables and exported to an independent OTLP endpoint.

Configuration#

Agent trajectory tracing is disabled by default. When enabled, it instruments agent runs with Dify-specific attributes (dify.trace_id, dify.tenant_id, dify.app_id, dify.agent_id, gen_ai.user.id, dify.invoke_from, dify.conversation.id, dify.workflow.id, dify.workflow.run_id, dify.node.id, dify.node.execution_id) and exports spans via OTLP.

The trajectory exporter is configured with environment isolation: inherited OTEL_* and LOGFIRE_* variables are temporarily stripped during agent pipeline setup to prevent platform endpoints, credentials, or sampling policies from leaking into the agent trajectory exporter.

Configuration Reference#

1

Related Topics#

• Trace Task Pipeline — TraceQueueManager batching, Celery dispatch, and retry mechanics
• Langfuse Integration — provider-specific trace adapter details 2
• Agent Log Event Pipeline — per-step SSE log events from agent strategy plugins to frontend UI 3