Agent Cost and Usage Tracking#

Dify tracks LLM token usage and monetary cost at two levels of agent execution:

1. Classic Agent Apps (Chatbot/Agent/Completion mode) — a runner loop accumulates usage per LLM call and publishes it with the terminal message event.
2. Workflow-embedded Agent Nodes — a message transformer extracts execution_metadata from the agent strategy output and attaches it to the NodeRunResult, feeding the workflow's cost accounting.

Accumulated usage propagates through an ops-trace event queue (TraceQueueManager) to observability providers. The first-party Langfuse provider converts it into GenerationUsage observations. Several bugs have been fixed in this pipeline — most notably usage silently dropped in Langfuse (no cost/latency) and usage overwritten instead of summed across agent rounds.

Usage Accumulation in Classic Agent Runners#

Both the CoT and Function-Calling agent runners accumulate LLMUsage across iteration rounds using a closure-based helper.

In cot_agent_runner.py, a llm_usage: dict[str, LLMUsage | None] dictionary is initialised before the loop, and an inner increase_usage() function mutates it each round — adding prompt_tokens, completion_tokens, total_tokens, prompt_price, completion_price, and total_price. The same pattern appears in fc_agent_runner.py for function-calling agents.

Prior bug: Before PR #33450, usage was overwritten on each round rather than accumulated, so only the final round's token counts were reported.

Usage Flow Through Workflow Agent Nodes#

For Agent Nodes inside a workflow, usage is handled by AgentMessageTransformer in api/core/workflow/nodes/agent/message_transformer.py:

1. Initialisation: llm_usage = LLMUsage.empty_usage() 1
2. Extraction: When the agent strategy emits a JSON-type ToolInvokeMessage, the transformer pops execution_metadata from the JSON body and converts it to LLMUsage via LLMUsage.from_metadata() 2.
3. Output: At stream completion, a StreamCompletedEvent is yielded with llm_usage set on NodeRunResult and agent_execution_metadata in the metadata dict, alongside TOOL_INFO and AGENT_LOG. The usage output key also carries a JSON-serialised copy of LLMUsage for downstream nodes.

This metadata is persisted on the WorkflowNodeExecution record, making it available to monitoring queries and historical trace views.

Persistence in Agent App Terminal Events#

For classic Agent Apps backed by the dify-agent backend, PR #38270 fixed a gap where the agent backend had usage data but never forwarded it to the API layer:

• AgentBackendRunSucceededInternalEvent (and AgentBackendDeferredToolCallInternalEvent) gained a usage: dict[str, JsonValue] | None field, populated by a new _agent_run_usage() serializer in event_adapter.py.
• app_runner.py gained _llm_usage_from_agent_backend() to convert the raw dict to LLMUsage, then threads it through _publish_terminal_answer() → publish_message_end(). Previously publish_message_end() always emitted LLMUsage.empty_usage(); it now uses usage or LLMUsage.empty_usage().

Without this fix, Message.answer_token_count and Message.total_price stayed at zero for agent apps using the backend runner.

PR #40937 extended this propagation to failed and cancelled runs. AgentBackendRunFailedInternalEvent and AgentBackendRunCancelledInternalEvent now carry usage: dict[str, JsonValue] | None fields, serialized by the same _agent_run_usage() helper. The agent runtime (runner.py) sets _terminal_usage from model.accumulated_usage in the finally block, ensuring usage accumulated before failure or cancellation is retained. The API workflow Agent v2 output adapter (output_adapter.py) reads this field from all four terminal event types (RunSucceededInternalEvent, DeferredToolCallInternalEvent, RunFailedInternalEvent, RunCancelledInternalEvent) and stores it in workflow execution metadata.

Agent App terminal events (app_runner.py) now call _persist_message_usage() immediately after receiving any terminal event, writing accumulated tokens and cost directly to the Message row. This ensures accounting does not depend on the client SSE stream staying connected until the terminal message event is emitted. For manual stops, the EasyUI pipeline (easy_ui_based_generate_task_pipeline.py) passes preserve_existing_usage=True to _save_message() to prevent overwriting provider-reported usage with local estimates.

Monitoring Statistics (Observability Service)#

The agent observability service (api/services/agent/observability_service.py) aggregates daily token, cost, and latency stats for the agent monitoring dashboard.

PR #39354 extended this to cover agents embedded in pure-workflow apps (which create workflow_runs rather than Message rows):

• _load_daily_statistics() was split into _load_webapp_daily_statistics() (queries Message) and _load_workflow_daily_statistics() (JOINs workflow_runs + workflow_node_executions via workflow_agent_node_bindings).
• A new _workflow_execution_metadata_numeric_sql() helper extracts numeric values from node execution_metadata JSON using database-specific syntax (PostgreSQL JSONB vs MySQL JSON path extraction).
• A _merge_daily_statistics() helper combines the two result sets, using a weighted average for latency to avoid double-counting.
• The _statistics_message_scope_sql() filter was also fixed (PR #38270) to only filter by invoke_from when explicitly specified, so source=all correctly includes debugger/preview runs.

Langfuse Trace Integration#

Usage data reaches Langfuse via LangFuseDataTrace in api/providers/trace/trace-langfuse/. Two trace paths carry cost data:

workflow_trace() — iterates all WorkflowNodeExecution rows and creates a LangfuseGeneration per LLM node, each with id=node_execution_id. Token and cost fields come from the persisted execution_metadata. Workflow apps have been unaffected by the ID bug below.

message_trace() — used by Chatbot, Agent, and Completion apps. Builds a GenerationUsage from trace_info.message_tokens, trace_info.answer_tokens, and message_data.total_price (the totalCost field), then attaches it to a single LangfuseGeneration observation 3.

Bug (v1.14.0–present, Issue #37824): The SDK v2→v3 migration (PR #34265) switched to the low-level ingestion API, which requires an explicit id — unlike v2 which auto-assigned a UUID. message_trace() constructed LangfuseGeneration without an id; filter_none_values stripped the None, causing Langfuse to silently drop the generation observation. Result: traces showed only the top-level trace node with no cost and 0.00 s latency for all non-workflow apps.

Fix (PR #37833): Added id=str(uuid.uuid4()) to LangfuseGeneration in message_trace() and the other affected handlers (suggested_question_trace(), moderation_trace(), dataset_retrieval_trace(), tool_trace()).

The ops-trace dispatch layer (TraceQueueManager in ops_trace_manager.py) routes MessageTraceInfo and WorkflowTraceInfo events to the registered providers asynchronously.