Experiment Tracing in Phoenix#

Phoenix captures per-run telemetry for experiments via OpenTelemetry spans and forwards them to a project-specific tracing endpoint. Two span kinds are used — CHAIN for task execution and EVALUATOR for evaluator runs. Cost, latency, and token metrics are derived from LLM SDK instrumentation attributes on those spans and surfaced in the Experiment Compare UI as per-run averages.

Primary source files:

• src/phoenix/experiments/functions.py — orchestrates span emission for task and evaluator runs
• src/phoenix/experiments/tracing.py — capture_spans context manager and resource injection

Span Emission: CHAIN and EVALUATOR#

Each task and evaluator execution opens an isolated root OTel span with a fresh Context() to prevent accidental parent-child relationships across concurrent runs.

Task runs (run_experiment) use span kind CHAIN 1. The root span is named Task: <func_name> and receives these attributes 2:

• INPUT_VALUE / INPUT_MIME_TYPE — JSON-encoded dataset example input
• OUTPUT_VALUE / OUTPUT_MIME_TYPE — JSON-encoded task output
• OPENINFERENCE_SPAN_KIND = "CHAIN"

Evaluator runs (evaluate_experiment) use span kind EVALUATOR 3. Each span is named Evaluation: <evaluator_name> and sets evaluation result fields via span.set_attributes(dict(flatten(jsonify(result), ...))) 4.

Errors in either path are recorded with span.record_exception(exc) and StatusCode.ERROR 5, and the span's trace_id is extracted and stored on the ExperimentRun / ExperimentEvaluationRun objects for linking back to Phoenix trace views 6.

Child Span Capture and Resource Injection#

The capture_spans(resource) context manager ensures that any spans created by the task's internal LLM SDK calls (e.g., OpenAI instrumentation) are tagged with the correct project resource, even though those spans are not explicitly parented to the root experiment span.

Mechanism: capture_spans monkey-patches ReadableSpan.__init__ using wrapt so that on every span initialization, SpanModifier.modify_resource merges the experiment's Resource (which carries PROJECT_NAME) into the span's existing resource attributes.

Thread safety: A reference-counted lock (_SPAN_INIT_MONKEY_PATCH_LOCK) ensures the patch is applied exactly once and removed only when all active capture_spans contexts have exited 7. A ContextVar (_ACTIVE_MODIFIER) isolates which modifier is active per async context 8.

Graceful degradation: modify_resource short-circuits if the span context is None or has INVALID_TRACE_ID, silently skipping malformed spans 9.

Tracer Setup and Dry-Run Degradation#

_get_tracer(project_name) creates a TracerProvider backed by a SimpleSpanProcessor + OTLPSpanExporter targeting {base_url}/v1/traces. The project name is set as a Resource attribute (PROJECT_NAME) so all spans from a given experiment are routed to the correct Phoenix project.

Dry-run mode: When project_name is None (i.e., dry_run=True), a _NoOpProcessor is used instead. This minimal SpanProcessor subclass only implements force_flush (returning True) — the SDK remains functional but no spans are exported 10. Evaluator runs also pass None as the project name when dry-running 11.

Task/evaluator code executes inside a try/except BaseException block within an ExitStack, so errors are caught, recorded on the span, and printed in red — but they do not abort the overall experiment run 12.

Cost Information: Attributes and UI Display#

Cost data flows from LLM SDK instrumentation attributes on child spans up through the experiment aggregation layer to the Experiment Compare UI.

Source attributes (OpenInference LLM span conventions):

• llm.token_count.prompt — prompt/input tokens
• llm.token_count.completion — completion/output tokens
• llm.cost.total — optionally emitted directly; otherwise computed from token counts × model pricing

Phoenix's ingestion pipeline multiplies token counts by model-specific rates from model_cost_manifest.json and propagates cumulative sums (cumulative_llm_token_count_*) up the span tree. Token counts are gated to LLM spans only during ingestion to prevent double-counting from wrapper spans.

Experiment Compare table headers display per-run averages, introduced in PR #8737 and extended in PR #10802:

• Latency — averageRunLatencyMs
• Token Count — costSummary.total.tokens / runCount
• Cost — costSummary.total.cost / runCount

Tooltip drill-downs show prompt vs. completion breakdowns by token type. The UI renders three states: idle (shows --), running (skeleton loaders), and complete (actual averages), preventing layout shifts during long-running experiments 13.