Annotation System#
Phoenix provides a dual-level annotation architecture that attaches structured feedback and evaluation data to both spans and traces. Annotations are the primary mechanism for recording human feedback, LLM evaluations, and programmatic scores against AI observability data.
Data Model#
Both annotation levels share an identical schema. The SpanAnnotation and TraceAnnotation ORM models differ only in their foreign key — span_rowid vs. trace_rowid. Core fields:
| Field | Type | Notes |
|---|---|---|
name | str | Annotation dimension (e.g., "helpfulness", "user_feedback") |
label | Optional[str] | Categorical value (e.g., "positive") |
score | Optional[float] | Numeric score |
explanation | Optional[str] | Free-text rationale |
annotator_kind | "LLM" | "CODE" | "HUMAN" | Who produced the annotation |
source | "API" | "APP" | How the annotation was submitted |
identifier | str | Deduplication key, defaults to "" |
user_id | Optional[int] | FK to the creating user |
Composite Key Deduplication (Upsert Semantics)#
The unique constraint on both tables is (name, *_rowid, identifier) . The create_span_annotations and create_trace_annotations mutations implement upsert logic: they select on this composite key and update in-place if a match is found, otherwise insert .
This means:
- Submitting the same
(name, target_id, identifier)tuple a second time overwrites the existing annotation rather than creating a duplicate. - Omitting
identifier(leaving it"") produces a single shared slot per(name, target)pair — useful for LLM evaluators that run once per span. - Supplying distinct
identifiervalues produces independent slots for the same annotation name — enabling multiple annotators or coding passes on the same span/trace.
Identifier Resolution and Per-User Upserts#
When identifier is not explicitly set, the mutation resolves it automatically based on source :
- Explicit identifier provided → used as-is.
source == APPand a user is logged in →get_user_identifier(user_id)is called, producingpx-app:{GlobalID}.- Otherwise → defaults to
"".
The get_user_identifier helper generates a user-scoped identifier (px-app:{user_global_id}). Combined with the composite key constraint, this gives per-user upsert semantics: each authenticated user has exactly one annotation slot per (name, target) pair when using the APP source — re-submitting feedback updates their record without creating duplicates and without affecting other users' annotations.
Annotation Configs#
Phoenix stores reusable annotation schemas in AnnotationConfig , separate from annotation instances. Three config types are supported :
- Categorical — discrete labels with associated scores and an optimization direction.
- Continuous — numeric range with bounds.
- Freeform — unstructured text.
A ProjectAnnotationConfig join table scopes configs to specific projects. On startup, Phoenix seeds a built-in "user_feedback" categorical config with positive=1.0 / negative=0.0 and MAXIMIZE direction .
GraphQL Mutations#
Both annotation levels expose three mutations via Strawberry, defined in their respective mutation mixin types:
| Mutation | Source |
|---|---|
createSpanAnnotations / createTraceAnnotations | span / trace |
patchSpanAnnotations / patchTraceAnnotations | span / trace |
deleteSpanAnnotations / deleteTraceAnnotations | span / trace |
Patch requires the annotation to be owned by the requesting user . Delete enforces the same ownership check but allows admins to delete any annotation .
After create/patch/delete, a SpanAnnotationInsertEvent or TraceAnnotationInsertEvent / DeleteEvent is placed on the event queue for downstream consumers .
Span Notes#
Span annotations include a special create_span_note mutation for free-text notes. Notes always use name="note" and receive a unique timestamp-based identifier (px-span-note:<iso-timestamp>), making them append-only — each call creates a new record rather than upserting . The create_span_annotations endpoint explicitly rejects inputs with name="note" .
Session-Tagged Identifiers (Coding Workflows)#
PR #13083 extended note and annotation REST endpoints (POST /v1/{trace,span,session}_notes) to accept an optional identifier field. When supplied, the note upserts on (entity, name='note', identifier) enabling deterministic open-coding / axial-coding workflows where re-running an analysis step overwrites the previous result. When omitted, a unique px-{kind}-note:<uuid> is generated for append-only behavior (preserving backward compatibility).
User Feedback UI#
The session turn UI exposes a TraceFeedbackActionToolbar component that renders thumbs-up/down buttons wired to createTraceAnnotations / deleteTraceAnnotations GraphQL mutations. Feedback is stored as a "user_feedback" trace annotation, and the px-app:{viewer_id} identifier ensures each viewer has exactly one active feedback record per trace.