Database Schema Management in Phoenix#

Phoenix uses Alembic to manage database schema migrations. Migration scripts live in src/phoenix/db/migrations/versions/, and the migrate() function in src/phoenix/db/migrate.py applies all pending migrations by calling command.upgrade(alembic_cfg, "head") at startup 1. A thread-safe variant, migrate_in_thread(), handles notebook environments where the main thread may block.

Schema Drift in Long-Lived Development Databases#

The core problem: Alembic tracks which migration files have been applied (via the alembic_version table) but cannot detect column-level drift introduced by running intermediate or experimental code against a shared local database. A development database can therefore report the current Alembic head revision while its actual table structure diverges from what the current codebase expects.

This was documented concretely in issue #14354: the llm_evaluators table in an affected local database still had an older schema (prompt_template_id, generative_model_id), while current main expects prompt_id, prompt_version_tag_id, output_configs, and updated_at. Because the Alembic revision appeared current, the mismatch was not immediately obvious.

How Stale Schema Surfaces as GraphQL Errors#

When a GraphQL field resolver queries a table with a missing column, SQLAlchemy raises a low-level sqlite3.OperationalError: no such column: llm_evaluators.prompt_id. Phoenix's GraphQL layer catches this and returns the generic message "an unexpected error occurred" — masking the underlying cause 2.

The coupling effect is the critical failure mode: EvaluatorByIdDataLoader uses SQLAlchemy's with_polymorphic to join all evaluator subtypes (LLM, code, built-in) in a single query. A stale column in the llm_evaluators table therefore breaks resolution of any evaluator kind, including CODE and BUILTIN evaluators whose own tables are perfectly healthy 2. The user sees evaluator creation succeed (the association row is written) but then fails when reading back through GraphQL.

Observed error pattern:

Error fetching GraphQL query 'datasetEvaluatorsLoaderQuery' with variables '{"id":"..."}':
[{"message":"an unexpected error occurred","locations":[...],"path":["dataset","datasetEvaluators","edges",0,"node","evaluator"]}]

Diagnosing Schema Drift#

1. Check Alembic revision — alembic current or inspect the alembic_version table. A current revision does not rule out column-level drift.
2. Check the actual table schema — for SQLite: PRAGMA table_info(llm_evaluators);. Compare against the model definition.
3. Look for OperationalError in server logs — the generic GraphQL error hides this; the raw Python traceback will show the missing column name.

Recovery and Hardening#

Recovering a drifted local database:

• Back up phoenix.db before any changes.
• Do not delete and recreate the database if you want to preserve existing traces and experiments.
• Manually apply the missing DDL changes (e.g., ALTER TABLE llm_evaluators ADD COLUMN prompt_id ...) or restore from a clean backup.

Preventing recurrence:

• Avoid running feature branches that modify ORM models against a shared long-lived local database.
• Use separate database files per feature branch, or a fresh database for each significant schema-changing branch.
• The proposed backend hardening 2 is to make EvaluatorByIdDataLoader load only the applicable subtype rather than all subtypes with with_polymorphic, so that a stale schema in one subtype cannot break resolution of others.

Key Source References#