Agent Session Persistence#

Agent session persistence enables durable storage of agent (assistant) chat sessions in Phoenix so conversations survive server restarts, scale across multiple backend replicas, and remain consistent across UI, CLI, Slack, and other client surfaces 1 2.

The system builds on three layers: database schema (three ORM tables) → server-authoritative mutations (GraphQL/REST endpoints for CRUD, truncate, branch, and compaction) → client concurrency protocol (turn-start locking with lastMessageId stale-write guards and busy-poll UX).

Data Model#

Three ORM tables support session persistence 1:

agent_sessions stores session metadata and model configuration. Key columns include:

• project_session_id and project_name (unique constraint) link sessions to assistant projects 3
• user_id (FK → users, nullable, ON DELETE CASCADE) for ownership
• title (String)
• expires_at (nullable timestamp) for temporary sessions with 24h TTL
• heartbeat_at (nullable UTC timestamp) serves as the turn-lock sentinel 4
• model_provider, model_name, custom_provider_id, builtin_provider — stamped at turn-start inside the lock transaction via stamp_session_model(); a customProviderDeleted flag handles the fallback state when a custom provider FK is set null 5
• created_at, updated_at (compound index with user_id for per-user recency queries)

agent_session_messages persists Vercel AI UIMessages with:

• position ordering
• message_id computed from message JSON; this invariant is enforced at the ORM layer 6
• is_compaction_message flag for synthetic checkpoint messages 7

agent_session_snapshots stores bashkit shell/filesystem snapshots keyed per session. Currently maintains only the most recent snapshot per session; per-message FK planned to enable correct fork/rewind behavior 8.

The foundation migration (e767d3c57f32) creates all three tables and their relationships 1. ORM models live in src/phoenix/db/models.py. The concurrency affordances design (lock, notify, fork, alert) is tracked in #14404. Revision/version management and idempotency key design is tracked in #14403.

Server-Authoritative Mutations#

GraphQL mutations in src/phoenix/server/api/mutations/agent_session_mutations.py enforce server control over session lifecycle 1:

• createAgentSession, deleteAgentSession, updateAgentSessionTitle manage basic CRUD operations
• truncateAgentSession(messageId) atomically deletes messages at or after a target position. If the target is a user message, it deletes the target plus everything after; if assistant, it keeps the target and deletes everything after. Single DB transaction 9
• branchAgentSession(sourceSessionId, messageId) copies a transcript prefix into a new AgentSession, regenerates all message IDs to break identity with the source, and flushes atomically 9

Fork and rewind are targeted to become fully server-authoritative, replacing the current local-only Zustand/AI SDK transforms that are overwritten only on the next chat request 10.

History projection via _load_agent_session_history() filters from the latest compaction point onward. Compaction is invoked via POST /compact, which calls the model to synthesize a structured XML checkpoint message (objectives, constraints, decisions, completed work, blockers, next steps) and persists it as a synthetic user-role message with is_compaction_message: true. Subsequent model requests see only the compaction message plus uncompacted turns 7.

Turn-Start Locking & Optimistic Concurrency#

The heartbeat_at column on agent_sessions implements the entire turn lock—no separate lock table 4.

Lock claim uses an atomic UPDATE agent_sessions SET heartbeat_at = now WHERE heartbeat_at IS NULL OR heartbeat_at < now - 60s. The update succeeds only if the condition matches, meaning no live lock is held. This happens before model or agent construction, so rejected requests burn zero LLM tokens.

Heartbeat refresh occurs every 15s via an in-request background task. The stream generator's finally block releases the lock. TURN_LOCK_STALENESS = 60s in src/phoenix/server/api/helpers/agent_sessions.py defines staleness; the is_turn_active() predicate derives busy state from this window. Crashed servers self-heal after 60s with no dedicated sweeper.

lastMessageId stale-write guard: every /chat request includes last_message_id (the client's believed tail of the persisted transcript). The server validates this before claiming the lock (expected = session_history[-1].message_id). A mismatch returns 409 {"code": "agent_session_stale"}. The value is the ID of the message just before the new user turn, or null for empty sessions.

409 responses distinguish between agent_session_busy (lock held by another client) and agent_session_stale (transcript diverged). The isTurnActive GraphQL field on AgentSession exposes lock state to clients 4.

Multi-Replica Consistency & Client UX#

Sessions stored in a shared database achieve multi-replica safety automatically—no distributed lock manager is needed. The concurrency protocol handles competing access from multiple browser tabs, CLI clients, and Slack integrations against the same session.

Busy-elsewhere UX: When a client receives a busy 409, it withdraws the optimistic user message back to the composer draft, disables the composer, shows "Session is being used elsewhere" notice, and polls every 3s using isTurnActive. When the lock clears, it swaps in the persisted transcript and re-enables the composer.

Stale-rejected sends: The client refetches the session, keeps the draft, and shows a one-shot refresh notice.

The CLI (PXI) implements identical 3s polling; slash commands remain usable while busy 4.

Session Retention & Sweeper#

The AgentSessionSweeper daemon (src/phoenix/server/daemons/agent_session_sweeper.py) runs hourly with random jitter, processing deletions in batches of 100 with row-level rechecks 1:

1. Deletes expired temporary sessions (expires_at < now)
2. Deletes idle persisted sessions (updated_at < now - max_idle_days, default 30 days)
3. Enforces per-user cap (default 30 sessions, keeps N most recent by updated_at DESC, id DESC)

Configuration is controlled via the setAgentSessionRetention GraphQL mutation and workspace SystemSettings. Temporary sessions set expires_at = now + 24h (configurable via TEMPORARY_AGENT_SESSION_TIME_TO_LIVE_HOURS); all sessions touch updated_at on every write, which drives idle-expiry and sweeper ordering.

Bashkit Snapshot Strategy#

Currently one snapshot per session (most-recent-only) trades correctness for space. This means fork and rewind operations use potentially stale shell state.

Investigation found bashkit snapshots are deterministic (same state → byte-identical bytes) and uncompressed, making them highly delta-friendly 8:

• zstd full-snapshot per message ≈ ~30 KB/message at ~1–3% of raw size (vs. ~1–3 MB raw)
• bsdiff/zstd delta chains reach ~200–300 bytes/step but add chain-replay complexity and cross-session reference risks

Recommendation: Move FK from agent_sessions to agent_session_messages, store one zstd-compressed full snapshot per message. Truncation and branching become O(1) reads of the target message's snapshot, with no chain reconstruction 8.

Key Source Files#