Sandbox Infrastructure#
The runner (services/runner/, Node.js/TypeScript) executes all agent workloads in isolated sandboxes, listening on port 8765 and invoked by the Python API container . Two sandbox providers exist: local (unconfined host process; default in dev) and Daytona (cloud-isolated remote sandbox). Provider selection is environment-driven β see Sandbox Provider Configuration; timeout strategy is covered in Agent Runner Timeout Enforcement.
This article covers the sandbox lifecycle: session states, persistence and reuse, durable mounts, failure modes, snapshot versioning, background job execution, and the Files-pane view of session storage.
Sandbox Lifecycle: States and Transitions#
A Daytona sandbox passes through five states :
| State | Description |
|---|---|
| Hot | Running with live harness process |
| Warm | Stopped on disk; filesystem preserved |
| Cold | Archived to object store |
| Dead | Deleted |
| New | Fresh create |
Transitions are driven by three Daytona idle timers β DAYTONA_AUTOSTOP (default 5 min), DAYTONA_AUTOARCHIVE (default 15 min), DAYTONA_AUTODELETE (default 30 min) β plus explicit runner pause/reconnect/delete calls dispatched from teardown.ts .
Teardown reason determines the action :
- Clean or resumable turn β stop (not delete); sandbox stays Warm.
- Kill / failed / aborted turn, or in-flight shutdown β delete.
Auto-archive was removed because restoring from archive was slower than a fresh create; the ladder is stop at 15 min idle, delete at 30 min .
During active Daytona turns, including sessionless runs, an activity heartbeat calls the sandbox's refreshActivity() to prevent autostop. Its interval is one third of the configured autostop interval, with a minimum of 10 seconds; overlapping refreshes are skipped, refresh failures are logged without failing the turn, and the timer stops when the turn settles (heartbeat, provider).
Top-level lifecycle wiring lives in sandbox_agent.ts (acquireEnvironment / runTurn) .
Session Persistence and Reuse#
Three layered mechanisms (all landed July 2026) progressively reduce session rebuild cost.
Keep-Alive Pool β PR #5156#
An opt-in, short-TTL pool in session-pool.ts holds harness sessions alive between turns . Before resuming, the runner validates three fingerprints :
| Fingerprint | Scope |
|---|---|
| Config | Harness, sandbox, model, provider, tools, permissions, system prompt (excludes per-turn volatiles) |
| History | Ordered user message texts + tool-call IDs; any edit or rewind trips a mismatch |
| Credential epoch | SHA256 over {secrets, toolCallbackAuth} + mount credential expiry |
All three must match and the tail must be a plain user message. Any mismatch falls back to cold replay β no turn is ever failed .
Configuration :
| Variable | Default | Effect |
|---|---|---|
AGENTA_RUNNER_SESSION_KEEPALIVE | false | Enable the pool |
AGENTA_RUNNER_SESSION_POOL_MAX | 8 | LRU cap |
AGENTA_RUNNER_SESSION_TTL_MS | 60 000 ms | Idle TTL after a clean turn |
AGENTA_RUNNER_SESSION_APPROVAL_TTL_MS | 600 000 ms | Extended TTL for HITL-gated sessions |
Daytona Warm Pooling β PR #5225#
Adds pause/reconnect to daytona-provider.ts, enabling two reuse tiers :
- Park-to-running β sandbox stays hot for
AGENTA_RUNNER_DAYTONA_SESSION_IDLE_TTL_MS(default 120 s). Continuation β 1.4 s vs β 12.5 s cold. - Park-to-stopped β after the window expires or the warm cap (
AGENTA_RUNNER_DAYTONA_SESSION_MAX_WARM, default 20) is full, the sandbox is stopped. Restart +session/loadβ 7.7 s.
On reconnect the runner reads the live sandbox's network policy and calls updateNetworkSettings only when policy differs, so parked sandboxes pick up permission changes without a rebuild . The sandbox_id pointer write is guarded by compare-and-set on latest_turn_index to prevent concurrent-turn races .
Durable Session Continuity β PR #5197#
Extends session memory across runner restarts and sandbox death using two stores :
session-continuity.tsβ in-memory(sessionId, harness)β{agentSessionId, turnIndex}map with turn-index staleness guard.session-continuity-durable.tsβ mirrors to the API'ssession_states.datarow via GET-then-PUT; survives runner restart.
sandbox-reconnect.ts implements the reconnect ladder: stored sandbox ID β try restart β fall back to fresh create on failure . Per-harness transcript mounts (~/.claude/projects, Pi sessions dir) get durable geesefs treatment so even a Dead sandbox can resume natively; Claude's .credentials.json is explicitly excluded . Disable with AGENTA_SESSION_HARNESS_MOUNTS=false.
Durable Storage Mounts#
Each agent gets a persistent S3-backed workspace that survives across sandbox sessions. The runner mounts it at <cwd>-agent/ via geesefs (FUSE) and creates an agent-files symlink in the session cwd. The symlink is rebuilt every run because geesefs silently downgrades symlinks to empty files across remounts .
Core logic lives in agent-mount.ts :
| Function | Purpose |
|---|---|
signAgentMountCredentials | Calls the sign endpoint; returns null on failure (never aborts the turn) |
seedAgentReadme | Seeds README.md on first run with wx flag (atomic, won't overwrite) |
linkAgentFiles | Creates/heals the agent-files symlink in cwd |
The runner container requires SYS_ADMIN capability, /dev/fuse device access, and apparmor:unconfined for FUSE support . The API sign endpoint (POST /mounts/agents/sign) uses a deterministic slug __ag__agent__{canonical_artifact_id}__{slugified_name} for idempotency . API-side mount logic: api/oss/src/core/mounts/service.py and api/oss/src/apis/fastapi/mounts/router.py.
Mount reachability checking and tunnel discovery live in mount.ts.
The runner passes --no-expire-multipart to geesefs, disabling mount-time cleanup of incomplete multipart uploads. Configure object-store bucket lifecycle rules to delete incomplete multipart uploads instead (mount arguments).
Known Failure Modes#
STS Credential Expiry β EACCES After ~15 Min (Issue #5516)#
Agent mounts are initialized with 900-second STS credentials passed as static environment variables to geesefs β no refresh mechanism exists . After expiry, geesefs receives HTTP 403 from SeaweedFS, mapped to EACCES. The mount appears healthy to liveness probes (which detect ENOTCONN, not EACCES), so all filesystem operations fail silently. Any turn exceeding ~15 minutes is deterministically broken . Status: open. Proposed fixes: detect EACCES in mount probes and trigger remount; refresh credentials at ~80% of TTL; assign unique RoleSessionName per mount. Raising duration_seconds is a temporary mitigation only.
Silent Mount Skip β Agent Files Lost (Issue #5342)#
When the object store is unreachable (no public endpoint or tunnel), the runner logs "tunnel discovery failed" and silently degrades: the durable mount is skipped but execution continues . Files written during that session are lost with no user-facing error. The relevant code is in mount.ts. Status: open β desired fix is a loud, actionable error when the required mount is unavailable.
Claude Harness Gets No Tools on Daytona (Issue #4984)#
The runner's internal MCP server listens on 127.0.0.1 of the runner host. Inside a Daytona remote sandbox, 127.0.0.1 resolves to the sandbox's own loopback β Claude receives zero tools and fails silently . An interim fail-loud error was added, then resolved: daytonaEnvVars() now explicitly configures the sandbox environment so tools behave identically to local .
Snapshot Versioning#
PR #7221 introduced a shared versioned recipe to prevent snapshot name drift between runner and snapshot builder :
services/runner/config/sandbox-recipe.jsonβ single source of truth for the snapshot version. Bothrunner-config.ts(exposingSANDBOX_RECIPE_VERSION/DEFAULT_DAYTONA_SNAPSHOT) andbuild_snapshot.pyderive snapshot names from this file .sandbox-recipe-fingerprint.json+test_recipe_fingerprint.pyβ a test hashes the build script SHA-256 plus pinned Pi, Codex ACP, and Claude ACP versions. Any change to build inputs without a version bump fails the test . Note: as of PR #7221, the recipe version itself is not part of the fingerprint payload; a follow-up PR #7304 adds it .- Actionable missing-snapshot errors β when Daytona creation fails because a snapshot is not found, the runner now reports the missing snapshot name and the
uv run build_snapshot.pycommand with the configured target .
Rule: when modifying sandbox build inputs (pinned tool versions, base image, install scripts), bump the recipe version and regenerate the fingerprint file.
Files Pane and Session Drive#
The frontend Files pane surfaces the agent's durable workspace and the session working directory. Several fixes landed to make the UI truthful about loading, empty, and error states.
Mount resolution and drive root (PR #5944): The overview page has no session cwd mount, so useSessionDrive.ts now treats the agent's own mount as the drive root when there is no session, or when a session's cwd mount has not been found after mount loading resolves. Sessions with a cwd mount retain that as the drive root; agent-mount files continue to appear under agent-files/ . The Files pane must wait for both the session working-directory query and the agent mount query to resolve before declaring the drive empty β each is independent.
Correct cwd mount selection (PR #5642): A shared pickCwdMount helper prefers name === "cwd", then a slug ending in __cwd, then falls back to the first mount. Mount queries are now ordered by created_at and id for deterministic results .
Loading states and partial failures (PR #5430): Session cwd and agent mounts now resolve independently β available files appear immediately with a "Loading moreβ¦" hint while a second mount is pending. Mount failures surface a warning badge with retry rather than silently appearing as an empty drive .
UI polish (PR #6016): Error states are styled to match the centered empty state; the search toolbar is hidden when there is no file tree to browse (but shown during loading); and ID menu entries are compact single-line copy items .
Empty session filtering (PR #5944): The sidebar hides sessions without a turn, title, message preview, or automation identity from the session list; pinned and waiting rows are retained. The sidebar request window was widened from 20 to 100 to prevent blank sessions from consuming recent-session slots .
Background Job Execution#
Background and scheduled agent jobs are dispatched through Celery, configured in api/oss/src/celery_config.py . Evaluation tasks run within the Python services container via evaluators_service.py β outside the runner's sandbox boundary and without Daytona. Agent turns triggered by background jobs route through http://runner:8765 and enter the same sandbox lifecycle described above. The Python API sandbox security model lives in api/oss/src/services/security/sandbox.py .
Key Files Reference#
| File | Purpose |
|---|---|
sandbox_agent.ts | Top-level lifecycle (acquireEnvironment / runTurn) |
teardown.ts | Teardown reason β stop vs delete dispatch |
session-pool.ts | Keep-alive pool; config/history/credential fingerprinting; TTL/LRU eviction |
session-continuity.ts | In-memory (sessionId, harness) β {agentSessionId, turnIndex} store |
session-continuity-durable.ts | Durable mirror to session_states.data; survives runner restart |
sandbox-reconnect.ts | Reconnect ladder: stored ID β restart β cold create |
daytona-provider.ts | Daytona pause/reconnect; warm pool; network policy convergence |
agent-mount.ts | Durable S3-backed mount; symlink healing; credential signing |
mount.ts | Mount reachability check; tunnel discovery; silent-skip bug (Issue #5342) |
config/sandbox-recipe.json | Versioned Daytona snapshot recipe β shared source of truth |
images/sandbox/daytona/build_snapshot.py | Snapshot build script; reads recipe version |
api/oss/src/core/mounts/service.py | Mount slug minting; get_or_create_agent_mount |
api/oss/src/services/security/sandbox.py | Python API sandbox security model |
api/oss/src/celery_config.py | Celery task queue configuration |
useSessionDrive.ts | Frontend drive root resolution (session cwd vs agent mount) |