Sandbox Infrastructure#

The runner (services/runner/) is a Node.js/TypeScript sidecar that executes agent workloads in isolated sandboxes. It exposes an HTTP API on port 8765 and is called by the Python services container for all sandboxed execution 1. Two sandbox providers are supported: local (an unconfined host process, default in dev) and Daytona (cloud-based, for remote isolation). Provider selection is controlled by AGENTA_RUNNER_DEFAULT_SANDBOX_PROVIDER and AGENTA_RUNNER_ENABLED_SANDBOX_PROVIDERS 1.

This article covers the full sandbox lifecycle — session persistence, reuse, reconnect, durable mounts, credential expiry, tool delivery, and background execution. For provider selection via env-vars, see Sandbox Provider Configuration. For timeout enforcement layers, see Agent Runner Timeout Enforcement.

Sandbox Lifecycle: Session States and Transitions#

A sandbox session passes through five states driven by Daytona idle timers and explicit runner actions 2:

State transitions are governed by three Daytona idle timers — DAYTONA_AUTOSTOP (default 5 min), DAYTONA_AUTOARCHIVE (default 15 min), and DAYTONA_AUTODELETE (default 30 min) — plus explicit runner pause/reconnect/delete calls from teardown.ts 3. Teardown reason determines action: kill/failed/aborted turns delete the sandbox; clean/resumable turns and idle-TTL expiry stop it; in-flight shutdown deletes 3.

Session Persistence and Keep-Alive#

Three layered PRs (all merged July 2026) progressively solve session memory loss across turns.

Local Sandbox Keep-Alive Pool (PR #5156)#

PR #5156 4 introduces a keep-alive pool in session-pool.ts that holds harness sessions alive for a short TTL after each turn ends. On the next turn, the runner checks three fingerprints:

• Config fingerprint — hash of harness, sandbox, model, provider, tools, permissions, and system prompt (excluding per-turn volatiles)
• History fingerprint — hash of ordered user message texts and tool-call IDs (assistant text excluded; edited history trips mismatch)
• Credential epoch — process-local SHA256 over canonical {secrets, toolCallbackAuth} plus mount credential expiry

If all three match and the new turn's tail is a plain user message, the runner continues the live session (session/load-style) instead of cold-replaying. Otherwise it falls back to cold start. The pool is off by default (AGENTA_RUNNER_SESSION_KEEPALIVE=true to enable) with an LRU cap (AGENTA_RUNNER_SESSION_POOL_MAX, default 8) and TTL (AGENTA_RUNNER_SESSION_TTL_MS, default 60 s; AGENTA_RUNNER_SESSION_APPROVAL_TTL_MS, default 600 s for sessions awaiting HITL approval) 4.

Daytona Sandbox Reuse and Warm Pooling (PR #5225)#

PR #5225 3 eliminates the ~15 s rebuild penalty on every Daytona turn by adding real pause/reconnect wrappers to daytona-provider.ts. Two reuse levels:

1. Park-to-running — sandbox stays running for AGENTA_RUNNER_DAYTONA_SESSION_IDLE_TTL_MS (default 120 s). Next turn continues live session at ~1.4 s vs ~12.5 s cold.
2. Park-to-stopped — after the window expires or warm cap is full (AGENTA_RUNNER_DAYTONA_SESSION_MAX_WARM, default 20), sandbox stops (not deletes). Restart + session/load takes ~7.7 s vs ~12.5 s cold.

On reconnect, the runner reads the live sandbox's networkBlockAll / networkAllowList and calls updateNetworkSettings only when policy differs — so parked sandboxes pick up permission changes without rebuild 3. The sandbox_id pointer write is guarded by compare-and-set on latest_turn_index to close the concurrent-turn race window 3.

Durable Session Continuity (PR #5197)#

PR #5197 2 extends session memory across runner restarts and sandbox death using two stores:

• session-continuity.ts — in-memory map of (sessionId, harness) → {agentSessionId, turnIndex} with a staleness guard (harness may session/load only if it authored the most recent turn)
• session-continuity-durable.ts — mirrors the in-memory store to the API's session_states.data row via GET-then-PUT read-modify-write; survives runner restart

sandbox-reconnect.ts implements the reconnect ladder: stored sandbox ID → try restart → fall back to fresh create on failure 2. Per-harness transcript mounts (~/.claude/projects, Pi sessions dir) get durable geesefs treatment so even a Dead sandbox can resume natively; Claude's .credentials.json is explicitly excluded 2. Controlled by AGENTA_SESSION_HARNESS_MOUNTS=false to disable 2.

Durable Storage Mounts#

Each agent gets a persistent S3-backed folder that survives across sessions. The runner mounts it at <cwd>-agent/ via geesefs (FUSE) and creates an agent-files symlink inside the session cwd. The symlink is rebuilt every run because geesefs silently downgrades symlinks to empty files across remounts 5. Core mount/symlink logic lives in agent-mount.ts 5:

The runner container requires SYS_ADMIN capability, /dev/fuse device access, and apparmor:unconfined to support FUSE mounts 1. The API layer (sign endpoint at POST /mounts/agents/sign) uses a deterministic slug __ag__agent__{canonical_artifact_id}__{slugified_name} for idempotency across repeated calls 5. See also: api/oss/src/core/mounts/service.py and api/oss/src/apis/fastapi/mounts/router.py 5.

Known Issues and Failure Modes#

STS Credential Expiry — EACCES After ~15 Min (Issue #5516)#

Agent mounts are initialized with 900-second STS credentials that are never refreshed. After expiry, geesefs continues running but receives 403 from SeaweedFS, mapped to EACCES. All filesystem operations fail with "Permission denied" for ~15 minutes; the mount appears healthy to liveness checks. Recovery only happens at turn boundaries when a remount occurs. Observed: 16 denial windows in one day, longest 14 min 10 s — deterministically breaking any turn longer than 15 min 6.

Proposed fixes 6: (1) detect EACCES in mount liveness probes to trigger immediate remount, (2) refresh/re-sign credentials before expiry at ~80% of TTL, (3) increase duration_seconds beyond expected turn length, (4) assign unique RoleSessionName per mount for better isolation.

Silent Mount Skip — Agent Files Lost (Issue #5342)#

When the object store is unreachable (no public endpoint or tunnel), the runner logs "tunnel discovery failed" and silently degrades: the durable workspace mount is skipped but the session continues. Agent files written during that turn are lost with no user-facing error 7. The relevant code is in mount.ts (reachability check and tunnel discovery). Desired behavior: fail loudly when the durable mount is required but unavailable, following patterns like PI_PERMISSION_EXTENSION_UNAVAILABLE_MESSAGE 7.

Claude Tools Missing on Daytona — Silent Failure (Issue #4984)#

The runner's internal MCP server (tools) listens on 127.0.0.1 of the runner host. On Daytona remote sandboxes, 127.0.0.1 resolves to the sandbox's own loopback, so the harness cannot reach the tool server — Claude receives zero tools and fails silently 8. Pi environments work because an in-sandbox extension handles the relay; Claude has no equivalent. The fix introduces an in-sandbox stdio shim (tool-mcp-stdio.ts) for Daytona transport, and added an interim fail-loud error for the Claude + Daytona + tools combination 8. For MCP transport selection logic, see MCP Tool Execution.

Background Job Execution#

Background and scheduled agent jobs are dispatched through Celery, configured in api/oss/src/celery_config.py 9. Evaluation tasks run within the Python services container via evaluators_service.py 10 — outside the runner's sandbox boundary and without Daytona. Agent turns triggered by background jobs route through http://runner:8765 and enter the same sandbox lifecycle described above. The sandbox security model at the Python API layer lives in api/oss/src/services/security/sandbox.py 11.

Key Files#