Agent Runtime (dify-agent-runtime)#

dify-agent-runtime is the Go-based sandboxed shell execution service that backs Dify's Agent V2 system. It provides isolated job execution — running shell commands, managing workspaces, and reading output — on behalf of agent runs orchestrated by the dify-agent FastAPI backend.

The runtime is a complete rewrite 1 of the original Python shellctl package (dify-agent/src/shellctl/ and dify-agent/src/shellctl_runtime/), replacing a FastAPI + shell-session-manager server with compiled Go binaries.

Motivation#

The rewrite targets two specific improvements 1:

• Faster startup time — Go compiled binaries eliminate Python interpreter overhead
• Smaller memory footprint — native execution with minimal runtime dependencies, no CGO required (CGO_ENABLED=0)

Architecture#

The runtime ships five binaries built from dify-agent-runtime/cmd/ 3:

Internally, dify-agent-runtime is organized as 5:

cmd/
  shellctl/ - main server binary (shellctl serve)
  sanitize-pty/ - tmux pipe-pane PTY sanitizer (stdin→stdout filter)
  runner-exit/ - post-drain SQLite exit recorder
  dify-agent-cli/ - cli tool talking to agent backend
  runner/ - process runner to bootstrap agent commands
internal/ - internal implementations

Dependencies include Go 1.26, modernc.org/sqlite (pure-Go SQLite driver, no CGO required), and tmux (runtime dependency) 6.

Path Isolation (Landlock)#

Each agent job runs inside a Landlock sandbox that restricts filesystem access per workspace 7:

Temp files stay isolated per workspace — the runner sets TMPDIR, TMP, and TEMP directly to the workspace directory (cwd), making the workspace both the command working directory and the temporary storage space 8.

Requires Linux ≥ 5.13; on unsupported kernels a warning is printed and isolation is skipped. Environment variables are documented in internal/envvar/envvar.go 9.

Docker Image#

The sandbox image (langgenius/dify-agent-local-sandbox) is built from dify-agent-runtime/docker/Dockerfile in a two-stage build:

1. Go build stage (golang:1.26): Compiles all five binaries with CGO_ENABLED=0 10
2. Runtime stage (python:3.12-slim-bookworm): Installs dev tooling for agent jobs — Node.js 24.20.0, pnpm 11.9.0, uv 0.8.9, tmux, git, ripgrep, jq, tini, iproute2 — then copies the Go binaries to /usr/local/bin/ 11

The container uses tini as its init process (PID 1) to supervise the shellctl server. This ensures proper reaping of orphaned child processes and clean signal handling inside the container. The entrypoint is ENTRYPOINT ["/usr/bin/tini", "-g", "--"] followed by CMD ["shellctl", "serve", "--listen", "0.0.0.0:5004"], which means tini spawns shellctl as a child process and manages its lifecycle 12.

The container runs as the non-root dify user (uid defaults to the created user) and exposes port 5004. The iproute2 package is required by the OpenShell supervisor; it is harmless for local and other backend deployments.

Build command 13:

docker build -f dify-agent-runtime/docker/Dockerfile \
  --build-context agent=./dify-agent \
  -t dify-agent-local-sandbox:local \
  dify-agent-runtime/

Supports linux/amd64 and linux/arm64. CI/CD in .github/workflows/build-push.yml was updated as part of PR #38841 to build from the new dify-agent-runtime/ context.

Job Execution Modes#

The POST /v1/jobs/run API accepts an optional mode field to control how the job's standard streams are wired:

The response models are identical in both modes. Use stdio for bounded, machine-readable control commands and pty for interactive jobs.

API Changes#

Request — POST /v1/jobs/run:

{
  "script": "echo hello",
  "mode": "stdio"
}

The mode parameter is optional and defaults to pty.

Job Artifacts:

• PTY mode: output.log (merged stdout/stderr)
• Stdio mode: output.log (stdout only), stderr.log (stderr only)

Input Endpoint — POST /v1/jobs/{job_id}/input:

Only works with PTY mode jobs. Stdio jobs return 409 with error code input_unsupported.

Home Snapshot API#

The runtime provides native save/restore of the sandbox home directory ($HOME) as tar+zstd streams 16. One snapshot operation runs at a time per runtime; concurrent requests return 409 snapshot_busy. Neither endpoint imposes a size limit — callers own size policy and must bound the streams in their own logic.

Each operation carries a total I/O deadline set by SHELLCTL_SNAPSHOT_TIMEOUT (Go duration string, e.g. 10m, 15m30s). It bounds how long a stalled peer can hold the single-operation gate; a peer that closes the connection releases it immediately. Unset or empty uses the default (45s); an unparseable or non-positive value fails startup.

POST /v1/snapshot/save#

Captures the $HOME directory as a streaming tar+zstd archive.

Request: Optional JSON body {"excludes": [...]} — gitignore syntax patterns matched at any depth. The runtime's own state directory (.local/share/shellctl) is always excluded and no pattern can re-include it; an excluded directory is not descended into. Malformed body returns 400 invalid_request. Empty body or {"excludes": []} captures everything except the state directory.

Response:

• Success: Streams application/octet-stream (chunked transfer). Success is signaled by HTTP trailers:

  • X-Snapshot-Status: ok
  • X-Snapshot-Sha256 — hex-encoded SHA256 of the archive
  • X-Snapshot-Bytes — total bytes written

  A cleanly terminated stream without the ok trailer, or an aborted connection, is a failure.

• Error:

  • 409 snapshot_busy — another snapshot operation is in progress
  • 500 snapshot_failed — failed before streaming began
  • Connection abort — failed mid-stream

An empty $HOME is not a special case: it streams an ordinary archive with no entries.

POST /v1/snapshot/restore#

Restores a tar+zstd archive into $HOME.

Request: Raw tar+zstd body (no parameters). Extracts under os.Root (path traversal, absolute names, and symlink escapes are refused). Archives are plain tar+zstd; the decoder caps the zstd window at 64 MiB, so foreign archives produced with long-window settings are rejected as malformed.

Response:

• Success: 200 OK with JSON body:
  ⧉

  {
    "entries": <number of files restored>,
    "bytes_written": <total bytes written>
  }
  

• Error:
  • 400 archive_malformed — invalid tar+zstd input
  • 409 snapshot_busy — another snapshot operation is in progress
  • 500 restore_failed — non-format failure (filesystem or environmental errors)

Restore is NOT transactional — a mid-stream failure can leave a partially restored $HOME, so callers must treat the sandbox as unusable and recreate it rather than retry into it.

Integration with dify-agent#

dify-agent-runtime (shellctl) is consumed by the dify-agent Python backend through the shell adapter layer. The ShellctlProvider implements ShellProviderProtocol and creates HTTP clients that connect to the shellctl server's REST API. Commands are run via ShellctlCommands, and files are transferred using base64-encoded payloads 14.

The shell layer in Agent V2 is controlled by the AGENT_SHELL_ENABLED config flag (default true; requires a runtime backend). When active, the dify.shell layer exposes tools like shell_run, shell_wait to the Pydantic AI agent engine 15.

Runtime Backends#

dify-agent supports four runtime backends for Home Snapshots and Execution Bindings. Select one through DIFY_AGENT_RUNTIME_BACKEND (local, enterprise, e2b, or openshell).

Local#

The local backend runs shellctl inside a dedicated dify-agent-local-sandbox container, separate from the dify-agent-backend FastAPI service. The backend connects to shellctl via DIFY_AGENT_SHELLCTL_ENTRYPOINT (or DIFY_AGENT_LOCAL_SANDBOX_ENDPOINT) 14. Home Snapshots and Workspaces are directories on the container's filesystem.

Enterprise#

Enterprise backend creates sandboxes on a Dify Enterprise Gateway. It supports default-Home Binding creation, acquisition, and coupled destroy. Explicit Home Snapshot materialization fails fast; there is no compatibility fallback to the retired Sandbox protocol.

E2B#

E2B backend provisions sandboxes on E2B cloud infrastructure. Configuration requires:

• DIFY_AGENT_E2B_API_KEY: E2B API key
• DIFY_AGENT_E2B_TEMPLATE: E2B template ID (e.g., difys-default-team/dify-agent-local-sandbox)
• DIFY_AGENT_E2B_ACTIVE_TIMEOUT_SECONDS: Maximum continuous active time for an E2B resource (default 3600s)
• DIFY_AGENT_E2B_SHELLCTL_PORT: Port shellctl listens on inside the E2B sandbox (default 5004)

One E2B resource represents both Binding and Workspace, so shared Workspace attachment is unsupported.

OpenShell#

OpenShell backend provisions one sandbox per Execution Binding on a self-hosted NVIDIA OpenShell gateway (≥ 0.0.106). Sandboxes are reached only through the gateway gRPC API. shellctl is bootstrapped through idempotent gateway exec on acquire and reached through an authenticated ForwardTcp tunnel.

Configuration:

• DIFY_AGENT_OPENSHELL_GATEWAY_ENDPOINT: Gateway gRPC endpoint (host, no scheme)
• DIFY_AGENT_OPENSHELL_WORKSPACE: OpenShell workspace name (default default)
• Auth: DIFY_AGENT_OPENSHELL_BEARER_TOKEN or mTLS bundle (DIFY_AGENT_OPENSHELL_TLS_CA_PATH, DIFY_AGENT_OPENSHELL_TLS_CLIENT_CERT_PATH, DIFY_AGENT_OPENSHELL_TLS_CLIENT_KEY_PATH)
• DIFY_AGENT_OPENSHELL_SANDBOX_IMAGE: Runtime image reference (must include shellctl and iproute2)
• DIFY_AGENT_OPENSHELL_DRIVER_CONFIG: JSON driver configuration mounting the shared Home Snapshot volume
• DIFY_AGENT_OPENSHELL_SHARED_MOUNT_PATH: In-sandbox mount path for the shared volume (default /mnt/dify-agent-shared)
• DIFY_AGENT_OPENSHELL_EGRESS_ALLOW: Optional comma-separated host egress allowlist (empty follows gateway/driver default)
• DIFY_AGENT_OPENSHELL_SHELLCTL_AUTH_TOKEN: Bearer token for shellctl data plane
• DIFY_AGENT_OPENSHELL_SHELLCTL_PORT: Port shellctl listens on (default 5004)
• DIFY_AGENT_OPENSHELL_READY_TIMEOUT_SECONDS: Sandbox startup timeout (default 300)
• DIFY_AGENT_OPENSHELL_EXEC_TIMEOUT_SECONDS: Gateway exec call timeout (default 120)

Home Snapshots are stored as directory copies on an operator-provided shared volume mounted into every sandbox at DIFY_AGENT_OPENSHELL_SHARED_MOUNT_PATH. Snapshots live under home-snapshots/<tenant-digest>/ on the volume. Production multi-tenant deployments must use one OpenShell workspace and one dedicated volume per tenant.

shellctl runs with SHELLCTL_ENABLE_PATH_ISOLATION=false because the OpenShell sandbox Landlock policy is authoritative. The runtime image must include the iproute2 package (an OpenShell supervisor requirement).

See the OpenShell Runtime Backend guide for deployment instructions, shared volume setup, and validation.

Key References#