Plugin Runtime Stability#

LangBot manages plugin execution in a dedicated plugin runtime subprocess that communicates with the main process over two transports: stdio (default on Linux/macOS) or WebSocket (Docker deployments and Windows). The same stability machinery also governs MCP (Model Context Protocol) tool-server sessions, which run as Box-sandboxed subprocesses connected over a WebSocket relay. Both layers share the same core patterns: heartbeat monitoring, reconnection callbacks, process-vs-transport distinction, and bounded retry budgets.

Primary source files:

• src/langbot/pkg/plugin/connector.py — PluginRuntimeConnector
• src/langbot/pkg/provider/tools/loaders/mcp.py — RuntimeMCPSession, MCPLoader
• src/langbot/pkg/provider/tools/loaders/mcp_stdio.py — BoxStdioSessionRuntime, MCPSessionErrorPhase, _ColdStartRetry
• src/langbot/pkg/utils/managed_runtime.py — ManagedRuntimeConnector (base class)

Transport Selection and Connection Setup#

PluginRuntimeConnector.initialize() selects a transport at startup based on runtime environment 1:

Auto-reconnect on WebSocket disconnects is possible; stdio mode requires a manual restart 2.

For MCP sessions, RuntimeMCPSession._lifecycle_loop() dispatches to a transport-specific initializer based on server_config['mode'] 3:

Heartbeat Monitoring#

Plugin runtime: heartbeat_loop() pings the plugin runtime every 20 seconds 4. Failures are logged at debug level but do not trigger reconnection — only transport-level disconnects invoke the runtime_disconnect_callback. The heartbeat task is cancelled in dispose() 5. This was introduced in PR #1698 to provide baseline visibility into runtime health 6.

MCP sessions (Box stdio): monitor_process_health() polls the managed Box process status every 5 seconds. It tolerates up to 3 consecutive polling errors before declaring the process gone and signaling the lifecycle loop. Concurrent with _shutdown_event.wait() via asyncio.wait(return_when=FIRST_COMPLETED), the lifecycle loop inspects whether the process is still alive when the monitor returns 3.

Distinguishing Process Crashes from Transient Issues#

The central design challenge is separating a dead process (requires full rebuild) from a live process with a broken transport (requires only reconnect). LangBot uses three special exception types that route around the fatal retry budget entirely 3:

Failure detected
    │
    ├── Is the managed process still running?
    │ │
    │ ├── YES + transport/WS dropped → raise _TransportReconnect
    │ │ └── Re-attach transport with 1s sleep, no retry budget consumed
    │ │
    │ ├── YES + handshake not answered (cold start) → raise _ColdStartRetry
    │ │ └── Wait 2s, retry handshake, reuse same process
    │ │
    │ └── NO → fatal error → exponential backoff retry (max 3, delays 2/4/8s)
    │
    └── Server session expired → raise _CallerReconnect
          └── Rebuild MCP session, wake all waiting callers via shared _reconnected_event

• _TransportReconnect 7: Raised when monitor_process_health() signals exit but the Box process is still running — i.e., only the WebSocket relay dropped. _managed_process_is_running() confirms liveness. The _preserve_managed_process flag prevents cleanup from killing the healthy process.

• _ColdStartRetry 8: Raised during BoxStdioSessionRuntime.initialize() when the handshake fails but the process is alive. Designed for slow npm/npx cold starts where package installation can take 20–30 seconds. _managed_process_has_exited() is the authoritative check — transient API errors are treated as "still coming up", not exited.

• _CallerReconnect 9: Raised when invoke_mcp_tool() or read_resource_envelope() encounters a server-side session expiry ("session terminated"/"session expired"). Multiple concurrent callers share a single _reconnected_event to avoid thundering-herd reconnects.

BOX_UNAVAILABLE short-circuits all retries immediately since Box is disabled at the config level — retrying is pointless 3.

Retry Budget and Reconnection Logic#

MCP session retries are managed by _lifecycle_loop_with_retry() 3:

• Fatal retry budget: _MAX_RETRIES = 3 (4 total attempts), delays [2, 4, 8] seconds
• Out-of-budget exceptions: _TransportReconnect, _ColdStartRetry, _CallerReconnect — these reset status to CONNECTING and loop without incrementing the attempt counter
• Startup timeout: _ready_event is awaited with a 30s timeout for remote/SSE/HTTP sessions; Box stdio uses box_config.startup_timeout_sec + 30

Plugin runtime reconnection follows a simpler model 2:

• On Docker/WebSocket: transport-level disconnect invokes runtime_disconnect_callback, which retries the full connection
• On stdio: disconnect logs an error and requires a manual restart (no auto-reconnect, since restarting a stdio subprocess safely requires human intervention)

Caller-initiated reconnect for session expiry 9:

• _trigger_reconnect() sets _reconnect_event and creates _reconnected_event
• Lifecycle loop detects _reconnect_event, rebuilds the session, then sets _reconnected_event
• Callers wait up to 30 seconds (_RECONNECT_WAIT_TIMEOUT) for reconnection

Timeout and Hang Prevention#

A hung MCP server with no per-call timeout can lock the entire session. With concurrency.session: 1, a single blocking tool call drops all subsequent messages — this caused a 9-hour production outage 10.

PR #2344 addressed this by wrapping session.call_tool() in asyncio.timeout(30) 11:

• TimeoutError is caught before the generic exception handler so it is not retried (a hung server won't recover in 30s)
• Raises: "MCP tool '<name>' on server '<server>' timed out after 30 seconds"

Session-expiry timeout: _trigger_reconnect() waits up to 30s (_RECONNECT_WAIT_TIMEOUT) for the lifecycle loop to rebuild the session 9.

Startup timeout: start() blocks on _ready_event with bounded wait — callers don't hang indefinitely waiting for a never-connecting server 3.

Plugin install readiness: _wait_for_installed_plugin_ready() polls for status == 'initialized' for up to 30 seconds before declaring the install failed, preventing silent failures when dependency installation crashes the plugin process 12.

Error Phase Tracking and Diagnostics#

MCPSessionErrorPhase 3 tags which lifecycle stage failed, surfaced via get_runtime_info_dict() in the admin UI:

For the plugin runtime itself, PluginRuntimeNotConnectedError is raised when operations are requested before connection is established 13. Use get_debug_info() to retrieve the runtime's debug key and WebSocket URL for troubleshooting 14.

Key PRs and Change History#