Workflow and Resource Management#

This article covers two interrelated production reliability concerns in the nnmbl backend: DB connection exhaustion caused by long-running LLM calls, and DBOS workflow lifecycle problems (shutdown races and replay non-determinism). These issues compound each other — the shutdown race generates failed workflows that must be replayed, and high replay frequency amplifies LLM non-determinism errors.

Primary source: Production Performance Hotspots: Stale Docs Check and DBOS Workflows

DB Connection Exhaustion During LLM Calls#

Root cause: coordinator_review_activity is a context manager in page_workflow.py that opens a DB review thread and holds it open for the entire duration of durable_update_coordinator_agent.run_sync() — an LLM agent call that can run for several minutes. With STALE_DOCS_CHECK_QUEUE concurrency set to 50, up to 50 long-lived DB sessions can overlap simultaneously.

The SQLAlchemy pool is configured at size=8, overflow=20 (28 total connections). At concurrency=50, the pool is exhausted and new acquisition attempts raise sqlalchemy.exc.TimeoutError: QueuePool limit of size 8 overflow 20 reached. Increasing pool size only delays exhaustion — the fix requires releasing connections before LLM calls begin.

Secondary driver: list_published_pages in review_steps.py fetches the complete set of published pages for a knowledge store on every PR, with no filtering by changed files. Combined with the above, this generates ~40 compute-hours/day from stale_docs_check workflows alone.

Cascading effect: Trivial tools (write_todos, grep, read_file) also hit their 60-second timeouts — not because the tools are slow, but because they're waiting on DB pool acquisition. This triggers pydantic_ai.ToolRetryError (751×/week) , and after DEFAULT_AGENT_RETRIES are exhausted, the coordinator fails with UnexpectedModelBehavior (182×/week).

DBOS Shutdown Race (Cloud Run SIGTERM)#

Signal: RuntimeError: cannot schedule new futures after shutdown — 210×/week.

On Cloud Run SIGTERM, shutdown_dbos() in dbos_boot.py calls DBOS.destroy(workflow_completion_timeout_sec=5), giving in-flight workflows only 5 seconds to finish. After destruction, agent_thread_pool.shutdown(wait=False, cancel_futures=True) cancels all pending futures — killing any mid-flight LLM call.

Long-running agent workflows (multi-minute) have near-zero chance of completing within the 5-second window. Killed workflows are then re-queued by the conductor's reschedule loop, which feeds directly into replay non-determinism.

Recovery mechanism: When an executor terminates, DBOS publishes a DBOSExecutorDeathEvent to a PubSub topic. A recovery endpoint acquires a Redis lock and resumes workflows that were in-progress on the terminated executor. 1

DBOS Replay Non-Determinism (DBOSUnexpectedStepError)#

Signal: DBOSUnexpectedStepError — 134×/week; DBOSWorkflowConflictIDError when concurrent fork/resume attempts collide.

Each agent tool call is recorded as a DBOS step named AgentTool:<tool_name> via to_step(_run_tool, name=f"AgentTool:{tool_name}") in tool.py. On recovery, the LLM independently re-selects tools — often in a different order — so the replayed step name diverges from the recorded step name, raising DBOSUnexpectedStepError.

Mitigation — get_result_with_auto_fork in core/dbos/utils.py: catches DBOSUnexpectedStepError and forks the workflow onto a fresh ID from the divergent step (max_forks=3). For nested sub-workflow failures it locates the ancestor step on the parent workflow before forking. When the fork budget is exhausted (3 forks), the workflow fails permanently. The _STEP_CACHE memoization in the same file prevents duplicate DBOS step registrations.

Issue Interaction#

Cloud Run SIGTERM
      │
      ▼
shutdown_dbos() — 5 s timeout
      │ kills in-flight LLM calls
      ▼
Workflow marked failed → re-queued
      │
      ▼
DBOS recovery replay
      │ LLM picks different tools
      ▼
DBOSUnexpectedStepError → auto_fork
      │ if forks exhausted
      ▼
Permanent workflow failure

Reducing the shutdown race (e.g., longer workflow_completion_timeout_sec, graceful draining) directly reduces replay frequency, which in turn reduces divergence probability.

Key Files#