LLM Request Timeout and Watchdog System#

A multi-layered reliability system ensures assistant responses are either delivered or fail gracefully. The layers stack from the HTTP socket level up to the browser UI:

1. HTTP timeout — caps how long the Ruby process waits for the OpenAI-compatible API.
2. Background job — enqueues work and provides base retry/discard behavior.
3. Browser watchdog — detects a permanently-spinning "Thinking…" bubble and reports back to the server.
4. Server-side age check — validates undelivered reports under a row lock; trusts no client clocks.
5. BackgroundJobHealth — surfaces worker queue problems in delivery-timeout diagnostics.

The diagram below shows how a failed response propagates through each layer:

Browser (chat_controller.js)
  └─ polls every 5s for pending bubbles older than 90s
  └─ POST /chats/:id/messages/:id/timeout
       └─ MessagesController#report_timeout
            └─ Chat#handle_undelivered_response!
                 └─ row lock + server-side 60s age check
                      └─ destroys/fails AssistantMessage
                      └─ broadcasts error UI
                      └─ captures DebugLogEntry w/ BackgroundJobHealth snapshot

Layer 1 — HTTP Request Timeout#

Provider::Openai sets request_timeout on the OpenAI Ruby client at initialization time:

client_options[:request_timeout] = ENV.fetch("OPENAI_REQUEST_TIMEOUT", 60).to_i

The .env.example documents this as: "HTTP timeout in seconds; raise for slow local models." This is the only timeout that applies at the network level — if the backend (OpenAI, Ollama, etc.) doesn't respond within this window, the Ruby client raises a timeout error which propagates to the background job.

This timeout applies to every call routed through the client: chat/assistant responses, auto-categorization, merchant detection, and PDF processing.

Layer 2 — Background Job and Error Handling#

When a user message is saved, Chat#ask_assistant_later immediately creates a pending AssistantMessage and enqueues AssistantResponseJob on the high_priority Sidekiq queue. The job calls message.request_response(...) — all error handling lives there and in the assistant classes.

ApplicationJob provides base-class retry/discard behavior:

• retry_on ActiveRecord::Deadlocked — auto-retries on DB deadlock.
• discard_on ActiveJob::DeserializationError — silently drops jobs with unresolvable arguments.
• enqueue_after_transaction_commit = true — prevents workers from picking up jobs before the triggering DB row is visible.

If the LLM call itself succeeds but returns an incomplete/failed stream, build_stream_error_message converts the raw streaming event into a human-readable error. The error is classified server-side in Chat#classify_error_message into one of: rate-limit, temporary provider error, misconfiguration, or generic — driving the localized message shown to users.

If the job never runs at all (worker down, queue not polled), no error is broadcast — the pending bubble simply never resolves. This is what the watchdog exists to detect.

Layer 3 — Browser Watchdog (chat_controller.js)#

The Stimulus chat_controller.js runs a client-side watchdog on every chat page. Two configurable values govern its behavior 1:

How it works:

1. On connect(), #startUndeliveredWatchdog() calls #checkUndeliveredResponses() immediately and then every pollInterval ms.
2. #checkUndeliveredResponses() iterates pendingResponseTarget elements (DOM nodes that only exist while a response is still pending). For each, it reads a data-pending-response-created-at timestamp and compares it to responseTimeout. Nothing triggers while the element is gone — no false positives once streaming starts.
3. When a bubble exceeds the threshold, #reportUndelivered(url) POSTs to data-pending-response-timeout-url. Reported and in-flight URLs are tracked in Sets to prevent duplicate calls. If the POST fails (network error), the URL stays un-reported so the next tick can retry.

The 90-second client threshold is intentionally longer than the 60-second server-side floor (see Layer 4) — it accounts for slow local models and long tool-call chains.

Layer 4 — Server-Side Age Check and Undelivered Response Handler#

The watchdog POST hits MessagesController#report_timeout, which calls Chat#handle_undelivered_response!.

Because the client clock is untrusted, the server enforces its own floor with UNDELIVERED_RESPONSE_TIMEOUT = 60.seconds. The full check runs inside a row-level lock to prevent races with a legitimate slow response 2:

1. Re-reads the AssistantMessage row under lock.
2. Bails if the message is no longer pending (a real response arrived).
3. Bails if created_at > 60.seconds.ago (not old enough yet).
4. If the message content is blank (job never started), destroys the row. If content is partially streamed, updates status to "failed" so history builders exclude it.

After resolution, the method records a DebugLogEntry with BackgroundJobHealth.snapshot included in metadata, and broadcasts the error UI via Turbo Streams.

The error payload type is "DeliveryTimeout" with a human-readable message key chat.errors.no_response and a technical message that includes the background job health summary 3.

Layer 5 — BackgroundJobHealth#

BackgroundJobHealth queries Sidekiq's Redis state from the web process (not the worker), so it can report a down worker even when the worker can't speak for itself 4.

Key constants 5:

healthy? returns true only when: at least one Sidekiq process is running and it polls high_priority and queue latency is under 60 seconds 6. On any Redis/Sidekiq error, it fails open (healthy: true) to avoid blocking the UI on unrelated infra blips 7.

BackgroundJobHealth.summary and .snapshot are embedded in the undelivered_error_payload technical message, giving operators immediate queue diagnostics when investigating a delivery timeout in debug logs 8.

Configuration Reference#

All timeout-related knobs are environment variables. See .env.example for the full list.

Tuning guidance: For slow self-hosted models (Ollama, LM Studio), raise OPENAI_REQUEST_TIMEOUT first. If the model reliably takes longer than 90 seconds from job enqueue to first token, also increase responseTimeout via the Stimulus value on the chat container element.