Queue Polling Backoff Strategy#

The queue polling backoff strategy is a dynamic, contention-aware interval management mechanism used in the DBOS queue system. Each queue worker thread continuously polls a PostgreSQL database for newly enqueued workflows, and the strategy automatically adjusts how frequently that poll occurs based on observed database contention 1. When contention is detected — specifically, when the database returns serialization or lock-acquisition errors — the worker immediately doubles its polling interval, reducing load on the database. As subsequent polls succeed, the interval gradually decays back toward the configured minimum, restoring responsiveness 2.

The strategy is designed to handle the inherent contention that arises when multiple DBOS worker processes independently poll the same queue database table. In DBOS's distributed architecture, all connected processes act as queue workers and periodically poll the database for new workflows to execute without any centralized coordination 3. This decentralized design improves horizontal scalability but concentrates concurrent write pressure on shared rows, making adaptive backoff necessary to prevent polling storms. Jitter is applied to every sleep interval to further stagger concurrent workers and prevent synchronized polling spikes 4.

This polling algorithm applies to both in-memory queues (now deprecated) and database-backed queues. Database-backed queues reload their configuration from the database at the start of each polling iteration, enabling dynamic reconfiguration without restart. In-memory queues use configuration that remains constant after initialization.

Backoff Algorithm#

The backoff algorithm operates on three interval bounds and two update rules applied per polling iteration.

Interval Bounds#

The worker thread initializes three values at startup 5:

Update Rules#

Two interval update rules execute on every iteration of the worker loop:

On contention detection (applied immediately, before sleep): the current interval doubles, clamped to the maximum 6:

polling_interval = min(max_polling_interval, polling_interval * 2.0)

After every iteration (applied unconditionally, whether the poll succeeded or failed): the interval decays by 10%, clamped to the minimum 7:

polling_interval = max(min_polling_interval, polling_interval * 0.9)

This combination produces an asymmetric response: contention causes an immediate doubling while recovery is gradual, taking multiple successful iterations to return to the base interval. For example, starting from a 1-second base interval, a single contention event raises the interval to 2 seconds; after that, it takes approximately 7 successive clean iterations to return to ~1 second (via repeated 0.9× decay).

Jitter#

The actual sleep duration on each iteration is not exactly polling_interval but a uniformly sampled value in the range [0.95 × polling_interval, 1.05 × polling_interval] 4:

stop_event.wait(timeout=polling_interval * random.uniform(0.95, 1.05))

This ±5% jitter prevents multiple worker threads or processes that started with similar intervals from polling in lockstep, mitigating the thundering-herd problem.

Interval Progression Example#

Starting from polling_interval_sec = 1.0:

Contention Detection#

The backoff is triggered exclusively by two specific PostgreSQL error conditions, surfaced through the SQLAlchemy/psycopg error hierarchy 8:

Both errors are wrapped by SQLAlchemy in an OperationalError. The worker catches OperationalError, inspects the underlying psycopg error, and only triggers backoff for these two specific subtypes 9. All other OperationalError subtypes and general exceptions are logged as warnings without modifying the polling interval.

When backoff triggers, a warning is logged that includes the queue name and the new polling interval value 10:

Contention detected in queue thread for <queue_name>. Increasing polling interval to <new_interval>.

Sources of Database Contention#

Contention arises inside start_queued_workflows() in dbos/_sys_db.py, the method called by each worker to dequeue and claim workflows. Three architectural choices within this method are the primary contention sources:

1. REPEATABLE READ Transaction Isolation#

For PostgreSQL, start_queued_workflows() opens a transaction at REPEATABLE READ isolation level only when a global concurrency limit (queue.concurrency) or a rate limiter (queue.limiter) is configured 11:

if self.engine.dialect.name == "postgresql" and (
    queue.concurrency is not None or queue.limiter is not None
):
    c.execute(sa.text("SET TRANSACTION ISOLATION LEVEL REPEATABLE READ"))

This ensures that concurrent workers see consistent snapshots when checking rate limits and global concurrency counts. For queues with only worker_concurrency set, the default READ COMMITTED isolation level is used, reducing the risk of serialization failures. When REPEATABLE READ is active, it introduces the risk of serialization failures when two workers operate on overlapping snapshots and one attempts to update rows that the other has already read.

2. Row-Level Locking: SKIP LOCKED vs. NOWAIT#

The workflow-selection query uses FOR UPDATE locking, but the mode depends on whether a global concurrency limit is configured 12:

When NOWAIT is in use, any two workers attempting to claim the same rows simultaneously will cause at least one of them to receive a LockNotAvailable error and back off.

3. Concurrent Read-Write Operations#

The method executes multiple SELECT queries (for rate-limit counting and, when global concurrency is configured, global concurrency counting) followed by per-row UPDATE operations, all within a single transaction 13. Worker concurrency (worker_concurrency) is now enforced using an in-memory map (ActiveWorkflowById.count_for_queue()) rather than a database query, eliminating that source of read-write overlap. The duration and read-write overlap of the transaction increases with queue depth and the number of configured limits, widening the window for snapshot conflicts between concurrent workers.

The UPDATE statement that transitions workflows from ENQUEUED to PENDING now includes a WHERE clause verifying the current status is ENQUEUED, preventing multiple workers from claiming the same workflow if they race on the same row.

Worker Thread Architecture#

The backoff logic is encapsulated entirely in the queue_worker_thread() function in dbos/_queue.py 1. A separate thread instance runs per queue, managed by the parent queue_thread() function 14.

Control Flow#

Key Implementation Details#

• Configuration reloading (database-backed queues): At the start of each polling loop iteration, the worker reloads the queue configuration from the database by calling dbos._sys_db.get_queue(). This allows queue parameters — concurrency, worker_concurrency, limiter, priority_enabled, partition_queue, and polling_interval_sec — to be modified dynamically via the Queue.set_* methods or the client API without restarting the worker. In-memory queues do not reload configuration and use their initial values for the lifetime of the worker thread.
• Queue deletion detection: For database-backed queues, if get_queue() returns None, the worker logs a message and exits the thread. This enables graceful worker shutdown when a database-backed queue is deleted from the queues table.
• Graceful shutdown: The worker uses stop_event.wait(timeout=...) rather than time.sleep(), so it responds to stop signals immediately without waiting for the full polling interval to elapse 15.
• Workflow execution: For each dequeued workflow ID, the thread calls execute_workflow_by_id() in a nested try/except, so a failure to launch one workflow does not abort the others or disrupt the backoff state 16.
• Partitioned queues: When partition_queue=True, the thread first fetches all active partition keys and issues a separate start_queued_workflows() call per partition key 17. Each such call is an independent transaction and can independently produce contention errors.
• Thread naming: Worker threads are named queue-worker-<queue_name> for observability 18.

Queue Manager Thread#

The parent queue_thread() function runs a single manager thread that monitors registered queues at a 1-second check interval, spawning a new queue_worker_thread for any queue whose worker thread is absent or has died 14. On shutdown, it joins all worker threads with a 10-second timeout per thread 19.

Configuration#

polling_interval_sec#

The polling_interval_sec parameter on the Queue constructor is the sole user-configurable input to the backoff algorithm 20:

Queue(
    name: str,
    concurrency: Optional[int] = None,
    limiter: Optional[QueueRateLimit] = None,
    *,
    worker_concurrency: Optional[int] = None,
    priority_enabled: bool = False,
    partition_queue: bool = False,
    polling_interval_sec: float = 1.0,
)

Parameters That Amplify Contention#

Certain other Queue parameters increase the likelihood that contention errors will be encountered, indirectly making the backoff more aggressive:

Related DBOS Polling and Backoff Mechanisms#

The queue polling backoff strategy is distinct from several other interval and retry mechanisms in the DBOS system:

Scheduler Polling#

The DBOS dynamic scheduler uses a separate scheduler_polling_interval_sec parameter (default 30 seconds) to control how frequently it polls for schedule changes 27 28. Its jitter strategy differs: it caps random delay at 10 seconds regardless of interval size 27, whereas queue workers apply a proportional ±5% jitter. The scheduler does not implement contention-driven backoff.

Step-Level Retry Backoff#

DBOS provides independent exponential backoff for step retries, configured per step via the @DBOS.step() decorator with interval_seconds, max_attempts, and backoff_rate parameters 29. The maximum retry interval is capped at 3600 seconds 30. This mechanism governs re-execution of individual workflow steps on application-level failures, not database polling frequency.

PostgreSQL LISTEN/NOTIFY#

DBOS supports a use_listen_notify configuration option as an event-driven alternative to polling-based queue detection 29. When enabled, workers are notified of new queue entries rather than discovering them via periodic polls, eliminating poll-induced database load and making contention-driven backoff unnecessary for that notification path.

Relevant Code Files#

Related Topics#

• DBOS Queue System — workflow queuing, concurrency limits, rate limiters, and partitioned queues
• PostgreSQL Transaction Isolation — REPEATABLE READ semantics and snapshot conflict behavior
• PostgreSQL Row-Level Locking — FOR UPDATE, SKIP LOCKED, and NOWAIT lock modes
• Exponential Backoff — general algorithm class; doubling on failure, multiplicative decay on success
• Thundering Herd Problem — the pathology that jitter and backoff jointly mitigate in distributed polling systems
• Thread Lifecycle Management — how DBOS manages daemon threads and graceful shutdown via threading.Event
• Database Connection Pooling — system database pool configuration (sys_db_pool_size, default 20) that bounds concurrent polling connections 33