Durable Queue Semantics#
DBOS queues are PostgreSQL-backed, workflow-centric primitives that provide exactly-once processing, durable state, and automatic recovery โ contrasting with standalone message brokers (Kafka, RabbitMQ) where these guarantees must be engineered separately.
Every queued task is a DBOS workflow. Queues provide flow control (concurrency limits, rate limits, ordering) over when those workflows run, not a separate transport layer.
Key entry points:
dbos/_queue.pyโQueueclass, worker threads, enqueue logicdbos/_schemas/system_database.pyโworkflow_statustable schemadbos/_sys_db.pyโstart_queued_workflows(), dequeue SQL- Queue & Concurrency tutorial โ official usage docs
How Enqueue Works (Durability Before Execution)#
When Queue.enqueue() is called, it invokes start_workflow(..., execute_workflow=False), which writes the workflow to the workflow_status table with status ENQUEUED (or DELAYED if a delay is set) before any execution begins . This write is the source of truth: the work is durable the moment it is enqueued. If the process crashes before the worker picks it up, the record remains in the database and will be dequeued on recovery.
The workflow_status table row includes queue_name, inputs (serialized args), status, priority, deduplication_id, queue_partition_key, and delay_until_epoch_ms .
Enqueue is also exposed via the DBOS Client (for cross-app enqueueing) and via the dbos.enqueue_workflow PL/pgSQL function (for Postgres triggers) .
Exactly-Once Delivery#
DBOS achieves exactly-once processing through two complementary mechanisms:
1. Workflow ID idempotency. Every workflow has a unique workflow_uuid. If the same UUID is submitted twice (e.g., after a crash and restart), the database insert is a no-op and the workflow executes exactly once. This is the mechanism used by the Kafka integration: it derives a deterministic ID from kafka-unique-id-{topic}-{partition}-{group.id}-{offset}, promoting Kafka's at-least-once delivery to exactly-once processing at the application layer .
2. Deduplication IDs. At enqueue time, callers can supply a deduplication_id via SetEnqueueOptions. The workflow_status table enforces a unique index on (queue_name, deduplication_id) for non-NULL values. If a workflow with that deduplication ID is currently ENQUEUED or PENDING, a subsequent enqueue raises DBOSQueueDeduplicatedError . The deduplication_id is cleared (set to NULL) once the workflow completes , so the same ID can be reused in future runs.
Deduplication IDs are not supported on partitioned queues .
Checkpoint and Recovery#
Every workflow step result is stored in operation_outputs . On failure and restart, the workflow engine replays from the last persisted step, skipping already-completed operations. Queued workflows in ENQUEUED status that were never picked up are simply dequeued by the next healthy worker โ no explicit replay trigger is needed.
The queue worker thread transitions DELAYED workflows to ENQUEUED on each polling cycle, then calls start_queued_workflows() which uses SELECT ... FOR UPDATE SKIP LOCKED to atomically claim workflows without double-dispatch across concurrent workers.
Workflow timeouts are durable: the workflow_deadline_epoch_ms is stored in the database and survives restarts .
Concurrency and Flow Control#
All flow control operates on the workflow_status table โ there is no in-process queue buffer.
| Parameter | Scope | Notes |
|---|---|---|
worker_concurrency | Per process | Limits workflows running on a single process |
concurrency | Global (all processes) | Counts all PENDING workflows on the queue |
limiter (limit/period) | Global | Rate limit: max starts in a sliding window; tracked via rate_limited flag |
priority_enabled | Per queue | Lower number = higher priority; FIFO within same priority |
partition_queue | Per partition | Applies concurrency limits independently per partition key |
Queue configuration lives in the system database and can be changed at runtime via set_concurrency(), set_limiter(), etc. without a redeploy . Workers pick up the new configuration on their next polling iteration .
The idx_workflow_status_in_flight index on (queue_name, status, priority, created_at) optimizes the hot dequeue path .
How DBOS Queues Differ from Standalone Message Brokers#
| Property | DBOS Queue | Kafka / RabbitMQ |
|---|---|---|
| Durability store | PostgreSQL (workflow_status) | Separate broker infrastructure |
| Exactly-once | Database-enforced via workflow IDs / dedup unique index | Requires broker transactions or manual offset management |
| Checkpoint / replay | Automatic via operation_outputs; resume from last step | Must be implemented by consumer |
| Ordering | FIFO (default) or priority, enforced by DB query ordering | Partition-level ordering; complex cross-partition guarantees |
| Recovery | Any healthy worker dequeues ENQUEUED records | Requires consumer group rebalancing |
| State | Workflow state and queue entry are one record | Separate message payload and consumer state |
The core architectural difference: DBOS wraps every queued message as a durable workflow record in the database before execution begins . This makes the database the single source of truth โ no separate broker, topic, or offset log is required for durability.