Concurrency Limit Management (v2)#

Prefect's v2 global concurrency limit system serializes slot acquisition and release through atomic SQL UPDATE statements, relying on PostgreSQL's READ COMMITTED isolation level for correctness. The primary implementation lives in two files:

• Model layer: src/prefect/server/models/concurrency_limits_v2.py — all slot math and DB mutations
• HTTP API layer: src/prefect/server/api/concurrency_limits_v2.py — REST endpoints that call the model layer
• DB configuration: src/prefect/server/database/configurations.py — isolation level enforcement

Slot Acquisition and Release#

bulk_increment_active_slots#

bulk_increment_active_slots() performs a single UPDATE … WHERE active_slots + requested ≤ limit. The condition is embedded in the WHERE clause, not in application code, so the check-and-increment is atomic within the transaction. It returns True only if rowcount == len(concurrency_limit_ids) — i.e., all named limits had room. 1

Decay is applied inline: active_slots_after_decay() 2 subtracts floor(slot_decay_per_second × elapsed_seconds) from stored active_slots before the comparison, floored at 0.

bulk_decrement_active_slots#

bulk_decrement_active_slots() issues a UPDATE that clamps active_slots - slots to 0, preventing negative counts. When occupancy_seconds is provided, it also updates avg_slot_occupancy_seconds as a weighted-average over the last limit × 2 samples — used later to compute Retry-After headers. 3

Denied-slot tracking#

bulk_update_denied_slots() increments denied_slots when acquisition fails. The API layer uses this counter — along with avg_slot_occupancy_seconds — to compute a jitter-aware Retry-After value (via clamped_poisson_interval) returned in HTTP 423 Locked responses. 4

PostgreSQL: READ COMMITTED Isolation#

AsyncPostgresConfiguration.engine() 5 sets isolation_level="READ COMMITTED" explicitly on engine creation. The inline comment explains the rationale: 6

Prefect's conditional concurrency-counter updates depend on PostgreSQL's READ COMMITTED behavior: a waiting writer re-reads the latest committed row and re-evaluates its predicate before applying the change. Enforce the isolation level explicitly so correctness does not depend on database or role defaults.

Under READ COMMITTED, when two concurrent UPDATE statements target the same row, PostgreSQL serializes them through an implicit row lock. The second writer re-reads the now-committed value before applying its own change, which means the capacity check (active_slots + slots ≤ limit) is re-evaluated against reality, not a stale snapshot. This was enforced explicitly in PR #22531 (merged 2026-07-20) after a race condition was reported in production. 7

SQLite: Uses BEGIN IMMEDIATE transactions via with_for_update=True to achieve equivalent serialization. See AioSqliteConfiguration.begin_transaction().

HTTP API Paths and the Race Condition#

Both /increment and /increment-with-lease call _acquire_concurrency_slots() inside db.session_context(begin_transaction=True), which wraps the acquisition in an atomic transaction. 8 9

Root cause (pre-PR #22531): If the PostgreSQL connection used a stricter isolation level than READ COMMITTED (e.g., REPEATABLE READ or SERIALIZABLE), waiting writers would re-evaluate their WHERE clause against a stale snapshot rather than re-reading the latest committed row. This allowed two concurrent transactions to both see active_slots < limit and both increment, overflowing the limit. The reported production case showed active_slots=2 on a limit=1 guard immediately after two flows woke from a wait. 7

PR #22531 hardcodes READ COMMITTED at the engine level, overriding any database or role defaults. 10 5

Remaining limitation: The HTTP API path does not use SELECT FOR UPDATE. READ COMMITTED serializes the actual UPDATE writers through an implicit row lock, but the preceding metadata read in bulk_read_concurrency_limits() is unlocked. Concurrent requests that arrive and read metadata simultaneously before either commits still represent a narrower-but-real race window. This is an acknowledged architectural constraint; the long-term fix is a move to Redis. 7

Lease-Based API#

/increment-with-lease adds a lease layer on top of slot acquisition: 9

1. Acquires slots atomically (same path as /increment)
2. Creates a lease record (TTL: 5 min – 24 hr, default 5 min) in lease storage
3. Returns a lease_id used by the client to release slots via /decrement-with-lease

/decrement-with-lease 11 reads the lease to recover slots and computes occupancy_seconds from lease.created_at, passing both to bulk_decrement_active_slots before revoking the lease. Leases can also be renewed via POST /leases/{lease_id}/renew. 12

Related Issues and Long-Term Direction#

• Production race condition report: Discussion #20520 — two flows acquired the same limit=1 slot when the slot was released
• PR #22531 enforced READ COMMITTED 10
• Long-term: The Prefect maintainer noted an intent to move concurrency slot management to Redis to eliminate the DB-level contention entirely
• AwaitingConcurrencySlot delays: On push work pools (e.g., Modal), flow runs in AwaitingConcurrencySlot state may see 12–15 minute delays before promotion even when slots are free 13