Rate Limiting and Concurrency Control#

Dify enforces per-app concurrent request quotas using a Redis-backed RateLimit class. Each app gets a cap on how many in-flight requests it can have simultaneously. When the limit is exceeded, AppInvokeQuotaExceededError is raised immediately — there is no queuing.

Entry points:

• api/core/app/features/rate_limiting/rate_limit.py — the RateLimit class
• api/services/app_generate_service.py — where RateLimit is instantiated and applied for every app invocation
• api/configs/feature/__init__.py — the two global config knobs

Redis Data Model#

RateLimit uses two Redis keys per client_id (the app UUID) 1:

Each request gets a UUID key inside the hash. The hash length (HLEN) is the live concurrency count. Requests that exceed _REQUEST_MAX_ALIVE_TIME (10 minutes) are evicted during periodic flush_cache runs 2.

Request Lifecycle#

RateLimit is a per-process singleton keyed on client_id 3. The full flow per request:

1. enter(request_id) — reads HLEN of the active-requests hash. If ≥ max_active_requests, raises AppInvokeQuotaExceededError; otherwise calls HSET to record the request 4.
2. exit(request_id) — calls HDEL to remove the request from the hash 5.
3. For streaming responses, exit is deferred: RateLimitGenerator wraps the generator and calls exit only when the generator is exhausted or closed 6.
4. For workflow/advanced-chat streaming, rate_limit_context (a contextmanager) covers the dispatch phase, after which the RateLimitGenerator wrapper holds the slot for the lifetime of the event stream 7.

In AppGenerateService._run_with_guardrails, quota billing and rate limiting are coordinated: billing is committed only after rate_limit.enter() succeeds, and refunded on any exception 8.

Quota Configuration#

_get_max_active_requests resolves the effective limit from two config values 9:

• APP_DEFAULT_ACTIVE_REQUESTS — per-app default; defaults to 0 (unlimited) 10
• APP_MAX_ACTIVE_REQUESTS — global ceiling; defaults to 0 (unlimited) 11

The logic takes the minimum of non-zero values. If both are 0, the limit is disabled entirely (RateLimit.disabled() returns True and all requests receive the sentinel _UNLIMITED_REQUEST_ID).

Apps can also override the limit via app.max_active_requests on the App model 12.

TOCTOU Race Condition in flush_cache#

The original flush_cache implementation had a Time-of-Check-Time-of-Use (TOCTOU) bug:

# Vulnerable pattern (before fix):
if redis_client.exists(max_active_requests_key): # key present here…
    val = redis_client.get(max_active_requests_key) # …may be gone here → None
    int(val.decode("utf-8")) # AttributeError on None

If the Redis key expired between exists() and get(), .decode("utf-8") was called on None, raising AttributeError and crashing the instance. The fix (PR #37548) collapses the two calls into a single get() and guards with a None check, falling back to the in-memory self.max_active_requests when the key is absent. This is the canonical fix pattern for this class of Redis TOCTOU bug.

The current implementation (flush_cache lines 50–71) reflects the fixed, single-get() approach 13.

Known Limitation: Non-Atomic Check-Then-Set#

The enter() path is not atomic. Between the HLEN check and the HSET, a concurrent worker on another process can read the same count and both workers may proceed past the limit. Under high concurrency this allows brief over-admission (by at most N_workers - 1 requests). The current design accepts this as a trade-off: the 10-minute stale-request eviction in flush_cache bounds drift, and the periodic recalculate (_ACTIVE_REQUESTS_COUNT_FLUSH_INTERVAL = 5 min) keeps the hash authoritative 14.

An atomic alternative would be a Lua script running HLEN + HSET in a single Redis round-trip, or using MULTI/EXEC with WATCH. Neither is currently implemented in this module.

Tests#

Unit tests live at api/tests/unit_tests/core/app/features/rate_limiting/test_rate_limit.py, covering singleton behavior, cache flush, enter/exit logic, the RateLimitGenerator wrapper, and concurrent access. After the TOCTOU fix, the exists.return_value mock was removed from the test suite 15.