Thread-Safe TTLCache#

cachetools.TTLCache is not thread-safe. On __setitem__, it calls an internal expire() method that walks a linked list. If another thread mutates the cache concurrently, the list can be corrupted, causing expire() to raise a KeyError on a key that was just inserted. Any shared TTLCache accessed from multiple threads must be protected by an external threading.Lock. 1

The Bug: _file_cache in settings/sources.py#

src/prefect/settings/sources.py maintains a module-level, unprotected cache 2:

_file_cache: TTLCache[str, dict[str, Any]] = TTLCache(maxsize=100, ttl=60)

_read_toml_file() reads and writes this cache without any lock. Because PrefectBaseSettings is instantiated at import time and in worker threads, any code path that creates settings from multiple threads concurrently can hit the race, surfacing as an intermittent KeyError from deep inside cachetools. 1

Real-world failure path (Prefect 3.7.4, AWS ECS worker): 1

EcsWorkerSettings() # called in a worker thread
  → ProfileSettingsTomlLoader.__init__() # settings/sources.py:133
  → _load_profile_settings() # sources.py:141
  → _read_toml_file(profiles_path) # sources.py:37
  → _file_cache[cache_key] = data # ← TTLCache.__setitem__ → expire() → KeyError

The issue is tracked as GitHub Issue #22569 1. As of the current main branch snapshot, _file_cache is still unprotected. 3

The Fix Pattern: Lock-Guarded TTLCache#

The correct pattern—already used in src/prefect/concurrency/services.py—pairs the cache with a threading.Lock and wraps every read and write in a with _cache_lock: block:

from threading import Lock
import cachetools

_no_limits_cache: cachetools.TTLCache[frozenset[str], bool] = cachetools.TTLCache(
    maxsize=1000, ttl=5.0
)
_cache_lock = Lock()

# read
with _cache_lock:
    if _no_limits_cache.get(cache_key, False):
        return _create_empty_limits_response()

# write
with _cache_lock:
    _no_limits_cache[cache_key] = True

See lines 26–29 and lines 148–151, 167–168 for the full implementation. This pattern was introduced in PR #19625 4 and explicitly noted to "work for both sync and async code paths."

Applying the same pattern to _file_cache in settings/sources.py would require:

1. Adding _file_cache_lock = Lock() alongside _file_cache.
2. Wrapping the cache read (_file_cache.get(...)) and write (_file_cache[cache_key] = data) in _read_toml_file() with with _file_cache_lock:.

Where TTLCache Appears in Prefect#

The _file_cache was introduced in PR #17121 5 as a performance optimization to avoid redundant TOML reads. It caches parsed TOML data keyed by f"toml_file:{path}:{st_mtime}", so the cache is invalidated automatically when the file changes. 6

Key References#

• Issue #22569 — bug report with full stack trace 1
• src/prefect/settings/sources.py — unprotected _file_cache and _read_toml_file() 3
• src/prefect/concurrency/services.py — reference implementation of the lock-guarded pattern 7
• PR #19625 — introduced the locked TTLCache in concurrency services 4
• PR #17121 — introduced the TOML file cache 5