Concurrency Safety

Concurrency Safety#

Kyverno's controller-based architecture spawns goroutines extensively — for background policy evaluation, report aggregation, context watching, and test/fuzz workloads. Several recurring concurrency defect patterns appear across the codebase: unbounded goroutines that ignore context cancellation, data races on shared variables, unsafe WaitGroup usage with early-return paths, and deadlocks from holding mutexes across blocking waits.

Goroutine Leaks#

Aggregate Report Controller#

pkg/controllers/report/aggregate/controller.go spawns an infinite background cleanup goroutine directly inside NewController . The goroutine runs a for {} loop polling every 10 seconds with no context parameter and no cancellation signal. Every invocation of NewController — which happens on leader-election restarts and frequently in tests — leaks one more goroutine permanently.

Correct pattern: Background work that runs on a schedule should be started in the controller's Run(ctx context.Context, workers int) method using wait.UntilWithContext, so the goroutine stops when the context is cancelled. The Run method already manages worker goroutines this way; the cleanup goroutine needs to be moved there.

Dynamic Resource Watcher Context Leak#

pkg/background/gpol/dynamic_watcher.go previously created watcher goroutines with context.Background() instead of propagating the parent context. PR #16087 fixed this by threading the passed ctx through so watchers terminate with the controller.

A related issue: on large clusters, watcher goroutines could exit without removing their entry from dynamicWatchers. Subsequent SyncWatchers calls saw the stale non-nil entry and skipped creating a replacement, permanently losing sync. The fix defers a cleanup step inside the goroutine that identity-checks (dynamicWatchers[gvr] == thisWatcher) before nulling the pointer.

Data Races#

Fuzz Utility — Unsafe Early Return Skipping `wg.Wait()`#

pkg/utils/fuzz/policy_spec.go createRules spawns up to 100 parallel goroutines (one per fuzz-generated rule) and collects results into a shared rules slice guarded by a sync.Mutex. However, if ff.GetBytes() fails mid-loop, the function returns immediately — skipping wg.Wait() . The still-running goroutines hold pointers to rules and m, and continue appending after the caller has received the returned slice, causing a data race.

Fix: Replace return rules with break inside the error branch so the loop exits into wg.Wait() before returning.

Perf-Testing — Shared `err` Variable in Closures#

docs/perf-testing/main.go had multiple pod-creation goroutines closing over the outer err variable from main(), creating a write-write race under the Go race detector. PR #16630 fixed it with a one-character change: _, err = → _, err :=, giving each goroutine its own local error variable.

Other Shared-State Races#

Component	Race	Fix
`WatchManager.GetDownstreams`	Reads `policyRefs`/`dynamicWatchers` maps without locking while other methods write	Added `wm.lock.Lock()`
TUF client initialization	Multiple image-verify goroutines raced on `tuf.Initialize()`	`sync.Once` for default init; `sync.Mutex` for custom TUF config
`configuration.IsExcluded()` / `ReportsBreaker`	Unprotected map read during config updates; unsynchronized global mutation	`RLock()` + `atomic.Value`
Reports controller background scan	Multiple goroutines wrote to the same properties map from informer cache	Defensive copy before mutation
Test handler (`Test_ValidateAuditWarn`)	Tests mutated shared policy objects concurrently	`DeepCopy()` before modification

Deadlocks#

GlobalContextEntry Store (`pkg/globalcontext`)#

Issue #16903 documents three concurrency defects in k8sresource.entry and externalapi.entry:

Self-join deadlock — SetWatchErrorHandler calls stop() → group.Wait() synchronously from the reflector goroutine. Since group.Wait() is waiting for informer.Run() to exit, and informer.Run() is executing the callback, the goroutine deadlocks on itself. Fix: run stop() asynchronously.
Mutex-guarded wait deadlock — Stop() holds e.Lock() while calling group.Wait() for the background polling goroutine. The polling goroutine calls setData() which blocks acquiring e.Lock(). Neither can proceed. Because the global-context controller runs with a single worker, this deadlocks the entire reconcile loop. Fix: release e.Lock() before group.Wait().
Non-atomic projection updates — setData writes e.dataMap[""] = jsonData before evaluating projections; a mid-loop projection failure leaves stale partial state in the map.

PR #16739 partially addressed the deadlock risk by making the watch error handler only cancel the context (deferring full stop() to after informer shutdown) and adding a 30-second timeout to WaitForCacheSync.

Webhook Recorder Deadlock#

Recorder.Record() held a mutex while sending to an unbuffered channel; if the receiver wasn't ready, it blocked forever. PR #15066 unlocked the mutex before the channel send.

Unsafe Fire-and-Forget Goroutines in Webhooks#

pkg/webhooks/resource/handlers.go calls handleBackgroundApplies with a dummy wait.Group that is never waited on, then returns ResponseSuccess immediately . The CEL-based gpol/handler.go runs UR creation entirely in a fire-and-forget goroutine . If UR creation fails or is slow, the trigger resource is admitted but no UpdateRequest is queued; recovery depends on the policy controller's periodic reconciliation tick rather than a guaranteed per-admission delivery.

Key Patterns and Fixes#

Anti-pattern	Correct Pattern
Spawn goroutines in `NewController` with no lifecycle	Start in `Run(ctx)` via `wait.UntilWithContext`
`context.Background()` / `context.TODO()` in long-lived goroutines	Propagate parent `ctx`; use `context.WithTimeout` for detached work
Early return inside a goroutine-spawning loop	`break` to fall through to `wg.Wait()` before returning
Closure capturing outer variable across goroutines	Declare with `:=` inside the goroutine
Mutate cached objects in place	`.DeepCopy()` before mutating
Hold mutex across blocking channel send or `group.Wait()`	Unlock before the blocking call

pkg/controllers/report/aggregate/controller.go — goroutine leak site
pkg/utils/fuzz/policy_spec.go — unsafe early-return in createRules
pkg/background/gpol/dynamic_watcher.go — dead watcher entry, cache mutation via direct pointer
pkg/globalcontext — deadlock-prone entry lifecycle
pkg/webhooks/resource/handlers.go — dummy WaitGroup, fire-and-forget UR creation
docs/perf-testing/main.go — shared err data race