Report Controller Goroutine Lifecycle#

Kyverno's report subsystem has a recurring class of goroutine lifecycle bugs: goroutines spawned during construction (rather than in Run), hardcoded context.Background() / context.TODO() in long-lived watchers, and missing cleanup paths on initialization failure. The three main sites are the aggregate report controller, the resource-report watcher, and the circuit-breaker initialization path.

1. Aggregate Report Controller — Leak in NewController#

pkg/controllers/report/aggregate/controller.go spawns an unbounded background cleanup goroutine directly inside NewController. The goroutine runs a bare for {} loop with a 10-second time.Sleep, carries no context, and never observes cancellation. Every call to NewController—including each leader-election restart and every test setup—leaks one goroutine permanently. 1

Correct pattern: The Run(ctx, workers) method already manages worker goroutines properly via group.StartWithContext(ctx, ...). The cleanup loop should be moved there and driven by wait.UntilWithContext, so it terminates when ctx is cancelled. 1

2. Resource-Report Watcher — Context Leak in startWatcher#

pkg/controllers/report/resource/controller.go's startWatcher previously passed context.Background() to the Kubernetes Watch call and context.TODO() to watchTools.NewRetryWatcherWithContext, ignoring the ctx parameter the method already received. The watchers therefore could not be cancelled when the controller shut down, leaving hanging connections and leaked goroutines. 2

PR #16087 fixed this by replacing both hardcoded contexts with ctx at lines 341 and 347, ensuring watch connections terminate with the controller lifecycle. 2

3. Circuit Breaker Initialization — Missing Retry / Goroutine Leak on Failure#

The circuit breaker package exposes a global ReportsBreaker via GetReportsBreaker() / SetReportsBreaker() backed by atomic.Value 3. The breaker's open predicate is driven by a counter that performs a list+watch against the ephemeral reports API at startup via StartAdmissionReportsCounter / StartBackgroundReportsCounter.

Before PR #13641, a failure in the counter startup caused an os.Exit(1), taking down the entire controller and blocking cluster admission. The PR replaced this with:

1. Temporary permissive breaker — on failure, SetReportsBreaker is called immediately with a breaker whose open func always returns false (allows all), so the controller starts normally.
2. Background retry goroutine — a goroutine retries the counter startup every 2 seconds; on success it calls SetReportsBreaker again with the real counter-backed breaker. 4

Each controller calls the appropriate counter startup function for its report type: cmd/kyverno/main.go and cmd/reports-controller/main.go use StartAdmissionReportsCounter (watching audit.kyverno.io/source==admission reports), while cmd/background-controller/main.go uses StartBackgroundReportsCounter (watching source==background-scan reports). This ensures each controller's circuit breaker monitors the ephemeral reports it actually creates. 4

For example, cmd/background-controller/main.go:

ephrs, err := breaker.StartBackgroundReportsCounter(signalCtx, setup.MetadataClient)
if err != nil {
    go func() {
        for {
            ephrs, err := breaker.StartBackgroundReportsCounter(signalCtx, setup.MetadataClient)
            if err != nil {
                setup.Logger.Error(err, "failed to start background scan reports watcher, retrying...")
                time.Sleep(2 * time.Second)
                continue
            }
            // ... set real breaker with counter
        }
    }()
}

The resource counter goroutine itself 5 is driven by the RetryWatcher's result channel; it exits naturally when the channel closes, so it does not need explicit context wiring.

Key Patterns#

Primary Source Files#