Policy Controller Reconciliation#
The policyController in pkg/policy/policy_controller.go runs a periodic reconciliation loop that re-enqueues all eligible policies for background scanning. This is the primary recovery mechanism for policies whose UpdateRequests may have been missed at admission time β for example, when fire-and-forget webhook goroutines fail to create URs .
forceReconciliation: The Periodic Ticker Loop#
forceReconciliation is launched as a goroutine from Run after the informer caches are synced:
go pc.forceReconciliation(ctx)
It creates a time.NewTicker(pc.reconcilePeriod) and on every tick calls requeuePolicies, which enqueues all eligible policies into the work queue . The reconcilePeriod is passed as a constructor parameter ; canBackgroundProcess references BACKGROUND_SCAN_INTERVAL (default: 1 hour) to gate mutateExisting policies that were recently created .
The loop exits cleanly on ctx.Done() .
requeuePolicies: What Gets Re-enqueued#
requeuePolicies iterates over all six policy types and calls enqueuePolicy for each eligible one:
| Policy Type | Eligibility Guard |
|---|---|
ClusterPolicy / Policy | canBackgroundProcess() β must have generate or mutateExisting rules and valid background variables |
GeneratingPolicy / NamespacedGeneratingPolicy | All instances re-queued unconditionally |
MutatingPolicy / NamespacedMutatingPolicy | Must have mutateExistingEnabled and no TargetMatchConstraints.Expression |
enqueuePolicy adds a prefixed key (kpol/, gpol/, ngpol/, mpol/, nmpol/) to the rate-limited work queue.
Work Queue Processing#
Worker goroutines started via wait.UntilWithContext drain the queue by calling syncPolicy. syncPolicy dispatches on the key prefix to the appropriate handler:
kpolβhandleMutate+handleGenerategpol/ngpolβcreateURForGeneratingPolicy/handleGenerateExisting(ifSynchronizationEnabled/GenerateExistingEnabled)mpol/nmpolβcreateURForMutatingPolicy(ifMutateExistingEnabled)
Failed items are retried up to maxRetries = 15 times using exponential backoff (5 ms Γ 2^n, up to ~82 s) before being dropped .
Known Issue: Missing ticker.Stop() on Exit#
Before PR #17027, forceReconciliation created a time.NewTicker but never called ticker.Stop(). When the context was cancelled and the function returned via case <-ctx.Done(), the underlying ticker goroutine and its associated resources were leaked. The fix adds defer ticker.Stop() immediately after ticker creation .
The same pattern β missing defer ticker.Stop() β was also found in the Prometheus metrics refresh goroutine in pkg/metrics/init.go, which additionally lacked a ctx.Done() case entirely and would loop forever after context cancellation .
Correct pattern:
ticker := time.NewTicker(period)
defer ticker.Stop() // always stop the ticker on return
for {
select {
case <-ticker.C:
// do work
case <-ctx.Done():
return
}
}
Related Components#
- Background Scan Report Reconciliation (
pkg/controllers/report/background/controller.go) β separate controller that decides whether to re-scan a resource based on hash/label/annotation changes; seeneedsReconcile. - UpdateRequest deduplication gap β each
forceReconciliationtick creates new URs for generate policies without checking for existing pending URs for the same policy/trigger pair . - Goroutine lifecycle anti-patterns β the aggregate report controller has a similar "goroutine spawned in
NewControllerwith no cancellation" bug; see Concurrency Safety.