CLI Worker Pool Management#

The WorkerPool in pkg/cli/loader/worker.go is the concurrency engine behind the kubectl-kyverno CLI's in-cluster resource loading. It dispatches Kubernetes API list calls across a configurable number of goroutines, used when Concurrency > 1 in ResourceFetcher 1. The pool is created and torn down within a single call to LoadResourcesConcurrent — it is not a long-lived singleton.

Architecture Overview#

ResourceFetcher.getFromCluster()
  └── loader.LoadResourcesConcurrent()
        └── ClusterLoader
              ├── NewWorkerPool(Workers=Concurrency, QueueSize=Concurrency*2)
              │ └── N goroutines started via go wp.worker(context.Background(), i)
              ├── executeTasks() — submits tasks, collects results
              └── Close(cancel) — signals shutdown, drains pool

Key types 2:

• WorkerPool — holds the task channel, result channel, sync.WaitGroup, and worker count.
• LoadTask — one Kubernetes list operation (GVK, GVR, namespace, ListOptions, dynamic client).
• LoadTaskResult — the result of that list operation (resources, error, duration).

Pool Sizing and Configuration#

NewClusterLoader configures the pool 3:

• Workers = resourceOptions.Concurrency (validated 1–32, 4)
• QueueSize = Concurrency * 2

Both channels (taskQueue, resultChan) are buffered to QueueSize. A queue depth of only 2×Concurrency is small relative to the number of GVK tasks that can be submitted — this is the root cause of the silent-drop bug described below.

Task Submission (SubmitTask)#

SubmitTask uses a double non-blocking select:

1. Check ctx.Done() — if cancelled, log and return.
2. Try to send on taskQueue — if the channel is full, fall through to default and silently drop the task.

This is the source of a known critical bug 5: when executeTasks submits all tasks synchronously in a tight loop before workers have drained the queue, any tasks beyond the QueueSize are discarded with only a debug-level log message. Since executeTasks still expects exactly len(tasks) results 6, dropped tasks cause the result-collection loop to block until the 5-minute timeout fires for each missing result.

Worker Lifecycle#

Workers are started in NewWorkerPool — each goroutine runs wp.worker(context.Background(), i). Workers receive context.Background(), not the cancellable context created by LoadResourcesConcurrent. This means:

• Calling cancel() during normal shutdown does not cause workers to exit via ctx.Done().
• Workers only stop when taskQueue is closed (the case task, ok := <-wp.taskQueue branch returns when ok == false, 7).
• Any in-flight processTask call that blocks on a Kubernetes API will not be interrupted by context cancellation 5.

processTask handles server-side pagination transparently, looping on list.GetContinue() until all pages are retrieved 8.

Graceful Shutdown (Close)#

WorkerPool.Close follows this sequence:

1. cancel() — cancels the context (affects executeTasks's ctx.Done() case, but not the workers themselves, per the issue above).
2. close(wp.taskQueue) — signals workers to drain and exit.
3. wp.wg.Wait() — blocks until all workers have returned.
4. close(wp.resultChan) — safe to close only after all writers (workers) are done.

ClusterLoader.Close is idempotent via a closed boolean guard and mutex 9. The cancel function propagates from LoadResourcesConcurrent's defer resourceLoader.Close(cancel) 10.

Known Bugs (as of v1.18.0)#

Tracking issue: #17053 — WorkerPool silently drops tasks causing deadlocks and massive timeouts, #17049 — Memory leak and loop hang in executeTasks.

Entry Points for Debugging / Modification#