Background Mutation Engine#

mutateExisting is Kyverno's mechanism for mutating resources that already exist in the cluster — as opposed to resources caught at admission time. Unlike standard admission-webhook mutation, which patches the object being admitted, mutateExisting rules specify explicit target resources that may be entirely unrelated to the trigger. All target mutations are deferred to a background controller and applied after the admission response has been returned.

Architecture Overview#

The two main stages are:

1. Webhook phase — Evaluates which mutateExisting rules pass background checks and creates an UpdateRequest CRD for each.
2. Background phase — The background controller processes each UpdateRequest, re-evaluates the mutation against live target resources, and writes the patched objects back via the dynamic client.

Stage 1 – UpdateRequest Creation (Webhook)#

When an admission request arrives, handleBackgroundApplies() is called asynchronously (30-second timeout). It invokes handleMutateExisting(), which:

1. Builds a PolicyContext from the admission request.
2. Calls engine.ApplyBackgroundChecks() to identify which mutateExisting rules pass without executing the actual mutation.
3. For each passing rule, calls applyUpdateRequest() with ruleType = kyvernov2.Mutate.
4. The transform() function converts each engine response into an UpdateRequestSpec carrying the policy name, rule name, trigger resource details, and the original AdmissionRequest.

The UpdateRequest object is written to the Kyverno namespace by generator.Apply(), which spawns a background goroutine with exponential-backoff retry. Requests are labeled using MutateLabelsSet(policy, resource) so they can be bulk-deleted if the policy is removed later 1.

Stage 2 – Background Controller Processing#

The background controller watches UpdateRequest objects via an informer. On Add/Update events it enqueues the request key, skipping those already in Completed or Skip state 2.

syncUpdateRequest() dequeues each key, fetches the UpdateRequest, and — if its state is Pending — calls processUR(). For kyvernov2.Mutate requests, processUR instantiates a mutateExistingController and calls its ProcessUR method.

ProcessUR drives the per-rule loop:

1. Trigger retrieval — Fetches the trigger resource from the API or falls back to the AdmissionRequest body for subresources 3.
2. PolicyContext construction — common.NewBackgroundContext() builds a context with the trigger and its namespace labels.
3. GVK resolution — If an AdmissionRequest is present, the trigger's GroupVersionKind is resolved from the GVR via discovery 4.
4. Engine evaluation — engine.Mutate() runs the rule against the policy context, which internally dispatches to mutate_existing.go for target loading and patch application.
5. Status update — The UpdateRequest is marked Completed or Failed via updateURStatus. Completed requests are then deleted by reconcileURStatus 5; failed ones are reset to Pending for retry (up to maxRetries = 10) 6.

Target Resource Discovery#

Inside the engine, mutate_existing.go calls loadTargets() for each rule's mutation.targets list. For each target spec:

1. Variable substitution — resolveSpec() marshals the TargetSelector to JSON, runs variables.SubstituteAll() against the policy context, and unmarshals back to a typed selector. This enables dynamic target selectors whose fields are JMESPath expressions.
2. Resource fetch — getTargets() calls client.GetResources() with the resolved group/version/kind/subresource and an optional LabelSelector. For namespaced policies, the target namespace is locked to the policy's own namespace 7.
3. Subresource tracking — Each returned object is wrapped in a resourceInfo struct that records the subresource name and parentResourceGVR, needed later for correct API routing 8.

The handler then iterates over the loaded targets, per-target context entries are loaded, per-target preconditions are evaluated, and the patch is computed via mutate.Mutate() or forEachMutator.mutateForEach() 9. The buildRuleResponse() helper attaches the PatchedTarget (including parentResourceGVR and subresource) to the rule response.

Subresource Routing and API Update#

After engine.Mutate() returns, ProcessUR iterates over each RuleResponse and writes the patched target back to Kubernetes via dclient.Interface. The routing logic branches on the patchedSubresource field 10:

The ResourceVersion of the original (pre-patch) object is copied onto the patched object before the update call 11, satisfying Kubernetes optimistic concurrency.

On error, the failure is appended to the error slice, an event is emitted, and the UpdateRequest is eventually retried. On success, an event is emitted via event.NewBackgroundSuccessEvent 12. If MutateExistingReportsEnabled() is true and the trigger resource is reportable, an ephemeral policy report is created 13.

Key Entry Points#