Kubernetes JSON Patch Concurrency in KubeVirt#

KubeVirt controllers use JSON Patch (types.JSONPatchType) rather than full Update() calls when modifying annotations, labels, finalizers, and other metadata fields on VMI/VM/Migration objects. This pattern avoids optimistic concurrency conflicts that occur in high-concurrency scenarios such as mass live migration.

Kubernetes Update() sends the full object with a resourceVersion; if any other actor modifies the object concurrently, the API server returns a 409 Conflict and the caller must retry. A targeted Patch narrows the conflict surface to only the field(s) being modified. With per-key map operations the surface shrinks further — an add for a new map key carries no test precondition and is inherently conflict-free against unrelated concurrent changes 1.

Patch Builder API#

All patch construction goes through pkg/apimachinery/patch/patch.go. The central type is PatchSet, built with a functional-options pattern:

patch.New(
    patch.WithTest("/metadata/finalizers", oldFinalizers),
    patch.WithReplace("/metadata/finalizers", newFinalizers),
).GeneratePayload()

GeneratePayload() serializes the patch set to RFC 6902 JSON bytes ready for the Kubernetes client. GenerateTestReplacePatch() is a convenience function for the common atomic test-then-replace case.

EscapeJSONPointer() must be called on any map key that contains / or ~ (e.g., annotation keys with domain prefixes) before using it in a path string.

Patterns by Field Type#

Map fields: labels and annotations#

Per-key approach (preferred): Operate on individual map entries rather than the whole map to eliminate false conflicts from concurrent unrelated writes. PR #18139 introduced this to fix a race during mass migration.

• Adding a new key → single add op on /metadata/labels/<key> — no test, so it cannot conflict with unrelated concurrent changes.
• Modifying an existing key → test + replace scoped to that specific key path.
• Removing a key → remove op on the specific key path.

SyncPodAnnotations in pkg/controller/controller.go demonstrates this: it iterates over changed keys and emits only add operations per key, never touching the whole annotation map.

Whole-map approach (legacy/simple cases): Some controllers still emit test("/metadata/labels", oldMap) + replace("/metadata/labels", newMap) (e.g., patchVMI in the migration controller). This is safe under low concurrency but prone to spurious failures when any unrelated key changes simultaneously.

Finalizers (slice field)#

Finalizer updates use a whole-slice test-then-replace because JSON Patch has no mechanism to address array elements by a stable key 1. The VM controller's getPatchFinalizerOps implements the canonical pattern:

patch.New(
    patch.WithTest("/metadata/finalizers", oldFinalizers),
    patch.WithReplace("/metadata/finalizers", newFinalizers),
).GeneratePayload()

This is consumed by removeVMIFinalizer, removeVMFinalizer, and addVMFinalizer.

Status sub-fields#

For fields that may not yet exist (e.g., migrationState), controllers emit an add when the existing value is nil and a test+replace otherwise 2. This prevents the patch from failing on a missing path while still using the narrower conflict surface.

Concurrency Problem That Drove This Pattern#

During mass live migration (2000+ VMs), the VMI controller and migration controller both issue patches on the same VMI objects, while virt-handler concurrently calls Update() on those same objects 1. The collision manifests as:

1. Controller builds a whole-map test("/metadata/labels", oldMap) + replace patch.
2. virt-handler Update() lands first and changes a single unrelated annotation.
3. The controller's test op fails because the observed map no longer matches — even though the labels the controller cared about are unchanged.
4. The controller requeues, retries, and in the worst case leaves VMIs with stale IP addresses in status.interfaces due to a never-applied activePods patch.

Switching to per-key add (no test) for new map entries eliminates the false conflict entirely 1. The fix was targeted: only Go map fields benefit from per-key patching; slice-typed fields (conditions, interfaces, finalizers, volumeStatus) still require whole-slice test+replace because JSON Patch has no stable array element addressing.

Server-side apply (SSA) was identified as the preferred long-term solution but deferred due to refactor scope 1.

Key Source References#