KubeVirt DaemonSet Upgrade Strategy#

Overview#

KubeVirt's virt-operator applies a multi-stage canary rollout when upgrading DaemonSets (primarily virt-handler). Rather than relying solely on Kubernetes' built-in rolling update, the operator controls MaxUnavailable progressively to validate a single canary pod before opening up a wider rollout. The logic lives in pkg/virt-operator/resource/apply/apps.go.

Rollout Phases#

The process is driven by processCanaryUpgrade and syncDaemonSet. There are five named phases 1:

Any phase can transition to failed on error (e.g., crash-looping canary pod).

Step-by-step#

1. started — When syncDaemonSet detects a spec change (specChanged=true), it calls processCanaryUpgrade with objectChanged=true. The DaemonSet is patched with MaxUnavailable=1 (daemonSetDefaultMaxUnavailable) 2, triggering Kubernetes to roll out exactly one new pod.

2. canary — On subsequent reconcile loops, if no updated+ready pods exist yet, the operator checks whether any new pod is crash-looping via PodIsCrashLooping. A crash triggers a FailedUpdate event and returns failed 3.

3. increasing — Once at least one updated pod is ready but the full rollout isn't complete, and the current strategy is still the default (MaxUnavailable=1), the operator patches to MaxUnavailable=10% (daemonSetFastMaxUnavailable) to accelerate the remaining nodes 4.

4. waiting — If MaxUnavailable is already 10% (not the default), the operator simply waits for all pods to become ready without re-patching 5.

5. successful — When updatedAndReadyPods == desiredReadyPods, MaxUnavailable is reverted to 1 and SetGeneration is recorded, marking the rollout complete 6.

Pod Validation: Annotations & Readiness#

A pod is considered "updated" only when its annotations match the target version, registry, and deployment ID from KubeVirt.Status — validated by PodIsUpToDate. This cross-check prevents pods from being counted as updated solely based on Kubernetes rollout status. The three annotations checked are:

• v1.InstallStrategyVersionAnnotation
• v1.InstallStrategyRegistryAnnotation
• v1.InstallStrategyIdentifierAnnotation

A pod is considered "ready" by PodIsReady only when it is in Running phase and all container statuses report Ready=true.

Similarly, DaemonSetIsUpToDate checks the same three annotations at the DaemonSet level to determine whether a spec change is needed at all.

Generation Tracking & Idempotency#

syncDaemonSet detects whether to enter processCanaryUpgrade by checking three flags 7:

• specChanged: DaemonSet annotations out of date OR object metadata changed
• generationUnknown: Recorded generation doesn't match API object
• ongoingRollout: MaxUnavailable != 1 (rollout is in progress)

Crucially, only specChanged (not generationUnknown) is passed as objectChanged to processCanaryUpgrade. This ensures an unknown generation alone does not restart the canary from scratch 8.

TLS Handling During Upgrades#

processCanaryUpgrade also manages a TLS migration path for virt-handler 9. If the cached DaemonSet has TLS enabled but the new spec does not, TLS args are preserved. Certificate secrets (VirtHandlerCertSecretName) are detached only after the rollout fully completes to avoid disrupting in-flight migrations 10.

Key Source Files#