KubeVirt DaemonSet Upgrade Strategy#
Overview#
KubeVirt's virt-operator applies a multi-stage canary rollout when upgrading DaemonSets (primarily virt-handler). Rather than relying solely on Kubernetes' built-in rolling update, the operator controls MaxUnavailable progressively to validate a single canary pod before opening up a wider rollout. The logic lives in pkg/virt-operator/resource/apply/apps.go.
Rollout Phases#
The process is driven by processCanaryUpgrade and syncDaemonSet. There are five named phases :
| Phase | MaxUnavailable | Condition to advance |
|---|---|---|
started | 1 (default) | Spec change detected; DaemonSet patched |
canary | 1 | Waiting for the single new pod to become ready |
increasing | 10% | Canary pod healthy; full rollout patch applied |
waiting | 10% | Full rollout in progress; waiting for all pods |
successful | 1 (reverted) | All pods updated and ready |
Any phase can transition to failed on error (e.g., crash-looping canary pod).
Step-by-step#
-
started— WhensyncDaemonSetdetects a spec change (specChanged=true), it callsprocessCanaryUpgradewithobjectChanged=true. The DaemonSet is patched withMaxUnavailable=1(daemonSetDefaultMaxUnavailable) , triggering Kubernetes to roll out exactly one new pod. -
canary— On subsequent reconcile loops, if no updated+ready pods exist yet, the operator checks whether any new pod is crash-looping viaPodIsCrashLooping. A crash triggers aFailedUpdateevent and returnsfailed. -
increasing— Once at least one updated pod is ready but the full rollout isn't complete, and the current strategy is still the default (MaxUnavailable=1), the operator patches toMaxUnavailable=10%(daemonSetFastMaxUnavailable) to accelerate the remaining nodes . -
waiting— IfMaxUnavailableis already10%(not the default), the operator simply waits for all pods to become ready without re-patching . -
successful— WhenupdatedAndReadyPods == desiredReadyPods,MaxUnavailableis reverted to1andSetGenerationis recorded, marking the rollout complete .
Pod Validation: Annotations & Readiness#
A pod is considered "updated" only when its annotations match the target version, registry, and deployment ID from KubeVirt.Status — validated by PodIsUpToDate. This cross-check prevents pods from being counted as updated solely based on Kubernetes rollout status. The three annotations checked are:
v1.InstallStrategyVersionAnnotationv1.InstallStrategyRegistryAnnotationv1.InstallStrategyIdentifierAnnotation
A pod is considered "ready" by PodIsReady only when it is in Running phase and all container statuses report Ready=true.
Similarly, DaemonSetIsUpToDate checks the same three annotations at the DaemonSet level to determine whether a spec change is needed at all.
Generation Tracking & Idempotency#
syncDaemonSet detects whether to enter processCanaryUpgrade by checking three flags :
specChanged: DaemonSet annotations out of date OR object metadata changedgenerationUnknown: Recorded generation doesn't match API objectongoingRollout:MaxUnavailable != 1(rollout is in progress)
Crucially, only specChanged (not generationUnknown) is passed as objectChanged to processCanaryUpgrade. This ensures an unknown generation alone does not restart the canary from scratch .
TLS Handling During Upgrades#
processCanaryUpgrade also manages a TLS migration path for virt-handler . If the cached DaemonSet has TLS enabled but the new spec does not, TLS args are preserved. Certificate secrets (VirtHandlerCertSecretName) are detached only after the rollout fully completes to avoid disrupting in-flight migrations .
Key Source Files#
| File | Purpose |
|---|---|
pkg/virt-operator/resource/apply/apps.go | syncDaemonSet, processCanaryUpgrade, phase constants, MaxUnavailable helpers |
pkg/virt-operator/util/readycheck.go | PodIsUpToDate, PodIsReady, PodIsCrashLooping, DaemonSetIsUpToDate |