KubeVirt Live Migration#

Live migration in KubeVirt moves a running VMI from a source node to a target node without downtime. The VirtualMachineInstanceMigration (VMIM) CR drives the lifecycle; virt-controller orchestrates phase transitions and pod scheduling; node-local virt-handler daemons on source and target execute the actual QEMU/libvirt handoff.

Migration Lifecycle#

The migration controller (pkg/virt-controller/watch/migration/migration.go) advances a VMIM through a state machine 1:

1. Unset → Pending – eligibility checks
2. Pending → Scheduling – target pod created via createTargetPod()
3. Scheduling → Scheduled – target pod becomes Ready
4. Scheduled → PreparingTarget – handleTargetPodHandoff() patches the VMI with MigrationTargetNodeNameLabel, handing control to the target virt-handler
5. PreparingTarget → TargetReady – target virt-handler signals listener address/ports via VMI status
6. TargetReady → Running – source virt-handler initiates QEMU live migration via migrateVMI()
7. Running → Succeeded/Failed – QEMU completes (or aborts); status propagated through MigrationState

VirtualMachineInstanceMigrationState fields of note 2:

• StartTimestamp, EndTimestamp, TargetNodeDomainReadyTimestamp — timing
• Completed, Failed, AbortRequested — terminal-state flags
• Mode (PreCopy / PostCopy / Paused) — active migration strategy

The per-node virt-handler files are migration-source.go (source side: proxy setup, QEMU invocation, progress tracking) and migration-target.go (target side: listener setup, domain arrival detection, finalization).

Workload-Update Migrations (Upgrade Gating)#

During a KubeVirt upgrade, the WorkloadUpdateController automatically migrates VMIs running on stale virt-launcher images. Before issuing any migrations, execute() enforces two hard gates:

kv.Status.Phase == KubeVirtPhaseDeployed
kv.Status.ObservedDeploymentID == kv.Status.TargetDeploymentID

3

Both conditions must be true before sync() is called. This prevents workload-update migrations from launching while the KubeVirt infrastructure components are still rolling out, avoiding migrations onto not-yet-upgraded virt-launcher images. The controller also caps concurrent migrations at ParallelMigrationsPerCluster 4 and annotates its migrations with WorkloadUpdateMigrationAnnotation for tracking 5.

isOutdated() identifies VMIs to migrate by comparing vmi.Status.LauncherContainerImageVersion against the current launcher image 6.

Source Cleanup / Target Finalization Race Condition#

What goes wrong#

After QEMU reports the source domain as Shutoff/Migrated, domainMigrated() returns true. The source virt-handler's execute() in vm.go reaches the check at 7 before any isMigrating or isMigrationSource guard:

if domainExists && (domainMigrated(domain) || domain.DeletionTimestamp != nil) {
    return c.deleteVM(vmi) // line 397
}

This calls deleteVM(), which tears down migration proxy listeners and removes the domain from the local cache, before the target virt-handler has finished its two-pass finalization:

1. ackMigrationCompletion() — sets MigrationState.EndTimestamp
2. finalizeMigration() (next reconcile loop) — sets MigrationState.Completed = true

If source cleanup races between these two passes, MigrationState.Completed is never written and the VMIM object stays in Running state indefinitely 8.

Observed failure modes#

Issue #18020 8: The TPM migration test hits the race — QEMU migration succeeds at 09:53:01, source cleanup fires at 09:53:01.614, and the VMIM object never transitions to Succeeded. virt-controller's recovery mechanism kicks in ~4 minutes later, well past the 240 s test timeout.

Issue #18320 9: During an upgrade, a workload-update migration succeeds but the source virt-launcher pod on node02 is never terminated. A subsequent test-initiated re-migration tries to schedule a third pod, and the migration controller refuses with:

"Waiting to schedule target pod for migration because there are already
multiple pods running for vmi"

API server contention under k8s 1.36 CI load widens the timing window, making the race more frequent 9.

Proposed fix#

Defer source cleanup when MigrationState indicates an in-flight, unfinalized migration 8:

if domainExists && domainMigrated(domain) {
    if vmi.Status.MigrationState != nil &&
        vmi.Status.MigrationState.StartTimestamp != nil &&
        !vmi.Status.MigrationState.Completed &&
        !vmi.Status.MigrationState.Failed {
        c.queue.AddAfter(controller.VirtualMachineInstanceKey(vmi), 5*time.Second)
        return nil
    }
    return c.deleteVM(vmi)
}

The DeletionTimestamp branch remains unchanged. This reorders the check so cleanup is deferred for 5 s while the target virt-handler finishes its second reconcile pass.

Key Source Files#