VM Incremental Backup (CBT-Based)#

Overview#

KubeVirt's CBT-based incremental backup lets you capture only changed disk blocks since the last checkpoint, using libvirt's native backup API backed by QCOW2 dirty bitmaps. It is gated by the IncrementalBackup feature gate (Alpha, v1.6.0) 1 and requires Changed Block Tracking to be active on the VM.

Core CRDs:

• VirtualMachineBackup — initiates and tracks a single backup run (full or incremental)
• VirtualMachineBackupTracker — persists LatestCheckpoint across backup runs, enabling incremental chains

Backup modes 2:

Key source files:

Backup Lifecycle#

The VMBackupController.sync() drives the state machine 3:

1. Prerequisites — checked in checkPrerequisites():

   • VM exists and VMI is running
   • CBT state == Enabled on the VMI
   • No live migration in progress
   • CheckpointRedefinitionRequired is not pending on the tracker
   • Backup target PVC exists and is filesystem mode (not block)

2. Full vs. incremental — isIncrementalBackup() returns true when !ForceFullBackup and the tracker has a non-empty LatestCheckpoint.Name. When incremental, backupOptions.Incremental is set to the checkpoint name 4.

3. Start — startBackup() attaches the target PVC as a utility volume, then issues a Backup gRPC command to virt-launcher 5.

4. virt-launcher execution — backup():

   • Calls generateDomainBackup() to produce backup and checkpoint XML
   • Checkpoint name format: {backupName}-{timestamp} (e.g. vmbackup-fedora-2026-07-04_15-41-59)
   • Each disk with a data store gets Checkpoint="bitmap" in the checkpoint XML 6
   • Optionally freezes the guest filesystem (quiesce) before calling dom.BackupBegin() 7

5. Completion — HandleBackupJobCompletedEvent() handles the libvirt DomainEventJobCompleted event, writing success/failure into the metadata cache. The controller then calls updateBackupTracker() to advance LatestCheckpoint with the new checkpoint name and disk info 8.

Checkpoint Lifecycle and VM Restart Recovery#

libvirt checkpoint metadata lives only in the running domain — a VM restart wipes it. KubeVirt works around this with checkpoint redefinition (added in PR #16448):

Normal flow:

• On backup completion, updateBackupTracker() stores the checkpoint name and per-disk volume info in the VirtualMachineBackupTracker status 8.
• On the next backup run, the controller reads LatestCheckpoint.Name and passes it to libvirt as the incremental base.

After VM restart:

1. The CBT state transition sets CheckpointRedefinitionRequired = true on the tracker.
2. The tracker reconciliation queue detects the flag and calls the RedefineCheckpoint REST/gRPC endpoint 9.
3. In virt-launcher, RedefineCheckpoint():
   • Queries QEMU via QMP query-named-block-nodes for current dirty bitmaps on each disk file 10
   • Builds checkpoint XML only for disks that still have the bitmap
   • Issues DOMAIN_CHECKPOINT_CREATE_REDEFINE | REDEFINE_VALIDATE to libvirt
4. On success: CheckpointRedefinitionRequired is cleared; pending backups proceed.
5. On corrupt/invalid bitmap (e.g. ERR_CHECKPOINT_INCONSISTENT): checkpoint metadata is cleared, a warning event is emitted, and the next backup falls back to full 11.

Backups using a tracker source are blocked (initializing state) until redefinition completes 12.

Utility Volumes: Backup Target PVC Isolation#

The backup target PVC (where libvirt writes the backup output) is attached to the virt-launcher pod via the UtilityVolumes mechanism (PR #15922), gated by UtilityVolumes (Alpha, v1.7.0) 13.

Why utility volumes? They let privileged KubeVirt service accounts hot-attach a PVC directly to the virt-launcher pod without exposing it as a VM disk. The attachment is out-of-band from the VM's own storage, so backup output goes to a separate PVC and never appears inside the guest.

How backup uses them:

• attachBackupTargetPVC() patches vmi.spec.utilityVolumes to add a UtilityVolume of type v1.Backup pointing to the user-provided PVC 14.
• detachBackupTargetPVC() removes the entry after the backup completes or is aborted, verified via cleanupVMIState() 15.
• The controller checks attachment readiness via backupTargetPVCAttached() (hotplug-mounted phase) before sending the start command.

Constraints:

• Target PVC must be a filesystem PVC; block mode is rejected 16.
• Utility volumes are not migrated with the VMI; live migration waits up to 150 s for them to detach 17.
• Only manageable by KubeVirt service accounts (privileged API).

Persistent-State PVC: Sizing for CBT#

CBT stores one QCOW2 overlay file per VM disk at subPath=cbt/ on the VM's persistent-state PVC (the same PVC used for TPM state and persistent EFI). The base PVC size is hardcoded at PVCSize = "10Mi" 18.

The problem (issue #18352): 10 Mi is not enough for multi-disk VMs or accumulated checkpoint bitmaps. A real-world 3-disk VM observed these overlay sizes:

• rootdisk.qcow2: 4.6 Mi (30 Gi virtual, 4 bitmaps)
• disk-black-worm-63.qcow2: 3.3 Mi (20 Gi virtual, 4 bitmaps)
• disk-coral-hummingbird-84.qcow2: partial/failed (creation error)

With the PVC at 93% utilization, adding a third disk's overlay fails with No space left on device, which puts the VM into a CrashLoopBackOff / CBT Initializing loop 19.

The fix (PR #18470, open): createPVC() checks cbt.HasCBTStateEnabled() and adds CBTBackendStateOverhead = "1Gi" to the requested storage when CBT is active. The 1 Gi default is described as a conservative value for common configurations; finer-grained sizing and bitmap retention policies are planned for future PRs 20.

Current workarounds (before PR #18470 merges):

• Manually expand the persistent-state PVC to 256 Mi–1 Gi.
• Configure a CDI storage profile with cdi.kubevirt.io/minimumSupportedPvcSize large enough — the PVC creation already labels the PVC with LabelApplyStorageProfile = "true" so the CDI webhook can override size 21.

IsBackendStorageNeeded() returns true for VMs that have CBT state initializing or enabled, in addition to those with persistent TPM or EFI, so the persistent-state PVC is created whenever CBT is active.

References#

• VEP #25: Incremental Backup
• VEP #90: Utility Volumes
• Issue #18352: persistent-state PVC too small for CBT multi-disk VMs
• PR #16448: Checkpoint redefinition after VM restart
• PR #18470: Increase PVC size for CBT-enabled VMs