Hotplug Volume Management#

Overview#

KubeVirt's hotplug volume system lets running VMs attach and detach PVCs without VM restart. The virt-handler component on each node is responsible for mounting hotplug volumes into the virt-launcher pod and keeping mount state in sync with the VMI spec. Two volume modes are supported: filesystem (bind-mount via disk.img) and block (device node creation + cgroup allowlist update).

Key source files:

• pkg/virt-handler/hotplug-disk/mount.go — VolumeMounter implementation
• pkg/virt-handler/hotplug-disk/findmnt.go — findmnt wrapper and JSON parsing
• pkg/virt-handler/vm.go — controller orchestration and retry scheduling

Mount Flow#

The VolumeMounter interface exposes Mount, MountFromPod, Unmount, UnmountAll, and IsMounted 1.

For each volume in vmi.Status.VolumeStatus with a non-nil HotplugVolume, mountFromPod dispatches to either:

• mountBlockHotplugVolume — creates a block device node via mknod, then updates cgroup device rules to grant the virt-launcher container rwm access 2.
• mountFileSystemHotplugVolume — bind-mounts disk.img from the attachment pod into the virt-launcher pod 3.

Filesystem volume mode paths are stored in a per-VMI vmiMountTargetRecord, persisted via the checkpoint manager (on disk at mountStateDir) so they survive virt-handler restarts 4 5.

Utility volumes (e.g. backup target PVCs) follow the same mount path but are always treated as directory mounts, and block-mode utility volumes are explicitly skipped with a warning 6.

Volume Discovery: findmnt-Based Polling#

Before bind-mounting, virt-handler must locate the actual backing file inside the attachment pod's namespace. This is done by:

1. Detecting the attachment pod via a socket-based isolation detector to get its PID.

2. Calling LookupFindmntInfoByVolume(volumeName, pid), which shells out to:

   ⧉

   /usr/bin/findmnt -T /<volumeName> -N <pid> -J
   

3. If the path is a block device or not found, falling back to LookupFindmntInfoByDevice(deviceName):

   ⧉

   /usr/bin/findmnt -S <deviceName> -N 1 -J
   

4. Parsing the JSON output into []FindmntInfo structs 7 and resolving the source path relative to the node's mount root.

If findmnt returns no results (volume not yet mounted in the attachment pod), getSourcePodFilePath returns an error wrapping ErrWaitingForHotplugMount 8.

Retry Scheduling and Timing Vulnerabilities#

There is no blocking retry loop. Instead, the controller re-queues the VMI with a fixed 1-second delay at three points in vm.go:

hotplugVolumesReady returns false when any hotplug volume status is not yet VolumeReady or HotplugVolumeMounted 9.

Timing vulnerability: The 1-second retry cadence assumes that storage provisioning and kubelet mount propagation complete well within that window. On slower or higher-latency storage backends (e.g., networked block storage with slow attach, distributed filesystems, or congested NFS), the attachment pod may take several seconds to have the volume visible in its namespace. Each failed findmnt call triggers another 1-second re-queue, but:

• There is no maximum retry limit or escalating backoff — virt-handler will re-queue indefinitely at 1 s/cycle.
• Each failed attempt incurs the overhead of spawning an exec.Command subprocess (/usr/bin/findmnt) synchronously inside the reconcile loop.
• If the controller queue is under pressure, the actual inter-attempt gap may exceed 1 second, further delaying mount acknowledgment.

The sentinel error ErrWaitingForHotplugMount 10 is distinguished from hard errors: a os.ErrNotExist emits a HotplugFailed Kubernetes event instead of re-queuing 11.

Unmount and State Cleanup#

Unmount iterates the in-memory and on-disk mount records, computes the set of still-desired hotplug volumes from vmi.Spec.Volumes, and tears down entries no longer present 12. Block devices have their cgroup rules revoked via removeBlockMajorMinor. UnmountAll is called on VMI termination and clears all tracked mounts without caring about desired state 13.

Key Entry Points for Debugging#