Install Strategy Generation#

virt-operator generates a one-time install strategy ConfigMap (named kubevirt-install-strategy-<hash>) that encodes the complete set of Kubernetes resources KubeVirt will create and manage. Once generated for a given deployment ID, this ConfigMap is effectively immutable — the operator never re-evaluates or regenerates it unless the deployment ID changes (e.g., on upgrade). 1

How Generation Works#

The dump is triggered by running virt-operator --dump-install-strategy, typically via a short-lived Kubernetes Job spawned during install or upgrade. The entry point is DumpInstallStrategyToConfigMap, which:

1. Reads the deployment config from the KUBEVIRT_DEPLOYMENT_CONFIG env var via GetConfigFromEnv.
2. Resolves the monitoring namespace by calling getMonitorNamespace, which looks up the prometheus-k8s ServiceAccount across a list of candidate namespaces.
3. Calls NewInstallStrategyConfigMap → GenerateCurrentInstallStrategy, which assembles a Strategy struct holding all resource lists.
4. Serializes the Strategy via dumpInstallStrategyToBytes, gzip+base64 encodes it, and stores it in the ConfigMap's manifests key.

The ConfigMap is annotated with InstallStrategyVersionAnnotation, InstallStrategyRegistryAnnotation, and InstallStrategyIdentifierAnnotation 2. The reconciler later loads it via LoadInstallStrategyFromCache, matching on deployment ID.

What Resources Are Generated#

GenerateCurrentInstallStrategy unconditionally generates CRDs, RBAC (ClusterRoles, RoleBindings, ServiceAccounts), Deployments, DaemonSets, Services, webhook configurations, API services, certificate secrets, ConfigMaps, SCCs, routes, instancetypes, and preferences. 3

ServiceMonitor and PrometheusRule are conditional — they are only added when the monitoring ServiceAccount is found 4:

isServiceAccountFound := monitorNamespace != ""
if isServiceAccountFound {
    strategy.serviceMonitors = append(...)
    strategy.prometheusRules = append(...)
} else {
    log.Warningf("failed to create ServiceMonitor resources ...")
}

If isServiceAccountFound is false, monitoring resources are silently omitted and the warning is logged — with no retry and no mechanism to re-trigger generation later. 5

The Race Condition#

Because the install-strategy Job runs concurrently with other operator bootstrap or upgrade tasks, the Prometheus monitoring ServiceAccount (prometheus-k8s) may not yet exist when getMonitorNamespace executes. In that case isServiceAccountFound is false, the ConfigMap is written without ServiceMonitor/PrometheusRule, and since the ConfigMap is immutable for that deployment ID, metrics scraping remains broken indefinitely until manual intervention. 6

This affects:

• Upgrades: the monitoring stack may still be reconciling when the new install-strategy Job runs.
• Early bootstrap: KubeVirt is installed before the Prometheus operator has created its ServiceAccount. 6

Setting serviceMonitorNamespace explicitly in the KubeVirt CR does not prevent the race — the SA check still gates resource inclusion. 6

Workaround: Inspect existing ConfigMaps with kubectl get configmap -n <namespace> -l kubevirt.io/install-strategy and check which one lacks ServiceMonitor. Manually delete the faulty ConfigMap and let virt-operator regenerate it after the monitoring SA is available. Manually triggering regeneration consistently restores metrics. 6

Fix in Progress — PR #18576#

PR #18576 addresses this by introducing getMonitorNamespaceWithRetry, which polls getMonitorNamespace every 5 seconds for up to 1 minute before giving up. DumpInstallStrategyToConfigMap is updated to call the retry wrapper instead of the one-shot lookup. If the SA appears within the retry window, monitoring resources are included; if not, behavior falls back to the current silent omission. 1

The retry uses virtwait.PollImmediately and wait.Interrupted to avoid treating context cancellations as hard failures, consistent with other wait patterns in the codebase. 1

Key Files#

Related Issues#

• Issue #16593 — upstream bug report tracking the ServiceMonitor race with at least 3 confirmed occurrences 6
• PR #18576 — proposed fix with retry logic 1