Webhook Certificate Rotation#

The kubeovn-webhook (packages/system/kubeovn-webhook) has a latent TLS certificate rotation gap: the webhook process loads its serving certificate exactly once at startup and never reloads it, even when cert-manager renews the backing Secret. This causes a hard outage roughly one year after install or the last pod restart. 1

The Problem#

main.go loads the cert/key pair with tls.LoadX509KeyPair and assigns it as a static slice in tls.Config.Certificates. There is no GetCertificate callback, no fsnotify file watcher, and no reload loop — once the in-memory certificate expires, every TLS handshake to the webhook fails. 2

The cert-manager Certificate object is configured with 3:

• duration: 8760h (1 year) for the serving certificate
• renewBefore: 24h — cert-manager rotates the Secret only ~24 hours before expiry

This 24-hour window is extremely tight: if the webhook pods are not restarted within that window (which is unlikely under normal operations), they continue to serve the expired in-memory certificate after expiry.

The Deployment has no reloader sidecar, no checksum annotation, and no secret.reloader.stakater.com/reload annotation — nothing triggers a rollout when the Secret changes. 4

Blast Radius#

The MutatingWebhookConfiguration has failurePolicy: Fail and intercepts CREATE on all pods in every namespace except those labeled cozystack.io/system=true or named kube-system. When the expired certificate causes TLS verification to fail, all pod creation in tenant namespaces is rejected — including virt-launcher pods, which blocks VM start operations. 1

Error seen in kube-apiserver logs:

Internal error occurred: failed calling webhook "kube-ovn-webhook.cozystack.io":
failed to call webhook: <TLS error: certificate has expired or is not yet valid>

1

Certificate Architecture#

cert-manager provisions certificates through two objects defined in certmanager.yaml:

1. Root CA Certificate — duration: 43800h (5 years), rotationPolicy: Never 5
2. Serving Certificate — duration: 8760h, renewBefore: 24h, signed by the CA issuer 3

The MutatingWebhookConfiguration carries cert-manager.io/inject-ca-from 6 so cert-manager keeps the webhook's caBundle up to date automatically — but this only controls what the API server uses to trust the webhook, not what the webhook itself serves.

Proposed Fixes#

Three mitigations are documented in Issue #3556 1:

Relevant Files#