Rolling Restart Resilience#

CloudNativePG's rolling restart machinery is implemented primarily in internal/controller/cluster_upgrade.go and coordinated by the main reconcile loop in internal/controller/cluster_controller.go. The goal is to replace pods with updated specs one at a time — replicas first, primary last — while never deleting a pod that isn't ready or while another rollout is already in flight.

Reconciliation Entry Point#

The upgrade path is reached near the end of each reconciliation cycle via handleRollingUpdate(), which calls rolloutRequiredInstances(). The pod list passed in is sorted by replication lag (primary first), so iterating from the tail upgrades the most-lagged replica first. Fenced instances are skipped 1.

Rollout Eligibility: Pod Readiness Guards#

isInstanceNeedingRollout() is the first gate. It returns a no-op rollout for any pod where:

• status.IsPodReady == false — the Kubernetes readiness probe has not passed 2
• status.MightBeUnavailable == true — the instance is still recovering from fencing or a previous restart 2

A separate guard in evaluatePodReadinessGuards() fires earlier in the reconcile loop:

• If the top-of-list instance has HTTP status but Kubernetes hasn't yet updated IsPodReady, the loop requeues for 10 s 3
• If the primary pod is Ready from Kubernetes' perspective but the operator's /pg/status call fails, failover is deferred and the loop requeues for 10 s to wait for Kubernetes to mark it not-Ready 4

Rollout Triggers#

isPodNeedingRollout() runs named checker functions. The first matching checker wins:

If the annotation cnpg.io/reconcilePodSpec: disabled is set on the cluster, all PodSpec-drift checks are skipped 10. A status.PendingRestart == true flag (Postgres config change needing a restart) is also eligible but marked as in-place-capable 11.

Rollout Coordination: The Rollout Manager#

Before any pod is deleted, rolloutRequiredInstances() calls r.rolloutManager.CoordinateRollout(). The rollout.Manager is a single global instance shared across all cluster reconciliations. It serializes rollouts using a single slot (lastCluster, lastInstance, lastUpdate):

• Same cluster: waits for instanceRolloutDelay between successive pods 12
• Different cluster: waits for clusterRolloutDelay before allowing a cross-cluster rollout 13

Both delays are configured at operator startup via configuration.Current.GetClustersRolloutDelay() / GetInstancesRolloutDelay() 14.

If the delay has not elapsed, CoordinateRollout returns RolloutAllowed: false, causing rolloutRequiredInstances() to emit a RolloutDelayed event and return errRolloutDelayed 15. The caller sets phase PhaseUpgradeDelayed and requeues for 15 s 16.

Primary Pod Handling#

updatePrimaryPod() chooses one of four strategies for the current primary:

Single-Instance Clusters#

When there is only one instance (cluster.Status.Instances == 1 or len(podList.Items) == 1), updatePrimaryPod() skips all switchover logic and directly sets PhaseUpgrade and calls upgradePod() on the primary 21. This is the only case where a primary is deleted without a prior switchover.

Streaming Replication Guard#

Before triggering a switchover, the code checks targetInstance.IsWalReceiverActive. If the target replica is not connected via streaming replication, the switchover is refused with errLogShippingReplicaElected and the loop requeues for 5 s 22 23. This prevents promoting a log-shipping replica that may be far behind.

WAL-Archiver Sidecar Missing Guard#

If a WAL-archiver plugin is enabled but the primary pod lacks the injected sidecar containers, archiverSidecarMissingOnPrimary() detects this and routes execution to recreatePrimaryInPlace() instead of a switchover. A switchover in this state would deadlock: demotion triggers ArchiveAllReadyWALs, which requires the missing sidecar 24.

PhaseInplaceDeletePrimaryRestart Wait Guard#

When the cluster is in PhaseInplaceDeletePrimaryRestart, handleSwitchover() blocks all further switchover attempts until ReadyInstances == Spec.Instances 25. Once all instances are ready, the phase transitions to PhaseHealthy.

Phase Reference#

Key Source Files#