CloudNativePG Primary Switchover & Failover#

Overview#

CloudNativePG distinguishes two primary-transition paths: switchover (planned, operator-initiated) and failover (unplanned, triggered when the current primary is unhealthy). Switchover is initiated during rolling updates or node drains; the old primary shuts down gracefully and a new primary is pre-selected (updatePrimaryPod()). Failover uses PendingFailoverMarker as an intermediate state, marking the old primary unhealthy and waiting for WAL receivers to drop before electing the most advanced replica 1. Both paths converge on the same WAL-receiver guard before promoting a replica: the new primary waits until IsWALReceiverActive() returns false, confirming the old primary has flushed all pending WAL.

Switchover (Planned)#

Switchover entry point: rolloutRequiredInstances() calls updatePrimaryPod().

Target selection: podList.Items[1] (the first non-primary in the LSN-sorted list) is chosen as targetInstance 2. A safety gate refuses to proceed if !targetInstance.IsWalReceiverActive, returning errLogShippingReplicaElected and requeuing for 5 seconds 3. This prevents promoting a log-shipping replica that may be far behind. When the guard passes, switchPrimary() writes targetPrimary to the cluster status and sets phase PhaseUpgrade.

Single-instance path: If cluster.Status.Instances == 1, no switchover is triggered; the primary pod is deleted directly via upgradePod() 4.

Special cases (all in updatePrimaryPod()):

• PrimaryUpdateMethod == Restart and in-place update is possible → annotation-based restart, no pod deletion 5
• PrimaryUpdateMethod == Restart and in-place update is not possible → PhaseInplaceDeletePrimaryRestart, pod deleted without switchover 6
• WAL-archiver sidecar missing on primary → archiverSidecarMissingOnPrimary() → recreatePrimaryInPlace() instead of switchover (avoids deadlock on ArchiveAllReadyWALs during demotion)

PhaseInplaceDeletePrimaryRestart wait guard: handleSwitchover() blocks new switchover attempts while in this phase until ReadyInstances == Spec.Instances.

Node drain: reconcileTargetPrimaryFromPods() checks if the primary's node is unschedulable or being drained via isNodeUnschedulableOrBeingDrained(). If true, setPrimaryOnSchedulableNode() waits for instances-1 replicas on other nodes, skips candidates on unschedulable nodes, requires IsWalReceiverActive on the candidate, then sets the new target.

Failover (Unplanned)#

Failover entry point: reconcileTargetPrimaryForNonReplicaCluster() in replicas.go.

PendingFailoverMarker sequence:

1. When TargetPrimary == CurrentPrimary and the current primary is unhealthy, set TargetPrimary = "pending" (PendingFailoverMarker) 7
2. Immediately call markOldPrimaryAsUnhealthy() — labels the old primary pod as unhealthy, removing it from -rw and -ro services to prevent replicas reconnecting to a stale primary via primary_conninfo. This is best-effort; failover continues even on label error.
3. Wait for AreWalReceiversDown(CurrentPrimary) — all WAL receivers (except the primary) must be inactive before proceeding.
4. When WAL receivers are down AND TargetPrimary == PendingFailoverMarker, elect mostAdvancedInstance (first in the LSN-sorted list) as the new primary 8.

Failover delay: enforceFailoverDelay() applies spec.failoverDelay before allowing election; during online upgrades, a fixed 30-second delay is enforced.

Quorum check: If IsFailoverQuorumActive(), evaluateQuorumCheck() is called to prevent unsafe failovers.

Instance-Level: WAL Receiver Wait & Demotion#

On the promoted replica: waitForWalReceiverDown() in instance_controller.go uses exponential backoff to poll IsWALReceiverActive() until the WAL receiver drops—meaning the old primary has flushed all pending WAL.

On the demoted old primary: verifyPgDataCoherenceForPrimary() in instance_startup.go runs at restart the full demotion sequence: wait for new primary confirmed → archive pending WAL → pg_rewind → Demote() (writes standby.signal).

MaxSwitchoverDelay (spec.switchoverDelay, default 3600s) bounds how long the old primary waits for its walsenders to drain before force-quitting.

Key Source Files#

Known Issues & Open Bugs#