CloudNativePG Primary Switchover & Failover#
Overview#
CloudNativePG distinguishes two primary-transition paths: switchover (planned, operator-initiated) and failover (unplanned, triggered when the current primary is unhealthy). Switchover is initiated during rolling updates or node drains; the old primary shuts down gracefully and a new primary is pre-selected (updatePrimaryPod()). Failover uses PendingFailoverMarker as an intermediate state, marking the old primary unhealthy and waiting for WAL receivers to drop before electing the most advanced replica . Both paths converge on the same WAL-receiver guard before promoting a replica: the new primary waits until IsWALReceiverActive() returns false, confirming the old primary has flushed all pending WAL.
Switchover (Planned)#
Switchover entry point: rolloutRequiredInstances() calls updatePrimaryPod().
Target selection: podList.Items[1] (the first non-primary in the LSN-sorted list) is chosen as targetInstance . A safety gate refuses to proceed if !targetInstance.IsWalReceiverActive, returning errLogShippingReplicaElected and requeuing for 5 seconds . This prevents promoting a log-shipping replica that may be far behind. When the guard passes, switchPrimary() writes targetPrimary to the cluster status and sets phase PhaseUpgrade.
Single-instance path: If cluster.Status.Instances == 1, no switchover is triggered; the primary pod is deleted directly via upgradePod() .
Special cases (all in updatePrimaryPod()):
PrimaryUpdateMethod == Restartand in-place update is possible β annotation-based restart, no pod deletionPrimaryUpdateMethod == Restartand in-place update is not possible βPhaseInplaceDeletePrimaryRestart, pod deleted without switchover- WAL-archiver sidecar missing on primary β
archiverSidecarMissingOnPrimary()βrecreatePrimaryInPlace()instead of switchover (avoids deadlock onArchiveAllReadyWALsduring demotion)
PhaseInplaceDeletePrimaryRestart wait guard: handleSwitchover() blocks new switchover attempts while in this phase until ReadyInstances == Spec.Instances.
Node drain: reconcileTargetPrimaryFromPods() checks if the primary's node is unschedulable or being drained via isNodeUnschedulableOrBeingDrained(). If true, setPrimaryOnSchedulableNode() waits for instances-1 replicas on other nodes, skips candidates on unschedulable nodes, requires IsWalReceiverActive on the candidate, then sets the new target.
Failover (Unplanned)#
Failover entry point: reconcileTargetPrimaryForNonReplicaCluster() in replicas.go.
PendingFailoverMarker sequence:
- When
TargetPrimary == CurrentPrimaryand the current primary is unhealthy, setTargetPrimary = "pending"(PendingFailoverMarker) - Immediately call
markOldPrimaryAsUnhealthy()β labels the old primary pod asunhealthy, removing it from-rwand-roservices to prevent replicas reconnecting to a stale primary viaprimary_conninfo. This is best-effort; failover continues even on label error. - Wait for
AreWalReceiversDown(CurrentPrimary)β all WAL receivers (except the primary) must be inactive before proceeding. - When WAL receivers are down AND
TargetPrimary == PendingFailoverMarker, electmostAdvancedInstance(first in the LSN-sorted list) as the new primary .
Failover delay: enforceFailoverDelay() applies spec.failoverDelay before allowing election; during online upgrades, a fixed 30-second delay is enforced.
Quorum check: If IsFailoverQuorumActive(), evaluateQuorumCheck() is called to prevent unsafe failovers.
Instance-Level: WAL Receiver Wait & Demotion#
On the promoted replica: waitForWalReceiverDown() in instance_controller.go uses exponential backoff to poll IsWALReceiverActive() until the WAL receiver dropsβmeaning the old primary has flushed all pending WAL.
On the demoted old primary: verifyPgDataCoherenceForPrimary() in instance_startup.go runs at restart the full demotion sequence: wait for new primary confirmed β archive pending WAL β pg_rewind β Demote() (writes standby.signal).
MaxSwitchoverDelay (spec.switchoverDelay, default 3600s) bounds how long the old primary waits for its walsenders to drain before force-quitting.
Key Source Files#
| File | Purpose |
|---|---|
internal/controller/cluster_upgrade.go | Switchover orchestration: updatePrimaryPod(), switchPrimary(), WAL receiver guard, archiver sidecar check |
internal/controller/replicas.go | Failover logic: PendingFailoverMarker, markOldPrimaryAsUnhealthy(), node drain switchover |
internal/controller/cluster_controller.go | handleSwitchover(), processUnschedulableInstances() |
internal/management/controller/instance_controller.go | waitForWalReceiverDown() |
internal/management/controller/instance_startup.go | verifyPgDataCoherenceForPrimary() β demotion/rewind sequence |
pkg/postgres/status.go | AreWalReceiversDown() |
api/v1/cluster_types.go | PendingFailoverMarker = "pending" |
Known Issues & Open Bugs#
| Issue | Summary | Status |
|---|---|---|
| #11114 | Target selected before old primary shuts down β non-sync replica can receive post-selection WAL β timeline divergence. Mitigated by Primary Lease in v1.30+. | Open |
| #10103 | Node drain: replica PDB NoPods race β zero active replica-labeled pods during role-label transition, unblocks evictions | Open |
| #11110 | Replay lag not checked pre-switchover; high-write clusters can leave primary down ~14 min waiting for replica to catch up | Open |
| #11230 | Single-instance primary roll: repeated recreate loop on pod eviction (e.g. memory pressure); no backoff, status looks identical to normal roll. Root cause: eviction, not a CNPG rollout bug . | Open |