Node Drain Switchover#

When Kubernetes drains a node, CNPG automatically triggers a switchover if the primary pod is running on that node. The operator detects drain taints (e.g., node.kubernetes.io/unschedulable) via isNodeUnschedulableOrBeingDrained and calls setPrimaryOnSchedulableNode to elect a new primary. Two open bugs in CNPG v1.28–1.30 cause this process to fail silently, leaving the cluster under-protected or causing WAL timeline divergence.

Bug 1: Replica PDB NoPods Race (#10103)#

Symptom: During a node drain, the replicas PDB fires a NoPods event and stops blocking evictions for ~60 seconds, leaving the cluster with zero replicas. With minSyncReplicas set, writes stall during this window. 1

Root cause: Three behaviors interact to create a label-transition gap:

1. Proactive switchover: Node cordon immediately triggers setPrimaryOnSchedulableNode, demoting the primary pod (which then restarts). 2
2. Inactive-pod skip in updateRoleLabels: After switchover, the demoted primary restarts (Pending/terminating). updateRoleLabels skips any pod where IsPodActive() is false — meaning pods with a DeletionTimestamp or in Pending phase are not relabeled. 2
3. Vacuously satisfied PDB: The replica PDB selects on cnpg.io/instanceRole: replica 3. When zero active pods carry that label, Kubernetes treats the PDB as satisfied (expectedPods=0) and does not block any eviction. 4

Example failure sequence (3-node, zone-spread, LRS PVCs on AKS) 5:

The problem is amplified with zone-pinned LRS storage (e.g., Azure Premium SSD v2), where evicted pods can be stuck in FailedScheduling for 5–8 minutes, extending the unprotected window. 6

Status: Open as of July 2026. No merged fix. Community workarounds: create a supplementary cluster-level PDB using only cnpg.io/cluster (no role filter); tune maxSurge/maxUnavailable on the node pool. 7

Bug 2: Target Selection Before Primary Shutdown (#11114)#

Symptom: After a switchover, a non-synchronous replica is promoted, while the true synchronous standby (which received more WAL) crash-loops with: This server's history forked from timeline 2 at 0/C000130. 8

Root cause: In updatePrimaryPod (cluster_upgrade.go), switchPrimary() writes targetPrimary to cluster status before sending a fast-shutdown signal to the old primary. The old primary remains accepting writes for hundreds of milliseconds. With quorum-based sync replication (method: any, number: 1), transactions committed in that window can be replicated to any replica — not necessarily the chosen target. 9

Observed in production (CNPG v1.29, GKE): pod-2 was primary; pod-1 selected as target (all LSNs equal at selection time); new WAL replicated to pod-3 only before shutdown; pod-1 promoted; pod-3 crash-looped. 10

Mitigations:

• method: first — makes the synchronous standby deterministic; the LSN-sorted target is nearly always the sync standby, reducing (but not eliminating) the race window. 11
• primaryUpdateMethod: restart — bypasses the switchover path entirely. 12
• Primary Lease (v1.30+, PR #10627) — introduces a Kubernetes Lease-based mutex at promotion time, ensuring the old primary completes shutdown before a new primary is promoted; closes most of the race window. 13

PDB Architecture and Label Lifecycle#

CNPG maintains two role-scoped PDBs per cluster 14:

Both PDBs are reconciled in reconcilePodDisruptionBudget. During a NodeMaintenanceWindow with reusePVC: true, the replica PDB is deleted to allow the drain to proceed 15.

Label states during switchover: CNPG uses three values for cnpg.io/instanceRole: primary, replica, and unhealthy. During failover (unplanned), the old primary is immediately labeled unhealthy 16, removing it from both the -rw service and the primary PDB. During a node-drain switchover, this immediate label strip does not occur — the pod restarts and updateRoleLabels is deferred until the pod is active again, creating the NoPods gap described in Bug 1.

The unhealthy label was introduced in PR #10409 (v1.28.3 / v1.29.1) to fix a separate service-routing split-brain during failover 17. It does not close the switchover NoPods gap, because the node-drain code path does not call markOldPrimaryAsUnhealthy before the pod restarts.

Key Source Files#

Related issues and PRs:

• #10103 — NoPods PDB race (open)
• #11114 — target-selection race (open)
• #7346 — earlier pod eviction reports during node maintenance
• PR #10409 (v1.28.3 / v1.29.1) — unhealthy label for failover isolation 17
• PR #10627 (v1.30.0) — Primary Lease mechanism 13