CloudNativePG Synchronous Replication#

Overview#

CloudNativePG's synchronous replication feature causes transaction commits to wait until WAL records have been replicated to a specified number of standby instances before returning to the client. It is disabled by default and configured via the .spec.postgresql.synchronous stanza.

The core API types live in api/v1/cluster_types.go:

• SynchronousReplicaConfiguration — the full configuration struct, nested under spec.postgresql.synchronous
• SynchronousReplicaConfigurationMethod — enum for any (quorum-based) or first (priority-based)
• DataDurabilityLevel — enum for required or preferred

Two fields are required when the stanza is present: method and number (the quorum size). Optional fields include maxStandbyNamesFromCluster, standbyNamesPre, standbyNamesPost, and failoverQuorum 1.

The legacy minSyncReplicas / maxSyncReplicas top-level fields are deprecated since v1.24 and are mutually exclusive with the synchronous stanza 2.

The operator translates the synchronous spec into the PostgreSQL synchronous_standby_names GUC. The logic that builds this string lives in pkg/postgres/replication/explicit.go.

Durability Modes: required vs preferred#

The dataDurability field controls how strictly synchronous replication is enforced when replicas are unavailable 3.

required (default)#

Write operations block until number synchronous standbys acknowledge the WAL record. If fewer than number healthy replicas are available, writes stall — RPO=0, but availability is reduced. The implementation uses all known instance names (including non-ready ones) to build synchronous_standby_names, and falls back to a placeholder name (<cluster>-placeholder) when the list would otherwise be empty to avoid a PostgreSQL syntax error 4.

preferred ("self-healing mode")#

The operator dynamically adjusts synchronous_standby_names to include only currently healthy non-primary replicas, capping number to however many are available 5. When no replicas are available the config resolves to an empty string — PostgreSQL falls back to asynchronous commits. This prevents write stalls at the cost of potential data loss.

Constraint: dataDurability: preferred is only permitted when both standbyNamesPre and standbyNamesPost are empty, enforced by a CEL validation rule on the struct 6.

Quorum (any) vs Priority (first)#

• method: any — commit acknowledged by any number of the listed standbys. With any 1 (pod1, pod2, pod3), any single replica satisfies the quorum.
• method: first — commit acknowledged by the first number standbys in priority order. The sync standby is deterministic, which matters for switchover safety (see Known Issues).

The sorted instance name list always places healthy non-primary replicas first, then non-ready instances, then the primary 7, making first 1 (...) reliably prefer the most up-to-date healthy replica.

Replica Lag Monitoring via Probes#

Replica readiness is surfaced via Kubernetes readiness/startup probes. Three strategies are available, selected by spec.probes.readiness.type / spec.probes.startup.type 8:

The streaming strategy is the most relevant for synchronous replication health. It is implemented in pgStreamingChecker.IsHealthy, which queries pg_stat_wal_receiver and computes lag as:

latest_end_lsn - pg_last_wal_replay_lsn()

The probe passes if the instance is a primary, a log-shipping replica (no primary_conninfo), or a streaming replica whose lag ≤ maximumLag 9. Without a maximumLag, it passes as long as the replica has connected at least once.

When the lag exceeds the limit, a ReplicaLaggingError is returned with detectedLag, configuredLag, and latestEndTime fields, which surface in pod events and logs.

The maximumLag field is of type resource.Quantity on ProbeWithStrategy, converted to uint64 bytes at runtime in getProbeRunnerFromCluster.

Connection to synchronous replication: when replicas fail the streaming readiness probe, the operator marks them unhealthy. Under dataDurability: preferred, this removes them from synchronous_standby_names, reducing the effective quorum — a useful self-healing mechanism. Under dataDurability: required, unhealthy replicas remain in the standby list, preserving durability guarantees at the cost of write availability.

Known Issues: Switchover Race Conditions#

Two open issues reveal gaps between the operator's view of replica readiness and true promotion-readiness.

Race condition: target selection before primary shutdown (issue #11114)#

Symptom: A replica is promoted that was not the synchronous standby for the most recent committed transactions, causing timeline divergence and crash-loops on other replicas 10.

Root cause: In updatePrimaryPod, the operator selects targetPrimary and writes it to the cluster status before the old primary receives its shutdown signal. With method: any, number: 1, PostgreSQL can satisfy the quorum by replicating to any replica. If new transactions are committed between target selection and primary shutdown, those WAL records may land on a non-target replica — not the designated new primary 10.

Example observed in production (CNPG v1.29): pod-2 was demoted, pod-1 selected as target, but commits after selection replicated to pod-3 only. Pod-1 was promoted, pod-3 crash-looped with:

This server's history forked from timeline 2 at 0/C000130.
new timeline 3 forked off current database system timeline 2 before current recovery point 0/D0000A0

10

Mitigations:

• Use method: first — the synchronous standby is deterministic, so the LSN-sorted target is nearly always the sync standby 10.
• Use primaryUpdateMethod: restart — avoids the switchover path entirely.
• v1.30 primary lease: ensures the old primary's shutdown completes before the new primary is allowed to promote, closing most of the window 10.

Replay-lag blind spot during rolling restarts (issue #11110)#

Symptom: A kubectl cnpg restart on a high-write cluster leaves the cluster without a writable primary for ~14 minutes 11.

Root cause: Target replica selection ranks by received LSN; replay LSN is only a tiebreaker. The sole pre-demotion gate is whether the target's WAL receiver is active — replay lag is never checked. At a write rate of 250 GB/hour WAL, replicas were 15–20 minutes behind on replay when the operator demoted the primary 11.

After demotion, the old primary's walsenders stay alive (in PM_SHUTDOWN_2) until the replica drains its backlog, bounded by switchoverDelay (default 3600s). Promotion fires only when waitForWalReceiverDown detects the receiver dropped — 14 minutes later in this case 11.

Mitigations:

• Enable synchronous replication: it throttles write throughput to keep replicas in-sync, preventing chronic replay lag 11.
• Proposed fix (not yet merged): add a pre-flight replay-lag check before demoting the primary; block and emit a status condition if no viable promotion target exists 11.

Key Source Files and References#

External references:

• Replication docs (v1.28) 2
• Bug #11114 — switchover race condition with method: any 10
• Bug #11110 — replay-lagged replica in rolling restart 11
• Discussion #10783 — proposal to default to synchronous replication 12