CloudNativePG Cluster Recovery#

CloudNativePG treats recovery as a bootstrap operation — recovery never happens in-place on a running cluster; instead, it creates or re-initializes an instance. 1 The three bootstrap methods are initdb, recovery, and pg_basebackup. Only one may be specified per cluster manifest. 2

Bootstrap Methods#

initdb — New cluster from scratch#

Creates a fresh PostgreSQL cluster via initdb. Entry point: InitInfo.Bootstrap() in pkg/management/postgres/initdb.go. Accepts locale, encoding, data checksums, and WAL segment size options passed directly to initdb. Supports post-init SQL via postInitSQL, postInitTemplateSQL, and postInitApplicationSQL. 3

recovery — Restore from backup#

Bootstraps from a physical base backup plus WAL replay. As of v1.26, native Barman Cloud is deprecated; the recommended path uses the Barman Cloud Plugin (barman-cloud.cloudnative-pg.io). Also supports VolumeSnapshot and Backup object references. 4

Replicas added after a VolumeSnapshot recovery fall back to pg_basebackup for synchronization — this can be very slow for large databases. 5

pg_basebackup — Clone from a live cluster#

Takes a physical snapshot of a running binary-compatible PostgreSQL instance via streaming replication. Implemented in ClonePgData() (pkg/management/postgres/join.go). wal_sender_timeout is explicitly disabled for join-related clones to tolerate slow I/O. 6 Binary compatibility (same major version, same OS/CPU) is required. 7

Point-in-Time Recovery (PITR)#

PITR is configured in spec.bootstrap.recovery.recoveryTarget. A valid WAL archive is mandatory. 8 Recovery targets:

9

By default, recovery proceeds to the latest available WAL on the latest timeline. Use barmanObjectStore.wal.maxParallel to parallelize WAL fetching from the object store. 10

PGDATA Safety: Pre-existing Directory Handling#

Before any bootstrap (initdb, pg_basebackup, recovery, join), EnsureTargetDirectoriesDoNotExist() is called to prevent overwriting existing data. Logic:

• If PGDATA exists and pg_controldata succeeds → rename directory with a timestamp suffix (preserves data, especially for statically-provisioned PVCs)
• If PGDATA exists but pg_controldata fails → delete directory (treat as a failed prior attempt)

11

Open bug #11005: The pg_basebackup bootstrap path did not call this check, causing crash loops when a Pod restarted mid-clone (pg_basebackup refuses to write into a non-empty directory). Fix: add EnsureTargetDirectoriesDoNotExist() to bootstrapUsingPgbasebackup(). 12

Replica Failure Patterns#

1. Timeline divergence crash loop#

After failover, if a replica's WAL history diverged from the promoted primary, it crash-loops with:

This server's history forked from timeline N at LSN X.
new timeline N+1 forked off current database system timeline N before current recovery point Y

Fix: Delete the replica's PVC and pod to trigger re-clone from the current primary. 13

2. WAL-restore infinite loop (issue #10419)#

After timeline divergence, a replica can get stuck in a loop: restore_command fetches WAL from archive → reaches end-of-WAL → streaming fails (timeline mismatch) → falls back to archive → repeats. With high maxParallel, this burns CPU indefinitely while the cluster reports healthy. 13

Immediate fix: Delete the replica PVC/pod to force re-clone. Configuration mitigation: set maxParallel: 1 and restore wal_receiver_timeout and wal_sender_timeout to the PostgreSQL default of 60s. 13

3. Startup probe timeout during WAL replay (issue #11024)#

On a replica restart where the primary's checkpointer is behind, the replica may need to replay hundreds of GB of WAL before the first restartpoint. The default startup probe timeout kills the pod before recovery completes, restarting it and losing all progress. 14

Workaround: Increase spec.startDelay (e.g., 36000 for 10 hours). Long-term fix requires WAL replay progress detection to bypass the startup probe during active recovery. 14

4. Replication-sensitive parameter removal (issue #10395)#

Removing a GUC like max_connections from the Cluster spec causes a replica crash-loop when the replica's pg_controldata records a higher value than the resulting PostgreSQL default:

FATAL: recovery aborted because of insufficient parameter settings
DETAIL: max_connections = 100 is a lower setting than on the primary server

The fix uses the pg_controldata value as a floor instead of skipping missing parameters. 15

PVC Deletion and Re-cloning#

Manual re-clone of a stuck replica:

kubectl delete pvc <cluster>-N -n <namespace>
kubectl delete pod <cluster>-N -n <namespace>

This is the recommended operator procedure for timeline divergence and WAL-restore loops. 13

Automatic re-clone (unrecoverable annotation):

The controller detects pods annotated with cnpg.io/unrecoverable: "true" and calls reconcileUnrecoverableInstances(), which deletes the pod's PVCs and pod via ensureInstanceIsDeleted(). Protected instances (current/target primary) are never auto-deleted. 16

After PVC deletion, the operator creates a new PVC and bootstrap job for the freed serial slot — the replica rejoins via pg_basebackup from the primary. Note: the annotation key uses utils.UnrecoverableInstanceAnnotationName 17; check the current codebase for the exact annotation string.

Key Source References#