PostgreSQL Major Version Upgrades in CloudNativePG#

Overview#

CloudNativePG orchestrates offline, in-place major PostgreSQL version upgrades using Kubernetes Job resources backed by pg_upgrade. The upgrade is triggered declaratively by setting a higher-major-version image on a Cluster object — no second cluster, no logical replication — but the cluster is unavailable for the duration 1.

Three upgrade strategies are supported; only the pg_upgrade path is covered here:

2

Reconciler Entry Point#

All major-upgrade logic lives in pkg/reconciler/majorupgrade/. The package-level doc describes the four-step lifecycle 3:

1. Delete all cluster Pods.
2. Create and run the upgrade Job (BackoffLimit=0 — no retries).
3. Wait for job completion.
4. On success, start new Pods for the upgraded version; on image rollback, delete the job and let the cluster restart.

Reconcile() is the main entrypoint. Its decision tree, in order:

1. Job already exists and completed → call majorVersionUpgradeHandleCompletion().
2. Job exists but not completed → check for user rollback via handleRollbackIfNeeded(); otherwise requeue after 30 s.
3. No job and no upgrade needed (requested major ≤ Status.PGDataImageInfo.MajorVersion) → call clearStaleUpgradeTarget() and return.
4. No job and upgrade needed → resolve extensions, patch status to PhaseMajorUpgrade, delete pods/jobs, create the upgrade Job.

Pod and Job Deletion#

Before creating the upgrade Job, deleteAllPodsInMajorUpgradePreparation() deletes all running instance Pods and any existing Jobs. Resources still terminating cause a 10-second requeue; only when the slate is fully clear does execution proceed to Job creation 4.

Upgrade Job Structure#

createMajorUpgradeJobDefinition() builds the Job with two containers:

• Init container (prepare) — runs manager instance upgrade prepare /controller/old using the old PostgreSQL image (Status.PGDataImageInfo.Image). It invokes pg_config to locate binaries/libraries, copies them to /controller/old, and writes /controller/old/bindir.txt 5.
• Main container (major-upgrade) — runs manager instance upgrade execute /controller/old/bindir.txt using the new PostgreSQL image 6.

BackoffLimit is hard-set to 0: a failed pg_upgrade cannot succeed on retry 7.

The Job is identified by label utils.JobRoleLabelName == "major-upgrade" 8.

For clusters with Image Volume extensions, both the source-version and target-version extension volumes are mounted simultaneously under separate trees (/extensions/<name> and /new-extensions/<name>) so the two sets never collide 9.

Execute Subcommand (instance upgrade execute)#

internal/cmd/manager/instance/upgrade/execute/cmd.go runs inside the main upgrade container and implements the upgrade in this order:

1. Extension environment setup — setupExtensionEnvironment() extends LD_LIBRARY_PATH and PATH for both source (Status.PGDataImageInfo) and target (Status.TargetPGDataImageInfo) extension sets, and applies per-extension env variables. Both status fields must be present or setup fails fast.
2. Fail-fast on previous failures — checks for *.failed_* directories left by a prior incomplete run 10.
3. initdb — creates PGDATA-new; propagates WAL segment size and data checksum settings from the old cluster's pg_controldata output 11.
4. Configuration — prepareConfigurationFiles() writes postgresql.conf includes and applies version-specific parameter overrides (see table below).
5. pg_upgrade --link — runs pg_upgrade with hard-linked files for speed 12.
6. Atomic directory swap — moveDataInPlace() renames PGDATA → PGDATA.old, then PGDATA-new → PGDATA; on failure, both directories are saved with a .failed_<timestamp> suffix to prevent data loss.
7. Extension update script — logs the location of PGDATA/update_extensions.sql if pg_upgrade emitted it 13.

Version-Specific Parameter Management#

Before pg_upgrade runs, the operator generates a valid postgresql.conf for the new major version. Key version-aware behavior 14:

The configuration generator is invoked with OperationType_TYPE_UPGRADE, which switches extension path GUCs (dynamic_library_path, extension_control_path) to point at the target-version /new-extensions mounts 16.

Post-Upgrade Status Handling#

majorVersionUpgradeHandleCompletion() fires after the Job reaches Completed:

1. Deletes non-primary PVCs — replica data will be re-cloned from scratch 17.
2. Carries forward TargetPGDataImageInfo.Extensions rather than re-resolving from the catalog, to avoid a PhaseImageCatalogError if the catalog was edited while the Job ran 18.
3. Patches cluster status: sets PGDataImageInfo to the new image/version, clears TargetPGDataImageInfo, and resets TimelineID to 1 — matching pg_upgrade's behavior and preventing replicas from fetching incompatible pre-upgrade timeline history files 19.
4. Deletes the upgrade Job and requeues 20.

Rollback Handling#

If the user reverts the cluster image to the old major version while the Job is still running, handleRollbackIfNeeded() detects this (requested major ≤ PGDataImageInfo.MajorVersion), deletes the Job with foreground propagation, resets Status.Image to the old image, and emits a MajorUpgradeRollback event 21.

Rollback is safe because the original PGDATA is not modified until the atomic directory swap in moveDataInPlace() succeeds. If the Job is deleted before that point, the cluster restarts cleanly on the original data 22.

Extension Resolution#

At upgrade-start, resolveExtensionsForMajorVersion() resolves the target extension list:

• ImageCatalog clusters: reads the catalog via imagecatalog.Get() and extensions.ResolveFromCatalog().
• Direct image clusters: validates extensions inline in the spec via extensions.ValidateWithoutCatalog().

The resolved list is stored in Status.TargetPGDataImageInfo before the Job is created, making it the authoritative source for completion handling 23.

Key Source Files#

Related PRs / Issues:

• #10366 — Added Image Volume extension support during major upgrades; introduced TargetPGDataImageInfo and setupExtensionEnvironment().
• #10790 — Known issue: fallback to Status.Image when Status.PGDataImageInfo is absent (affects clusters reconciled by older operator versions).