PostgreSQL Major Version Upgrades in CloudNativePG#
Overview#
CloudNativePG orchestrates offline, in-place major PostgreSQL version upgrades using Kubernetes Job resources backed by pg_upgrade. The upgrade is triggered declaratively by setting a higher-major-version image on a Cluster object — no second cluster, no logical replication — but the cluster is unavailable for the duration .
Three upgrade strategies are supported; only the pg_upgrade path is covered here:
| Method | Type | Downtime |
|---|---|---|
| Logical dump/restore | Blue/green | Offline |
| Native logical replication | Blue/green | Online |
pg_upgrade (this doc) | In-place | Offline |
Reconciler Entry Point#
All major-upgrade logic lives in pkg/reconciler/majorupgrade/. The package-level doc describes the four-step lifecycle :
- Delete all cluster Pods.
- Create and run the upgrade
Job(BackoffLimit=0— no retries). - Wait for job completion.
- On success, start new Pods for the upgraded version; on image rollback, delete the job and let the cluster restart.
Reconcile() is the main entrypoint. Its decision tree, in order:
- Job already exists and completed → call
majorVersionUpgradeHandleCompletion(). - Job exists but not completed → check for user rollback via
handleRollbackIfNeeded(); otherwise requeue after 30 s. - No job and no upgrade needed (requested major ≤
Status.PGDataImageInfo.MajorVersion) → callclearStaleUpgradeTarget()and return. - No job and upgrade needed → resolve extensions, patch status to
PhaseMajorUpgrade, delete pods/jobs, create the upgrade Job.
Pod and Job Deletion#
Before creating the upgrade Job, deleteAllPodsInMajorUpgradePreparation() deletes all running instance Pods and any existing Jobs. Resources still terminating cause a 10-second requeue; only when the slate is fully clear does execution proceed to Job creation .
Upgrade Job Structure#
createMajorUpgradeJobDefinition() builds the Job with two containers:
- Init container (
prepare) — runsmanager instance upgrade prepare /controller/oldusing the old PostgreSQL image (Status.PGDataImageInfo.Image). It invokespg_configto locate binaries/libraries, copies them to/controller/old, and writes/controller/old/bindir.txt. - Main container (
major-upgrade) — runsmanager instance upgrade execute /controller/old/bindir.txtusing the new PostgreSQL image .
BackoffLimit is hard-set to 0: a failed pg_upgrade cannot succeed on retry .
The Job is identified by label utils.JobRoleLabelName == "major-upgrade" .
For clusters with Image Volume extensions, both the source-version and target-version extension volumes are mounted simultaneously under separate trees (/extensions/<name> and /new-extensions/<name>) so the two sets never collide .
Execute Subcommand (instance upgrade execute)#
internal/cmd/manager/instance/upgrade/execute/cmd.go runs inside the main upgrade container and implements the upgrade in this order:
- Extension environment setup —
setupExtensionEnvironment()extendsLD_LIBRARY_PATHandPATHfor both source (Status.PGDataImageInfo) and target (Status.TargetPGDataImageInfo) extension sets, and applies per-extensionenvvariables. Both status fields must be present or setup fails fast. - Fail-fast on previous failures — checks for
*.failed_*directories left by a prior incomplete run . initdb— createsPGDATA-new; propagates WAL segment size and data checksum settings from the old cluster'spg_controldataoutput .- Configuration —
prepareConfigurationFiles()writespostgresql.confincludes and applies version-specific parameter overrides (see table below). pg_upgrade --link— runspg_upgradewith hard-linked files for speed .- Atomic directory swap —
moveDataInPlace()renamesPGDATA→PGDATA.old, thenPGDATA-new→PGDATA; on failure, both directories are saved with a.failed_<timestamp>suffix to prevent data loss. - Extension update script — logs the location of
PGDATA/update_extensions.sqlifpg_upgradeemitted it .
Version-Specific Parameter Management#
Before pg_upgrade runs, the operator generates a valid postgresql.conf for the new major version. Key version-aware behavior :
| Parameter | Behavior |
|---|---|
max_slot_wal_keep_size | Forced to -1 for all upgrades (workaround for a PG 17.0–17.5 bug) |
idle_replication_slot_timeout | Forced to 0 for PG ≥ 18 |
| WAL segment size | Propagated from old cluster's pg_controldata |
| Data checksums | Propagated; --no-data-checksums added for PG ≥ 18 if disabled |
The configuration generator is invoked with OperationType_TYPE_UPGRADE, which switches extension path GUCs (dynamic_library_path, extension_control_path) to point at the target-version /new-extensions mounts .
Post-Upgrade Status Handling#
majorVersionUpgradeHandleCompletion() fires after the Job reaches Completed:
- Deletes non-primary PVCs — replica data will be re-cloned from scratch .
- Carries forward
TargetPGDataImageInfo.Extensionsrather than re-resolving from the catalog, to avoid aPhaseImageCatalogErrorif the catalog was edited while the Job ran . - Patches cluster status: sets
PGDataImageInfoto the new image/version, clearsTargetPGDataImageInfo, and resetsTimelineIDto 1 — matchingpg_upgrade's behavior and preventing replicas from fetching incompatible pre-upgrade timeline history files . - Deletes the upgrade Job and requeues .
Rollback Handling#
If the user reverts the cluster image to the old major version while the Job is still running, handleRollbackIfNeeded() detects this (requested major ≤ PGDataImageInfo.MajorVersion), deletes the Job with foreground propagation, resets Status.Image to the old image, and emits a MajorUpgradeRollback event .
Rollback is safe because the original PGDATA is not modified until the atomic directory swap in moveDataInPlace() succeeds. If the Job is deleted before that point, the cluster restarts cleanly on the original data .
Extension Resolution#
At upgrade-start, resolveExtensionsForMajorVersion() resolves the target extension list:
- ImageCatalog clusters: reads the catalog via
imagecatalog.Get()andextensions.ResolveFromCatalog(). - Direct image clusters: validates extensions inline in the spec via
extensions.ValidateWithoutCatalog().
The resolved list is stored in Status.TargetPGDataImageInfo before the Job is created, making it the authoritative source for completion handling .
Key Source Files#
| File | Purpose |
|---|---|
pkg/reconciler/majorupgrade/reconciler.go | Main reconciliation loop and completion/rollback logic |
pkg/reconciler/majorupgrade/job.go | Upgrade Job definition builder |
internal/cmd/manager/instance/upgrade/execute/cmd.go | In-pod pg_upgrade execution |
docs/src/postgres_upgrades.md | User-facing documentation |
Related PRs / Issues: