iframe-proxy

leonardoce · 2026-08-20T13:11:54Z

When a failover is in progress and the old primary's PostgreSQL is unresponsive, skip the checkpoint and fast-shutdown attempts and shut it down immediately instead. Before, an unreachable former primary was never shut down at all, so it never rejoined as a replica of the new primary.

The shutdown is requested without blocking the reconciler, since a wedged postmaster may never finish it. Failover no longer stalls forever waiting on an unreachable primary, though demotion can still take up to a minute or more if postgres stays unresponsive to signals.

Closes: #11348

github-actions · 2026-08-20T13:12:35Z

leonardoce · 2026-08-20T15:36:08Z

/test

github-actions · 2026-08-20T15:36:17Z

@leonardoce, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/32387115979

armru · 2026-09-04T16:27:14Z

/test

github-actions · 2026-09-04T16:27:28Z

@armru, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/33895225818

armru · 2026-09-07T10:01:56Z

/test

github-actions · 2026-09-07T10:02:05Z

… failover When a failover is in progress and the old primary's PostgreSQL is unresponsive, skip the checkpoint and fast-shutdown attempts and shut it down immediately instead, so demoting the old primary doesn't stall waiting on timeouts and the failover can complete without delay. Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>

The demotion of an unreachable old primary sends the shutdown request over `instanceCommandChan`, which is unbuffered, with a blocking send. PostgreSQL being unreachable says nothing about the lifecycle manager still selecting on that channel: a postmaster wedged long enough to stop answering `pg_isready` while the lifecycle manager is already shutting it down on its own, on the SIGTERM path, leaves nobody receiving. The send then parks the reconcile goroutine, and no context cancellation can interrupt a send on an unbuffered channel. That park is permanent, and its cost is not limited to the demotion: the instance manager runs the `instance-cluster` controller with a single worker, so one parked goroutine stops every later reconciliation of that controller for the rest of the process's life. Add a non-blocking `TryRequestImmediateShutdown` and use it instead. A request that cannot be delivered right now is dropped, and the caller's one second requeue tries again. While there, collapse the two demotion paths into a single `reconcileOldPrimary` taking an `unreachable` flag: they shared their guards, their wait on the reconcile context and their return, and only differed in the shutdown request and the log timestamp. Also correct what the code and the documentation claim about that path. The comment on `TryShuttingDownImmediate` said a "fast" attempt needs a working connection, which is not true: only the preceding checkpoint does, `pg_ctl -m fast` just signals the postmaster. The real reason to skip fast is that a postmaster which stopped answering `pg_isready` is not expected to complete the graceful shutdown fast asks for. `ErrNoConnectionEstablished` carried `ErrPgRejectingConnection`'s description, stating the opposite of what it means. Finally, both documents that describe the switchover shutdown sequence only covered the reachable case: add the unreachable one, where `.spec.switchoverDelay` has no effect and pending WALs are archived on the next start instead. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

`RequestImmediateShutdown` was left behind when the demotion of an unreachable old primary moved to the non-blocking request: nothing calls it, and a blocking send on the unbuffered command channel is exactly what that path must not do. Remove it and fold its description into `TryRequestImmediateShutdown`, which is now the only way to ask. Cover the request with unit tests. The two outcomes are indistinguishable without asserting the return value, since a declined send and a delivered one both leave the caller running: the tests check that a declined request reports false and leaves nothing on the channel for a later receiver to find, and that a delivered one carries `shutDownImmediate`. Also correct what the documentation says about the unreachable path. Both documents framed it as skipping the `.spec.switchoverDelay` window, which reads as if tuning that setting used to shorten the wait. There was no wait: the demotion never ran at all, because it sits below the `pg_isready` check that an unresponsive primary fails, so the old primary was never shut down and never restarted as a replica of the new primary. `.spec.switchoverDelay` is the timeout of the fast shutdown, and the fast shutdown is skipped. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

Demoting an unreachable old primary requested an immediate shutdown and then waited on the reconcile context, the way the reachable path waits for its fast shutdown. That wait does not end. A postmaster that has stopped answering `pg_isready` may also have stopped processing signals, and then `pg_ctl -m immediate stop` never completes: it gives up on its own after a minute, the lifecycle manager logs the failure and carries on, and the instance manager is never asked to terminate, so the context is never cancelled. The instance manager runs the `instance-cluster` controller with a single worker, so the parked goroutine took every later reconciliation with it, including the retries of the demotion itself. Return right after making the request, and let the caller requeue the way it already does for any instance that is down. The reachable path keeps its wait, where the fast shutdown does complete. Observed on a three instance cluster with the primary's postmaster and its children stopped with SIGSTOP, so that `pg_isready` reports no response and no signal can be processed, and `.status.targetPrimary` moved to another instance. Before, the demotion log line was the last thing the instance manager ever logged from that controller: no further reconciliation in six minutes, no restart, and the operator still reporting a healthy cluster while the failover sat unfinished. After, the reconciler keeps looping, requesting the shutdown again whenever the lifecycle manager is free to take it, and the demotion completes as soon as the postmaster can process the request. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>

It explains a past behavior without saying which version changed it. Backported as-is, "Previously" would mean a different thing on every release branch's own docs page. The explanation already lives in the PR description. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>

The caller already logs the same error, and pg_ctl already reports success on its own. The sibling TryShuttingDownFastImmediate logs neither. Also tidies the doc comment above TryRequestImmediateShutdown and merges two near-identical test cases into one. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>

… failover (#11347) When a failover is in progress, and the old primary's PostgreSQL is unresponsive, skip the checkpoint and fast-shutdown attempts and shut it down immediately instead. Before, an unreachable former primary was never shut down at all, so it never rejoined as a replica of the new primary. The shutdown is requested without blocking the reconciler, since a wedged postmaster may never finish it. Failover no longer stalls forever waiting on an unreachable primary, though demotion can still take up to a minute or more if Postgres stays unresponsive to signals. Closes: #11348 Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com> Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com> Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com> Co-authored-by: Armando Ruocco <armando.ruocco@enterprisedb.com> Co-authored-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com> (cherry picked from commit 44edfd5)

The isolation check only failed the liveness probe, leaving the shutdown to the kubelet's ordinary termination path. That is a smart shutdown: it lets already-open sessions keep committing for up to smartShutdownTimeout (180s default), while a replacement primary may already be promoting. On the first isolation failure, request an immediate shutdown of PostgreSQL directly, instead of waiting for the kubelet's SIGTERM. Immediate shutdown skips the checkpoint and the fast phase, so nothing delays it. The instance is already headed for a rewind, so the crash recovery this costs on the next start is free. Reuses the non-blocking request PR #11347 added for the same reason, a primary that can't be trusted to shut down cleanly. Firing on the first failure, rather than matching the kubelet's own failureThreshold, means a transient blip a retry would have cleared can trigger a shutdown too. Accepted trade-off: a truly isolated primary should stop immediately. This does not order detection against replica promotion. Detection follows the kubelet's probe cadence; promotion follows the primary lease's LeaseDuration on branches that have one (1.30+), with no time floor at all on branches that don't (1.28/1.29). At the defaults the two are roughly a tie, not a guaranteed order. Only PR #11357 closes that gap, by detecting isolation off the lease's own renewal clock instead of the probe. PR #11283 took the same approach as this patch. It was closed as partially superseded by #11357 on main. This is a backport-sized version for the release branches #11357 won't reach. Closes #11280 Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>

@Jamstah

##### [\`v1.30.1\`](https://github.com/cloudnative-pg/cloudnative-pg/releases/tag/v1.30.1) **Release date:** Sep 23, 2026 ##### Enhancements - Added `restoreAdditionalCommandArgs` to `.barmanObjectStore.data` for in-tree Barman Cloud recoveries, so custom flags are forwarded to `barman-cloud-restore` during recovery. ([#11278](cloudnative-pg/cloudnative-pg#11278))  - Plugin authors no longer need to re-derive whether an empty WAL archive should be treated as an error: the operator now computes the `check_empty_wal_archive` decision once and passes it to CNPG-I plugins on both WAL archiving and restore hooks. ([#11216](cloudnative-pg/cloudnative-pg#11216))  - Allowed setting the `ssl_ecdh_curve` PostgreSQL parameter in `.spec.postgresql.parameters`. It was previously rejected by the validating webhook even though the operator does not manage it. Contributed by [@Jamstah](https://github.com/Jamstah). ([#11308](cloudnative-pg/cloudnative-pg#11308))  - Added support for setting `auth_user` in `spec.pgbouncer.parameters` to override the user PgBouncer uses for its `auth_query` connection, instead of always deriving it from the `authQuerySecret`. Useful with a custom `authQuerySecret` where the role to authenticate as differs from the one implied by the Secret. Contributed by [@jonasbadstuebner](https://github.com/jonasbadstuebner). ([#10899](cloudnative-pg/cloudnative-pg#10899))  - Fixed PgBouncer metrics silently reporting zero when PgBouncer adds or reorders columns: `SHOW STATS` and `SHOW POOLS` results are now matched by column name instead of by position. Contributed by [@AdzerKI](https://github.com/AdzerKI) (`SHOW STATS`). ([#11366](cloudnative-pg/cloudnative-pg#11366), [#11464](cloudnative-pg/cloudnative-pg#11464))  - `cnpg` plugin: - Added a `--dry-run` option to the `backup` command that prints the `Backup` resource that would be created, without creating it. ([#11233](cloudnative-pg/cloudnative-pg#11233))  ##### Security and Supply Chain - Fixed a webhook validation bypass where an absolute extension path with an embedded `..` traversal (e.g. `/a/../../../../etc`) was accepted because only the raw path was checked; such paths still escaped the extension's mount point once resolved at runtime. Affected entries are now resolved and rejected/skipped consistently across admission, the major-upgrade job, and already-persisted Cluster specs. Contributed by [@Anand-240](https://github.com/Anand-240). ([#11210](cloudnative-pg/cloudnative-pg#11210))  - Updated `google.golang.org/grpc` to v1.83.1, fixing [CVE-2026-84304](https://nvd.nist.gov/vuln/detail/CVE-2026-84304) / GHSA-vp52-pcj8-j9qc (gRPC-Go heap memory exhaustion via HTTP/2 DATA frame fragmentation). ([#11394](cloudnative-pg/cloudnative-pg#11394))  ##### Changes - Updated the default PostgreSQL operand image to 18.6-system-trixie. ([#11257](cloudnative-pg/cloudnative-pg#11257))  - Moved the operator's own base image from Debian 12 (bookworm) to Debian 13 (trixie), now that Debian 12 has moved to LTS-only maintenance. ([#11193](cloudnative-pg/cloudnative-pg#11193))  ##### Fixes - Demoted an unreachable old primary immediately during failover, instead of waiting on checkpoint and fast-shutdown attempts that could never complete. Previously, an old primary whose PostgreSQL had become unresponsive was never shut down, so it could never rejoin the cluster as a replica of the new primary. The shutdown is now requested without blocking the reconciler, though demotion can still take up to a minute or more if PostgreSQL keeps ignoring shutdown signals. ([#11347](cloudnative-pg/cloudnative-pg#11347))  - Fixed failover and Pod recreation getting stuck while any instance in the cluster was fenced. A fenced instance kept reporting itself as healthy on its status endpoint even though its Pod was permanently not Ready, which blocked the readiness guard, the primary-election logic, and the Pod-reattachment gate from reacting to an unrelated failure elsewhere in the cluster, leaving it stalled until someone manually removed the fencing. All three now recognize and skip fenced instances. ([#11274](cloudnative-pg/cloudnative-pg#11274))  - Fixed a failover that could stall forever or get silently reverted. While a failover was pending, the reconciler could skip checking whether it was ready to complete, or mistake the pending state for a missing Pod and reset the target primary back to the unhealthy one. It now keeps progressing a pending failover and completes it once the WAL receivers are down, rechecking quorum safety right before promoting in case it changed while waiting. ([#11336](cloudnative-pg/cloudnative-pg#11336))  - Fixed the designated-primary sort in `PostgresqlStatusList` so a tie on LSN no longer sorts it out of order. If the designated primary's Pod later disappeared from the status list, this could have led the operator to promote the wrong standby. ([#11494](cloudnative-pg/cloudnative-pg#11494))  - Gated PostgreSQL startup on the instance manager actually holding the primary `Lease`, closing a window where a restarting or rescheduled primary could start accepting writes before acquiring the lease. Lease acquisition now happens during instance initialization, strictly before PostgreSQL is allowed to start, instead of only later in the reconciliation loop once the instance was already reported ready. ([#11356](cloudnative-pg/cloudnative-pg#11356)) - Bounded the primary lease take-over `Get` call to the lease's `RenewDeadline`, so a call against an unreachable API server now fails fast and retries instead of blocking indefinitely. Left unbounded, the same stalled call could also delay a primary's own lease renewal after a transient API server blip, risking an unwanted takeover by another instance once the lease appeared to expire. ([#11350](cloudnative-pg/cloudnative-pg#11350)) - Stopped abandoning a fast shutdown after 60 seconds. The request carried no explicit timeout, so `pg_ctl` fell back to its own implicit 60-second limit, and the instance manager, being PID 1 in the container, took PostgreSQL down mid-shutdown regardless of a larger configured `stopDelay`. The fast phase now waits for the full `stopDelay` instead. ([#11263](cloudnative-pg/cloudnative-pg#11263))  - Waited for certificate status before refreshing secrets during bootstrap. ([#11450](cloudnative-pg/cloudnative-pg#11450))  - Fixed in-tree Barman Cloud recovery ignoring `wal.maxParallel` and `wal.restoreAdditionalCommandArgs`, fetching WAL one segment at a time instead of using the configured parallelism. ([#11105](cloudnative-pg/cloudnative-pg#11105))  - Honored the `unrecoverable` annotation on pending and terminating instances. ([#11241](cloudnative-pg/cloudnative-pg#11241))  - Honored `Backup.spec.online` when finalizing VolumeSnapshot-based backups, so a cold backup is no longer reported as online in status. ([#11276](cloudnative-pg/cloudnative-pg#11276))  - Set `--no-data-checksums` explicitly when `.spec.postgresql.dataChecksums` is `false` on PostgreSQL 18+, which now enables checksums by default in `initdb`. Contributed by [@evan314159](https://github.com/evan314159). ([#11488](cloudnative-pg/cloudnative-pg#11488))  - Stopped discarding `.spec.inheritedMetadata` labels on PVC reconciliation; they were applied before, and then overwritten by, the operator's own labels. Contributed by [@GellertBabel](https://github.com/GellertBabel). ([#11499](cloudnative-pg/cloudnative-pg#11499))  - Stopped `Pooler` and `DatabaseRole` from recreating their owned resources after being marked for deletion, which could block an external finalizer (e.g. ArgoCD foreground deletion) waiting for those resources to disappear. Contributed by [@mjlshen](https://github.com/mjlshen). ([#11359](cloudnative-pg/cloudnative-pg#11359))  - Handled a `NULL` `stats_reset` value in the `pg_stat_wal` metric, which could occasionally break Prometheus scraping on PostgreSQL 18 clusters. Contributed by [@HoniSanders](https://github.com/HoniSanders). ([#11207](cloudnative-pg/cloudnative-pg#11207))  - Recovered log streaming when a stale non-FIFO file blocked the log pipe. ([#11246](cloudnative-pg/cloudnative-pg#11246))  - Stopped sampling logs of the Pods' processes, so no log lines are silently dropped. ([#11131](cloudnative-pg/cloudnative-pg#11131))  - Stopped caching Helm release Secrets in the operator, which could spike memory usage and crashloop the operator on clusters with large Helm releases installed alongside it. ([#11211](cloudnative-pg/cloudnative-pg#11211))  - Re-ran owned-resource finalizer cleanup after an operator restart, so a namespace deleted while the operator was down doesn't hang in `Terminating` forever. ([#11215](cloudnative-pg/cloudnative-pg#11215))  - Fixed rejoining replicas taking a long time to complete `pg_rewind` when WAL prefetching (`wal.maxParallel > 1`) was enabled. ([#11204](cloudnative-pg/cloudnative-pg#11204))  - Treated a PostgreSQL instance rejecting connections as started up in probes. ([#11152](cloudnative-pg/cloudnative-pg#11152))  - Stopped issuing a redundant PATCH on every reconciliation when a cluster's `pvcTemplate` set `volumeAttributesClass`. Contributed by [@andrew-farries](https://github.com/andrew-farries). ([#11196](cloudnative-pg/cloudnative-pg#11196))  - Stopped halting replication slot reconciliation at the first failing slot. ([#11119](cloudnative-pg/cloudnative-pg#11119))  - During rollouts, compared the running configuration hash, rather than the on-disk one, avoiding unnecessary switchovers. ([#11130](cloudnative-pg/cloudnative-pg#11130))  - Retried `VolumeSnapshot` errors until the deadline, instead of giving up early. ([#11132](cloudnative-pg/cloudnative-pg#11132))  - Symlinked `pg_wal` to the WAL volume during plugin-based recovery, instead of replaying WAL onto the data volume, which could exhaust it on a large point-in-time recovery. ([#11127](cloudnative-pg/cloudnative-pg#11127))  - Corrected grammar in `PhaseUpgradeDelayed` status messages. Contributed by [@Utkarsh-sharma47](https://github.com/Utkarsh-sharma47). ([#11065](cloudnative-pg/cloudnative-pg#11065))  - CloudNativePG Interface (CNPG-I): - Reported the Postgres capabilities of a plugin correctly. ([#11447](cloudnative-pg/cloudnative-pg#11447))  - Prevented the plugin status field from flapping on Cluster reconciliation. Contributed by [@sharifmshaker](https://github.com/sharifmshaker). ([#11386](cloudnative-pg/cloudnative-pg#11386))  - Replaced a finalizer with in-memory cleanup on Service deletion. This avoids namespaces hanging in `Terminating` state when the operator and its plugins run in the same namespace. Contributed by [@maxlengdell](https://github.com/maxlengdell). ([#10940](cloudnative-pg/cloudnative-pg#10940))  - `cnpg` plugin: - Used UTC time zone for the default backup name. ([#11329](cloudnative-pg/cloudnative-pg#11329))  - Deleted PVCs before Pods in `cnpg destroy`, avoiding a race that could leave dangling PVCs needing a second invocation to clean up. Contributed by [@daanvinken](https://github.com/daanvinken). ([#10847](cloudnative-pg/cloudnative-pg#10847))

leonardoce requested a review from a team as a code owner August 20, 2026 13:11

dosubot Bot added size:M This PR changes 30-99 lines, ignoring generated files. release-1.28 release-1.29 release-1.30 labels Aug 20, 2026

cnpg-bot added the backport-requested ◀️ This pull request should be backported to all supported releases label Aug 20, 2026

leonardoce force-pushed the old-unreachable branch from c9e3989 to 4067765 Compare August 20, 2026 13:31

leonardoce changed the title ~~fix(controller): promote a new primary immediately when the old one is unreachable~~ fix(controller): demote an unreachable old primary immediately during failover Aug 20, 2026

cnpg-bot added the ok to merge 👌 This PR can be merged label Aug 20, 2026

This was referenced Aug 21, 2026

feat(instance-manager): move primary isolation check to the primary lease #11357

Merged

feat(instance-manager): turn the liveness probe into a watchdog #11358

Open

fix(instance-manager): promote a replica without waiting for WAL receivers to stop #11367

Open

leonardoce force-pushed the old-unreachable branch from 4067765 to 4e7f39f Compare August 26, 2026 06:55

NiccoloFei force-pushed the old-unreachable branch from 4e7f39f to ec2abdf Compare August 28, 2026 10:24

NiccoloFei approved these changes Sep 1, 2026

View reviewed changes

armru approved these changes Sep 7, 2026

View reviewed changes

armru force-pushed the old-unreachable branch 3 times, most recently from aa2e3df to b35e622 Compare September 7, 2026 09:58

gbartolini mentioned this pull request Sep 14, 2026

[Bug]: Failover blocked indefinitely when walsender survives on zombie node (storage failure with kernel alive) #10591

Open

4 tasks done

mnencia force-pushed the old-unreachable branch from b35e622 to 28286e6 Compare September 15, 2026 14:05

leonardoce and others added 2 commits September 16, 2026 15:09

armru added 2 commits September 16, 2026 15:09

mnencia force-pushed the old-unreachable branch from 28286e6 to 464a9f2 Compare September 16, 2026 13:09

mnencia added 2 commits September 16, 2026 18:50

mnencia approved these changes Sep 17, 2026

View reviewed changes

gbartolini approved these changes Sep 17, 2026

View reviewed changes

gbartolini merged commit 44edfd5 into cloudnative-pg:main Sep 17, 2026
29 checks passed

mnencia mentioned this pull request Sep 18, 2026

fix(probes): stop an isolated primary with an immediate shutdown #11504

Closed

github-actions Bot mentioned this pull request Oct 4, 2026

Update Helm release cloudnative-pg to v0.29.1 claytono/infra#3087

Merged

1 task

PeterZaitsev mentioned this pull request Oct 14, 2026

[Bug]: the primary isolation check does not stop an isolated primary from accepting writes #11280

Open

4 tasks done

Sunbelt Computer Software

PL/B Language Development and Support

Navigation Menu

Repository navigation

fix(controller): demote an unreachable old primary immediately during failover - #11347

fix(controller): demote an unreachable old primary immediately during failover#11347
gbartolini merged 6 commits into
cloudnative-pg:mainfrom
leonardoce:old-unreachable

leonardoce commented Aug 20, 2026 •

edited by mnencia

Loading

Uh oh!

github-actions Bot commented Aug 20, 2026

Uh oh!

leonardoce commented Aug 20, 2026

Uh oh!

github-actions Bot commented Aug 20, 2026

Uh oh!

armru commented Sep 4, 2026

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

armru commented Sep 7, 2026

Uh oh!

github-actions Bot commented Sep 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Sunbelt Computer Software

PL/B Language Development and Support

Conversation

leonardoce commented Aug 20, 2026 • edited by mnencia Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 20, 2026

Uh oh!

leonardoce commented Aug 20, 2026

Uh oh!

github-actions Bot commented Aug 20, 2026

Uh oh!

armru commented Sep 4, 2026

Uh oh!

github-actions Bot commented Sep 4, 2026

Uh oh!

armru commented Sep 7, 2026

Uh oh!

github-actions Bot commented Sep 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

leonardoce commented Aug 20, 2026 •

edited by mnencia

Loading