fix(wal-restore): improve WAL restoration during pg_rewind by armru · Pull Request #11204 · cloudnative-pg/cloudnative-pg · GitHub
Skip to content

fix(wal-restore): improve WAL restoration during pg_rewind - #11204

Merged
leonardoce merged 12 commits into
mainfrom
dev/11200
Jul 21, 2026
Merged

leonardoce merged 12 commits into
mainfrom
dev/11200

Conversation

@armru

@armru armru commented Jul 15, 2026 •

Copy link
Copy Markdown
Member

When a demoted primary rejoins the cluster after a failover, the instance manager runs pg_rewind --restore-target-wal so the WAL segments no longer in its pg_wal can be fetched from the archive. The restore_command handed to pg_rewind was the same manager wal-restore invocation used during normal recovery, WAL prefetching included. pg_rewind walks the timeline backwards, fetches every file it needs exactly once, and treats any restore failure as fatal: with wal.maxParallel > 1 a prefetch miss is practically guaranteed, the miss records the end-of-wal-stream flag on disk, and the next invocation consumes the flag and fails without even contacting the object store, aborting the whole rewind. A flag left over from the pod's previous life aborts even the first invocation. To make things worse, a failed pg_rewind was only retried by restarting the whole join under the reconcile exponential backoff, which grows to roughly 1000 seconds, so a transient restore failure could stall the rejoin for hours.

manager wal-restore now has a --rewind flag that disables prefetching and the end-of-wal-stream flag machinery. The instance manager writes a rewind-specific restore_command into override.conf right before running pg_rewind, and the demotion path rewrites the standard configuration immediately after, so the flag never leaks into normal recovery. pg_rewind itself is now retried in-process with a bounded backoff before the error surfaces, which is safe because pg_rewind collects every WAL segment it needs before writing anything into the target data directory: a failed run leaves the target unchanged.

The same context is conveyed to WAL restore plugins through the new mode field of WALRestoreRequest (cloudnative-pg/cnpg-i#351), and plugin-barman-cloud honors it in cloudnative-pg/plugin-barman-cloud#1007. Plugins that ignore the field keep their current behavior, helped by the in-process retry advancing one segment per attempt. The cnpg-i dependency points to a pseudo-version of that pull request and will move to the next tagged release once it is available.

Related to #10419, which shares the flag machinery but is a distinct failure mode and is not closed here.

Closes #11200

@armru
armru requested a review from a team July 15, 2026 13:26
@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. bug 🐛 Something isn't working release-1.28 release-1.29 release-1.30 labels Jul 15, 2026
@cnpg-bot cnpg-bot added the backport-requested ◀️ This pull request should be backported to all supported releases label Jul 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

@armru armru changed the title fix: reliable WAL restoration during pg_rewind fix: improve WAL restoration during pg_rewind Jul 15, 2026
@armru

armru commented Jul 15, 2026

Copy link
Copy Markdown
Member Author

/test

@github-actions

Copy link
Copy Markdown
Contributor

@armru, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/29419409135

@mnencia mnencia changed the title fix: improve WAL restoration during pg_rewind fix(wal-restore): improve WAL restoration during pg_rewind Jul 16, 2026
@mnencia

mnencia commented Jul 16, 2026

Copy link
Copy Markdown
Member

/test

@github-actions

Copy link
Copy Markdown
Contributor

@cnpg-bot cnpg-bot added the ok to merge 👌 This PR can be merged label Jul 16, 2026
@dosubot dosubot Bot added the lgtm This PR has been approved by a maintainer label Jul 17, 2026
armru added 6 commits July 21, 2026 11:41
When pg_rewind runs with --restore-target-wal, it fetches the WAL
segments it needs by walking the WAL backward, and any failure of
restore_command aborts the whole run. The wal-restore command, however,
applies the end-of-wal-stream flag machinery designed for streaming
standbys: it prefetches the segments that follow the requested one,
sets an on-disk flag when any of them is missing, and makes the next
invocation fail outright so that PostgreSQL switches to streaming
replication.

During a rewind on a short post-failover timeline this aborts the run
deterministically: the segments following the divergence point do not
exist in the archive yet, so a successful restore sets the flag and the
following request fails even though its segment is available. A stale
flag left over from before the promotion has the same effect on the
first request. Since pg_rewind cannot fall back to streaming
replication, the rejoin of the demoted primary fails and is retried
from scratch by the reconciliation loop under exponential backoff.

Introduce a --rewind option in wal-restore that disables prefetching
and the flag machinery, and use it in the restore_command written into
override.conf while running pg_rewind. The standard replica
configuration is rewritten by the demotion that follows the rewind.

Closes #11200

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
pg_rewind does not modify the target data directory until it has
collected all the WAL segments it needs, so a run aborted by a failed
WAL restoration can be retried from scratch, reusing the segments
already fetched into pg_wal. Without a retry, a transient failure such
as an object store hiccup is handed back to the reconciliation loop,
whose exponential backoff can keep the instance down for a long time
with no postmaster running.

Retry the pg_rewind invocation a few times with a short backoff before
returning the error, and fix the Rewind documentation comment that
described a PostgreSQL version gate on --restore-target-wal that no
longer exists.

Part of #11200

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Move the computation of how many WAL files a single wal-restore
invocation may fetch into a dedicated function, so that the rewind-mode
override disabling prefetching is covered by unit tests instead of
living inline in run().

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
The TestFileExist and TestDirectoryEmpty helpers run their commands via
kubectl exec with no shell on either side, so glob patterns are passed
literally to test(1) and never match: assertions like "no wals in the
spool directory" could not fail regardless of the actual spool state.
List the spool directory through the pod exec API and compare its exact
contents instead, so the rewind-mode steps really verify that nothing
is prefetched and that the end-of-wal-stream flag is left untouched.

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
The retry comment in Rewind claimed pg_rewind performs no writes before
collecting the WAL segments it needs. That is only exact for a cleanly
shut down target: on an unclean one, pg_rewind first runs single-user
crash recovery, which does write to PGDATA. That recovery is ordinary
and repeatable, so retrying remains safe, but the comment should not
overstate the read-only phase.

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
The WAL restore protocol gained a mode field stating the context a WAL
file is being restored in (cloudnative-pg/cnpg-i#351). Set it on every
plugin restore request, so WAL restore plugins can drop the prefetching
and end-of-wal-stream optimizations when the restore runs on behalf of
pg_rewind, like the in-tree restore path already does.

The cnpg-i dependency points to a pseudo-version of that pull request
and will be moved to the next tagged release once it is available.

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
armru and others added 6 commits July 21, 2026 11:41
Diagnosing a stalled rejoin from pod logs requires telling a rewind-mode
WAL restoration apart from a normal recovery one, and an in-process
pg_rewind retry apart from a fresh run started by the reconciliation
loop. Add the rewindMode field to the wal-restore completion log and an
attempt counter to the pg_rewind failure log.

Part of #11200

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
The pg_control backup/restore was only run once, before the first
pg_rewind attempt. A failed attempt is exactly the condition it
exists to repair, so subsequent retries could still see an empty
control file. Move the call inside the retry loop so it runs before
every attempt.

Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
The retry predicate ignores the error and only checks the context, so
it retries any pg_rewind failure, including permanent ones like a bad
connection string, not only the transient WAL-restore case the
surrounding comment describes.

Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Extract the retry predicate into a named function and add unit tests
for it, since Rewind() shells out to a real pg_rewind binary and had
no test coverage for its retry logic at all.

Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Move the pg_rewind write-ordering rationale back to the retry call
site; pgRewindShouldRetry only checks context cancellation, so it
isn't the right place to explain why retrying the whole operation is
safe. Also trim a new e2e test comment that restated the existing
"Expected outcome" trailer in different words.

Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
@leonardoce
leonardoce merged commit ad1d52c into main Jul 21, 2026
39 checks passed
@leonardoce
leonardoce deleted the dev/11200 branch July 21, 2026 12:02
cnpg-bot pushed a commit that referenced this pull request Jul 21, 2026
When a demoted primary rejoins the cluster after a failover, the
instance manager runs `pg_rewind --restore-target-wal` so the WAL
segments no longer in its `pg_wal` can be fetched from the archive. The
`restore_command` handed to pg_rewind was the same `manager wal-restore`
invocation used during normal recovery, WAL prefetching included.
pg_rewind walks the timeline backwards, fetches every file it needs
exactly once, and treats any restore failure as fatal: with
`wal.maxParallel > 1` a prefetch miss is practically guaranteed, the
miss records the end-of-wal-stream flag on disk, and the next invocation
consumes the flag and fails without even contacting the object store,
aborting the whole rewind. A flag left over from the pod's previous life
aborts even the first invocation. To make things worse, a failed
pg_rewind was only retried by restarting the whole join under the
reconcile exponential backoff, which grows to roughly 1000 seconds, so a
transient restore failure could stall the rejoin for hours.

`manager wal-restore` now has a `--rewind` flag that disables
prefetching and the end-of-wal-stream flag machinery. The instance
manager writes a rewind-specific `restore_command` into `override.conf`
right before running pg_rewind, and the demotion path rewrites the
standard configuration immediately after, so the flag never leaks into
normal recovery. pg_rewind itself is now retried in-process with a
bounded backoff before the error surfaces, which is safe because
pg_rewind collects every WAL segment it needs before writing anything
into the target data directory: a failed run leaves the target
unchanged.

The same context is conveyed to WAL restore plugins through the new
`mode` field of `WALRestoreRequest` (cloudnative-pg/cnpg-i#351), and
plugin-barman-cloud honors it in
cloudnative-pg/plugin-barman-cloud#1007. Plugins that ignore the field
keep their current behavior, helped by the in-process retry advancing
one segment per attempt. The cnpg-i dependency points to a
pseudo-version of that pull request and will move to the next tagged
release once it is available.

Related to #10419, which shares the flag machinery but is a distinct
failure mode and is not closed here.

Closes #11200

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
Co-authored-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Co-authored-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
(cherry picked from commit ad1d52c)
cnpg-bot pushed a commit that referenced this pull request Jul 21, 2026
When a demoted primary rejoins the cluster after a failover, the
instance manager runs `pg_rewind --restore-target-wal` so the WAL
segments no longer in its `pg_wal` can be fetched from the archive. The
`restore_command` handed to pg_rewind was the same `manager wal-restore`
invocation used during normal recovery, WAL prefetching included.
pg_rewind walks the timeline backwards, fetches every file it needs
exactly once, and treats any restore failure as fatal: with
`wal.maxParallel > 1` a prefetch miss is practically guaranteed, the
miss records the end-of-wal-stream flag on disk, and the next invocation
consumes the flag and fails without even contacting the object store,
aborting the whole rewind. A flag left over from the pod's previous life
aborts even the first invocation. To make things worse, a failed
pg_rewind was only retried by restarting the whole join under the
reconcile exponential backoff, which grows to roughly 1000 seconds, so a
transient restore failure could stall the rejoin for hours.

`manager wal-restore` now has a `--rewind` flag that disables
prefetching and the end-of-wal-stream flag machinery. The instance
manager writes a rewind-specific `restore_command` into `override.conf`
right before running pg_rewind, and the demotion path rewrites the
standard configuration immediately after, so the flag never leaks into
normal recovery. pg_rewind itself is now retried in-process with a
bounded backoff before the error surfaces, which is safe because
pg_rewind collects every WAL segment it needs before writing anything
into the target data directory: a failed run leaves the target
unchanged.

The same context is conveyed to WAL restore plugins through the new
`mode` field of `WALRestoreRequest` (cloudnative-pg/cnpg-i#351), and
plugin-barman-cloud honors it in
cloudnative-pg/plugin-barman-cloud#1007. Plugins that ignore the field
keep their current behavior, helped by the in-process retry advancing
one segment per attempt. The cnpg-i dependency points to a
pseudo-version of that pull request and will move to the next tagged
release once it is available.

Related to #10419, which shares the flag machinery but is a distinct
failure mode and is not closed here.

Closes #11200

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
Co-authored-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Co-authored-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
(cherry picked from commit ad1d52c)
leonardoce added a commit that referenced this pull request Jul 21, 2026
When a demoted primary rejoins the cluster after a failover, the
instance manager runs `pg_rewind --restore-target-wal` so the WAL
segments no longer in its `pg_wal` can be fetched from the archive. The
`restore_command` handed to pg_rewind was the same `manager wal-restore`
invocation used during normal recovery, WAL prefetching included.
pg_rewind walks the timeline backwards, fetches every file it needs
exactly once, and treats any restore failure as fatal: with
`wal.maxParallel > 1` a prefetch miss is practically guaranteed, the
miss records the end-of-wal-stream flag on disk, and the next invocation
consumes the flag and fails without even contacting the object store,
aborting the whole rewind. A flag left over from the pod's previous life
aborts even the first invocation. To make things worse, a failed
pg_rewind was only retried by restarting the whole join under the
reconcile exponential backoff, which grows to roughly 1000 seconds, so a
transient restore failure could stall the rejoin for hours.

`manager wal-restore` now has a `--rewind` flag that disables
prefetching and the end-of-wal-stream flag machinery. The instance
manager writes a rewind-specific `restore_command` into `override.conf`
right before running pg_rewind, and the demotion path rewrites the
standard configuration immediately after, so the flag never leaks into
normal recovery. pg_rewind itself is now retried in-process with a
bounded backoff before the error surfaces, which is safe because
pg_rewind collects every WAL segment it needs before writing anything
into the target data directory: a failed run leaves the target
unchanged.

The same context is conveyed to WAL restore plugins through the new
`mode` field of `WALRestoreRequest` (cloudnative-pg/cnpg-i#351), and
plugin-barman-cloud honors it in
cloudnative-pg/plugin-barman-cloud#1007. Plugins that ignore the field
keep their current behavior, helped by the in-process retry advancing
one segment per attempt. The cnpg-i dependency points to a
pseudo-version of that pull request and will move to the next tagged
release once it is available.

Related to #10419, which shares the flag machinery but is a distinct
failure mode and is not closed here.

Closes #11200

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
Co-authored-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Co-authored-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
(cherry picked from commit ad1d52c)
(cherry picked from commit f5f1b8a)
leonardoce pushed a commit to cloudnative-pg/plugin-barman-cloud that referenced this pull request Jul 21, 2026
…ry (#1007)

When a demoted primary rejoins the cluster after a failover, the
instance manager runs `pg_rewind --restore-target-wal`, and the
resulting `restore_command` invocations are served by this sidecar
exactly as if an instance in recovery were asking. pg_rewind walks the
timeline backwards, fetches every WAL file it needs exactly once, and
treats any restore failure as fatal, so two optimizations meant for
recovery break it: prefetching the following segments is wasted work
that, past the end of the timeline, is guaranteed to end in an archive
miss, and the end-of-wal-stream flag recorded by that miss makes a later
invocation fail without contacting the object store, on a segment the
archive actually has, aborting the whole rewind.

The restore request now carries the context it is made in
(cloudnative-pg/cnpg-i#351). When it says `MODE_REWIND`, the sidecar
restores exactly the requested file: no prefetching, no
end-of-wal-stream flag check, no flag recording. Requests from operators
predating the field keep the current behavior.

This is the plugin-side counterpart of
cloudnative-pg/cloudnative-pg#11204. The cnpg-i dependency points to a
pseudo-version of the protocol pull request and will move to the next
tagged release once it is available.

Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Co-authored-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
renovate Bot added a commit to sdwilsh/ansible-playbooks that referenced this pull request Sep 24, 2026
##### [\`v1.30.1\`](https://github.com/cloudnative-pg/cloudnative-pg/releases/tag/v1.30.1)

**Release date:** Sep 23, 2026

##### Enhancements

- Added `restoreAdditionalCommandArgs` to `.barmanObjectStore.data` for in-tree Barman Cloud recoveries, so custom flags are forwarded to `barman-cloud-restore` during recovery. ([#11278](cloudnative-pg/cloudnative-pg#11278)) <!-- 1.29 1.28 -->

- Plugin authors no longer need to re-derive whether an empty WAL archive should be treated as an error: the operator now computes the `check_empty_wal_archive` decision once and passes it to CNPG-I plugins on both WAL archiving and restore hooks. ([#11216](cloudnative-pg/cloudnative-pg#11216)) <!-- 1.29 1.28 -->

- Allowed setting the `ssl_ecdh_curve` PostgreSQL parameter in `.spec.postgresql.parameters`. It was previously rejected by the validating webhook even though the operator does not manage it. Contributed by [@Jamstah](https://github.com/Jamstah). ([#11308](cloudnative-pg/cloudnative-pg#11308)) <!-- 1.29 1.28 -->

- Added support for setting `auth_user` in `spec.pgbouncer.parameters` to override the user PgBouncer uses for its `auth_query` connection, instead of always deriving it from the `authQuerySecret`. Useful with a custom `authQuerySecret` where the role to authenticate as differs from the one implied by the Secret. Contributed by [@jonasbadstuebner](https://github.com/jonasbadstuebner). ([#10899](cloudnative-pg/cloudnative-pg#10899)) <!-- 1.29 1.28 -->

- Fixed PgBouncer metrics silently reporting zero when PgBouncer adds or reorders columns: `SHOW STATS` and `SHOW POOLS` results are now matched by column name instead of by position. Contributed by [@AdzerKI](https://github.com/AdzerKI) (`SHOW STATS`). ([#11366](cloudnative-pg/cloudnative-pg#11366), [#11464](cloudnative-pg/cloudnative-pg#11464)) <!-- 1.29 1.28 -->

- `cnpg` plugin:

  - Added a `--dry-run` option to the `backup` command that prints the `Backup` resource that would be created, without creating it. ([#11233](cloudnative-pg/cloudnative-pg#11233)) <!-- 1.29 1.28 -->

##### Security and Supply Chain

- Fixed a webhook validation bypass where an absolute extension path with an embedded `..` traversal (e.g. `/a/../../../../etc`) was accepted because only the raw path was checked; such paths still escaped the extension's mount point once resolved at runtime. Affected entries are now resolved and rejected/skipped consistently across admission, the major-upgrade job, and already-persisted Cluster specs. Contributed by [@Anand-240](https://github.com/Anand-240). ([#11210](cloudnative-pg/cloudnative-pg#11210)) <!-- 1.29 1.28 -->

- Updated `google.golang.org/grpc` to v1.83.1, fixing [CVE-2026-84304](https://nvd.nist.gov/vuln/detail/CVE-2026-84304) / GHSA-vp52-pcj8-j9qc (gRPC-Go heap memory exhaustion via HTTP/2 DATA frame fragmentation). ([#11394](cloudnative-pg/cloudnative-pg#11394)) <!-- 1.29 1.28 -->

##### Changes

- Updated the default PostgreSQL operand image to 18.6-system-trixie. ([#11257](cloudnative-pg/cloudnative-pg#11257)) <!-- 1.29 1.28 -->

- Moved the operator's own base image from Debian 12 (bookworm) to Debian 13 (trixie), now that Debian 12 has moved to LTS-only maintenance. ([#11193](cloudnative-pg/cloudnative-pg#11193)) <!-- 1.29 1.28 -->

##### Fixes

- Demoted an unreachable old primary immediately during failover, instead of waiting on checkpoint and fast-shutdown attempts that could never complete. Previously, an old primary whose PostgreSQL had become unresponsive was never shut down, so it could never rejoin the cluster as a replica of the new primary. The shutdown is now requested without blocking the reconciler, though demotion can still take up to a minute or more if PostgreSQL keeps ignoring shutdown signals. ([#11347](cloudnative-pg/cloudnative-pg#11347)) <!-- 1.29 1.28 -->

- Fixed failover and Pod recreation getting stuck while any instance in the cluster was fenced. A fenced instance kept reporting itself as healthy on its status endpoint even though its Pod was permanently not Ready, which blocked the readiness guard, the primary-election logic, and the Pod-reattachment gate from reacting to an unrelated failure elsewhere in the cluster, leaving it stalled until someone manually removed the fencing. All three now recognize and skip fenced instances. ([#11274](cloudnative-pg/cloudnative-pg#11274)) <!-- 1.29 1.28 -->

- Fixed a failover that could stall forever or get silently reverted. While a failover was pending, the reconciler could skip checking whether it was ready to complete, or mistake the pending state for a missing Pod and reset the target primary back to the unhealthy one. It now keeps progressing a pending failover and completes it once the WAL receivers are down, rechecking quorum safety right before promoting in case it changed while waiting. ([#11336](cloudnative-pg/cloudnative-pg#11336)) <!-- 1.29 1.28 -->

- Fixed the designated-primary sort in `PostgresqlStatusList` so a tie on LSN no longer sorts it out of order. If the designated primary's Pod later disappeared from the status list, this could have led the operator to promote the wrong standby. ([#11494](cloudnative-pg/cloudnative-pg#11494)) <!-- 1.29 1.28 -->

- Gated PostgreSQL startup on the instance manager actually holding the primary `Lease`, closing a window where a restarting or rescheduled primary could start accepting writes before acquiring the lease. Lease acquisition now happens during instance initialization, strictly before PostgreSQL is allowed to start, instead of only later in the reconciliation loop once the instance was already reported ready. ([#11356](cloudnative-pg/cloudnative-pg#11356))

- Bounded the primary lease take-over `Get` call to the lease's `RenewDeadline`, so a call against an unreachable API server now fails fast and retries instead of blocking indefinitely. Left unbounded, the same stalled call could also delay a primary's own lease renewal after a transient API server blip, risking an unwanted takeover by another instance once the lease appeared to expire. ([#11350](cloudnative-pg/cloudnative-pg#11350))

- Stopped abandoning a fast shutdown after 60 seconds. The request carried no explicit timeout, so `pg_ctl` fell back to its own implicit 60-second limit, and the instance manager, being PID 1 in the container, took PostgreSQL down mid-shutdown regardless of a larger configured `stopDelay`. The fast phase now waits for the full `stopDelay` instead. ([#11263](cloudnative-pg/cloudnative-pg#11263)) <!-- 1.29 1.28 -->

- Waited for certificate status before refreshing secrets during bootstrap. ([#11450](cloudnative-pg/cloudnative-pg#11450)) <!-- 1.29 1.28 -->

- Fixed in-tree Barman Cloud recovery ignoring `wal.maxParallel` and `wal.restoreAdditionalCommandArgs`, fetching WAL one segment at a time instead of using the configured parallelism. ([#11105](cloudnative-pg/cloudnative-pg#11105)) <!-- 1.29 1.28 -->

- Honored the `unrecoverable` annotation on pending and terminating instances. ([#11241](cloudnative-pg/cloudnative-pg#11241)) <!-- 1.29 1.28 -->

- Honored `Backup.spec.online` when finalizing VolumeSnapshot-based backups, so a cold backup is no longer reported as online in status. ([#11276](cloudnative-pg/cloudnative-pg#11276)) <!-- 1.29 1.28 -->

- Set `--no-data-checksums` explicitly when `.spec.postgresql.dataChecksums` is `false` on PostgreSQL 18+, which now enables checksums by default in `initdb`. Contributed by [@evan314159](https://github.com/evan314159). ([#11488](cloudnative-pg/cloudnative-pg#11488)) <!-- 1.29 1.28 -->

- Stopped discarding `.spec.inheritedMetadata` labels on PVC reconciliation; they were applied before, and then overwritten by, the operator's own labels. Contributed by [@GellertBabel](https://github.com/GellertBabel). ([#11499](cloudnative-pg/cloudnative-pg#11499)) <!-- 1.29 1.28 -->

- Stopped `Pooler` and `DatabaseRole` from recreating their owned resources after being marked for deletion, which could block an external finalizer (e.g. ArgoCD foreground deletion) waiting for those resources to disappear. Contributed by [@mjlshen](https://github.com/mjlshen). ([#11359](cloudnative-pg/cloudnative-pg#11359)) <!-- 1.29 1.28 -->

- Handled a `NULL` `stats_reset` value in the `pg_stat_wal` metric, which could occasionally break Prometheus scraping on PostgreSQL 18 clusters. Contributed by [@HoniSanders](https://github.com/HoniSanders). ([#11207](cloudnative-pg/cloudnative-pg#11207)) <!-- 1.29 1.28 -->

- Recovered log streaming when a stale non-FIFO file blocked the log pipe. ([#11246](cloudnative-pg/cloudnative-pg#11246)) <!-- 1.29 1.28 -->

- Stopped sampling logs of the Pods' processes, so no log lines are silently dropped. ([#11131](cloudnative-pg/cloudnative-pg#11131)) <!-- 1.29 1.28 -->

- Stopped caching Helm release Secrets in the operator, which could spike memory usage and crashloop the operator on clusters with large Helm releases installed alongside it. ([#11211](cloudnative-pg/cloudnative-pg#11211)) <!-- 1.29 1.28 -->

- Re-ran owned-resource finalizer cleanup after an operator restart, so a namespace deleted while the operator was down doesn't hang in `Terminating` forever. ([#11215](cloudnative-pg/cloudnative-pg#11215)) <!-- 1.29 1.28 -->

- Fixed rejoining replicas taking a long time to complete `pg_rewind` when WAL prefetching (`wal.maxParallel > 1`) was enabled. ([#11204](cloudnative-pg/cloudnative-pg#11204)) <!-- 1.29 1.28 -->

- Treated a PostgreSQL instance rejecting connections as started up in probes. ([#11152](cloudnative-pg/cloudnative-pg#11152)) <!-- 1.29 1.28 -->

- Stopped issuing a redundant PATCH on every reconciliation when a cluster's `pvcTemplate` set `volumeAttributesClass`. Contributed by [@andrew-farries](https://github.com/andrew-farries). ([#11196](cloudnative-pg/cloudnative-pg#11196)) <!-- 1.29 1.28 -->

- Stopped halting replication slot reconciliation at the first failing slot. ([#11119](cloudnative-pg/cloudnative-pg#11119)) <!-- 1.29 1.28 -->

- During rollouts, compared the running configuration hash, rather than the on-disk one, avoiding unnecessary switchovers. ([#11130](cloudnative-pg/cloudnative-pg#11130)) <!-- 1.29 1.28 -->

- Retried `VolumeSnapshot` errors until the deadline, instead of giving up early. ([#11132](cloudnative-pg/cloudnative-pg#11132)) <!-- 1.29 1.28 -->

- Symlinked `pg_wal` to the WAL volume during plugin-based recovery, instead of replaying WAL onto the data volume, which could exhaust it on a large point-in-time recovery. ([#11127](cloudnative-pg/cloudnative-pg#11127)) <!-- 1.29 1.28 -->

- Corrected grammar in `PhaseUpgradeDelayed` status messages. Contributed by [@Utkarsh-sharma47](https://github.com/Utkarsh-sharma47). ([#11065](cloudnative-pg/cloudnative-pg#11065)) <!-- 1.29 1.28 -->

- CloudNativePG Interface (CNPG-I):

  - Reported the Postgres capabilities of a plugin correctly. ([#11447](cloudnative-pg/cloudnative-pg#11447)) <!-- 1.29 1.28 -->
  - Prevented the plugin status field from flapping on Cluster reconciliation. Contributed by [@sharifmshaker](https://github.com/sharifmshaker). ([#11386](cloudnative-pg/cloudnative-pg#11386)) <!-- 1.29 1.28 -->
  - Replaced a finalizer with in-memory cleanup on Service deletion. This avoids namespaces hanging in `Terminating` state when the operator and its plugins run in the same namespace. Contributed by [@maxlengdell](https://github.com/maxlengdell). ([#10940](cloudnative-pg/cloudnative-pg#10940)) <!-- 1.29 -->

- `cnpg` plugin:

  - Used UTC time zone for the default backup name. ([#11329](cloudnative-pg/cloudnative-pg#11329)) <!-- 1.29 1.28 -->
  - Deleted PVCs before Pods in `cnpg destroy`, avoiding a race that could leave dangling PVCs needing a second invocation to clean up. Contributed by [@daanvinken](https://github.com/daanvinken). ([#10847](cloudnative-pg/cloudnative-pg#10847)) <!-- 1.29 1.28 -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-requested ◀️ This pull request should be backported to all supported releases bug 🐛 Something isn't working lgtm This PR has been approved by a maintainer ok to merge 👌 This PR can be merged release-1.28 release-1.30 size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: pg_rewind's single-shot restore_command abort + full reconcile restart can stall crash-failover rejoin for hours

4 participants