{{ message }}
Repository navigation
fix(wal-restore): improve WAL restoration during pg_rewind - #11204
Merged
Merged
Conversation
Contributor
Member
Author
|
/test |
Contributor
|
@armru, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/29419409135 |
Member
|
/test |
Contributor
mnencia
approved these changes
Jul 17, 2026
leonardoce
approved these changes
Jul 21, 2026
When pg_rewind runs with --restore-target-wal, it fetches the WAL segments it needs by walking the WAL backward, and any failure of restore_command aborts the whole run. The wal-restore command, however, applies the end-of-wal-stream flag machinery designed for streaming standbys: it prefetches the segments that follow the requested one, sets an on-disk flag when any of them is missing, and makes the next invocation fail outright so that PostgreSQL switches to streaming replication. During a rewind on a short post-failover timeline this aborts the run deterministically: the segments following the divergence point do not exist in the archive yet, so a successful restore sets the flag and the following request fails even though its segment is available. A stale flag left over from before the promotion has the same effect on the first request. Since pg_rewind cannot fall back to streaming replication, the rejoin of the demoted primary fails and is retried from scratch by the reconciliation loop under exponential backoff. Introduce a --rewind option in wal-restore that disables prefetching and the flag machinery, and use it in the restore_command written into override.conf while running pg_rewind. The standard replica configuration is rewritten by the demotion that follows the rewind. Closes #11200 Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
pg_rewind does not modify the target data directory until it has collected all the WAL segments it needs, so a run aborted by a failed WAL restoration can be retried from scratch, reusing the segments already fetched into pg_wal. Without a retry, a transient failure such as an object store hiccup is handed back to the reconciliation loop, whose exponential backoff can keep the instance down for a long time with no postmaster running. Retry the pg_rewind invocation a few times with a short backoff before returning the error, and fix the Rewind documentation comment that described a PostgreSQL version gate on --restore-target-wal that no longer exists. Part of #11200 Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Move the computation of how many WAL files a single wal-restore invocation may fetch into a dedicated function, so that the rewind-mode override disabling prefetching is covered by unit tests instead of living inline in run(). Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
The TestFileExist and TestDirectoryEmpty helpers run their commands via kubectl exec with no shell on either side, so glob patterns are passed literally to test(1) and never match: assertions like "no wals in the spool directory" could not fail regardless of the actual spool state. List the spool directory through the pod exec API and compare its exact contents instead, so the rewind-mode steps really verify that nothing is prefetched and that the end-of-wal-stream flag is left untouched. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
The retry comment in Rewind claimed pg_rewind performs no writes before collecting the WAL segments it needs. That is only exact for a cleanly shut down target: on an unclean one, pg_rewind first runs single-user crash recovery, which does write to PGDATA. That recovery is ordinary and repeatable, so retrying remains safe, but the comment should not overstate the read-only phase. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
The WAL restore protocol gained a mode field stating the context a WAL file is being restored in (cloudnative-pg/cnpg-i#351). Set it on every plugin restore request, so WAL restore plugins can drop the prefetching and end-of-wal-stream optimizations when the restore runs on behalf of pg_rewind, like the in-tree restore path already does. The cnpg-i dependency points to a pseudo-version of that pull request and will be moved to the next tagged release once it is available. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
Diagnosing a stalled rejoin from pod logs requires telling a rewind-mode WAL restoration apart from a normal recovery one, and an in-process pg_rewind retry apart from a fresh run started by the reconciliation loop. Add the rewindMode field to the wal-restore completion log and an attempt counter to the pg_rewind failure log. Part of #11200 Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
The pg_control backup/restore was only run once, before the first pg_rewind attempt. A failed attempt is exactly the condition it exists to repair, so subsequent retries could still see an empty control file. Move the call inside the retry loop so it runs before every attempt. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
The retry predicate ignores the error and only checks the context, so it retries any pg_rewind failure, including permanent ones like a bad connection string, not only the transient WAL-restore case the surrounding comment describes. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Extract the retry predicate into a named function and add unit tests for it, since Rewind() shells out to a real pg_rewind binary and had no test coverage for its retry logic at all. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Move the pg_rewind write-ordering rationale back to the retry call site; pgRewindShouldRetry only checks context cancellation, so it isn't the right place to explain why retrying the whole operation is safe. Also trim a new e2e test comment that restated the existing "Expected outcome" trailer in different words. Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com>
cnpg-bot
pushed a commit
that referenced
this pull request
Jul 21, 2026
When a demoted primary rejoins the cluster after a failover, the instance manager runs `pg_rewind --restore-target-wal` so the WAL segments no longer in its `pg_wal` can be fetched from the archive. The `restore_command` handed to pg_rewind was the same `manager wal-restore` invocation used during normal recovery, WAL prefetching included. pg_rewind walks the timeline backwards, fetches every file it needs exactly once, and treats any restore failure as fatal: with `wal.maxParallel > 1` a prefetch miss is practically guaranteed, the miss records the end-of-wal-stream flag on disk, and the next invocation consumes the flag and fails without even contacting the object store, aborting the whole rewind. A flag left over from the pod's previous life aborts even the first invocation. To make things worse, a failed pg_rewind was only retried by restarting the whole join under the reconcile exponential backoff, which grows to roughly 1000 seconds, so a transient restore failure could stall the rejoin for hours. `manager wal-restore` now has a `--rewind` flag that disables prefetching and the end-of-wal-stream flag machinery. The instance manager writes a rewind-specific `restore_command` into `override.conf` right before running pg_rewind, and the demotion path rewrites the standard configuration immediately after, so the flag never leaks into normal recovery. pg_rewind itself is now retried in-process with a bounded backoff before the error surfaces, which is safe because pg_rewind collects every WAL segment it needs before writing anything into the target data directory: a failed run leaves the target unchanged. The same context is conveyed to WAL restore plugins through the new `mode` field of `WALRestoreRequest` (cloudnative-pg/cnpg-i#351), and plugin-barman-cloud honors it in cloudnative-pg/plugin-barman-cloud#1007. Plugins that ignore the field keep their current behavior, helped by the in-process retry advancing one segment per attempt. The cnpg-i dependency points to a pseudo-version of that pull request and will move to the next tagged release once it is available. Related to #10419, which shares the flag machinery but is a distinct failure mode and is not closed here. Closes #11200 Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com> Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com> Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com> Co-authored-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com> Co-authored-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com> (cherry picked from commit ad1d52c)
cnpg-bot
pushed a commit
that referenced
this pull request
Jul 21, 2026
When a demoted primary rejoins the cluster after a failover, the instance manager runs `pg_rewind --restore-target-wal` so the WAL segments no longer in its `pg_wal` can be fetched from the archive. The `restore_command` handed to pg_rewind was the same `manager wal-restore` invocation used during normal recovery, WAL prefetching included. pg_rewind walks the timeline backwards, fetches every file it needs exactly once, and treats any restore failure as fatal: with `wal.maxParallel > 1` a prefetch miss is practically guaranteed, the miss records the end-of-wal-stream flag on disk, and the next invocation consumes the flag and fails without even contacting the object store, aborting the whole rewind. A flag left over from the pod's previous life aborts even the first invocation. To make things worse, a failed pg_rewind was only retried by restarting the whole join under the reconcile exponential backoff, which grows to roughly 1000 seconds, so a transient restore failure could stall the rejoin for hours. `manager wal-restore` now has a `--rewind` flag that disables prefetching and the end-of-wal-stream flag machinery. The instance manager writes a rewind-specific `restore_command` into `override.conf` right before running pg_rewind, and the demotion path rewrites the standard configuration immediately after, so the flag never leaks into normal recovery. pg_rewind itself is now retried in-process with a bounded backoff before the error surfaces, which is safe because pg_rewind collects every WAL segment it needs before writing anything into the target data directory: a failed run leaves the target unchanged. The same context is conveyed to WAL restore plugins through the new `mode` field of `WALRestoreRequest` (cloudnative-pg/cnpg-i#351), and plugin-barman-cloud honors it in cloudnative-pg/plugin-barman-cloud#1007. Plugins that ignore the field keep their current behavior, helped by the in-process retry advancing one segment per attempt. The cnpg-i dependency points to a pseudo-version of that pull request and will move to the next tagged release once it is available. Related to #10419, which shares the flag machinery but is a distinct failure mode and is not closed here. Closes #11200 Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com> Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com> Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com> Co-authored-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com> Co-authored-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com> (cherry picked from commit ad1d52c)
leonardoce
added a commit
that referenced
this pull request
Jul 21, 2026
When a demoted primary rejoins the cluster after a failover, the instance manager runs `pg_rewind --restore-target-wal` so the WAL segments no longer in its `pg_wal` can be fetched from the archive. The `restore_command` handed to pg_rewind was the same `manager wal-restore` invocation used during normal recovery, WAL prefetching included. pg_rewind walks the timeline backwards, fetches every file it needs exactly once, and treats any restore failure as fatal: with `wal.maxParallel > 1` a prefetch miss is practically guaranteed, the miss records the end-of-wal-stream flag on disk, and the next invocation consumes the flag and fails without even contacting the object store, aborting the whole rewind. A flag left over from the pod's previous life aborts even the first invocation. To make things worse, a failed pg_rewind was only retried by restarting the whole join under the reconcile exponential backoff, which grows to roughly 1000 seconds, so a transient restore failure could stall the rejoin for hours. `manager wal-restore` now has a `--rewind` flag that disables prefetching and the end-of-wal-stream flag machinery. The instance manager writes a rewind-specific `restore_command` into `override.conf` right before running pg_rewind, and the demotion path rewrites the standard configuration immediately after, so the flag never leaks into normal recovery. pg_rewind itself is now retried in-process with a bounded backoff before the error surfaces, which is safe because pg_rewind collects every WAL segment it needs before writing anything into the target data directory: a failed run leaves the target unchanged. The same context is conveyed to WAL restore plugins through the new `mode` field of `WALRestoreRequest` (cloudnative-pg/cnpg-i#351), and plugin-barman-cloud honors it in cloudnative-pg/plugin-barman-cloud#1007. Plugins that ignore the field keep their current behavior, helped by the in-process retry advancing one segment per attempt. The cnpg-i dependency points to a pseudo-version of that pull request and will move to the next tagged release once it is available. Related to #10419, which shares the flag machinery but is a distinct failure mode and is not closed here. Closes #11200 Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com> Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com> Signed-off-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com> Co-authored-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com> Co-authored-by: Leonardo Cecchi <leonardo.cecchi@enterprisedb.com> (cherry picked from commit ad1d52c) (cherry picked from commit f5f1b8a)
leonardoce
pushed a commit
to cloudnative-pg/plugin-barman-cloud
that referenced
this pull request
Jul 21, 2026
…ry (#1007) When a demoted primary rejoins the cluster after a failover, the instance manager runs `pg_rewind --restore-target-wal`, and the resulting `restore_command` invocations are served by this sidecar exactly as if an instance in recovery were asking. pg_rewind walks the timeline backwards, fetches every WAL file it needs exactly once, and treats any restore failure as fatal, so two optimizations meant for recovery break it: prefetching the following segments is wasted work that, past the end of the timeline, is guaranteed to end in an archive miss, and the end-of-wal-stream flag recorded by that miss makes a later invocation fail without contacting the object store, on a segment the archive actually has, aborting the whole rewind. The restore request now carries the context it is made in (cloudnative-pg/cnpg-i#351). When it says `MODE_REWIND`, the sidecar restores exactly the requested file: no prefetching, no end-of-wal-stream flag check, no flag recording. Requests from operators predating the field keep the current behavior. This is the plugin-side counterpart of cloudnative-pg/cloudnative-pg#11204. The cnpg-i dependency points to a pseudo-version of the protocol pull request and will move to the next tagged release once it is available. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com> Signed-off-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com> Co-authored-by: Marco Nenciarini <marco.nenciarini@enterprisedb.com>
renovate Bot
added a commit
to sdwilsh/ansible-playbooks
that referenced
this pull request
Sep 24, 2026
##### [\`v1.30.1\`](https://github.com/cloudnative-pg/cloudnative-pg/releases/tag/v1.30.1) **Release date:** Sep 23, 2026 ##### Enhancements - Added `restoreAdditionalCommandArgs` to `.barmanObjectStore.data` for in-tree Barman Cloud recoveries, so custom flags are forwarded to `barman-cloud-restore` during recovery. ([#11278](cloudnative-pg/cloudnative-pg#11278)) <!-- 1.29 1.28 --> - Plugin authors no longer need to re-derive whether an empty WAL archive should be treated as an error: the operator now computes the `check_empty_wal_archive` decision once and passes it to CNPG-I plugins on both WAL archiving and restore hooks. ([#11216](cloudnative-pg/cloudnative-pg#11216)) <!-- 1.29 1.28 --> - Allowed setting the `ssl_ecdh_curve` PostgreSQL parameter in `.spec.postgresql.parameters`. It was previously rejected by the validating webhook even though the operator does not manage it. Contributed by [@Jamstah](https://github.com/Jamstah). ([#11308](cloudnative-pg/cloudnative-pg#11308)) <!-- 1.29 1.28 --> - Added support for setting `auth_user` in `spec.pgbouncer.parameters` to override the user PgBouncer uses for its `auth_query` connection, instead of always deriving it from the `authQuerySecret`. Useful with a custom `authQuerySecret` where the role to authenticate as differs from the one implied by the Secret. Contributed by [@jonasbadstuebner](https://github.com/jonasbadstuebner). ([#10899](cloudnative-pg/cloudnative-pg#10899)) <!-- 1.29 1.28 --> - Fixed PgBouncer metrics silently reporting zero when PgBouncer adds or reorders columns: `SHOW STATS` and `SHOW POOLS` results are now matched by column name instead of by position. Contributed by [@AdzerKI](https://github.com/AdzerKI) (`SHOW STATS`). ([#11366](cloudnative-pg/cloudnative-pg#11366), [#11464](cloudnative-pg/cloudnative-pg#11464)) <!-- 1.29 1.28 --> - `cnpg` plugin: - Added a `--dry-run` option to the `backup` command that prints the `Backup` resource that would be created, without creating it. ([#11233](cloudnative-pg/cloudnative-pg#11233)) <!-- 1.29 1.28 --> ##### Security and Supply Chain - Fixed a webhook validation bypass where an absolute extension path with an embedded `..` traversal (e.g. `/a/../../../../etc`) was accepted because only the raw path was checked; such paths still escaped the extension's mount point once resolved at runtime. Affected entries are now resolved and rejected/skipped consistently across admission, the major-upgrade job, and already-persisted Cluster specs. Contributed by [@Anand-240](https://github.com/Anand-240). ([#11210](cloudnative-pg/cloudnative-pg#11210)) <!-- 1.29 1.28 --> - Updated `google.golang.org/grpc` to v1.83.1, fixing [CVE-2026-84304](https://nvd.nist.gov/vuln/detail/CVE-2026-84304) / GHSA-vp52-pcj8-j9qc (gRPC-Go heap memory exhaustion via HTTP/2 DATA frame fragmentation). ([#11394](cloudnative-pg/cloudnative-pg#11394)) <!-- 1.29 1.28 --> ##### Changes - Updated the default PostgreSQL operand image to 18.6-system-trixie. ([#11257](cloudnative-pg/cloudnative-pg#11257)) <!-- 1.29 1.28 --> - Moved the operator's own base image from Debian 12 (bookworm) to Debian 13 (trixie), now that Debian 12 has moved to LTS-only maintenance. ([#11193](cloudnative-pg/cloudnative-pg#11193)) <!-- 1.29 1.28 --> ##### Fixes - Demoted an unreachable old primary immediately during failover, instead of waiting on checkpoint and fast-shutdown attempts that could never complete. Previously, an old primary whose PostgreSQL had become unresponsive was never shut down, so it could never rejoin the cluster as a replica of the new primary. The shutdown is now requested without blocking the reconciler, though demotion can still take up to a minute or more if PostgreSQL keeps ignoring shutdown signals. ([#11347](cloudnative-pg/cloudnative-pg#11347)) <!-- 1.29 1.28 --> - Fixed failover and Pod recreation getting stuck while any instance in the cluster was fenced. A fenced instance kept reporting itself as healthy on its status endpoint even though its Pod was permanently not Ready, which blocked the readiness guard, the primary-election logic, and the Pod-reattachment gate from reacting to an unrelated failure elsewhere in the cluster, leaving it stalled until someone manually removed the fencing. All three now recognize and skip fenced instances. ([#11274](cloudnative-pg/cloudnative-pg#11274)) <!-- 1.29 1.28 --> - Fixed a failover that could stall forever or get silently reverted. While a failover was pending, the reconciler could skip checking whether it was ready to complete, or mistake the pending state for a missing Pod and reset the target primary back to the unhealthy one. It now keeps progressing a pending failover and completes it once the WAL receivers are down, rechecking quorum safety right before promoting in case it changed while waiting. ([#11336](cloudnative-pg/cloudnative-pg#11336)) <!-- 1.29 1.28 --> - Fixed the designated-primary sort in `PostgresqlStatusList` so a tie on LSN no longer sorts it out of order. If the designated primary's Pod later disappeared from the status list, this could have led the operator to promote the wrong standby. ([#11494](cloudnative-pg/cloudnative-pg#11494)) <!-- 1.29 1.28 --> - Gated PostgreSQL startup on the instance manager actually holding the primary `Lease`, closing a window where a restarting or rescheduled primary could start accepting writes before acquiring the lease. Lease acquisition now happens during instance initialization, strictly before PostgreSQL is allowed to start, instead of only later in the reconciliation loop once the instance was already reported ready. ([#11356](cloudnative-pg/cloudnative-pg#11356)) - Bounded the primary lease take-over `Get` call to the lease's `RenewDeadline`, so a call against an unreachable API server now fails fast and retries instead of blocking indefinitely. Left unbounded, the same stalled call could also delay a primary's own lease renewal after a transient API server blip, risking an unwanted takeover by another instance once the lease appeared to expire. ([#11350](cloudnative-pg/cloudnative-pg#11350)) - Stopped abandoning a fast shutdown after 60 seconds. The request carried no explicit timeout, so `pg_ctl` fell back to its own implicit 60-second limit, and the instance manager, being PID 1 in the container, took PostgreSQL down mid-shutdown regardless of a larger configured `stopDelay`. The fast phase now waits for the full `stopDelay` instead. ([#11263](cloudnative-pg/cloudnative-pg#11263)) <!-- 1.29 1.28 --> - Waited for certificate status before refreshing secrets during bootstrap. ([#11450](cloudnative-pg/cloudnative-pg#11450)) <!-- 1.29 1.28 --> - Fixed in-tree Barman Cloud recovery ignoring `wal.maxParallel` and `wal.restoreAdditionalCommandArgs`, fetching WAL one segment at a time instead of using the configured parallelism. ([#11105](cloudnative-pg/cloudnative-pg#11105)) <!-- 1.29 1.28 --> - Honored the `unrecoverable` annotation on pending and terminating instances. ([#11241](cloudnative-pg/cloudnative-pg#11241)) <!-- 1.29 1.28 --> - Honored `Backup.spec.online` when finalizing VolumeSnapshot-based backups, so a cold backup is no longer reported as online in status. ([#11276](cloudnative-pg/cloudnative-pg#11276)) <!-- 1.29 1.28 --> - Set `--no-data-checksums` explicitly when `.spec.postgresql.dataChecksums` is `false` on PostgreSQL 18+, which now enables checksums by default in `initdb`. Contributed by [@evan314159](https://github.com/evan314159). ([#11488](cloudnative-pg/cloudnative-pg#11488)) <!-- 1.29 1.28 --> - Stopped discarding `.spec.inheritedMetadata` labels on PVC reconciliation; they were applied before, and then overwritten by, the operator's own labels. Contributed by [@GellertBabel](https://github.com/GellertBabel). ([#11499](cloudnative-pg/cloudnative-pg#11499)) <!-- 1.29 1.28 --> - Stopped `Pooler` and `DatabaseRole` from recreating their owned resources after being marked for deletion, which could block an external finalizer (e.g. ArgoCD foreground deletion) waiting for those resources to disappear. Contributed by [@mjlshen](https://github.com/mjlshen). ([#11359](cloudnative-pg/cloudnative-pg#11359)) <!-- 1.29 1.28 --> - Handled a `NULL` `stats_reset` value in the `pg_stat_wal` metric, which could occasionally break Prometheus scraping on PostgreSQL 18 clusters. Contributed by [@HoniSanders](https://github.com/HoniSanders). ([#11207](cloudnative-pg/cloudnative-pg#11207)) <!-- 1.29 1.28 --> - Recovered log streaming when a stale non-FIFO file blocked the log pipe. ([#11246](cloudnative-pg/cloudnative-pg#11246)) <!-- 1.29 1.28 --> - Stopped sampling logs of the Pods' processes, so no log lines are silently dropped. ([#11131](cloudnative-pg/cloudnative-pg#11131)) <!-- 1.29 1.28 --> - Stopped caching Helm release Secrets in the operator, which could spike memory usage and crashloop the operator on clusters with large Helm releases installed alongside it. ([#11211](cloudnative-pg/cloudnative-pg#11211)) <!-- 1.29 1.28 --> - Re-ran owned-resource finalizer cleanup after an operator restart, so a namespace deleted while the operator was down doesn't hang in `Terminating` forever. ([#11215](cloudnative-pg/cloudnative-pg#11215)) <!-- 1.29 1.28 --> - Fixed rejoining replicas taking a long time to complete `pg_rewind` when WAL prefetching (`wal.maxParallel > 1`) was enabled. ([#11204](cloudnative-pg/cloudnative-pg#11204)) <!-- 1.29 1.28 --> - Treated a PostgreSQL instance rejecting connections as started up in probes. ([#11152](cloudnative-pg/cloudnative-pg#11152)) <!-- 1.29 1.28 --> - Stopped issuing a redundant PATCH on every reconciliation when a cluster's `pvcTemplate` set `volumeAttributesClass`. Contributed by [@andrew-farries](https://github.com/andrew-farries). ([#11196](cloudnative-pg/cloudnative-pg#11196)) <!-- 1.29 1.28 --> - Stopped halting replication slot reconciliation at the first failing slot. ([#11119](cloudnative-pg/cloudnative-pg#11119)) <!-- 1.29 1.28 --> - During rollouts, compared the running configuration hash, rather than the on-disk one, avoiding unnecessary switchovers. ([#11130](cloudnative-pg/cloudnative-pg#11130)) <!-- 1.29 1.28 --> - Retried `VolumeSnapshot` errors until the deadline, instead of giving up early. ([#11132](cloudnative-pg/cloudnative-pg#11132)) <!-- 1.29 1.28 --> - Symlinked `pg_wal` to the WAL volume during plugin-based recovery, instead of replaying WAL onto the data volume, which could exhaust it on a large point-in-time recovery. ([#11127](cloudnative-pg/cloudnative-pg#11127)) <!-- 1.29 1.28 --> - Corrected grammar in `PhaseUpgradeDelayed` status messages. Contributed by [@Utkarsh-sharma47](https://github.com/Utkarsh-sharma47). ([#11065](cloudnative-pg/cloudnative-pg#11065)) <!-- 1.29 1.28 --> - CloudNativePG Interface (CNPG-I): - Reported the Postgres capabilities of a plugin correctly. ([#11447](cloudnative-pg/cloudnative-pg#11447)) <!-- 1.29 1.28 --> - Prevented the plugin status field from flapping on Cluster reconciliation. Contributed by [@sharifmshaker](https://github.com/sharifmshaker). ([#11386](cloudnative-pg/cloudnative-pg#11386)) <!-- 1.29 1.28 --> - Replaced a finalizer with in-memory cleanup on Service deletion. This avoids namespaces hanging in `Terminating` state when the operator and its plugins run in the same namespace. Contributed by [@maxlengdell](https://github.com/maxlengdell). ([#10940](cloudnative-pg/cloudnative-pg#10940)) <!-- 1.29 --> - `cnpg` plugin: - Used UTC time zone for the default backup name. ([#11329](cloudnative-pg/cloudnative-pg#11329)) <!-- 1.29 1.28 --> - Deleted PVCs before Pods in `cnpg destroy`, avoiding a race that could leave dangling PVCs needing a second invocation to clean up. Contributed by [@daanvinken](https://github.com/daanvinken). ([#10847](cloudnative-pg/cloudnative-pg#10847)) <!-- 1.29 1.28 -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

When a demoted primary rejoins the cluster after a failover, the instance manager runs
pg_rewind --restore-target-walso the WAL segments no longer in itspg_walcan be fetched from the archive. Therestore_commandhanded to pg_rewind was the samemanager wal-restoreinvocation used during normal recovery, WAL prefetching included. pg_rewind walks the timeline backwards, fetches every file it needs exactly once, and treats any restore failure as fatal: withwal.maxParallel > 1a prefetch miss is practically guaranteed, the miss records the end-of-wal-stream flag on disk, and the next invocation consumes the flag and fails without even contacting the object store, aborting the whole rewind. A flag left over from the pod's previous life aborts even the first invocation. To make things worse, a failed pg_rewind was only retried by restarting the whole join under the reconcile exponential backoff, which grows to roughly 1000 seconds, so a transient restore failure could stall the rejoin for hours.manager wal-restorenow has a--rewindflag that disables prefetching and the end-of-wal-stream flag machinery. The instance manager writes a rewind-specificrestore_commandintooverride.confright before running pg_rewind, and the demotion path rewrites the standard configuration immediately after, so the flag never leaks into normal recovery. pg_rewind itself is now retried in-process with a bounded backoff before the error surfaces, which is safe because pg_rewind collects every WAL segment it needs before writing anything into the target data directory: a failed run leaves the target unchanged.The same context is conveyed to WAL restore plugins through the new
modefield ofWALRestoreRequest(cloudnative-pg/cnpg-i#351), and plugin-barman-cloud honors it in cloudnative-pg/plugin-barman-cloud#1007. Plugins that ignore the field keep their current behavior, helped by the in-process retry advancing one segment per attempt. The cnpg-i dependency points to a pseudo-version of that pull request and will move to the next tagged release once it is available.Related to #10419, which shares the flag machinery but is a distinct failure mode and is not closed here.
Closes #11200