{{ message }}
Repository navigation
fix: delete PVCs before pod in cnpg destroy to prevent race condition - #10847
Merged
litaocdl merged 3 commits intoAug 12, 2026
Merged
Conversation
Contributor
daanvinken
force-pushed
the
fix/destroy-delete-order
branch
from
June 16, 2026 16:58
941bb4b to
57a7892
Compare
|
Would be great to get in. I got confused today when using |
NiccoloFei
force-pushed
the
fix/destroy-delete-order
branch
from
July 21, 2026 08:53
57a7892 to
9a58c8b
Compare
armru
force-pushed
the
fix/destroy-delete-order
branch
from
July 23, 2026 09:00
9839099 to
81d2286
Compare
Member
|
/test |
Contributor
|
@armru, here's the link to the E2E on CNPG workflow run: https://github.com/cloudnative-pg/cloudnative-pg/actions/runs/29993491415 |
armru
approved these changes
Jul 23, 2026
NiccoloFei
force-pushed
the
fix/destroy-delete-order
branch
from
August 5, 2026 11:39
81d2286 to
50256d1
Compare
litaocdl
force-pushed
the
fix/destroy-delete-order
branch
from
August 11, 2026 10:21
50256d1 to
29945a1
Compare
Collaborator
|
/test |
Contributor
The destroy command deleted the pod first, then the PVCs. Between those calls, the operator's reconcile loop could see dangling PVCs and recreate the pod, requiring a second invocation of destroy to clean up. Delete PVCs before the pod to match the deletion order used by the operator's internal ensureInstanceIsDeleted in cluster_scale.go. Signed-off-by: Daan Vinken <daanvinken@tythus.com>
…pg destroy The previous commit's restructure skipped job cleanup when --keep-pvc is used, even though that cleanup was introduced by cloudnative-pg#5298 to remove jobs left over from the instance. The keep-pvc path also still deleted the pod before detaching the PVCs, leaving the same reattach window the previous commit closes for the delete path. Unify both paths so PVCs are handled first, then the pod, then the jobs, regardless of --keep-pvc, and cover the command with unit tests. Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com>
pvc usually have finalizer to have pod deleted first. Delete the PVC here works because we are using kubernetes client API, which sends the delete request and returns after kubernetes accepts it, add comments here and unit test, to make sure no one in future update the code to ensure the PVC is deleted before deleting pod Signed-off-by: Tao Li <tao.li@enterprisedb.com>
litaocdl
force-pushed
the
fix/destroy-delete-order
branch
from
August 12, 2026 03:04
522c5b0 to
6654be9
Compare
litaocdl
approved these changes
Aug 12, 2026
cnpg-bot
pushed a commit
that referenced
this pull request
Aug 12, 2026
`cnpg destroy` deleted the pod before deleting its PVCs. Between those operations, the operator could reconcile, see dangling PVCs, and recreate the pod. That made a second `destroy` invocation necessary to finish cleanup. Delete the PVCs first so the operator no longer observes reusable dangling PVCs for the instance before the pod is removed. PVCs may have the `kubernetes.io/pvc-protection` finalizer while still mounted by the pod. This ordering is still safe because the plugin uses the Kubernetes client-go API directly: the delete call sends a DELETE request and returns once the request is accepted. It does not wait for the PVC to disappear. Protected PVCs remain in `Terminating` until the pod is deleted, then Kubernetes can remove the finalizer and complete PVC deletion. Closes #10846 Signed-off-by: Daan Vinken <daanvinken@tythus.com> Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com> Signed-off-by: Tao Li <tao.li@enterprisedb.com> Co-authored-by: Armando Ruocco <armando.ruocco@enterprisedb.com> Co-authored-by: Tao Li <tao.li@enterprisedb.com> (cherry picked from commit 791af39)
cnpg-bot
pushed a commit
that referenced
this pull request
Aug 12, 2026
`cnpg destroy` deleted the pod before deleting its PVCs. Between those operations, the operator could reconcile, see dangling PVCs, and recreate the pod. That made a second `destroy` invocation necessary to finish cleanup. Delete the PVCs first so the operator no longer observes reusable dangling PVCs for the instance before the pod is removed. PVCs may have the `kubernetes.io/pvc-protection` finalizer while still mounted by the pod. This ordering is still safe because the plugin uses the Kubernetes client-go API directly: the delete call sends a DELETE request and returns once the request is accepted. It does not wait for the PVC to disappear. Protected PVCs remain in `Terminating` until the pod is deleted, then Kubernetes can remove the finalizer and complete PVC deletion. Closes #10846 Signed-off-by: Daan Vinken <daanvinken@tythus.com> Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com> Signed-off-by: Tao Li <tao.li@enterprisedb.com> Co-authored-by: Armando Ruocco <armando.ruocco@enterprisedb.com> Co-authored-by: Tao Li <tao.li@enterprisedb.com> (cherry picked from commit 791af39)
cnpg-bot
pushed a commit
that referenced
this pull request
Aug 12, 2026
`cnpg destroy` deleted the pod before deleting its PVCs. Between those operations, the operator could reconcile, see dangling PVCs, and recreate the pod. That made a second `destroy` invocation necessary to finish cleanup. Delete the PVCs first so the operator no longer observes reusable dangling PVCs for the instance before the pod is removed. PVCs may have the `kubernetes.io/pvc-protection` finalizer while still mounted by the pod. This ordering is still safe because the plugin uses the Kubernetes client-go API directly: the delete call sends a DELETE request and returns once the request is accepted. It does not wait for the PVC to disappear. Protected PVCs remain in `Terminating` until the pod is deleted, then Kubernetes can remove the finalizer and complete PVC deletion. Closes #10846 Signed-off-by: Daan Vinken <daanvinken@tythus.com> Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com> Signed-off-by: Tao Li <tao.li@enterprisedb.com> Co-authored-by: Armando Ruocco <armando.ruocco@enterprisedb.com> Co-authored-by: Tao Li <tao.li@enterprisedb.com> (cherry picked from commit 791af39)
litaocdl
added a commit
that referenced
this pull request
Aug 13, 2026
`cnpg destroy` deleted the pod before deleting its PVCs. Between those operations, the operator could reconcile, see dangling PVCs, and recreate the pod. That made a second `destroy` invocation necessary to finish cleanup. Delete the PVCs first so the operator no longer observes reusable dangling PVCs for the instance before the pod is removed. PVCs may have the `kubernetes.io/pvc-protection` finalizer while still mounted by the pod. This ordering is still safe because the plugin uses the Kubernetes client-go API directly: the delete call sends a DELETE request and returns once the request is accepted. It does not wait for the PVC to disappear. Protected PVCs remain in `Terminating` until the pod is deleted, then Kubernetes can remove the finalizer and complete PVC deletion. Closes #10846 Signed-off-by: Daan Vinken <daanvinken@tythus.com> Signed-off-by: Armando Ruocco <armando.ruocco@enterprisedb.com> Signed-off-by: Tao Li <tao.li@enterprisedb.com> Co-authored-by: Armando Ruocco <armando.ruocco@enterprisedb.com> Co-authored-by: Tao Li <tao.li@enterprisedb.com>
renovate Bot
added a commit
to sdwilsh/ansible-playbooks
that referenced
this pull request
Sep 24, 2026
##### [\`v1.30.1\`](https://github.com/cloudnative-pg/cloudnative-pg/releases/tag/v1.30.1) **Release date:** Sep 23, 2026 ##### Enhancements - Added `restoreAdditionalCommandArgs` to `.barmanObjectStore.data` for in-tree Barman Cloud recoveries, so custom flags are forwarded to `barman-cloud-restore` during recovery. ([#11278](cloudnative-pg/cloudnative-pg#11278)) <!-- 1.29 1.28 --> - Plugin authors no longer need to re-derive whether an empty WAL archive should be treated as an error: the operator now computes the `check_empty_wal_archive` decision once and passes it to CNPG-I plugins on both WAL archiving and restore hooks. ([#11216](cloudnative-pg/cloudnative-pg#11216)) <!-- 1.29 1.28 --> - Allowed setting the `ssl_ecdh_curve` PostgreSQL parameter in `.spec.postgresql.parameters`. It was previously rejected by the validating webhook even though the operator does not manage it. Contributed by [@Jamstah](https://github.com/Jamstah). ([#11308](cloudnative-pg/cloudnative-pg#11308)) <!-- 1.29 1.28 --> - Added support for setting `auth_user` in `spec.pgbouncer.parameters` to override the user PgBouncer uses for its `auth_query` connection, instead of always deriving it from the `authQuerySecret`. Useful with a custom `authQuerySecret` where the role to authenticate as differs from the one implied by the Secret. Contributed by [@jonasbadstuebner](https://github.com/jonasbadstuebner). ([#10899](cloudnative-pg/cloudnative-pg#10899)) <!-- 1.29 1.28 --> - Fixed PgBouncer metrics silently reporting zero when PgBouncer adds or reorders columns: `SHOW STATS` and `SHOW POOLS` results are now matched by column name instead of by position. Contributed by [@AdzerKI](https://github.com/AdzerKI) (`SHOW STATS`). ([#11366](cloudnative-pg/cloudnative-pg#11366), [#11464](cloudnative-pg/cloudnative-pg#11464)) <!-- 1.29 1.28 --> - `cnpg` plugin: - Added a `--dry-run` option to the `backup` command that prints the `Backup` resource that would be created, without creating it. ([#11233](cloudnative-pg/cloudnative-pg#11233)) <!-- 1.29 1.28 --> ##### Security and Supply Chain - Fixed a webhook validation bypass where an absolute extension path with an embedded `..` traversal (e.g. `/a/../../../../etc`) was accepted because only the raw path was checked; such paths still escaped the extension's mount point once resolved at runtime. Affected entries are now resolved and rejected/skipped consistently across admission, the major-upgrade job, and already-persisted Cluster specs. Contributed by [@Anand-240](https://github.com/Anand-240). ([#11210](cloudnative-pg/cloudnative-pg#11210)) <!-- 1.29 1.28 --> - Updated `google.golang.org/grpc` to v1.83.1, fixing [CVE-2026-84304](https://nvd.nist.gov/vuln/detail/CVE-2026-84304) / GHSA-vp52-pcj8-j9qc (gRPC-Go heap memory exhaustion via HTTP/2 DATA frame fragmentation). ([#11394](cloudnative-pg/cloudnative-pg#11394)) <!-- 1.29 1.28 --> ##### Changes - Updated the default PostgreSQL operand image to 18.6-system-trixie. ([#11257](cloudnative-pg/cloudnative-pg#11257)) <!-- 1.29 1.28 --> - Moved the operator's own base image from Debian 12 (bookworm) to Debian 13 (trixie), now that Debian 12 has moved to LTS-only maintenance. ([#11193](cloudnative-pg/cloudnative-pg#11193)) <!-- 1.29 1.28 --> ##### Fixes - Demoted an unreachable old primary immediately during failover, instead of waiting on checkpoint and fast-shutdown attempts that could never complete. Previously, an old primary whose PostgreSQL had become unresponsive was never shut down, so it could never rejoin the cluster as a replica of the new primary. The shutdown is now requested without blocking the reconciler, though demotion can still take up to a minute or more if PostgreSQL keeps ignoring shutdown signals. ([#11347](cloudnative-pg/cloudnative-pg#11347)) <!-- 1.29 1.28 --> - Fixed failover and Pod recreation getting stuck while any instance in the cluster was fenced. A fenced instance kept reporting itself as healthy on its status endpoint even though its Pod was permanently not Ready, which blocked the readiness guard, the primary-election logic, and the Pod-reattachment gate from reacting to an unrelated failure elsewhere in the cluster, leaving it stalled until someone manually removed the fencing. All three now recognize and skip fenced instances. ([#11274](cloudnative-pg/cloudnative-pg#11274)) <!-- 1.29 1.28 --> - Fixed a failover that could stall forever or get silently reverted. While a failover was pending, the reconciler could skip checking whether it was ready to complete, or mistake the pending state for a missing Pod and reset the target primary back to the unhealthy one. It now keeps progressing a pending failover and completes it once the WAL receivers are down, rechecking quorum safety right before promoting in case it changed while waiting. ([#11336](cloudnative-pg/cloudnative-pg#11336)) <!-- 1.29 1.28 --> - Fixed the designated-primary sort in `PostgresqlStatusList` so a tie on LSN no longer sorts it out of order. If the designated primary's Pod later disappeared from the status list, this could have led the operator to promote the wrong standby. ([#11494](cloudnative-pg/cloudnative-pg#11494)) <!-- 1.29 1.28 --> - Gated PostgreSQL startup on the instance manager actually holding the primary `Lease`, closing a window where a restarting or rescheduled primary could start accepting writes before acquiring the lease. Lease acquisition now happens during instance initialization, strictly before PostgreSQL is allowed to start, instead of only later in the reconciliation loop once the instance was already reported ready. ([#11356](cloudnative-pg/cloudnative-pg#11356)) - Bounded the primary lease take-over `Get` call to the lease's `RenewDeadline`, so a call against an unreachable API server now fails fast and retries instead of blocking indefinitely. Left unbounded, the same stalled call could also delay a primary's own lease renewal after a transient API server blip, risking an unwanted takeover by another instance once the lease appeared to expire. ([#11350](cloudnative-pg/cloudnative-pg#11350)) - Stopped abandoning a fast shutdown after 60 seconds. The request carried no explicit timeout, so `pg_ctl` fell back to its own implicit 60-second limit, and the instance manager, being PID 1 in the container, took PostgreSQL down mid-shutdown regardless of a larger configured `stopDelay`. The fast phase now waits for the full `stopDelay` instead. ([#11263](cloudnative-pg/cloudnative-pg#11263)) <!-- 1.29 1.28 --> - Waited for certificate status before refreshing secrets during bootstrap. ([#11450](cloudnative-pg/cloudnative-pg#11450)) <!-- 1.29 1.28 --> - Fixed in-tree Barman Cloud recovery ignoring `wal.maxParallel` and `wal.restoreAdditionalCommandArgs`, fetching WAL one segment at a time instead of using the configured parallelism. ([#11105](cloudnative-pg/cloudnative-pg#11105)) <!-- 1.29 1.28 --> - Honored the `unrecoverable` annotation on pending and terminating instances. ([#11241](cloudnative-pg/cloudnative-pg#11241)) <!-- 1.29 1.28 --> - Honored `Backup.spec.online` when finalizing VolumeSnapshot-based backups, so a cold backup is no longer reported as online in status. ([#11276](cloudnative-pg/cloudnative-pg#11276)) <!-- 1.29 1.28 --> - Set `--no-data-checksums` explicitly when `.spec.postgresql.dataChecksums` is `false` on PostgreSQL 18+, which now enables checksums by default in `initdb`. Contributed by [@evan314159](https://github.com/evan314159). ([#11488](cloudnative-pg/cloudnative-pg#11488)) <!-- 1.29 1.28 --> - Stopped discarding `.spec.inheritedMetadata` labels on PVC reconciliation; they were applied before, and then overwritten by, the operator's own labels. Contributed by [@GellertBabel](https://github.com/GellertBabel). ([#11499](cloudnative-pg/cloudnative-pg#11499)) <!-- 1.29 1.28 --> - Stopped `Pooler` and `DatabaseRole` from recreating their owned resources after being marked for deletion, which could block an external finalizer (e.g. ArgoCD foreground deletion) waiting for those resources to disappear. Contributed by [@mjlshen](https://github.com/mjlshen). ([#11359](cloudnative-pg/cloudnative-pg#11359)) <!-- 1.29 1.28 --> - Handled a `NULL` `stats_reset` value in the `pg_stat_wal` metric, which could occasionally break Prometheus scraping on PostgreSQL 18 clusters. Contributed by [@HoniSanders](https://github.com/HoniSanders). ([#11207](cloudnative-pg/cloudnative-pg#11207)) <!-- 1.29 1.28 --> - Recovered log streaming when a stale non-FIFO file blocked the log pipe. ([#11246](cloudnative-pg/cloudnative-pg#11246)) <!-- 1.29 1.28 --> - Stopped sampling logs of the Pods' processes, so no log lines are silently dropped. ([#11131](cloudnative-pg/cloudnative-pg#11131)) <!-- 1.29 1.28 --> - Stopped caching Helm release Secrets in the operator, which could spike memory usage and crashloop the operator on clusters with large Helm releases installed alongside it. ([#11211](cloudnative-pg/cloudnative-pg#11211)) <!-- 1.29 1.28 --> - Re-ran owned-resource finalizer cleanup after an operator restart, so a namespace deleted while the operator was down doesn't hang in `Terminating` forever. ([#11215](cloudnative-pg/cloudnative-pg#11215)) <!-- 1.29 1.28 --> - Fixed rejoining replicas taking a long time to complete `pg_rewind` when WAL prefetching (`wal.maxParallel > 1`) was enabled. ([#11204](cloudnative-pg/cloudnative-pg#11204)) <!-- 1.29 1.28 --> - Treated a PostgreSQL instance rejecting connections as started up in probes. ([#11152](cloudnative-pg/cloudnative-pg#11152)) <!-- 1.29 1.28 --> - Stopped issuing a redundant PATCH on every reconciliation when a cluster's `pvcTemplate` set `volumeAttributesClass`. Contributed by [@andrew-farries](https://github.com/andrew-farries). ([#11196](cloudnative-pg/cloudnative-pg#11196)) <!-- 1.29 1.28 --> - Stopped halting replication slot reconciliation at the first failing slot. ([#11119](cloudnative-pg/cloudnative-pg#11119)) <!-- 1.29 1.28 --> - During rollouts, compared the running configuration hash, rather than the on-disk one, avoiding unnecessary switchovers. ([#11130](cloudnative-pg/cloudnative-pg#11130)) <!-- 1.29 1.28 --> - Retried `VolumeSnapshot` errors until the deadline, instead of giving up early. ([#11132](cloudnative-pg/cloudnative-pg#11132)) <!-- 1.29 1.28 --> - Symlinked `pg_wal` to the WAL volume during plugin-based recovery, instead of replaying WAL onto the data volume, which could exhaust it on a large point-in-time recovery. ([#11127](cloudnative-pg/cloudnative-pg#11127)) <!-- 1.29 1.28 --> - Corrected grammar in `PhaseUpgradeDelayed` status messages. Contributed by [@Utkarsh-sharma47](https://github.com/Utkarsh-sharma47). ([#11065](cloudnative-pg/cloudnative-pg#11065)) <!-- 1.29 1.28 --> - CloudNativePG Interface (CNPG-I): - Reported the Postgres capabilities of a plugin correctly. ([#11447](cloudnative-pg/cloudnative-pg#11447)) <!-- 1.29 1.28 --> - Prevented the plugin status field from flapping on Cluster reconciliation. Contributed by [@sharifmshaker](https://github.com/sharifmshaker). ([#11386](cloudnative-pg/cloudnative-pg#11386)) <!-- 1.29 1.28 --> - Replaced a finalizer with in-memory cleanup on Service deletion. This avoids namespaces hanging in `Terminating` state when the operator and its plugins run in the same namespace. Contributed by [@maxlengdell](https://github.com/maxlengdell). ([#10940](cloudnative-pg/cloudnative-pg#10940)) <!-- 1.29 --> - `cnpg` plugin: - Used UTC time zone for the default backup name. ([#11329](cloudnative-pg/cloudnative-pg#11329)) <!-- 1.29 1.28 --> - Deleted PVCs before Pods in `cnpg destroy`, avoiding a race that could leave dangling PVCs needing a second invocation to clean up. Contributed by [@daanvinken](https://github.com/daanvinken). ([#10847](cloudnative-pg/cloudnative-pg#10847)) <!-- 1.29 1.28 -->
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

cnpg destroydeletes the pod before PVCs. Between those calls, the operator can reconcile, see dangling PVCs, and recreate the pod. This requires a second invocation ofdestroyto clean up.Reorder to delete PVCs first, matching
ensureInstanceIsDeletedincluster_scale.go.Reproducible with node-local storage (e.g. OpenEBS LVM): cordon a replica's node, delete the pod so it goes Pending, then run
cnpg destroy.Fixes #10846