Activity

1. dosubot commented on Jul 3, 2026

   

   dosubotBot

   on Jul 3, 2026 – with DosuBot

   More actions

   Thanks for the detailed write-up, @magnus-longva-bouvet. Your analysis is accurate — this is a real gap in the current implementation.

   To confirm your findings: reconcileCheckWalArchiveFile only removes the .check-empty-wal-archive flag when ContinuousArchiving becomes True [1], and the flag is only ever created during bootstrap operations (initdb, restore, pg_basebackup) [2] [3]. There is no mechanism to re-create it when a WAL-archiver plugin is added to a running cluster.

   On a plugin-less cluster, the built-in archiver returns success as a no-op when no backup is configured [4], which causes the ContinuousArchiving condition to become True and removes the flag file within minutes of bootstrap. By the time the plugin is added, the sentinel is long gone, and the plugin's CheckBackupDestination gate (IsEmptyWalArchiveCheckEnabled && checkFileExisting) [5] is never satisfied.

   The related thread plugin-barman-cloud#350 describes the inverse scenario (check fires when it shouldn't), and your comment there correctly identifies that the fix in #457 addressed the annotation behavior but not this path. PR #11032 (merged 2026-06-26) fixes the sidecar injection deadlock when enabling the plugin on an existing cluster, but does not address re-creating the sentinel file for the empty-archive check.

   A potential fix would likely involve either:

   • Re-creating .check-empty-wal-archive when a WAL-archiver plugin is added (e.g., in the operator's plugin reconciliation path), or
   • Having the plugin itself detect that it's being invoked for the first time (no prior successful archive) and run the check regardless of the flag file.

   To reply, just mention @dosu.

   ---

   Share context across your team and agents. Try Dosu.

   
2. ichibahn commented on Jul 6, 2026

   

   ichibahn

   on Jul 6, 2026

   Last edited by ichibahn

   More actions

   I can confirm this reproduces on current main (025a9be53) with plugin-barman-cloud v0.13.0 (kind + MinIO), and I'd like to work on it.

   Reproduction summary — two clusters with the identical final spec, differing only in ordering:

   	Plugin at bootstrap	Plugin added later
   Empty-archive check	barman-cloud-check-wal-archive runs, fails with Expected empty archive, archiving blocked	Never runs — sidecar log goes straight to barman-cloud-wal-archive
   Result	Operator refuses to co-mingle	New WALs written silently next to pre-existing objects, ContinuousArchiving=True throughout

   Repro steps and logs

   1. Cluster Manifests

   cluster-buggy (Bootstrapped without plugins, patched later):

   apiVersion: postgresql.cnpg.io/v1
   kind: Cluster
   metadata:
     name: cluster-buggy
     namespace: repro-11126
   spec:
     instances: 1
     storage:
       size: 1Gi

   Patch applied to cluster-buggy later (11-add-plugin-patch.yaml):

     plugins:
       - name: barman-cloud.cloudnative-pg.io
         enabled: true
         isWALArchiver: true
         parameters:
           barmanObjectName: minio-store
   

   cluster-control (Control group - bootstrapped with the plugin from the start):

   kind: Cluster
   metadata:
     name: cluster-control
     namespace: repro-11126
   spec:
     instances: 1
     storage:
       size: 1Gi
     plugins:
       - name: barman-cloud.cloudnative-pg.io
         enabled: true
         isWALArchiver: true
         parameters:
           barmanObjectName: minio-store
   

   2. Timeline & Conditions Transition

   cluster-buggy
   05:35:25Z: Initial bootstrap completes (Initialized=True).

   05:45:38Z: Reaches ContinuousArchiving=True with reason ContinuousArchivingSuccess via the no-op path. This prematurely deletes the .check-empty-wal-archive sentinel file.

   After Patch: The plugin sidecar is added, but the sentinel is already gone, leaving no trace for the empty check to fire:

   ls: cannot access '/var/lib/postgresql/data/pgdata/.check-empty-wal-archive': No such file or directory
   

   cluster-control
   05:49:43Z: Bootstrap completes.

   05:50:04Z: Correctly runs the empty archive check, detects existing data, and blocks archiving by transitioning to ContinuousArchiving=False (ContinuousArchivingFailing). The sentinel file remains intact:

   -rw------- 1 postgres tape 0 Jul  6 05:49 /var/lib/postgresql/data/pgdata/.check-empty-wal-archive
   

   3. Sidecar Log Snippets

   cluster-buggy-1 (Skips validation, archives directly)

   {"level":"info","ts":"2026-07-06T05:45:38.151381737Z","msg":"Executing barman-cloud-wal-archive","logging_pod":"cluster-buggy-1","walName":"/var/lib/postgresql/data/pgdata/pg_wal/000000010000000000000004","options":["--endpoint-url","http://minio:9000","--cloud-provider","aws-s3","s3://backups/","cluster-buggy",...]}
   {"level":"info","ts":"2026-07-06T05:45:38.534089566Z","msg":"Archived WAL file","walName":"/var/lib/postgresql/data/pgdata/pg_wal/000000010000000000000004",...}
   

   cluster-control-1 (Executes check and catches the collision)

   {"level":"info","ts":"2026-07-06T05:50:04.434819892Z","logger":"barman-cloud-check-wal-archive","msg":"2026-07-06 05:50:04,434 [40] ERROR: WAL archive check failed for server cluster-control: Expected empty archive","pipe":"stderr","logging_pod":"cluster-control-1"}
   {"level":"error","ts":"2026-07-06T05:50:04.463555309Z","msg":"Error invoking barman-cloud-check-wal-archive","logging_pod":"cluster-control-1","error":"exit status 1"}
   

   4. MinIO Bucket Listing
      The destination bucket already contained a pre-existing object (...0001). cluster-buggy silently co-mingled its own new WAL segments (...0004, ...0005) right next to it without warning:

   [2026-07-06 05:45:38 UTC]  16MiB STANDARD wals/0000000100000000/000000010000000000000004  # Overwritten/Added silently by buggy cluster
   [2026-07-06 05:46:38 UTC]  16MiB STANDARD wals/0000000100000000/000000010000000000000005  # Overwritten/Added silently by buggy cluster
   

   Root cause, as I traced it: the check is gated on the .check-empty-wal-archive sentinel, which is removed by reconcileCheckWalArchiveFile once ContinuousArchiving=True. But archiver-less clusters also reach ContinuousArchiving=True (same Success reason) via the no-op wal-archive path, so the sentinel is consumed within minutes of bootstrap — long before any real archiver exists. When a WAL-archiver plugin is enabled later, nothing re-creates the sentinel, so the plugin-side check (IsEmptyWalArchiveCheckEnabled && sentinel exists) can never fire.

   Proposed approach (two changes that need to land together):

   1. Re-create/keep the sentinel from the instance reconciler whenever the cluster has no WAL archiver configured. Detection would be conservative — treat any enabled plugin as a potential archiver, since archiveWALViaPlugins also invokes legacy plugins that don't declare isWALArchiver. A false "has archiver" only means the check runs once more; a false "no archiver" could clobber a working setup.

   2. Distinguish no-op success in the condition reason: the no-op path would report ContinuousArchiving=True with a new reason (e.g. ContinuousArchivingNotConfigured) instead of ContinuousArchivingSuccess, and sentinel removal would be gated on True && reason==Success.

   Change (1) alone is insufficient: the stale Success condition left over from the no-op era deletes the re-created sentinel on the next reconcile, before a newly enabled archiver runs its first check — I could observe this ordering in the repro. (2) closes that race. Since the condition is overwritten on every reconcile, existing archiver-less clusters self-heal to the new reason after upgrade with no migration needed.

   Compatibility: Status stays True for archiver-less clusters, so kubectl wait --for=condition=ContinuousArchiving, alerting rules, and the e2e helpers are unaffected; only reason/message change. The estimated non-test diff is ~40 lines across 4 files, plus new unit tests for reconcileCheckWalArchiveFile (currently uncovered), including a regression test for the race above.

   Two questions before I open a PR:

   1. Are you comfortable with the reason change for archiver-less clusters, or would you prefer keeping Success and gating sentinel removal some other way (e.g. persisting archiver-configured state elsewhere)?
   2. Should removing an archiver deliberately re-arm the empty-archive check for a later re-add? The proposed reconciler logic would do this as a side effect; cnpg.io/skipEmptyWalArchiveCheck remains the escape hatch for intentional re-use. Happy to make it either way — just want to confirm which behavior is intended.

   Thanks !
3. ichibahn commented on Jul 21, 2026

   

   ichibahn

   on Jul 21, 2026

   More actions

   Following up after reading #11216 / #11217 and its plugin-side counterpart cloudnative-pg/plugin-barman-cloud#1009 — since they touch the same decision path, I checked how they interact with this issue.

   As far as I can tell, the refactoring centralizes where the decision is computed, but doesn't change what it evaluates: shouldCheckEmptyWalArchive is still annotation && marker-file-present, and the marker's lifecycle (reconcileCheckWalArchiveFile) is untouched. So for the scenario in this issue — the marker was already consumed via the no-op archiving path before the plugin existed — the operator will compute false and the plugin will now obey it as-is (per #1009, the plugin no longer re-inspects the marker). The bug remains, just with a single owner for the decision.

   That single ownership is actually good news for fixing this: once #11216 lands, the fix for this issue becomes operator-only (no plugin change needed). The proposal in my earlier comment — (1) re-create/keep the marker while no WAL archiver is configured, and (2) report the no-op archiving success with a distinct reason (e.g. ContinuousArchivingNotConfigured) so that marker removal is gated on a real archiving success — layers directly on top of the refactoring, feeding the corrected marker state into shouldCheckEmptyWalArchive.

   @mnencia — if you're already planning to address this issue as a follow-up to #11216, happy to defer or help review (I have a kind+MinIO reproduction with a control cluster I can re-run against any candidate fix). Otherwise I'd like to take it, building on your refactoring. I have a working implementation of the earlier proposal, verified against the reproduction: with the fix, the marker survives on archiver-less clusters (condition reports True with a distinct NotConfigured reason), enabling the plugin later now triggers "Expected empty archive" as designed, and the bootstrap-with-plugin control behaves identically to today. Notably it works with the unmodified released plugin (v0.13.0), and I've confirmed it merges cleanly with #11216 (build and unit tests pass on the combined tree). Before opening a PR, the two open questions from my earlier comment (whether the reason change for archiver-less clusters is acceptable, and whether removing an archiver should re-arm the check) are the main design points I'd like a nod on.
4. ichibahn commented on Jul 28, 2026

   

   ichibahn

   on Jul 28, 2026

   More actions

   #11216, cloudnative-pg/plugin-barman-cloud#1009 and cloudnative-pg/cnpg-i#353 have all landed since my last comment, so I re-ran my reproduction against main at 8046d243d (which includes #11216) to see where that leaves this issue.

   On my side the scenario still reproduces: a cluster created without an archiver reports ContinuousArchiving=True(ContinuousArchivingSuccess) from the no-op archiving path, and the .check-empty-wal-archive marker was already gone 23 seconds after the cluster became Ready. Adding the barman-cloud plugin afterwards, with a destination that already
   contained an object, the sidecar log shows no check at all and two 16 MiB WAL segments went straight into that destination. The control cluster — same final spec, plugin present at bootstrap — failed with Expected empty archive as designed. That matches what I'd expect from reading the refactoring: shouldCheckEmptyWalArchive centralises the decision, but it still evaluates the annotation and the marker, and the marker's lifecycle in reconcileCheckWalArchiveFile is unchanged.

   I've opened PR #11258 with the approach from my earlier comments, rebased on top of #11216. It's operator-only — verified against the released plugin v0.13.0 with no plugin changes — and in the same reproduction the check now fires 12 seconds after the sidecar is injected, with nothing written into the non-empty destination. I also added a control for the case I was most unsure about: with an empty destination the check still runs exactly once, passes, and the marker is consumed as before, so the check doesn't start repeating.

   The two design questions from my earlier comment are still the parts I'd most like your read on: whether the distinct condition reason for archiver-less clusters is acceptable, and whether removing an archiver should re-arm the check. I answered both in the PR description so there's something concrete to react to, but I'm happy to go the other way on either — and if you'd rather fold this into your own follow-up to #11216, that's fine too, just tell me which is easier to review.
5. khrisrichardson commented on Oct 10, 2026

   

   khrisrichardson

   on Oct 10, 2026

   More actions

   Another consequence of the same no-op archiving path: a "completed" base backup that cannot be restored.

   Versions: CloudNativePG 1.30.1 (operator and instance manager), ghcr.io/cloudnative-pg/postgresql:18.6, in-tree barmanObjectStore; kind.

   1. Create a Cluster without spec.backup and let it write WAL. ContinuousArchiving=True (ContinuousArchivingSuccess) is reported, as described above.
   2. Add spec.backup.barmanObjectStore and take an immediate Backup.
   3. The Backup reports completed, and its backup_label starts in segment N. But segments up to N are absent from the bucket, even though pg_stat_archiver counts them as archived: they were handled by the no-op path before the store was configured.
   4. Recovery from that backup fails with WAL file …000N for server <name> does not exist.

   So beyond the skipped empty-archive check, the no-op path can leave a backup that is reported complete but unrestorable, with the cluster reporting healthy archiving throughout. We reproduced it repeatedly across our acceptance runs (listing the bucket with the source's own credentials). Our workaround is to create the Cluster only after its object store is ready, so it has spec.backup from bootstrap and never takes the no-op path. We haven't yet checked whether the barman-cloud plugin path (added later) produces the same unrestorable backup, but the marker analysis above suggests the window exists there too. A fix that keeps ContinuousArchiving from reporting True while no archiver is configured would close both.