fix(metrics): handle NULL stats_reset in pg_stat_wal by HoniSanders · Pull Request #11207 · cloudnative-pg/cloudnative-pg · GitHub
Skip to content

fix(metrics): handle NULL stats_reset in pg_stat_wal - #11207

Merged
mnencia merged 2 commits into
cloudnative-pg:mainfrom
HoniSanders:dev/9106
Jul 30, 2026
Merged

mnencia merged 2 commits into
cloudnative-pg:mainfrom
HoniSanders:dev/9106

Conversation

@HoniSanders

@HoniSanders HoniSanders commented Jul 15, 2026 •

Copy link
Copy Markdown
Contributor

Since PostgreSQL 18, pg_stat_wal.stats_reset can be NULL. The metrics exporter scanned it into a non-nullable Go string, so metric collection failed with "converting NULL to string is unsupported", breaking Prometheus scrapes on otherwise-healthy clusters.

Coalesce stats_reset to '-infinity' in both pg_stat_wal queries in TryGetPgStatWAL, matching how fillArchiverStatus already handles the nullable last_archived_time and last_failed_time columns in this same file. Keeping the column as timestamptz rather than casting to ::text leaves the metric label format unchanged for non-NULL values.

Closes #9106


Summary

Since PostgreSQL 18, pg_stat_wal.stats_reset can be NULL. The metrics
exporter scanned it into a non-nullable Go string, so metric collection
failed with:

sql: Scan error on column index 4, name "stats_reset": converting NULL to string is unsupported

This broke Prometheus scrapes on otherwise-healthy clusters running PostgreSQL 18.

Fix

Coalesce stats_reset to '-infinity' in both pg_stat_wal queries in
TryGetPgStatWAL. This matches how fillArchiverStatus already handles the
nullable last_archived_time / last_failed_time columns in this same file
(COALESCE(col, '-infinity') into a plain string), so TryGetPgStatWAL is no
longer the odd one out scanning a nullable timestamptz directly.

Keeping the column as a timestamptz (rather than casting to ::text) leaves
the metric label format unchanged for non-NULL values and is consistent with
the sibling archiver fields.

Thanks to @mnencia for pointing to the existing pattern.

Testing

  • Unit tests cover the coalesced stats_reset value scanning into the string
    field for both the pre-18 and 18+ query shapes.
  • go build, go vet, and go test pass for pkg/management/postgres and
    pkg/management/postgres/webserver/metricserver.

@HoniSanders
HoniSanders requested a review from a team July 15, 2026 15:18
@cnpg-bot cnpg-bot added backport-requested ◀️ This pull request should be backported to all supported releases release-1.28 release-1.29 release-1.30 labels Jul 15, 2026
@github-actions

Copy link
Copy Markdown
Contributor

@dosubot dosubot Bot added size:S This PR changes 10-29 lines, ignoring generated files. bug 🐛 Something isn't working labels Jul 15, 2026
@mnencia

mnencia commented Jul 15, 2026

Copy link
Copy Markdown
Member

One thought on the approach here: pkg/postgres/status.go's LastArchivedWALTime/LastFailedWALTime handle the same situation (nullable last_archived_time/last_failed_time in pg_stat_archiver) by coalescing at the SQL level (COALESCE(last_archived_time, '-infinity')) and keeping the Go field a plain string. That's the pattern used everywhere else in this file for nullable Postgres columns; TryGetPgStatWAL is the only place scanning directly instead.

COALESCE(stats_reset::text, '') in both queries here would get the same result without touching PgStatWal.StatsReset's type or wal.go at all, and it sidesteps the infinity/-infinity concern entirely since you'd be coalescing text, not a timestamptz that has to round-trip through a time.Time-shaped driver value.

Not a blocker, sql.NullString fixes the bug correctly either way. Just flagging the smaller, more consistent option in case you'd rather go that route.

@dosubot dosubot Bot added size:XS This PR changes 0-9 lines, ignoring generated files. and removed size:S This PR changes 10-29 lines, ignoring generated files. labels Jul 16, 2026
@HoniSanders

Copy link
Copy Markdown
Contributor Author

Thanks for the suggestion! I see that fillArchiverStatus in this same file is the precedent, and TryGetPgStatWAL was the outlier. Switched to COALESCE(stats_reset, '-infinity') in both queries and reverted PgStatWal.StatsReset back to a plain string (wal.go is now untouched).

I went without the ::text cast on purpose: keeping it a timestamptz preserves the exact label format the exporter emits today and matches how last_archived_time renders, whereas stats_reset::text would switch non-NULL values to Postgres's native text format.

  • Current behavior (non-NULL values today): the exporter scans stats_reset — a timestamptz — directly into a Go string. pgx hands it back as a time.Time, which database/sql renders as RFC3339Nano (2024-01-01T12:00:00.123456Z). That's the label format clusters emit right now, and it's also how the sibling last_archived_time field renders, since COALESCE(col, '-infinity') keeps the value a timestamptz. COALESCE(stats_reset, '-infinity') preserves that format exactly.
  • COALESCE(stats_reset::text, ''): the ::text cast runs server-side, so Postgres emits its native text format (2024-01-01 12:00:00.123456+00). That changes the label string for every non-NULL value relative to what's emitted today — a one-time series-identity change for anyone already scraping the metric — and is inconsistent with how last_archived_time renders.

Happy to use COALESCE(stats_reset::text, '') instead if you'd prefer the empty-string sentinel over -infinity — let me know which you'd like.

@HoniSanders

Copy link
Copy Markdown
Contributor Author

@mnencia What are the next steps?

@HoniSanders

Copy link
Copy Markdown
Contributor Author

@jsilvela Can you help push this version of the PR forward? It is more closely tailored to @mnencia 's suggestions.

@mnencia

mnencia commented Jul 29, 2026

Copy link
Copy Markdown
Member

/test feature_type=observability

@github-actions

Copy link
Copy Markdown
Contributor

@dosubot dosubot Bot added the lgtm This PR has been approved by a maintainer label Jul 29, 2026
@cnpg-bot cnpg-bot added the ok to merge 👌 This PR can be merged label Jul 29, 2026
Since PostgreSQL 18, pg_stat_wal.stats_reset can be NULL. The metrics
exporter scanned it into a non-nullable Go string, so collection failed
with "converting NULL to string is unsupported", breaking Prometheus
scrapes for otherwise healthy clusters.

Coalesce stats_reset to '-infinity' in both pg_stat_wal queries, matching
how fillArchiverStatus already handles the nullable last_archived_time and
last_failed_time columns in this same file. This keeps StatsReset a plain
string and leaves the metric label format unchanged for non-NULL values.

Closes cloudnative-pg#9106

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Honi Sanders <6538616+HoniSanders@users.noreply.github.com>
Signed-off-by: Niccolò Fei <niccolo.fei@enterprisedb.com>
@mnencia
mnencia merged commit 953536e into cloudnative-pg:main Jul 30, 2026
33 checks passed
cnpg-bot pushed a commit that referenced this pull request Jul 30, 2026
Since PostgreSQL 18, pg_stat_wal.stats_reset can be NULL. The metrics
exporter scanned it into a non-nullable Go string, so metric collection
failed with "converting NULL to string is unsupported", breaking
Prometheus scrapes on otherwise-healthy clusters.

Coalesce stats_reset to '-infinity' in both pg_stat_wal queries in
TryGetPgStatWAL, matching how fillArchiverStatus already handles the
nullable last_archived_time and last_failed_time columns in this same
file. Keeping the column as timestamptz rather than casting to ::text
leaves the metric label format unchanged for non-NULL values.

Closes #9106

Assisted-by: Claude
Signed-off-by: Honi Sanders <6538616+HoniSanders@users.noreply.github.com>
Signed-off-by: Niccolò Fei <niccolo.fei@enterprisedb.com>
Co-authored-by: Niccolò Fei <niccolo.fei@enterprisedb.com>
(cherry picked from commit 953536e)
cnpg-bot pushed a commit that referenced this pull request Jul 30, 2026
Since PostgreSQL 18, pg_stat_wal.stats_reset can be NULL. The metrics
exporter scanned it into a non-nullable Go string, so metric collection
failed with "converting NULL to string is unsupported", breaking
Prometheus scrapes on otherwise-healthy clusters.

Coalesce stats_reset to '-infinity' in both pg_stat_wal queries in
TryGetPgStatWAL, matching how fillArchiverStatus already handles the
nullable last_archived_time and last_failed_time columns in this same
file. Keeping the column as timestamptz rather than casting to ::text
leaves the metric label format unchanged for non-NULL values.

Closes #9106

Assisted-by: Claude
Signed-off-by: Honi Sanders <6538616+HoniSanders@users.noreply.github.com>
Signed-off-by: Niccolò Fei <niccolo.fei@enterprisedb.com>
Co-authored-by: Niccolò Fei <niccolo.fei@enterprisedb.com>
(cherry picked from commit 953536e)
cnpg-bot pushed a commit that referenced this pull request Jul 30, 2026
Since PostgreSQL 18, pg_stat_wal.stats_reset can be NULL. The metrics
exporter scanned it into a non-nullable Go string, so metric collection
failed with "converting NULL to string is unsupported", breaking
Prometheus scrapes on otherwise-healthy clusters.

Coalesce stats_reset to '-infinity' in both pg_stat_wal queries in
TryGetPgStatWAL, matching how fillArchiverStatus already handles the
nullable last_archived_time and last_failed_time columns in this same
file. Keeping the column as timestamptz rather than casting to ::text
leaves the metric label format unchanged for non-NULL values.

Closes #9106

Assisted-by: Claude
Signed-off-by: Honi Sanders <6538616+HoniSanders@users.noreply.github.com>
Signed-off-by: Niccolò Fei <niccolo.fei@enterprisedb.com>
Co-authored-by: Niccolò Fei <niccolo.fei@enterprisedb.com>
(cherry picked from commit 953536e)
renovate Bot added a commit to sdwilsh/ansible-playbooks that referenced this pull request Sep 24, 2026
##### [\`v1.30.1\`](https://github.com/cloudnative-pg/cloudnative-pg/releases/tag/v1.30.1)

**Release date:** Sep 23, 2026

##### Enhancements

- Added `restoreAdditionalCommandArgs` to `.barmanObjectStore.data` for in-tree Barman Cloud recoveries, so custom flags are forwarded to `barman-cloud-restore` during recovery. ([#11278](cloudnative-pg/cloudnative-pg#11278)) <!-- 1.29 1.28 -->

- Plugin authors no longer need to re-derive whether an empty WAL archive should be treated as an error: the operator now computes the `check_empty_wal_archive` decision once and passes it to CNPG-I plugins on both WAL archiving and restore hooks. ([#11216](cloudnative-pg/cloudnative-pg#11216)) <!-- 1.29 1.28 -->

- Allowed setting the `ssl_ecdh_curve` PostgreSQL parameter in `.spec.postgresql.parameters`. It was previously rejected by the validating webhook even though the operator does not manage it. Contributed by [@Jamstah](https://github.com/Jamstah). ([#11308](cloudnative-pg/cloudnative-pg#11308)) <!-- 1.29 1.28 -->

- Added support for setting `auth_user` in `spec.pgbouncer.parameters` to override the user PgBouncer uses for its `auth_query` connection, instead of always deriving it from the `authQuerySecret`. Useful with a custom `authQuerySecret` where the role to authenticate as differs from the one implied by the Secret. Contributed by [@jonasbadstuebner](https://github.com/jonasbadstuebner). ([#10899](cloudnative-pg/cloudnative-pg#10899)) <!-- 1.29 1.28 -->

- Fixed PgBouncer metrics silently reporting zero when PgBouncer adds or reorders columns: `SHOW STATS` and `SHOW POOLS` results are now matched by column name instead of by position. Contributed by [@AdzerKI](https://github.com/AdzerKI) (`SHOW STATS`). ([#11366](cloudnative-pg/cloudnative-pg#11366), [#11464](cloudnative-pg/cloudnative-pg#11464)) <!-- 1.29 1.28 -->

- `cnpg` plugin:

  - Added a `--dry-run` option to the `backup` command that prints the `Backup` resource that would be created, without creating it. ([#11233](cloudnative-pg/cloudnative-pg#11233)) <!-- 1.29 1.28 -->

##### Security and Supply Chain

- Fixed a webhook validation bypass where an absolute extension path with an embedded `..` traversal (e.g. `/a/../../../../etc`) was accepted because only the raw path was checked; such paths still escaped the extension's mount point once resolved at runtime. Affected entries are now resolved and rejected/skipped consistently across admission, the major-upgrade job, and already-persisted Cluster specs. Contributed by [@Anand-240](https://github.com/Anand-240). ([#11210](cloudnative-pg/cloudnative-pg#11210)) <!-- 1.29 1.28 -->

- Updated `google.golang.org/grpc` to v1.83.1, fixing [CVE-2026-84304](https://nvd.nist.gov/vuln/detail/CVE-2026-84304) / GHSA-vp52-pcj8-j9qc (gRPC-Go heap memory exhaustion via HTTP/2 DATA frame fragmentation). ([#11394](cloudnative-pg/cloudnative-pg#11394)) <!-- 1.29 1.28 -->

##### Changes

- Updated the default PostgreSQL operand image to 18.6-system-trixie. ([#11257](cloudnative-pg/cloudnative-pg#11257)) <!-- 1.29 1.28 -->

- Moved the operator's own base image from Debian 12 (bookworm) to Debian 13 (trixie), now that Debian 12 has moved to LTS-only maintenance. ([#11193](cloudnative-pg/cloudnative-pg#11193)) <!-- 1.29 1.28 -->

##### Fixes

- Demoted an unreachable old primary immediately during failover, instead of waiting on checkpoint and fast-shutdown attempts that could never complete. Previously, an old primary whose PostgreSQL had become unresponsive was never shut down, so it could never rejoin the cluster as a replica of the new primary. The shutdown is now requested without blocking the reconciler, though demotion can still take up to a minute or more if PostgreSQL keeps ignoring shutdown signals. ([#11347](cloudnative-pg/cloudnative-pg#11347)) <!-- 1.29 1.28 -->

- Fixed failover and Pod recreation getting stuck while any instance in the cluster was fenced. A fenced instance kept reporting itself as healthy on its status endpoint even though its Pod was permanently not Ready, which blocked the readiness guard, the primary-election logic, and the Pod-reattachment gate from reacting to an unrelated failure elsewhere in the cluster, leaving it stalled until someone manually removed the fencing. All three now recognize and skip fenced instances. ([#11274](cloudnative-pg/cloudnative-pg#11274)) <!-- 1.29 1.28 -->

- Fixed a failover that could stall forever or get silently reverted. While a failover was pending, the reconciler could skip checking whether it was ready to complete, or mistake the pending state for a missing Pod and reset the target primary back to the unhealthy one. It now keeps progressing a pending failover and completes it once the WAL receivers are down, rechecking quorum safety right before promoting in case it changed while waiting. ([#11336](cloudnative-pg/cloudnative-pg#11336)) <!-- 1.29 1.28 -->

- Fixed the designated-primary sort in `PostgresqlStatusList` so a tie on LSN no longer sorts it out of order. If the designated primary's Pod later disappeared from the status list, this could have led the operator to promote the wrong standby. ([#11494](cloudnative-pg/cloudnative-pg#11494)) <!-- 1.29 1.28 -->

- Gated PostgreSQL startup on the instance manager actually holding the primary `Lease`, closing a window where a restarting or rescheduled primary could start accepting writes before acquiring the lease. Lease acquisition now happens during instance initialization, strictly before PostgreSQL is allowed to start, instead of only later in the reconciliation loop once the instance was already reported ready. ([#11356](cloudnative-pg/cloudnative-pg#11356))

- Bounded the primary lease take-over `Get` call to the lease's `RenewDeadline`, so a call against an unreachable API server now fails fast and retries instead of blocking indefinitely. Left unbounded, the same stalled call could also delay a primary's own lease renewal after a transient API server blip, risking an unwanted takeover by another instance once the lease appeared to expire. ([#11350](cloudnative-pg/cloudnative-pg#11350))

- Stopped abandoning a fast shutdown after 60 seconds. The request carried no explicit timeout, so `pg_ctl` fell back to its own implicit 60-second limit, and the instance manager, being PID 1 in the container, took PostgreSQL down mid-shutdown regardless of a larger configured `stopDelay`. The fast phase now waits for the full `stopDelay` instead. ([#11263](cloudnative-pg/cloudnative-pg#11263)) <!-- 1.29 1.28 -->

- Waited for certificate status before refreshing secrets during bootstrap. ([#11450](cloudnative-pg/cloudnative-pg#11450)) <!-- 1.29 1.28 -->

- Fixed in-tree Barman Cloud recovery ignoring `wal.maxParallel` and `wal.restoreAdditionalCommandArgs`, fetching WAL one segment at a time instead of using the configured parallelism. ([#11105](cloudnative-pg/cloudnative-pg#11105)) <!-- 1.29 1.28 -->

- Honored the `unrecoverable` annotation on pending and terminating instances. ([#11241](cloudnative-pg/cloudnative-pg#11241)) <!-- 1.29 1.28 -->

- Honored `Backup.spec.online` when finalizing VolumeSnapshot-based backups, so a cold backup is no longer reported as online in status. ([#11276](cloudnative-pg/cloudnative-pg#11276)) <!-- 1.29 1.28 -->

- Set `--no-data-checksums` explicitly when `.spec.postgresql.dataChecksums` is `false` on PostgreSQL 18+, which now enables checksums by default in `initdb`. Contributed by [@evan314159](https://github.com/evan314159). ([#11488](cloudnative-pg/cloudnative-pg#11488)) <!-- 1.29 1.28 -->

- Stopped discarding `.spec.inheritedMetadata` labels on PVC reconciliation; they were applied before, and then overwritten by, the operator's own labels. Contributed by [@GellertBabel](https://github.com/GellertBabel). ([#11499](cloudnative-pg/cloudnative-pg#11499)) <!-- 1.29 1.28 -->

- Stopped `Pooler` and `DatabaseRole` from recreating their owned resources after being marked for deletion, which could block an external finalizer (e.g. ArgoCD foreground deletion) waiting for those resources to disappear. Contributed by [@mjlshen](https://github.com/mjlshen). ([#11359](cloudnative-pg/cloudnative-pg#11359)) <!-- 1.29 1.28 -->

- Handled a `NULL` `stats_reset` value in the `pg_stat_wal` metric, which could occasionally break Prometheus scraping on PostgreSQL 18 clusters. Contributed by [@HoniSanders](https://github.com/HoniSanders). ([#11207](cloudnative-pg/cloudnative-pg#11207)) <!-- 1.29 1.28 -->

- Recovered log streaming when a stale non-FIFO file blocked the log pipe. ([#11246](cloudnative-pg/cloudnative-pg#11246)) <!-- 1.29 1.28 -->

- Stopped sampling logs of the Pods' processes, so no log lines are silently dropped. ([#11131](cloudnative-pg/cloudnative-pg#11131)) <!-- 1.29 1.28 -->

- Stopped caching Helm release Secrets in the operator, which could spike memory usage and crashloop the operator on clusters with large Helm releases installed alongside it. ([#11211](cloudnative-pg/cloudnative-pg#11211)) <!-- 1.29 1.28 -->

- Re-ran owned-resource finalizer cleanup after an operator restart, so a namespace deleted while the operator was down doesn't hang in `Terminating` forever. ([#11215](cloudnative-pg/cloudnative-pg#11215)) <!-- 1.29 1.28 -->

- Fixed rejoining replicas taking a long time to complete `pg_rewind` when WAL prefetching (`wal.maxParallel > 1`) was enabled. ([#11204](cloudnative-pg/cloudnative-pg#11204)) <!-- 1.29 1.28 -->

- Treated a PostgreSQL instance rejecting connections as started up in probes. ([#11152](cloudnative-pg/cloudnative-pg#11152)) <!-- 1.29 1.28 -->

- Stopped issuing a redundant PATCH on every reconciliation when a cluster's `pvcTemplate` set `volumeAttributesClass`. Contributed by [@andrew-farries](https://github.com/andrew-farries). ([#11196](cloudnative-pg/cloudnative-pg#11196)) <!-- 1.29 1.28 -->

- Stopped halting replication slot reconciliation at the first failing slot. ([#11119](cloudnative-pg/cloudnative-pg#11119)) <!-- 1.29 1.28 -->

- During rollouts, compared the running configuration hash, rather than the on-disk one, avoiding unnecessary switchovers. ([#11130](cloudnative-pg/cloudnative-pg#11130)) <!-- 1.29 1.28 -->

- Retried `VolumeSnapshot` errors until the deadline, instead of giving up early. ([#11132](cloudnative-pg/cloudnative-pg#11132)) <!-- 1.29 1.28 -->

- Symlinked `pg_wal` to the WAL volume during plugin-based recovery, instead of replaying WAL onto the data volume, which could exhaust it on a large point-in-time recovery. ([#11127](cloudnative-pg/cloudnative-pg#11127)) <!-- 1.29 1.28 -->

- Corrected grammar in `PhaseUpgradeDelayed` status messages. Contributed by [@Utkarsh-sharma47](https://github.com/Utkarsh-sharma47). ([#11065](cloudnative-pg/cloudnative-pg#11065)) <!-- 1.29 1.28 -->

- CloudNativePG Interface (CNPG-I):

  - Reported the Postgres capabilities of a plugin correctly. ([#11447](cloudnative-pg/cloudnative-pg#11447)) <!-- 1.29 1.28 -->
  - Prevented the plugin status field from flapping on Cluster reconciliation. Contributed by [@sharifmshaker](https://github.com/sharifmshaker). ([#11386](cloudnative-pg/cloudnative-pg#11386)) <!-- 1.29 1.28 -->
  - Replaced a finalizer with in-memory cleanup on Service deletion. This avoids namespaces hanging in `Terminating` state when the operator and its plugins run in the same namespace. Contributed by [@maxlengdell](https://github.com/maxlengdell). ([#10940](cloudnative-pg/cloudnative-pg#10940)) <!-- 1.29 -->

- `cnpg` plugin:

  - Used UTC time zone for the default backup name. ([#11329](cloudnative-pg/cloudnative-pg#11329)) <!-- 1.29 1.28 -->
  - Deleted PVCs before Pods in `cnpg destroy`, avoiding a race that could leave dangling PVCs needing a second invocation to clean up. Contributed by [@daanvinken](https://github.com/daanvinken). ([#10847](cloudnative-pg/cloudnative-pg#10847)) <!-- 1.29 1.28 -->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-requested ◀️ This pull request should be backported to all supported releases bug 🐛 Something isn't working lgtm This PR has been approved by a maintainer ok to merge 👌 This PR can be merged release-1.28 release-1.30 size:XS This PR changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: 1.27.1 stats query bug

4 participants