fix(gateway): keep last-known-good max_connections on bad GUC read by morch23mj · Pull Request #723 · documentdb/documentdb · GitHub
Skip to content

fix(gateway): keep last-known-good max_connections on bad GUC read - #723

Open
morch23mj wants to merge 1 commit into
documentdb:mainfrom
wetransform:fix/max-connections-last-known-good
Open

morch23mj wants to merge 1 commit into
documentdb:mainfrom
wetransform:fix/max-connections-last-known-good

Conversation

@morch23mj

Copy link
Copy Markdown

Description

Fixes #720. Supersedes #721 (see "Why this instead of #721").

max_connections is a startup GUC — it cannot change without a Postgres restart — yet PgConfiguration::max_connections() (configuration/pg_configuration.rs) re-reads it on every dynamic-config refresh and silently substitutes 25 on a bad/missing read. Because max_connections is part of the PgPoolSettings pool-cache key, that fabricated 25 flips the key mid-run with no real change behind it; get_data_pool then permanently hard-fails every request for that user ("Connection pool missing for user.") — the pool is healthy but orphaned. Full root cause and log evidence: #720.

This fixes the root cause: on a bad read, max_connections() returns the last successfully-read value instead of fabricating a divergent one, so the pool key stays stable and the spurious miss never happens.

Why this instead of #721

#721 made get_data_pool fall back to reusing the user's existing pool on a settings-key miss. As @AndrewKhoma noted in review, that is unsafe: if max_connections had genuinely dropped to 25, the reused (100-sized) pool would let requests use connections Postgres no longer has (over-subscription); and the user-pool hard-fail is intentional precisely because max_connections is a startup GUC. This PR takes that feedback directly — it does not reuse pools and leaves the intentional hard-fail guard in get_data_pool untouched. It only removes the spurious key flip at its source (the bad read). A genuine max_connections change (via restart → fresh read) still updates the value and flips the key as designed, so the intended rebuild path is preserved.

What changed

max_connections() caches the last valid read in an AtomicUsize (stored on every valid read); on a bad read it returns that last-known-good value, falling back to 25 only on the first-read edge case. Scope: one field on PgConfiguration + the getter.

Testing

  • cargo test -p documentdb_gateway_core → 434 passed, 0 failed, including two new tests (last-known-good on bad read; falls back to 25 with no prior good value) and the existing validate_max_conn_change (a real change still flips the key → pool rebuild preserved).
  • rustfmt and clippy --workspace --all-targets --all-features -- -D warnings clean on the repo's pinned toolchain (Rust 1.95).

Type of Change

  • Bug fix

AI Disclosure

  • AI tools were used. Tool: Claude (Anthropic). Root-cause tracing, the fix, and the tests were prepared with substantial AI assistance, then reviewed and verified against the source by the author.

Checklist

  • I can explain every change if asked
  • Tested locally (cargo test — 434 passed)
  • Tests added/updated
  • Commits signed off (DCO)
  • Read CONTRIBUTING.md

Signed-off-by: Morchid Chellali mc@wetransform.to

Signed-off-by: Morchid Chellali <mc@wetransform.to>
@morch23mj
morch23mj requested a review from a team as a code owner August 15, 2026 16:24
@documentdb-triage-tool documentdb-triage-tool Bot added bug gateway for gateway related issues labels Aug 15, 2026
@documentdb-triage-tool

Copy link
Copy Markdown

@AndrewKhoma AndrewKhoma left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for opening this PR, but I need a better explanation on how you've been able to reproduce and execute this state with the running postgresql

}),
Err(e) => {
tracing::warn!("Failed to get connection for pg_file_settings: {e}");
Vec::new()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this line and the line 102 (Vec::new() in unwrap_or_else) can be a potential bug you've faced, since we will drop all of the settings, not just the max_connections one in case of any failure instead of returning error

seems wrong to me, since in case of the error it's better to have stale GUCs then none

@morch23mj

morch23mj commented Aug 18, 2026 •

Copy link
Copy Markdown
Author

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug gateway for gateway related issues

Projects

Status: Todo

Development

Successfully merging this pull request may close these issues.

Gateway permanently fails all requests for a user after a dynamic-config refresh flips the connection-pool key ("Connection pool missing for user.")

2 participants