Description

Fixes a regression, introduced in v0.114-0, where the gateway stops serving already-authenticated clients after 2h05m of process uptime, returning internal errors with Connection pool missing for user. in the gateway log.

Reported by an external user whose 12–48 hour soak harness passed consistently on 0.112 and began failing at ~2 hours on 0.114, reproducibly across multiple machines.

Root cause

ConnectionPool::last_used_nanos holds nanoseconds since the process epoch, and that is how last_used() decodes it:

pub fn last_used(&self) -> Instant {
    time::u64_to_instant(self.last_used_nanos.load(Ordering::Relaxed))
    // == EpochClock::epoch() + Duration::from_nanos(v)
}

f02641fe ("[RustGW] Correctly migrate to EpochClock in connection pool") changed both acquisition paths to write it with EpochClock::almost_now_timestamp(), which returns seconds since the UNIX epoch (~1.785e9). Decoded as nanoseconds that lands ~1.785 s after process start and then advances one nanosecond per elapsed second — effectively frozen. The two functions look interchangeable at a call site and are not.

So last_used().elapsed() stopped measuring pool idleness and started measuring process uptime. The reaper wakes every 300 s and disposes pools idle beyond 7200 s, so with a frozen stamp:

• tick at t=7200 s reads 7198.2 s → spares the pool
• tick at t=7500 s reads 7498.2 s → evicts it

From then on every user data pool is evicted regardless of load, and re-evicted on each subsequent tick. get_data_pool cannot rebuild an evicted pool because the user's password is not retained past authentication, so requests from an authenticated client fail until it re-authenticates.

Ancestry confirms the reported release boundary: f02641fe is not an ancestor of v0.112-0 or v0.113-0, and is an ancestor of v0.114-0.

Fix

Both acquisition paths now route through a single touch_last_used() helper that writes the encoding last_used() decodes. Two deliberate choices:

1. One writer. The field is written only by touch_last_used and read only by last_used, so the encoding is owned in one place. Both carry doc comments naming the trap.

2. Precise Instant::now(), not the coarse clock. The minimal fix preserving the original commit's intent would be instant_to_u64(EpochClock::almost_now()). That was tried first and the new tests still failed, at 5.004 s of reported idleness: EpochClock's coarse updater is a tokio task spawned on whichever runtime first initialises the global clock, so almost_now() stops advancing when that runtime goes away. The gateway daemon has one long-lived runtime so it does advance in production, but the reaper's only input shouldn't depend on a background task's liveness. Instant::now() is ~20 ns on a path that then checks out a pooled Postgres connection.

The other almost_now_timestamp() callers (ttl_cache.rs, transaction_store.rs) were audited and are correct — both compare unix-seconds to unix-seconds. The connection pool was the only site mixing bases.

Validation: 8-hour three-arm soak

Beyond the unit tests, this was validated on documentdb-local:pg17-0.114.0 (PG17 + documentdb extensions). Three arms, same image, environment, network and load, differing only in the gateway binary — each verified by md5 inside its container rather than assumed:

The fixed arm completed 28,680 operations with zero client errors while its reaper ticked 97 times — so the reaper ran and evaluated the pool every cycle and simply found nothing to evict. Both broken arms were still failing continuously at cutoff.

The unfixed arm validates the method rather than the bug: it comes through the same build-and-swap pipeline as fixed, so its failing on schedule establishes that a clean fixed result is attributable to the fix and not to an artifact of how the binary was produced. Predictions (including the t=7500 s tick) were recorded before the run.

Recovery and recurrence were demonstrated directly. A fresh client pointed at the already-broken control arm ran 226 clean operations — authenticating rebuilds the pool — then failed at the very next reaper tick:

[05:39:53.202Z] OK   i=225 ok=226 bad=0
[05:39:55.212Z] FAIL i=227 ok=227 bad=1 msg=An unexpected internal error has occurred.

That gateway started at 02:19:54.5 Z, so the tick was due at 05:39:54.5 Z — the transition brackets it within two seconds.

Two notes for anyone reproducing this: the client sees An unexpected internal error has occurred., never the pool message, so a client-side detector is blind — grep the gateway log. And the failure can present as the workload hanging rather than erroring, matching the original report of the database "appearing to become unresponsive."

Tests

Three regression tests, all failing on the unfixed code:

Making them deterministic took some care — a stamp in the wrong unit still decodes to some instant near the epoch, so it only becomes distinguishable once the process has been up longer than the offset the bad stamp decodes to (~1.785 s). The new testing::wait_for_epoch_uptime helper pins uptime past that point first, making it a controllable input rather than something to wait out.

The pre-existing test_last_used_with_fresh_pool_returns_recent_instant covers only a freshly constructed pool, and the regressing commit left the constructor writing the correct encoding — which is why it kept passing throughout.

Also included

• Changelog entry under v0.117-0.
• An RCA at docs/rca/2026-07-27-gateway-connection-pool-last-used-regression.md, including two follow-ups deliberately not addressed here: (a) even with correct timestamps, a >2h-idle authenticated client hits the same error because the pool can't be rebuilt without the password — pre-existing behaviour, also present in 0.112; (b) the EpochClock updater's runtime-lifetime coupling, which ttl_cache and transaction_store also depend on. Happy to drop the RCA file from this PR if maintainers would rather it live outside the repo.

Verification

• 462/462 documentdb_gateway_core unit tests pass
• cargo fmt --all --check and cargo clippy -p documentdb_gateway_core --all-targets clean
• 8h three-arm soak as above

Type of Change

• Bug fix
• New feature
• Documentation update
• Refactoring / code cleanup
• DevOps / tooling
• Test improvement
• Other: ___________

AI Disclosure

• No AI tools were used substantially in this PR
• AI tools were used to assist with this PR (please complete below)
  • Tool(s) used: Claude Code
  • How it was used: Used to trace the regression from the customer report to the introducing commit, build a deterministic reproduction of an uptime-dependent failure, design and run the three-arm soak, and draft the fix, tests, and RCA. All findings were verified against the code and by running the tests and the soak locally; the first candidate fix was rejected after its tests failed, which is what surfaced the EpochClock updater lifetime issue documented above.

Checklist

• I can explain every change in this PR if asked during review
• I have tested these changes locally
• I have added or updated tests as appropriate
• All commits are signed off (DCO: git commit -s)
• I have read the CONTRIBUTING.md guidelines