{{ message }}
Repository navigation
Commit 8a1914f
committed
fix(gw): stop reaping in-use connection pools after 2h of uptime
`ConnectionPool::last_used_nanos` holds nanoseconds since the process
epoch, and that is how `last_used()` decodes it. Both acquisition paths
were writing it with `EpochClock::almost_now_timestamp()`, which returns
seconds since the UNIX epoch. Decoded as nanoseconds, that lands ~1.785s
after process start and then advances a nanosecond per second, so the
stamp was effectively frozen.
`last_used().elapsed()` therefore measured process uptime rather than
pool idleness. The reaper wakes every 300s and disposes pools idle beyond
POSTGRES_POOL_DISPOSE_INTERVAL_SEC (7200s); with a frozen stamp the tick
at t=7200s reads 7198.2s and spares the pool, and the tick at t=7500s
reads 7498.2s and evicts it. From then on every user data pool is evicted
regardless of how heavily it is in use, and re-evicted on each subsequent
tick. `get_data_pool` cannot rebuild an evicted pool because the user
password is not retained past authentication, so requests from an
already-authenticated client fail from 2h05m of uptime onward. The client
sees "An unexpected internal error has occurred."; only the gateway log
carries "Connection pool missing for user."
Route both acquisition paths through a single `touch_last_used` helper
that writes the encoding `last_used` decodes. Use the precise
`Instant::now()` rather than either coarse `EpochClock` reading: the
coarse updater is a tokio task bound to whichever runtime first
initialises the global clock, and the reaper's only input should not
depend on that task still being alive. The syscall is ~20ns on a path
that then checks out a pooled connection.
Regression introduced in v0.114-0; v0.112-0 and v0.113-0 are unaffected.
Adds three regression tests that fail on the unfixed code, a
`wait_for_epoch_uptime` test helper that makes process uptime a
controllable input, and an RCA under docs/rca/.
Validated by an 8h three-arm soak on documentdb-local:pg17-0.114.0 in
which the arms differ only by this fix: stock v0.114.0 first failed at
2h05m00.3s and a locally built unmodified v0.114-0 at 2h05m00.1s, while
the fixed build ran 8h and 28,680 operations with zero occurrences and
its reaper ticking 97 times. Details in the RCA.
Signed-off-by: Guanzhou Song <guanzhousong@microsoft.com>1 parent fd46318 commit 8a1914f
5 files changed
Lines changed: 137 additions & 9 deletions
File tree
- pg_documentdb_gw/documentdb_gateway_core/src
- postgres/conn_mgmt
- testing
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
45 | | - | |
| 45 | + | |
46 | 46 | | |
47 | 47 | | |
48 | 48 | | |
| |||
147 | 147 | | |
148 | 148 | | |
149 | 149 | | |
150 | | - | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
151 | 157 | | |
152 | 158 | | |
153 | 159 | | |
| |||
278 | 284 | | |
279 | 285 | | |
280 | 286 | | |
281 | | - | |
282 | | - | |
| 287 | + | |
283 | 288 | | |
284 | 289 | | |
285 | 290 | | |
| |||
306 | 311 | | |
307 | 312 | | |
308 | 313 | | |
309 | | - | |
310 | | - | |
| 314 | + | |
311 | 315 | | |
312 | 316 | | |
313 | 317 | | |
| |||
333 | 337 | | |
334 | 338 | | |
335 | 339 | | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
336 | 360 | | |
337 | 361 | | |
338 | 362 | | |
| |||
391 | 415 | | |
392 | 416 | | |
393 | 417 | | |
394 | | - | |
| 418 | + | |
395 | 419 | | |
396 | 420 | | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
397 | 426 | | |
398 | 427 | | |
399 | 428 | | |
| |||
525 | 554 | | |
526 | 555 | | |
527 | 556 | | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
| 574 | + | |
| 575 | + | |
| 576 | + | |
| 577 | + | |
| 578 | + | |
| 579 | + | |
| 580 | + | |
| 581 | + | |
| 582 | + | |
| 583 | + | |
| 584 | + | |
| 585 | + | |
| 586 | + | |
| 587 | + | |
| 588 | + | |
| 589 | + | |
| 590 | + | |
| 591 | + | |
| 592 | + | |
| 593 | + | |
| 594 | + | |
| 595 | + | |
| 596 | + | |
528 | 597 | | |
529 | 598 | | |
530 | 599 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
342 | 342 | | |
343 | 343 | | |
344 | 344 | | |
| 345 | + | |
345 | 346 | | |
346 | 347 | | |
347 | 348 | | |
| |||
749 | 750 | | |
750 | 751 | | |
751 | 752 | | |
| 753 | + | |
| 754 | + | |
| 755 | + | |
| 756 | + | |
| 757 | + | |
| 758 | + | |
| 759 | + | |
| 760 | + | |
| 761 | + | |
| 762 | + | |
| 763 | + | |
| 764 | + | |
| 765 | + | |
| 766 | + | |
| 767 | + | |
| 768 | + | |
| 769 | + | |
| 770 | + | |
| 771 | + | |
| 772 | + | |
| 773 | + | |
| 774 | + | |
| 775 | + | |
| 776 | + | |
| 777 | + | |
| 778 | + | |
| 779 | + | |
| 780 | + | |
| 781 | + | |
| 782 | + | |
| 783 | + | |
| 784 | + | |
| 785 | + | |
| 786 | + | |
| 787 | + | |
| 788 | + | |
| 789 | + | |
| 790 | + | |
752 | 791 | | |
753 | 792 | | |
754 | 793 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
27 | | - | |
| 27 | + | |
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| |||

0 commit comments