Repository navigation
GSS encryption fallback (GssEncryptionMode=Prefer) can retry on a second socket without the connection Timeout — Open blocks until the peer closes (~2%) #6655
Description
Summary
With the Npgsql 10 default GssEncryptionMode=Prefer, opening a connection to a server that accepts TCP but never
answers sometimes blocks far past Timeout. The Timeout fires during GSS-encryption negotiation, Npgsql logs that it
is retrying without GSS encryption and opens a second socket — and in roughly 2% of attempts that second attempt runs
with no effective timeout, so Open() blocks until the peer closes the connection. GssEncryptionMode=Disable removes it
entirely.
This looks like the same family as #6524 (the fallback path after the GSS attempt not honouring the timeout), but with a
different trigger: Kerberos libraries are present here (macOS), and the server is simply unresponsive.
Environment
- Npgsql 10.0.3 (via Npgsql.EntityFrameworkCore.PostgreSQL 10.0.3), .NET 10, macOS (arm64), Kerberos tooling present (
klist) - Connection string:
Host=127.0.0.1;Port=<fake>;Timeout=1;Pooling=false(+ database/user)
Reproduction
A TCP listener that accepts connections and never writes a byte. Loop single NpgsqlConnection.Open() calls against it,
sequentially, each with a watchdog of a few seconds:
Every non-blocked attempt failed promptly as expected (NpgsqlException wrapping TimeoutException), and latency on the
timeout path was unchanged by the setting. SslMode made no difference (Prefer 4/400 vs Disable 5/400 blocked with GSS
still at its default) — consistent with GSS negotiation happening before SSL.
What the trace shows
With Npgsql's trace logging enabled (a capturing ILoggerProvider via NpgsqlLoggingConfiguration.InitializeLogging),
trials run sequentially so every line in a window belongs to one trial:
- "Attempting to connect" → "Socket connected"
- "Negotiating GSS encryption" → the Timeout fires
- the fallback message ("… retrying without it")
- a second "Attempting to connect" → "Socket connected"
- then nothing until the listener closes, then an
IOException
In the hung trials step 4's socket never gets a timeout applied; in the other ~98% the fallback attempt times out normally.
Ruled out (each by measurement)
- exception classification (all prompt failures were the timeout)
- CPU load (timeout held at ~21 ms under load average 44)
- thread-pool starvation (0/10 with every worker parked and the pool capped)
- cold start (the hang occurred at iteration 3, not 0)
SslMode(see above)
Impact / workaround
For us it hung an end-of-run cleanup pass that must give up on an unresponsive server. Workaround: set
GssEncryptionMode=Disable on connections that must never block past Timeout.
Happy to share the reproduction harness.
Activity
vonzshik commentedon Sep 23, 2026
Oh my, that's a funny one.
So, most of the time whenever we try a second time to open a physical connection we'll either have a big enough value for the timeout (at least a few seconds), or it will be already expired, at which point we'll just immediately throw an exception.
But there's a corner case, where the timeout has not expired but its value is very small (for example, 400 microseconds). And it looks like we do not really handle that well because the API for the socket doesn't accept anything less a millisecond, so by casting
TimeSpan.TotalMillisecondstointreturns0, which socket treats as an infinite timeout.npgsql/src/Npgsql/Internal/NpgsqlReadBuffer.cs
Line 45 in af6676e
The fix should be pretty simple, we're just going to consider any timeout below 1 millisecond as if it has already triggered.
zarkomatejic commentedon Sep 23, 2026
Thanks — that explains the shape of our data precisely. We saw it at
Timeout=1(second), so a sub-millisecond remainder on the second attempt is rare, which matches the ~2% rate we measured (16/800 Opens blocked withGssEncryptionMode=Prefer, 0/800 withDisable). The trace fits too: the secondAttempting to connect→Socket connectedwas followed by silence until the listener closed the socket, rather than by a timeout.Worth noting for anyone who finds this by searching: the GSS fallback is just the most common way to get a second physical connection attempt — by your explanation, any reconnect inside one
Timeoutbudget can hit the same(int)TotalMilliseconds == 0→ infinite socket timeout. SoGssEncryptionMode=Disableworks by removing the second attempt, not because GSS is at fault.Happy to run our 800-iteration harness against a nightly once the fix lands, if a confirmation on macOS / .NET 10 is useful.
vonzshik commentedon Sep 23, 2026
Yep yep, that's actually a long standing issue. Technically you can reproduce it with any timeout, so long as connecting to the socket takes most of the timeout (like, out of 5 seconds let's say it took 4.999501 seconds, so we're only left with 0.499 milliseconds, which by casting to int will give us 0 = infinite). Doesn't really have to be only
Socket.Connect, it can also be at some later point, just the simplest example. So yeah, no GSS fallback is actually necessary, just something that makes it easier to trigger.I did run it for a bit on my windows machine and it did stop reproducing. You can either just build npgsql from my PR, or just wait for us to merge and release the fix (hopefully it wouldn't take too long 😄 )
lahma commentedon Oct 2, 2026
A related one on the cancel-request path.
DoCancelRequestbuilds a freshNpgsqlTimeout(ConnectionTimeout)for the GSS attempt (L2048) and another one for the fallback (L2063). It also falls back on any exception, a timeout included (L2056). Against a server that accepts TCP but never answers, one cancel attempt therefore costs 2 × Timeout. All of it runs synchronously, underCancelLock, on the thread that canceled the token.I measured this with a freezing TCP proxy (Npgsql 10.0.3 and main, Windows SSPI), token canceled at 3 s,
No Reset On Close=true;Timeout=5:Cancel()callback return atOn the Command Timeout path with defaults this comes to 30 s + 15 s + 15 s = 60 s (45 s with GSS disabled).
A small fix is to share one deadline across both attempts, as
OpenCoredoes, and to skip the fallback once that deadline has expired:catch (Exception e) when (gssEncMode == GssEncryptionMode.Prefer && !timeout.HasExpired). Sharing the deadline makes the sub-millisecond remainder from this issue reachable on the cancel path too, so it should land together with the clamp fix here. The broader fix is #3277 / #5070. Found while investigating #6662.