Activity

1. vonzshik commented on Mar 31, 2026

   

   vonzshik

   on Mar 31, 2026

   Contributor

   More actions

   This one's certainly interesting. For one, we did have a similar issue #6416, where user also had a container without kerberos library, but in their case there was no hang, just annoying stderr errors. In addition, we have #6467, where user reported hangs, but in their case it seems to be something related to NegotiateAuthentication.GetOutgoingBlob rather than SslStream.
   Are you absolutely sure it hangs while establishing SSL connection to PG? It's just whenever that library is missing, we catch TypeInitializationException from the runtime and gracefully continue with SSL.
2. netijoe commented on Mar 31, 2026

   

   netijoe

   on Mar 31, 2026

   Author

   More actions

   Thanks for the quick response and the related issues.

   To clarify: yes, we're quite confident the hang is in SslStream.ForceAuthenticationAsync. The GC root chain from dotnet-dump was:

   Timer → TimerCallback → [Marten HighWaterAgent] → NpgsqlConnector.ConnectAsync
     → RawOpen → NegotiateEncryption → SetupEncryption → SslStream.ForceAuthenticationAsync
   

   This is consistent across multiple heap dumps taken at different intervals.

   Regarding #6467 — our stack trace differs slightly. We don't see NegotiateAuthentication.GetOutgoingBlob, the hang is further down in the SSL path. This suggests the GSS negotiation itself may complete (or be skipped), but something in the SSL handshake that follows gets stuck.

   Environment details that may be relevant:

   • Alpine Linux (musl libc, not glibc)
   • libgssapi_krb5.so.2 is not present
   • PostgreSQL server has SSL enabled and requires it
   • The hang is 100% reproducible — every unpooled connection created under these conditions hangs indefinitely
3. vonzshik commented on Mar 31, 2026

   

   vonzshik

   on Mar 31, 2026

   Contributor

   More actions

   Tried to repro with mcr.microsoft.com/dotnet/sdk:9.0-alpine3.22 container (I'm pretty sure it's musl?), and so far no issue whatsoever. I can see in the output window that there is an attempt to load libgssapi_krb5.so.2 (unsuccessfully of course) and then it retries without GSS encryption but with ssl.
4. brianpursley commented on Apr 1, 2026

   

   brianpursley

   on Apr 1, 2026

   Contributor

   More actions

   I had a few thoughts:

   1. I wonder if Kubernetes is a factor in reproducing this. Are you using any kind of proxy or kubectl port-forward? Or using PgBouncer or anything like that? Is your postgres instance inside your cluster or outside?

   2. Some async calls in NpgsqlConnector use cancellationToken without combining it with the timeout, so those calls can possibly hang indefinitely if they are unable to time out. This may not explain this issue exactly, but could explain the indefinite hanging.

   Specifically, this call (which eventually calls SslStream.ForceAuthenticationAsync if you drill into its implementation):
   

   npgsql/src/Npgsql/Internal/NpgsqlConnector.cs

   Line 1177 in 2fea281

   await sslStream.AuthenticateAsClientAsync(sslStreamOptions, cancellationToken).ConfigureAwait(false);

   In contrast, a couple of other async calls (DNS lookup and socket connection) combine the cancellationToken with the timeout like this:
   

   npgsql/src/Npgsql/Internal/NpgsqlConnector.cs

   Lines 1299 to 1315 in 2fea281

   	using var combinedCts = timeout.IsSet ? CancellationTokenSource.CreateLinkedTokenSource(cancellationToken) : null;
   	combinedCts?.CancelAfter(timeout.CheckAndGetTimeLeft());
   	var combinedToken = combinedCts?.Token ?? cancellationToken;
   	try
   	{
   	ipAddresses = await Dns.GetHostAddressesAsync(Host, combinedToken).ConfigureAwait(false);
   	}
   	catch (OperationCanceledException)
   	{
   	cancellationToken.ThrowIfCancellationRequested();
   	Debug.Assert(timeout.HasExpired);
   	ThrowHelper.ThrowNpgsqlExceptionWithInnerTimeoutException("The operation has timed out");
   	}
   	catch (SocketException ex)
   	{
   	throw new NpgsqlException(ex.Message, ex);
   	}

   Should that combinedToken approach also be used when calling sslStream.AuthenticateAsClientAsync()?

   Or potentially the cancellationToken could be combined with the timeout higher up in the stack and used throughout, instead of needing to create combinedToken before each call.
5. TLS-Terminator commented on Apr 2, 2026

   

   TLS-Terminator

   on Apr 2, 2026

   Last edited by TLS-Terminator

   More actions

   We have the same issue. Alpine .NET 10 container running in k8s started showing constant unmanaged memory growth after upgrading to npgsql 10.

   Our postgres is managed AWS Aurora, and connections go through AWS RDS proxy which serves as a connection pooler.

   
6. netijoe commented on Apr 3, 2026

   

   netijoe

   on Apr 3, 2026

   Last edited by netijoe

   Author

   More actions

   Follow-up: raw TCP probing of managed PostgreSQL (Aiven)

   Adding some data that may explain why this reproduces only against managed PostgreSQL services (Aiven with PgBouncer in our case, AWS Aurora/RDS Proxy for @TLS-Terminator) and not against standard Postgres.

   We probed what Aiven responds to various pre-startup messages using raw TCP sockets (no Npgsql involved):

   GSSENCRequest  (magic 0x04D2162F) → S
   SSLRequest     (magic 0x04D2162E) → connection closed
   CancelRequest  (magic 0x04D2162D) → E
   Garbage        (magic 0x04D216FF) → E
   

   For comparison, standard Postgres (postgres:14-alpine in Docker) responds N to both GSSENCRequest and SSLRequest (no SSL/GSS configured).

   Aiven returns S (the SSLRequest response byte) to GSSENCRequest — this is not a valid GSSENCRequest response (G or N expected per the protocol). Their proxy layer appears to treat the two similar magic numbers non-standardly.

   This likely explains the interaction with @brianpursley's finding about the missing timeout on AuthenticateAsClientAsync: the proxy's unexpected S response to GSSENCRequest puts the connection into a state where the subsequent SSL handshake blocks indefinitely — and since there's no combined timeout on that call, it hangs forever.

   Our observations with Npgsql 10 on Alpine (mcr.microsoft.com/dotnet/aspnet:9.0-alpine, no libgssapi_krb5.so.2):

   • Simple sequential NpgsqlConnection.OpenAsync() succeeds against Aiven with both GssEncryptionMode=Prefer and Disable
   • In production, a long-running NpgsqlDataSource with timer-based polling (Marten HighWaterAgent) caused NpgsqlConnector accumulation and OOM — resolved by setting GssEncryptionMode=Disable
   • We were unable to reproduce the exact leak pattern in isolation with simple connection tests

   Hope it helps.

   👍React with 👍3Reacted by Brian Pursley, TLS-Terminator and Josh Baldwin
7. jbaldwin-vizio commented on Apr 30, 2026

   

   jbaldwin-vizio

   on Apr 30, 2026

   More actions

   I can confirm setting GssEncryptionMode = GssEncryptionMode.Disable, has resolved this memory leak issue for our project that connects to an AWS RDS Postgres instance. Thank you for the guidance on how to resolve this issue for now.
8. alen-z commented on May 28, 2026

   

   alen-z

   on May 28, 2026

   Last edited by alen-z

   More actions

   We can confirm this is a massive unmanaged memory leak vector in high-throughput environments using .NET 10 and Npgsql 10 inside Linux containers (e.g. mcr.microsoft.com/dotnet/aspnet:10.0-noble) lacking native Kerberos libraries.

   Because our distributed database configuration requires client-side pooling to be disabled (Pooling=false), every query initiates a fresh connection. This forced the GSS-to-SSL fallback loop to execute thousands of times per minute, silently triggering unmanaged exception storms and throwing/catching native TypeInitializationException frames under the hood.

   Our resident memory (VmRSS) climbed linearly at ~620 MiB/hour until OOM-killed. Explicitly setting GssEncryptionMode=Disable on our connection strings completely flatlined this portion of the native memory leak.

   
9. mastersign commented on May 28, 2026

   

   mastersign

   on May 28, 2026

   More actions

   I have a similar issue after upgrading to Npgsql 10 with unmanaged memory growth of 260MB per hour. After changing my implementation from creating a new NpgsqlDataSource and then calling CreateConnection() for each query, to keeping the NpgsqlDataSource for app life time and only calling CreateConnection() for every query, the leak went to 55MB per day.
10. vonzshik commented on May 28, 2026

    

    vonzshik

    on May 28, 2026

    Contributor

    More actions

    We can confirm this is a massive unmanaged memory leak vector in high-throughput environments using .NET 10 and Npgsql 10 inside Linux containers (e.g. mcr.microsoft.com/dotnet/aspnet:10.0-noble) lacking native Kerberos libraries.

    That's, honestly, very concerning and quite unexpected for me. From what I've seen in .NET all the necessary logic to load the native library happens in static ctor, from which InvalidOperationException is thrown if it's unable to load the library. In addition, that specific exception should have been cached by the runtime, so overall it should have been just getting an exception from a few stack frames with close to 0 allocations.
    That's probably something I'll look into later, but in the meantime what we can do is to add a boolean field to check beforehand whether gss library is actually installed or not, and set it to false whenever we hit TypeInitializationException. That way, while the first few requests might still allocate and fail, anything afterwards will completely skip that path.
11. TLS-Terminator commented on May 28, 2026

    

    TLS-Terminator

    on May 28, 2026

    More actions

    @vonzshik have you looked at research above from @brianpursley and @netijoe regarding missing timeouts? The root issue might not be an exception storm
12. vonzshik commented on May 28, 2026

    

    vonzshik

    on May 28, 2026

    Contributor

    More actions

    While the fact that we do not have a timeout for SslStream is certainly an issue, so far it seems more of a fallout from whatever happens with GSS encryption, given that disabling it seems to help. Which is why minimizing any calls to it is better solution, compared to just adding timeout (I'll also do that, just in a separate pr).
    What I'm really confused about is how gss encryption can even affect ssl negotiation. If we fail during getting credentials due to missing native library, then we didn't sent anything over the wire and we just fallback to ssl encryption, so everything happens the exact same way as if gss encryption was disabled. If we fail during gss negotiation, then we scrap everything up to the socket and start anew with gss disabled, so it also should be fine.