Npgsql 7.0.2 blocks threads NpgsqlConnector.PerformUserCancellation causing connection pool exhaustion · Issue #5032 · npgsql/npgsql · GitHub
Skip to content

Npgsql 7.0.2 blocks threads NpgsqlConnector.PerformUserCancellation causing connection pool exhaustion #5032

Closed
Bug
#5033
Closed
Npgsql 7.0.2 blocks threads NpgsqlConnector.PerformUserCancellation causing connection pool exhaustion#5032
Bug
#5033
Assignees
vonzshik
Milestone
7.0.4

Description

@JColeGeotab
JColeGeotab
opened on Apr 12, 2023

Steps to reproduce

I have not been able to create a sample to reproduce yet, and I'm not able to share our source code. I will keep trying and share if I can reproduce.

The issue

After upgrading Npgsql NuGet package rom 7.0.1 to 7.0.2 we are seeing idle connections not getting closed after several days. The number of open connections gradually increased until hitting our max pool size (1000), after which every query failed since the connection pool was exhausted.

Our process thread count also increased until it plateued around 1000 when the max pool size was reached.

Reverting the Npgsql NuGet package back to 7.0.1 fixed the issue.

Stack trace:
 System.Private.CoreLib!System.Threading.ManualResetEventSlim.Wait(int32,value class System.Threading.CancellationToken)
  System.Private.CoreLib!System.Threading.ManualResetEventSlim.Wait()
  Npgsql!Npgsql.Internal.NpgsqlConnector.PerformUserCancellation()
  Npgsql!Npgsql.Internal.NpgsqlConnector+<>c.<StartNestedCancellableOperation>b__262_0(class System.Object)
  System.Private.CoreLib!System.Threading.CancellationTokenSource.Register(class System.Delegate,class System.Object,class System.Threading.SynchronizationContext,class System.Threading.ExecutionContext)
  Npgsql!Npgsql.NpgsqlDataReader+<NextResult>d__47.MoveNext()
  System.Private.CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start(!!0&)
  Npgsql!Npgsql.NpgsqlDataReader.NextResultAsync(value class System.Threading.CancellationToken)
  Npgsql!Npgsql.NpgsqlCommand+<ExecuteReader>d__124.MoveNext()
  System.Private.CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start(!!0&)
  Npgsql!Npgsql.NpgsqlCommand.ExecuteReader(value class System.Data.CommandBehavior,bool,value class System.Threading.CancellationToken)
  Npgsql!Npgsql.NpgsqlCommand.ExecuteReaderAsync(value class System.Data.CommandBehavior,value class System.Threading.CancellationToken)
  Npgsql!Npgsql.NpgsqlCommand+<ExecuteDbDataReaderAsync>d__117.MoveNext()
  System.Private.CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start(!!0&)
  Npgsql!Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(value class System.Data.CommandBehavior,value class System.Threading.CancellationToken)

I noticed there is a new ManualResetEventSlim added and used in the PerformUserCancellation method that was not there in 7.0.1 as well as some other changes in this method.

https://github.com/npgsql/npgsql/blob/v7.0.2/src/Npgsql/Internal/NpgsqlConnector.cs#L1720

Further technical details

Npgsql version: 7.0.2 (issue is not present in 7.0.1)
PostgreSQL version: 13
Operating system: Ubuntu

Activity

  1. vonzshik commented on Apr 12, 2023

    @vonzshik
    vonzshik
    on Apr 12, 2023
    Contributor

    Yeah, had to add ManualResetEventSlim there to fix #4906. Just in case, are there any other stuck threads? I'm going to take a look in the next few days but any info will be helpful.

  2. vonzshik commented on Apr 12, 2023

    @vonzshik
    vonzshik
    on Apr 12, 2023
    Contributor

    Looking a bit more at the code, I think I know where's the issue... gonna test tomorrow to confirm.

  3. JColeGeotab commented on Apr 12, 2023

    @JColeGeotab
    JColeGeotab
    on Apr 12, 2023
    Author

    Nope, the threads stuck in that stack are the only threads stuck in Npgsql code.

  4. vonzshik commented on Apr 12, 2023

    @vonzshik
    vonzshik
    on Apr 12, 2023
    Contributor

    Yeah, OK. The logic there is, that we first reset mres, then register a callback on CancellationToken, and then read from the socket (and setting mres at some point). Now, registering a callback on CancellationToken can immediately trigger that callback synchronously in case if CancellationToken is already cancelled. So I guess that's what happens. The thread which is supposed to read from the socket actually stuck on mres.

  5. JColeGeotab commented on Apr 12, 2023

    @JColeGeotab
    JColeGeotab
    on Apr 12, 2023
    Author

    If it helps, there were a few threads that had a slightly different stack.

    In the dotnet-stack output we captured from a process with 446 blocked threads, there were 16 with the stack I pasted above, and 430 with the shorter stack below, missing the StartNestedCancellableOperation line.

      System.Private.CoreLib!System.Threading.ManualResetEventSlim.Wait(int32,value class System.Threading.CancellationToken)
  Npgsql!Npgsql.Internal.NpgsqlConnector.PerformUserCancellation()
  System.Private.CoreLib!System.Threading.CancellationTokenSource.Register(class System.Delegate,class System.Object,class System.Threading.SynchronizationContext,class System.Threading.ExecutionContext)
  Npgsql!Npgsql.NpgsqlDataReader+<NextResult>d__47.MoveNext()
  System.Private.CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start(!!0&)
  Npgsql!Npgsql.NpgsqlDataReader.NextResultAsync(value class System.Threading.CancellationToken)
  Npgsql!Npgsql.NpgsqlCommand+<ExecuteReader>d__124.MoveNext()
  System.Private.CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start(!!0&)
  Npgsql!Npgsql.NpgsqlCommand.ExecuteReader(value class System.Data.CommandBehavior,bool,value class System.Threading.CancellationToken)
  Npgsql!Npgsql.NpgsqlCommand.ExecuteReaderAsync(value class System.Data.CommandBehavior,value class System.Threading.CancellationToken)
  Npgsql!Npgsql.NpgsqlCommand+<ExecuteDbDataReaderAsync>d__117.MoveNext()
  System.Private.CoreLib!System.Runtime.CompilerServices.AsyncMethodBuilderCore.Start(!!0&)
  Npgsql!Npgsql.NpgsqlCommand.ExecuteDbDataReaderAsync(value class System.Data.CommandBehavior,value class System.Threading.CancellationToken)

    Diff:
    image

  6. vonzshik commented on Apr 12, 2023

    @vonzshik
    vonzshik
    on Apr 12, 2023
    Contributor

    The stack might be different due to jit recompiling that method from tier 0 (fast compilation, minimum of optimizations) to tier 1 (longer compilation, more optimization options, like method inlining). But looking at the stack trace it does indeed confirm that I was right. If the callback isn't called synchronously, we would have gotten a stack trace like this:

    at ConsoleApp30.Program.<>c.<MyMethod>b__1_0()
at System.Threading.CancellationToken.<>c.<Register>b__12_0(Object obj)
at System.Threading.CancellationTokenSource.Invoke(Delegate d, Object state, CancellationTokenSource source)
at System.Threading.CancellationTokenSource.CallbackNode.<>c.<ExecuteCallback>b__9_0(Object s)
at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state)
  7. vonzshik
    added this to the 7.0.4 milestone on Apr 12, 2023
  8. vonzshik
    added
    bug
    on Apr 12, 2023
  9. vonzshik
    self-assigned this
    on Apr 12, 2023
  10. vonzshik
    added 2 commits that reference this issue on Apr 13, 2023

    Fix deadlock while cancelling a query

    844c629

    Fix deadlock while cancelling a query

    79f235d
  11. vonzshik
    closed this as completedin #5033on Apr 23, 2023
  12. vonzshik
    added 4 commits that reference this issue on Apr 23, 2023

    Fix deadlock while cancelling a query (#5033)

    276bb7d

    Fix deadlock while cancelling a query (#5033)

    72bef2b

    Fix deadlock while cancelling a query (#5033)

    106a012

    Fix deadlock while cancelling a query (#5033)

    3881c6c

  13. JColeGeotab commented on Apr 24, 2023

    @JColeGeotab
    JColeGeotab
    on Apr 24, 2023
    Author

    Thanks for the quick turnaround! As soon as 7.0.4 is available we should be able to quickly confirm that our issue is resolved.

  14. vonzshik commented on Apr 24, 2023

    @vonzshik
    vonzshik
    on Apr 24, 2023
    Contributor

    @JColeGeotab you can also test using myget feed.

  15. roji
    added the
    Bug
    issue type on Jun 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Bug

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions