I have an interesting challenge where HashSet operations are the bottleneck for my simulation. I decided to see if I could go faster using an int64 array as the backing store and bit operations for updating. It appears that you can if you can make assumptions about your domain.
// * Summary *
BenchmarkDotNet=v0.13.1, OS=Windows 10.0.22000
AMD Ryzen 9 3900X, 1 CPU, 24 logical and 12 physical cores
.NET SDK=6.0.101
[Host] : .NET 6.0.2 (6.0.222.6406), X64 RyuJIT DEBUG
DefaultJob : .NET 6.0.2 (6.0.222.6406), X64 RyuJIT
