The other two are obvious but I was curious about the removal of substring and use of span. This benchark: https://gist.github.com/cartermp/da23761991bd1cf0f179edfe1d00b737

yielded these results:

BenchmarkDotNet=v0.12.1, OS=Windows 10.0.19041.388 (2004/?/20H1)
Intel Core i7-6700K CPU 4.00GHz (Skylake), 1 CPU, 8 logical and 4 physical cores
.NET Core SDK=5.0.100-preview.6.20318.15
  [Host]        : .NET Core 3.1.6 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.31603), X64 RyuJIT DEBUG
  .NET 4.7.2    : .NET Framework 4.8 (4.8.4180.0), X64 RyuJIT
  .NET Core 3.1 : .NET Core 3.1.6 (CoreCLR 4.700.20.26901, CoreFX 4.700.20.31603), X64 RyuJIT

(note: github struggles to render the actual proper values in tables)

To note: The only reason why I made the comment that I did was because we have to use AsSpan and depending on the case, can make me a little nervous in terms of readability if we start doing that everywhere.

I was just browsing the repo and noticed these things could be improved, so that's what I did thinking any performance uplift (and I'm pretty sure this change will have a completely negligible impact on its own) is better than nothing, admittedly with no regard to readability. That's all there is to say, really ;)

These changes may not actually be miniscule - one of these cases removes a new string allocation for every single span of text we classify in a document. For large documents that could be meaningful.

@cartermp, how would one go about measuring the difference on a sizeable file/codebase?

Measuring the difference is best done with PerfView and capturing a trace before/after your changes for the same codebase and operations done in the codebase. It's kind of annoying and time-consuming work. In general I think this is fine to take without advanced analysis since the improvements are pretty obvious. But for less obvious things we would at least need to see some more information about the current performance characteristics and a writeup about how the new code improves on it.