Repository navigation
Merge runs of consecutive simple rules in RegexLexer into one regex - #3155
eendebakpt wants to merge 6 commits into
Conversation
RegexLexer.get_tokens_unprocessed tries each rule's compiled regex in turn at every position, so a state with N rules can cost up to N match() calls per character. For most lexers the common tokens (whitespace, names, numbers, operators, ...) sit in a long run of plain-token rules that are all attempted on every identifier character. Merge maximal runs of *consecutive* "simple" rules -- a plain _TokenType action, no state transition, shared flags, and a foldable pattern (no named group, backreference, or global inline flag) -- into a single combined regex ``(?P<g0>r0)|(?P<g1>r1)|...``. Python's alternation is leftmost-match, so this is exactly equivalent to trying those rules in order; non-simple rules stay in place as barriers, preserving every rule's relative order. Dispatch to the matched rule's token via the capturing-group index, which is robust even when a rule has inner groups. The transformation is output-preserving and on by default; set ``RegexLexer.merge_simple_rules = False`` to disable. This roughly halves the number of per-position match attempts (PythonLexer's root state: 56 -> 32 entries) for ~1.2x faster lexing, with no change to the emitted token stream (verified against the full test suite and a new parity test across bundled lexers). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
|
Thanks for the PR, this is very interesting! Reducing the amount of match calls seems like an easy win. I'll probably not able to review very soon, but I'll get to it (or maybe @Anteru of course) |
|
I like this. I think we'll want that in the next major release. In general, there's lots of potential in cleaning up the regex rules, either during build time, or at run time. That's really the main question here, if this should be a compile/build time transformation which emits "optimized" lexers, and you can opt to use the optimized one or the base one. |
I don't think we should offer a choice, since it forces users into a tradeoff that's hard to make in many cases. That's why I find the "build time" option intriguing. Of course there's also always the option to offload the glue to something like Rust, although I think the majority of time is spent matching regexes, and the engine itself is not particularly inefficient. |
|
My preference (for maintainability) would be "this is on by default" and "it's baked in during the build before tests run", so the only situation in which you can turn it off is during development if you suspect it's broken. Not sure how much effort it is to make this a build-time thing, similar to mapfiles and other things (in general I find Python build scripts a bit finicky.) |
|
Yep, agreed. (I don't know how to run things pre-build with uv, but at least with the mapfiles we have chosen to update those independent of the build process...) |
I looked into this a bit, but creating a build time option (e.g. optimize the rules and only ship the optimized set to the final pip package) turns out to be a bit messy. Maybe someone more familiar with pygments can do a better job, or point me in the right direction. |

RegexLexer.get_tokens_unprocessedtries each rule's regex in turn at every position, so a state with N rules costs up to Nre.match()calls per character. In practice the common tokens (whitespace, names, numbers, operators, …) sit in a long run of plain-token rules that are attempted on every identifier character.This merges maximal runs of consecutive simple rules in a state into one combined regex
(?P<g0>r0)|(?P<g1>r1)|…. A rule is simple when it has a plain_TokenTypeaction, no state transition, shared flags, and a foldable pattern(no named group, backreference, or global inline flag). Python's alternation is leftmost-match, so the combined regex is exactly equivalent to trying those rules in order; non-simple rules stay in place as barriers, preserving every rule's relative order. The matched rule is recovered from the capturing-group index, so it works even when a rule has inner groups.
The transformation is output-preserving and on by default; set
RegexLexer.merge_simple_rules = Falseto disable it. ForPythonLexerit halves the per-position match attempts in therootstate (56 → 32 entries).Benchmark
pyperf compare_to, lexing a representative multi-screen source file with eachlexer (script below):
bench_lexers.py
Impact on IPython
This optimization came out of investigations of the latency of the python/ipython REPL. IPython highlights its input prompt with
PygmentsLexer(Python3Lexer), re-lexing the visible buffer on every keystroke (prompt_toolkit keeps no cross-keystroke token cache). This change cuts that per-keystroke lexing cost by ~1.27x with identical highlighting, so typing latency in the terminal REPL improves proportionally — most noticeably while editing larger multi-line cells, where lexing dominates the per-keystroke work.