RegexLexer.get_tokens_unprocessed tries each rule of the current state in
turn at every position, so a state with N rules costs up to N re.match()
calls per token.  A run of consecutive simple rules (a regex and a plain
token type, no state change) is equivalent to the single rule

    ('(?P<g0>r0)|(?P<g1>r1)|...', merged(t0, t1, ...))

because regex alternation is tried in order, like the rule loop.

Rather than merging at runtime in the metaclass (PR pygments#3155), this does it
at build time, as the maintainers preferred: scripts/optimize_lexers.py
rewrites the lexer modules with an AST-guided text edit that only touches
the merged rules, and the hatchling hook in hatch_build.py runs it when a
wheel is built.  The sources in the repository and editable installs are
unchanged, so development runs the unmerged rules; PYGMENTS_OPTIMIZE_LEXERS=0
builds a wheel without the optimization.

Rules whose regex uses a named group, backreference, conditional or global
inline flag are left alone, as are lexers using re.VERBOSE and the classes
whose rules other code inspects (PostgresLexer, TeraTermLexer,
Inform6Lexer).  The merged action class lives in pygments.lexer.

The result: 6692 rules merged into 1527 regexes in 216 modules, and with
the benchmark from pygments#3155 (pyperf, Python 3.13) lexing is 1.17x to 1.42x
faster, 1.25x geometric mean.  A new test-wheel CI job runs the whole test
suite against the built wheel.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>