TableGenLexer is Ocassionally Hanging during Random Input Unit Tests · Issue #2940 · pygments/pygments · GitHub
Skip to content

TableGenLexer is Ocassionally Hanging during Random Input Unit Tests #2940

Closed
Closed
TableGenLexer is Ocassionally Hanging during Random Input Unit Tests#2940

Description

@Spill-Tea
Spill-Tea
opened on Jul 28, 2025

Seven workflows thus far have failed unit testing by timing out after 6 hours. This occurs during random testing of the lexers.

Specifically the unit test: test_basic_api.py::test_random_input. These appear to be attributed to the TableGenLexer. This lexer appears to have been added a year ago, with the latest code change happening ~ 9 months ago. These specific failures started popping up ~ 2 months ago, independent of OS and version.

I don't think this can be attributed to any specific recent change to this lexer. It is likely worth debugging this lexer, using snippets of this random input as bait, to see which regex expression is causing issue.

Activity

  1. Spill-Tea commented on Jul 28, 2025

    @Spill-Tea
    Spill-Tea
    on Jul 28, 2025
    ContributorAuthor

    For reference:

    # example 1 (Windows, 3.11)
'T>Mvcz/CE6R+A;HJ\'^0D[6x<uW-7gpMOBY{yp1kmGVg)g:T:?XcFr5EL*7QM&KSI\x7f_Pi#4)o*@E]m|\\Nn%s\\T&}bXPoG$4YYr=-XSf.Z&2`u>9B[{yDFy^D9,aBkm,Hu)3:y9Z;cepO#?*n2|5]+`LD5mboj=+eL=h~bQi^7l/<>@jN~&"]lnUeCRH(}`\'0_JgbV\\Ck1vf{<M%w0MXK<V-\x7f%PAQV"L7I-?_pr,#Uj1(=0]UGTltFLz/=zW_1+^A.sKaKkuZZSBbd%I[fw*OT6it.@$C~Y\x7f$z2Z~S\\}+\']@?XtxFD`"8}A0aH4"i)lf6q4!Ah3{W3rF!5qxh9sx-<JPsaq2)t8nh}6@uV#;!\'yw|[[~r`vaoO\'j(^GHW/Q3:8R;eI!\\cGkIf9*pUc:hBN>4xlE"J{moO8JgqK.#EqSvNdd3?w&.,$(;\x7f8|N\x7fsU7Y/w(!P1%id5_vt2R>jzdne|,CRWQ$\n'

# example 2 (Windows, 3.13)
'I\'h>Az45X3WIeE/"ZYSCFK\x7fxYKq,9M][/~N^rm/p0jaiCj<$u+6Y*{DU;mzF-j!{]rbb~v;(d~P%:Q"7zo*mT&#c&AR?97"*wkvgnGAj3T14fQh*tGiw<)$wo{E018c_`"qD7aMRQ.YUru_09}N>]%W\x7fB8|GO=^m\\p}y\\sx$T.2%V6v~eB?Pet8-kh,Ez-qlM$z?i3eWRuV\'S5DxRA-[xNq\\\x7f](,F\'+MS-+_*oqdHk+rC:!<QOfL|3IYgcBHRb?@c}EV)D>\x7f65>kJL8V4Z2Ox,Tw[{@U2/5P![#P^}1"@jQb4JG\\uNCXWGSgVX72rcs.t&1)o\\%Zw#`O<PKFl|\x7f~Z2H/^4Z8lbd;i;{_Ast)#JypdyK=U,%0&:=`Nf9dgaB(a:;nDLs$n!KLIEIyC|gMSohlp6vp.3_[.(Umf`^50)>1Tf=]<huteyLa6:XFB\'sk`@W9X7\'iJl(#JHnn+v!|O@&H?}=\n'

# example 3 (Windows, 3.11)
')n!Cmh<d\x7fhr]]w,6m7Cbq.|VS[{[QA8[5$.qrJtM~:+9HEz07]5>UVqG403H#PYXT|\x7fY4}1&oaR8E*e*u%y",A9Zc31![%>14fw\\Fj^0#x@nTn\\iyKLuK+OxGDd&q1$K>:wu+X5TUrO:/]/B?^\x7fNVg:M.G}sMj"xJf2OLWL<,$3lkqQrz.^_Wbjf9`%bV5C"lc;etg\'i(vycZ*sOppI^YCnT7[D2|{Qbff(BPk7I7_@8E~:\'*B\'Go}DxLt#)zJ--ec?\x7f,<);w4)/6"5=vF-ECiN`r$omQ`N>~h2a0v9MN.\'d@@Yw62`4n=I%^Ds*URkoFv-P}#/G`N}A{dA?gX\\6(mQeo{?BR$tp\\{B8LOj|t&y_>s\x7fUW(|-?1+HvZ&3)zZ(S6=FpxyH3u,J&]_Vechi_u#XiRW~08IZ2X@bPSa!JlMEH"Rj~!TY;S;<gldI\'\\=KSUFm<+9DAlWza/=ag;KpPhk!sk%\n'

# example 4 (Windows, 3.10)
'_RYYSd\'B}lv8qVSw\x7fNSZX\\}(4Tba\'+k3YZe9=Oxb/"!R%\x7fwImH-HLrD>k\'^q-G10:`}xev3Aw2yD|e@?jo4L?m1**Mg[oW="h`rjV$?sp%Xz-ALu/6z31h$gi5cCU;l[{%aW&LP#!6*\\\'D;~mrj_#;DZv:>nCq#YTucY)A<r8.L=N}pv,8P|KiuNnj^,i)JgKt~$\\QB4o,F\x7fth_%kW<2\x7f];em.,f]e5G^znfyiHMHl\\fKq(&/QO$|20xJH(z/87{A)~0cR.t)gUE6B(E]bE8k6&95h"2Q>a;&J#xFMmd"":q(W}2UQM9^<y3o*?I$O`FA{9uXl.@!_\\%Vp`_bGZnIKOgt@[Jfuk,sCsdBy#hrNpSvs`-]{3dtFcEj65S+l4I9&QU:[U1*fPx7+\'0?5ET]7n+@!Xid=R^VXT7<pWOz71cF~G)=<P{P\x7f>/Z:bB|DysaR>IJM+NCG!0@-Vw|aTC.o~4[Kw\n'

    We also had failures on Ubuntu versions 3.10 and 3.13. These don't provide the same output, but fail during test_basic_api.py. I suspect this is a difference in displaying raised error messages with pytest between OS.

    It may also be worth implementing a timeout context manager for this specific unit test, to prevent 6 hour runs.

  2. Spill-Tea commented on Aug 1, 2025

    @Spill-Tea
    Spill-Tea
    on Aug 1, 2025
    ContributorAuthor

    I've at least come the the conclusion, that the issue is the code-block section

    "root": [
...
(r"\[\{", Text, "codeblock"),
...
]
"codeblock": [
    (r"\}\]", Text, "#pop"),
    (r"([^}]+|\}[^]])*", using(CppLexer)),
],

    The lexer is entering this context with these 4 examples, and is unable to escape because no final "}]" is observed.

    This easiest way to reproduce this error is using the text snippet "[{".

  3. kurtmckee commented on Sep 25, 2025

    @kurtmckee
    kurtmckee
    on Sep 25, 2025
    Contributor

    Copying a bit of text from the duplicate issue I opened.

    In addition to the reproducer @Spill-Tea posted ("[{"), this can be reproduced in master with the following command:

    tox run -- --randomly-seed=2325091130

    @frasercrmck, I'm tagging you as the original author of the code from #2751, in case you've got availability and expertise to review this.

  4. Spill-Tea commented on Sep 25, 2025

    @Spill-Tea
    Spill-Tea
    on Sep 25, 2025
    ContributorAuthor

    The easiest way to fix this is to modify the codeblock context using the CppLexer to #pop from the context.

    (r"([^}]+|\}[^]])*", using(CppLexer), "#pop"),

    The only problem, that I couldn't resolve, is it adds an extra new line. Such that it fails the input text -> tokens -> reserialize output text test, where input != output.

    Example:
    Input: "[{}]"
    Output: "[{}]\n"

  5. kurtmckee commented on Sep 25, 2025

    @kurtmckee
    kurtmckee
    on Sep 25, 2025
    Contributor

    I believe that's expected behavior.

    pygments/pygments/lexer.py

    Lines 90 to 92 in 0328cfa

    Are you able to submit a PR to address the freeze issue?

  6. Spill-Tea
    added a commit that references this issue on Sep 26, 2025

    fix(lexers.tablegen): Correct potential infinite loop of TableGenLexe…

    1ae3c76

  7. Spill-Tea commented on Sep 26, 2025

    @Spill-Tea
    Spill-Tea
    on Sep 26, 2025
    ContributorAuthor

    I went ahead and pushed a PR for this fix. Noting the caveat above regarding the extra new line included in output of a complete code block, unless the default behavior is overridden by the ensurenl=False keyword.

    This can be important, because instead of hanging for six hours, the same unit test test_basic_api.py::test_random_input may stochastically fail, because it tries to confirm both input and output of lexing are identical.

  8. birkenfeld
    added a commit that references this issue on Feb 20, 2026

    Tablegen lexer fix (#2972)

    96fff34

  9. Spill-Tea commented on Feb 20, 2026

    @Spill-Tea
    Spill-Tea
    on Feb 20, 2026
    ContributorAuthor

    PR #2972 addressing this issue has been merged. Closing Issue.

  10. Spill-Tea
    closed this as completedon Feb 20, 2026
  11. penberg
    added a commit that references this issue on Apr 22, 2026

    Merge 'Bump pygments from 2.19.1 to 2.20.0' from app/dependabot

    ce76ce7
  12. orrinfrazier
    added a commit that references this issue on Jul 28, 2026

    Merge 'Bump pygments from 2.19.1 to 2.20.0' from app/dependabot

    d83582a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions