Rewrite _tokenize with 2-phase model · RustPython/RustPython@55737ed · GitHub
Skip to content

Commit 55737ed

Browse files
committed
Rewrite _tokenize with 2-phase model
Replace per-line reparsing with single-pass tokenization: - Read all lines via readline, parse once, yield tokens - Fix token type values (COMMENT=65, NL=66, OP=55) - Fix NEWLINE/NL end positions and implicit newline handling - Fix DEDENT positions via look-ahead to next non-DEDENT token - Handle FSTRING_MIDDLE brace unescaping ({{ → {, }} → }) - Emit implicit NL before ENDMARKER when source lacks trailing newline - Raise IndentationError from lexer errors - Remove 13 expectedFailure marks for now-passing tests
1 parent bf2b993 commit 55737ed

5 files changed

Lines changed: 750 additions & 407 deletions

File tree

Lib/test/test_tabnanny.py

Lines changed: 0 additions & 1 deletion

Lib/test/test_tokenize.py

Lines changed: 0 additions & 13 deletions

0 commit comments

Comments
 (0)