What

Fixes #3418 — the TNTLexer never terminates on malformed input without a trailing newline when ensurenl=False.

The outer except IndexError: handler in get_tokens_unprocessed() called self.error_till_line_end(start, text) but dropped the position that call returns. start and end therefore stayed where they were, and the while start <= end < len(text) loop came back to the same offset forever. Every other call site in the lexer assigns the returned position.

The fix is that one line:

-                self.error_till_line_end(start, text)
+                start = end = self.error_till_line_end(start, text)

error_till_line_end() always returns a position past start (it either stops at the newline or runs to the end of the text), so the recovery now always makes progress.

Tests

tests/test_tnt.py gains test_error_recovery_progress, which is the test sketched in the issue: it replaces error_till_line_end on the instance with a wrapper that raises when the same (start, source) recovery state is seen twice. The unpatched lexer therefore fails the test instead of hanging, and the test also checks that the token stream still round-trips to the input.

Verified locally:

• the new test fails on the unpatched lexer (4 cases, 0.22 s) and passes with the fix
• pytest -q --ignore=tests/contrast — 5333 passed, 16 skipped
• pytest tests/test_tnt.py — 73 passed
• ruff check --ignore UP031 ., scripts/check_crlf.py, scripts/detect_missing_analyse_text.py --skip-no-aliases, scripts/check_sources.py, scripts/count_token_references.py — all clean
• a differential run over 4368 inputs of one to three characters: for every input the unpatched lexer can terminate on, the token stream is identical to the patched one; the unpatched lexer does not terminate on 273 of them; the patched lexer terminates on all of them

@archetelos — your report asked whether such a pull request would be welcome, so this is the one-line change plus the test as you sketched it. The open policy question from the issue (whether ensurenl=False should be supported for this lexer, or whether returning error tokens is the right answer) is left to you and the maintainers: the change here only makes the existing error path terminate and does not alter the tokens produced for well-formed input.

AI usage disclosure: the fix and the test were prepared with AI assistance, based on the analysis in the issue; all test results above were produced locally and reviewed.