gh-125588: Teach the python PEG generator the new f-string tokens (#1… · python/cpython@9dfef4e · GitHub
Skip to content

Commit 9dfef4e

Browse files
authored
gh-125588: Teach the python PEG generator the new f-string tokens (#125589)
Signed-off-by: Pablo Galindo <pablogsal@gmail.com>
1 parent 0e45b1f commit 9dfef4e

6 files changed

Lines changed: 61 additions & 2 deletions

File tree

Lib/test/test_peg_generator/test_pegen.py

Lines changed: 8 additions & 0 deletions
Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
The Python PEG generator can now use f-strings in the grammar actions. Patch
2+
by Pablo Galindo

Tools/peg_generator/pegen/grammar_parser.py

Lines changed: 16 additions & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

Tools/peg_generator/pegen/metagrammar.gram

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -126,6 +126,9 @@ target_atom[str]:
126126
| NAME { name.string }
127127
| NUMBER { number.string }
128128
| STRING { string.string }
129+
| FSTRING_START { fstring_start.string }
130+
| FSTRING_MIDDLE { fstring_middle.string }
131+
| FSTRING_END { fstring_end.string }
129132
| "?" { "?" }
130133
| ":" { ":" }
131134
| !"}" !"]" OP { op.string }

Tools/peg_generator/pegen/parser.py

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -205,6 +205,36 @@ def string(self) -> Optional[tokenize.TokenInfo]:
205205
return self._tokenizer.getnext()
206206
return None
207207

208+
@memoize
209+
def fstring_start(self) -> Optional[tokenize.TokenInfo]:
210+
FSTRING_START = getattr(token, "FSTRING_START")
211+
if not FSTRING_START:
212+
return None
213+
tok = self._tokenizer.peek()
214+
if tok.type == FSTRING_START:
215+
return self._tokenizer.getnext()
216+
return None
217+
218+
@memoize
219+
def fstring_middle(self) -> Optional[tokenize.TokenInfo]:
220+
FSTRING_MIDDLE = getattr(token, "FSTRING_MIDDLE")
221+
if not FSTRING_MIDDLE:
222+
return None
223+
tok = self._tokenizer.peek()
224+
if tok.type == FSTRING_MIDDLE:
225+
return self._tokenizer.getnext()
226+
return None
227+
228+
@memoize
229+
def fstring_end(self) -> Optional[tokenize.TokenInfo]:
230+
FSTRING_END = getattr(token, "FSTRING_END")
231+
if not FSTRING_END:
232+
return None
233+
tok = self._tokenizer.peek()
234+
if tok.type == FSTRING_END:
235+
return self._tokenizer.getnext()
236+
return None
237+
208238
@memoize
209239
def op(self) -> Optional[tokenize.TokenInfo]:
210240
tok = self._tokenizer.peek()

Tools/peg_generator/pegen/python_generator.py

Lines changed: 2 additions & 1 deletion

0 commit comments

Comments
 (0)