Repository navigation
Python: keep quoted named-arg values as single tokens in CodeTokenizer - #14461
Manohar Paturi (ManoharPaturi) wants to merge 1 commit into
Conversation
GitHub Actions
Failure
GitHub Actions
Failure
A template like {{ plugin.function arg1='value with spaces' }} failed with
TemplateSyntaxError because the tokenizer never entered value mode for the
quote after name=: the space inside the quoted value ended the token early,
producing the invalid block key='value instead of a NamedArgBlock.
Enter VALUE mode when a quote directly follows the = of a named argument and
flush the token as a NamedArgBlock, matching how positional quoted values
with spaces already worked.
Signed-off-by: Manohar Paturi <186662190+ManoharPaturi@users.noreply.github.com>
f20e8c1 to
c2e51dc
Compare
GitHub Actions
Failure
PRABHU KIRAN VANDRANKI (VANDRANKI)
left a comment
PRABHU KIRAN VANDRANKI (VANDRANKI)
left a comment
There was a problem hiding this comment.
Community review (does not clear the merge gate).
I read the full diff, applied it on upstream main (5b4a326) in a separate worktree and ran it from python/ with Python 3.13.
The bug is real and the fix works end to end. On main, a quoted value with a space in a named argument fails to tokenize. I rendered templates through KernelPromptTemplate with a plugin function p.echo(arg1, arg2="-"):
{{p.echo arg1='a b'}}: main raisesTemplateSyntaxError(wrappingNamedArgBlockSyntaxError); the PR renders[a b|-].{{p.echo arg1="a b" arg2='c d'}}: PR renders[a b|c d].{{p.echo arg1='a b' arg2=$v}}withv="VV": PR renders[a b|VV].{{p.echo arg1='ab'}}and a positional{{p.echo 'x y'}}: same on both, so the existing paths are unchanged.
Tokenizer-level checks (main versus PR): p.f 'pos val' a='x y' and p.f a='x y' 'pos2' now give FUNCTION_ID, VALUE and NAMED_ARG blocks in order; p.f a='x y' b="q r" (two spaces) gives two NAMED_ARG blocks. tests/unit/template_engine: 454 passed with the PR applied. I did not run the new tests against main; the script above is my evidence for main's behavior.
Things that still fail after the PR (all fail on main too, so not regressions):
- Empty quoted value,
p.f a='' b='z':NamedArgBlockSyntaxError. - An escaped quote inside a named value,
arg1='a\'b c': the tokenizer drops the backslash and yieldsarg1='a'b c', whichNamedArgBlockrejects. The positional equivalent'a\'b c'tokenizes to a VALUE block, so named and positional values still differ here. - Spaces around
=,a = 'x': error.
Small point: the diff changes an existing test case from ("fo.o ", "fo.o") to ("a.b ", "a.b") in test_it_parses_function_id_blocks. Both inputs tokenize the same way on main and on the PR in my run (one FUNCTION_ID block), so I do not see why that edit is needed.
The change looks correct for the reported case.
Fixes #14460.
the tokenizer now carries a quote-aware state while scanning tokens so a quoted value stays one token even when it contains spaces, matching the ValBlock grammar. template tests for both quote styles fail on main and pass here.