gh-119182: Use strict error handler in PyUnicode_FromFormat() by vstinner · Pull Request #120307 · python/cpython · GitHub
Skip to content

gh-119182: Use strict error handler in PyUnicode_FromFormat() #120307

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
vstinner wants to merge 1 commit into from
Closed

gh-119182: Use strict error handler in PyUnicode_FromFormat() #120307

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion Doc/c-api/unicode.rst
View file
Open in desktop
5 changes: 5 additions & 0 deletions Doc/whatsnew/3.14.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -261,6 +261,11 @@ New Features
Porting to Python 3.14
----------------------

* :c:func:`PyUnicode_FromFormat` now decodes the ``"%s"`` format argument from
UTF-8 with the "strict" error handler, instead of the "replace" error
handler.
(Contributed by Victor Stinner in :gh:`119182`.)

Deprecated
----------

Expand Down
3 changes: 1 addition & 2 deletions Lib/test/test_capi/test_getargs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1298,8 +1298,7 @@ def test_nonascii_keywords(self):
self.assertEqual(parse((), {}, '|O', [invalid]), (NULL,))
self.assertEqual(parse((1,), {'b': 2}, 'O|O', [invalid, 'b']),
(1, 2))
with self.assertRaisesRegex(TypeError,
f"function missing required argument '{name}\ufffd'"):
with self.assertRaises(UnicodeDecodeError):
parse((), {}, 'O', [invalid])
with self.assertRaisesRegex(UnicodeDecodeError,
f"'utf-8' codec can't decode bytes? "):
Expand Down
29 changes: 15 additions & 14 deletions Lib/test/test_capi/test_unicode.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -384,12 +384,11 @@ def check_format(expected, format, *args):
check_format('ascii\x7f=unicode\xe9',
b'ascii\x7f=%U', 'unicode\xe9')

# non-ascii format, ascii argument: ensure that PyUnicode_FromFormatV()
# raises an error
self.assertRaisesRegex(ValueError,
r'^PyUnicode_FromFormatV\(\) expects an ASCII-encoded format '
'string, got a non-ASCII byte: 0xe9$',
PyUnicode_FromFormat, b'unicode\xe9=%s', 'ascii')
# "%s" format decodes its argument from UTF-8/strict
check_format('value=\u20ac',
b'value=%s', '\u20ac'.encode())
with self.assertRaises(UnicodeDecodeError):
PyUnicode_FromFormat(b'value=%s', b'invalid\xe9')

# test "%c"
check_format('\uabcd',
Expand All @@ -412,11 +411,13 @@ def check_format(expected, format, *args):
check_format('%abc',
b'%%%s', b'abc')

# truncated string
# test "%s" format with precision
check_format('abc',
b'%.3s', b'abcdef')
check_format('abc[\ufffd',
b'%.5s', 'abc[\u20ac]'.encode('utf8'))
with self.assertRaises(UnicodeDecodeError):
PyUnicode_FromFormat(b'%.5s', 'abc[\u20ac]'.encode('utf8'))
check_format('abc[\u20ac',
b'%.7s', 'abc[\u20ac]'.encode('utf8'))
check_format("'\\u20acABC'",
b'%A', '\u20acABC')
check_format("'\\u20",
Expand All @@ -431,8 +432,8 @@ def check_format(expected, format, *args):
b'%.3U', '\u20acABCDEF')
check_format('\u20acAB',
b'%.3V', '\u20acABCDEF', None)
check_format('abc[\ufffd',
b'%.5V', None, 'abc[\u20ac]'.encode('utf8'))
with self.assertRaises(UnicodeDecodeError):
PyUnicode_FromFormat(b'%.5V', None, 'abc[\u20ac]'.encode('utf8'))

# following tests comes from #7330
# test width modifier and precision modifier with %S
Expand Down Expand Up @@ -723,9 +724,9 @@ class LocalType:
check_format('repr=\u4eba\u6c11',
b'repr=%V', None, b'\xe4\xba\xba\xe6\xb0\x91')

#Test replace error handler.
check_format('repr=abc\ufffd',
b'repr=%V', None, b'abc\xff')
# Test replace the "strict" error handler.
with self.assertRaises(UnicodeDecodeError):
PyUnicode_FromFormat(b'repr=%V', None, b'abc\xff')

# Issue #33817: empty strings
check_format('',
Expand Down
3 changes: 3 additions & 0 deletions Misc/NEWS.d/next/C API/2024-06-07-22-38-08.gh-issue-119182.P3nXBm.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
:c:func:`PyUnicode_FromFormat` now decodes the ``"%s"`` format argument from
UTF-8 with the "strict" error handler, instead of the "replace" error handler.
Patch by Victor Stinner.
20 changes: 6 additions & 14 deletions Objects/unicodeobject.c
View file
Open in desktop