[3.13] gh-155292: Don't consider Unicode codepoint attributes outside… · python/cpython@c28b121 · GitHub
Skip to content

Commit c28b121

Browse files
miss-islingtonencukousethmlarsonStanFromIreland
authored
[3.13] gh-155292: Don't consider Unicode codepoint attributes outside RFC 3454 (GH-155293) (GH-156020) (GH-156922)
Due to a bug, some Unicode codepoint attributes were considered for characters not yet defined in Unicode 3.2.0 or attributes which changed in later Unicode versions. RFC 3454 (StringPrep) requires using Unicode 3.2.0 strictly. (cherry picked from commit 7e109d0) The cherry-pick needed reworking as GH-144815 wasn't backported to 3.14 and below, so unassigned characters don't have bidi values. (cherry picked from commit 1e54caa) Co-authored-by: Petr Viktorin <encukou@gmail.com> Co-authored-by: Seth Larson <seth@python.org> Co-authored-by: Stan Ulbrych <89152624+stanfromireland@users.noreply.github.com> Co-authored-by: Petr Viktorin <encukou@gmail.com> --------- Co-authored-by: Petr Viktorin <encukou@gmail.com> Co-authored-by: Seth Larson <seth@python.org> Co-authored-by: Stan Ulbrych <89152624+stanfromireland@users.noreply.github.com>
1 parent a4ab2c1 commit c28b121

9 files changed

Lines changed: 428 additions & 183 deletions

File tree

‎Lib/stringprep.py‎

Lines changed: 327 additions & 150 deletions
Large diffs are not rendered by default.

‎Lib/test/test_codecs.py‎

Lines changed: 9 additions & 0 deletions

‎Lib/test/test_unicodedata.py‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -319,6 +319,15 @@ def test_bidirectional(self):
319319
self.assertRaises(TypeError, self.db.bidirectional)
320320
self.assertRaises(TypeError, self.db.bidirectional, 'xx')
321321

322+
def test_bidirectional_unassigned(self):
323+
self.assertEqual(self.db.bidirectional('\u0378'), '')
324+
self.assertEqual(self.db.bidirectional('\u077F'), '' if self.old else 'AL')
325+
self.assertEqual(self.db.bidirectional('\u20CF'), '')
326+
self.assertEqual(self.db.bidirectional('\u0590'), '')
327+
self.assertEqual(self.db.bidirectional('\uFFFF'), '')
328+
self.assertEqual(self.db.bidirectional('\U0001FFFE'), '')
329+
self.assertEqual(self.db.bidirectional('\U00010D01'), '' if self.old else 'AL')
330+
322331
def test_decomposition(self):
323332
self.assertEqual(self.db.decomposition('\uFFFE'),'')
324333
self.assertEqual(self.db.decomposition('\u00bc'), '<fraction> 0031 2044 0034')

‎Misc/NEWS.d/next/Security/2026-08-06-11-43-20.gh-issue-155292.j4pHBO.rst‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
Change the :mod:`stringprep` module and :mod:`encodings.idna` codec to not
2+
consider Unicode codepoint attributes beyond those defined in :rfc:`3454`.

‎Modules/unicodedata_db.h‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Modules/unicodename_db.h‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Objects/unicodetype_db.h‎

Lines changed: 1 addition & 1 deletion
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Tools/unicode/makeunicodedata.py‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@
2828

2929
import dataclasses
3030
import os
31+
import subprocess
3132
import sys
3233
import zipfile
3334

@@ -126,6 +127,7 @@ def maketables(trace=0):
126127
makeunicodename(unicode, trace)
127128
makeunicodedata(unicode, trace)
128129
makeunicodetype(unicode, trace)
130+
makestringprep()
129131

130132

131133
# --------------------------------------------------------------------
@@ -711,6 +713,19 @@ def makeunicodename(unicode, trace):
711713
fprint(' "%s",' % prefix)
712714
fprint('};')
713715

716+
717+
def makestringprep():
718+
FILE = "Lib/stringprep.py"
719+
720+
print("--- Preparing", FILE, "...")
721+
722+
MKSTRINGPREP = "Tools/unicode/mkstringprep.py"
723+
724+
with open(FILE, "w") as f:
725+
f.truncate()
726+
subprocess.check_call([sys.executable, MKSTRINGPREP], stdout=f)
727+
728+
714729
def merge_old_version(version, new, old):
715730
# Changes to exclusion file not implemented yet
716731
if old.exclusions != new.exclusions:

‎Tools/unicode/mkstringprep.py‎

Lines changed: 63 additions & 30 deletions

0 commit comments

Comments
 (0)