gh-95555: Support Unicode property escapes \p{...} in regular express… · python/cpython@794b42f · GitHub
Skip to content

Commit 794b42f

Browse files
serhiy-storchakaclaude
serhiy-storchaka
and
claude
authored
gh-95555: Support Unicode property escapes \p{...} in regular expressions (GH-151969)
Add support for \p{property} and \P{property} escapes in Unicode (str) regular expressions, for the properties the engine can resolve without the unicodedata database. They are matched as CATEGORY opcodes or as fixed sets of character ranges. Supported in this change: many General_Category values (the groups L, N, Z, C and the values Lu, Lt, Lm, Nd, Nl, No, Zs, Zl, Zp, Cc, Cf, Cs, Co and Cn); the binary properties Alphabetic, Lowercase, Uppercase, Numeric, Printable, XID_Start, XID_Continue, Cased and Case_Ignorable; the POSIX compatibility classes; the code-point classes ASCII, Any, Assigned, Noncharacter_Code_Point, Join_Control, Pattern_Syntax and Pattern_White_Space; and Regional_Indicator, ASCII_Hex_Digit and Hex_Digit. Property and value names use loose matching (UAX #44 UAX44-LM3), so a property may be spelled \p{Lu}, \p{gc=Lu} or \p{name=yes}. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1 parent 908f438 commit 794b42f

10 files changed

Lines changed: 854 additions & 7 deletions

File tree

‎Doc/library/re.rst‎

Lines changed: 46 additions & 1 deletion

‎Doc/whatsnew/3.16.rst‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -192,6 +192,13 @@ re
192192
matches an ASCII lowercase consonant.
193193
(Contributed by Serhiy Storchaka in :gh:`152100`.)
194194

195+
* Regular expressions now support Unicode property escapes ``\p{...}`` and
196+
``\P{...}``, which match a character by a Unicode property -- for example
197+
``\p{Lu}`` (an uppercase letter), ``\p{Cased}`` or ``\p{ASCII}``. See
198+
:ref:`the regular expression syntax <re-syntax>` for the supported
199+
properties.
200+
(Contributed by Serhiy Storchaka in :gh:`95555`.)
201+
195202

196203
shlex
197204
-----

‎Lib/re/_constants.py‎

Lines changed: 63 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,7 @@
1313

1414
# update when constants are added or removed
1515

16-
MAGIC = 20230612
16+
MAGIC = 20260622
1717

1818
from _sre import MAXREPEAT, MAXGROUPS # noqa: F401
1919

@@ -150,6 +150,35 @@ def _makecodes(*names):
150150
'CATEGORY_UNI_SPACE', 'CATEGORY_UNI_NOT_SPACE',
151151
'CATEGORY_UNI_WORD', 'CATEGORY_UNI_NOT_WORD',
152152
'CATEGORY_UNI_LINEBREAK', 'CATEGORY_UNI_NOT_LINEBREAK',
153+
154+
# Unicode property categories. These are not affected by the ASCII,
155+
# LOCALE or UNICODE flags.
156+
'CATEGORY_ALPHA', 'CATEGORY_NOT_ALPHA',
157+
'CATEGORY_LOWER', 'CATEGORY_NOT_LOWER',
158+
'CATEGORY_UPPER', 'CATEGORY_NOT_UPPER',
159+
'CATEGORY_NUMERIC', 'CATEGORY_NOT_NUMERIC',
160+
'CATEGORY_PRINTABLE', 'CATEGORY_NOT_PRINTABLE',
161+
'CATEGORY_ALNUM', 'CATEGORY_NOT_ALNUM',
162+
'CATEGORY_XID_START', 'CATEGORY_NOT_XID_START',
163+
'CATEGORY_XID_CONTINUE', 'CATEGORY_NOT_XID_CONTINUE',
164+
'CATEGORY_TITLE', 'CATEGORY_NOT_TITLE',
165+
'CATEGORY_CASED', 'CATEGORY_NOT_CASED',
166+
'CATEGORY_CASE_IGNORABLE', 'CATEGORY_NOT_CASE_IGNORABLE',
167+
# Compound categories: Lu = uppercase letter, N = number.
168+
'CATEGORY_LU', 'CATEGORY_NOT_LU',
169+
'CATEGORY_N', 'CATEGORY_NOT_N',
170+
'CATEGORY_LM', 'CATEGORY_NOT_LM',
171+
'CATEGORY_NL', 'CATEGORY_NOT_NL',
172+
'CATEGORY_NO', 'CATEGORY_NOT_NO',
173+
'CATEGORY_CF', 'CATEGORY_NOT_CF',
174+
'CATEGORY_Z', 'CATEGORY_NOT_Z',
175+
'CATEGORY_ZS', 'CATEGORY_NOT_ZS',
176+
'CATEGORY_C', 'CATEGORY_NOT_C',
177+
'CATEGORY_CN', 'CATEGORY_NOT_CN',
178+
'CATEGORY_ASSIGNED', 'CATEGORY_NOT_ASSIGNED',
179+
'CATEGORY_BLANK', 'CATEGORY_NOT_BLANK',
180+
'CATEGORY_GRAPH', 'CATEGORY_NOT_GRAPH',
181+
'CATEGORY_PRINT', 'CATEGORY_NOT_PRINT',
153182
)
154183

155184

@@ -206,6 +235,39 @@ def _makecodes(*names):
206235
CATEGORY_NOT_LINEBREAK: CATEGORY_UNI_NOT_LINEBREAK
207236
}
208237

238+
# The Unicode property categories are the same regardless of the flags.
239+
CH_PROPERTY = (
240+
CATEGORY_ALPHA, CATEGORY_NOT_ALPHA,
241+
CATEGORY_LOWER, CATEGORY_NOT_LOWER,
242+
CATEGORY_UPPER, CATEGORY_NOT_UPPER,
243+
CATEGORY_NUMERIC, CATEGORY_NOT_NUMERIC,
244+
CATEGORY_PRINTABLE, CATEGORY_NOT_PRINTABLE,
245+
CATEGORY_ALNUM, CATEGORY_NOT_ALNUM,
246+
CATEGORY_XID_START, CATEGORY_NOT_XID_START,
247+
CATEGORY_XID_CONTINUE, CATEGORY_NOT_XID_CONTINUE,
248+
CATEGORY_TITLE, CATEGORY_NOT_TITLE,
249+
CATEGORY_CASED, CATEGORY_NOT_CASED,
250+
CATEGORY_CASE_IGNORABLE, CATEGORY_NOT_CASE_IGNORABLE,
251+
CATEGORY_LU, CATEGORY_NOT_LU,
252+
CATEGORY_N, CATEGORY_NOT_N,
253+
CATEGORY_LM, CATEGORY_NOT_LM,
254+
CATEGORY_NL, CATEGORY_NOT_NL,
255+
CATEGORY_NO, CATEGORY_NOT_NO,
256+
CATEGORY_CF, CATEGORY_NOT_CF,
257+
CATEGORY_Z, CATEGORY_NOT_Z,
258+
CATEGORY_ZS, CATEGORY_NOT_ZS,
259+
CATEGORY_C, CATEGORY_NOT_C,
260+
CATEGORY_CN, CATEGORY_NOT_CN,
261+
CATEGORY_ASSIGNED, CATEGORY_NOT_ASSIGNED,
262+
CATEGORY_BLANK, CATEGORY_NOT_BLANK,
263+
CATEGORY_GRAPH, CATEGORY_NOT_GRAPH,
264+
CATEGORY_PRINT, CATEGORY_NOT_PRINT,
265+
)
266+
for _cat in CH_PROPERTY:
267+
CH_LOCALE[_cat] = _cat
268+
CH_UNICODE[_cat] = _cat
269+
del _cat
270+
209271
CH_NEGATE = dict(zip(CHCODES[::2] + CHCODES[1::2], CHCODES[1::2] + CHCODES[::2]))
210272

211273
# flags

‎Lib/re/_parser.py‎

Lines changed: 29 additions & 3 deletions

0 commit comments

Comments
 (0)