PEP 489: Multi-phase extension module initialization · pythoncapi/cpython@d5cacbb · GitHub
Skip to content

Commit d5cacbb

Browse files
committed
PEP 489: Multi-phase extension module initialization
Known limitations of the current implementation: - documentation changes are incomplete - there's a reference leak I haven't tracked down yet The leak is most visible by running: ./python -m test -R3:3 test_importlib However, you can also see it by running: ./python -X showrefcount Importing the array or _testmultiphase modules, and then deleting them from both sys.modules and the local namespace shows significant increases in the total number of active references each cycle. By contrast, with _testcapi (which continues to use single-phase initialisation) the global refcounts stabilise after a couple of cycles.
1 parent ec219ba commit d5cacbb

34 files changed

Lines changed: 4451 additions & 3113 deletions

Doc/c-api/module.rst

Lines changed: 66 additions & 26 deletions

Doc/library/importlib.rst

Lines changed: 19 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -55,6 +55,12 @@ generically as an :term:`importer`) to participate in the import process.
5555
:pep:`451`
5656
A ModuleSpec Type for the Import System
5757

58+
:pep:`488`
59+
Elimination of PYO files
60+
61+
:pep:`489`
62+
Multi-phase extension module initialization
63+
5864
:pep:`3120`
5965
Using UTF-8 as the Default Source Encoding
6066

@@ -756,9 +762,9 @@ find and load modules.
756762
Only class methods are defined by this class to alleviate the need for
757763
instantiation.
758764

759-
.. note::
760-
Due to limitations in the extension module C-API, for now
761-
BuiltinImporter does not implement :meth:`Loader.exec_module`.
765+
.. versionchanged:: 3.5
766+
As part of :pep:`489`, the builtin importer now implements
767+
:meth:`Loader.create_module` and :meth:`Loader.exec_module`
762768

763769

764770
.. class:: FrozenImporter
@@ -973,14 +979,18 @@ find and load modules.
973979

974980
Path to the extension module.
975981

976-
.. method:: load_module(name=None)
982+
.. method:: create_module(spec)
983+
984+
Creates the module object from the given specification in accordance
985+
with :pep:`489`.
986+
987+
.. versionadded:: 3.5
988+
989+
.. method:: exec_module(module)
977990

978-
Loads the extension module if and only if *fullname* is the same as
979-
:attr:`name` or is ``None``.
991+
Initializes the given module object in accordance with :pep:`489`.
980992

981-
.. note::
982-
Due to limitations in the extension module C-API, for now
983-
ExtensionFileLoader does not implement :meth:`Loader.exec_module`.
993+
.. versionadded:: 3.5
984994

985995
.. method:: is_package(fullname)
986996

Doc/whatsnew/3.5.rst

Lines changed: 17 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,7 @@ Implementation improvements:
9393
(:issue:`19977`).
9494

9595
* :pep:`488`, the elimination of ``.pyo`` files.
96+
* :pep:`489`, multi-phase initialization of extension modules.
9697

9798
Significantly Improved Library Modules:
9899

@@ -265,6 +266,21 @@ updated API to help with this change.
265266
:pep:`488` -- Elimination of PYO files
266267

267268

269+
PEP 489: Multi-phase extension module initialization
270+
----------------------------------------------------
271+
272+
:pep:`489` updates extension module initialization to take advantage of the
273+
two step module loading mechanism introduced by :pep:`451` in Python 3.4.
274+
275+
This change brings the import semantics of extension modules that opt-in to
276+
using the new mechanism much closer to those of Python source and bytecode
277+
modules, including the ability to any valid identifier as a module name,
278+
rather than being restricted to ASCII.
279+
280+
.. seealso::
281+
282+
:pep:`488` -- Multi-phase extension module initialization
283+
268284
Other Language Changes
269285
======================
270286

@@ -667,7 +683,7 @@ time
667683
tkinter
668684
-------
669685

670-
* The :module:`tkinter._fix` module used for setting up the Tcl/Tk environment
686+
* The :mod:`tkinter._fix` module used for setting up the Tcl/Tk environment
671687
on Windows has been replaced by a private function in the :module:`_tkinter`
672688
module which makes no permanent changes to environment variables.
673689
(Contributed by Zachary Ware in :issue:`20035`.)
@@ -1012,7 +1028,6 @@ Changes in the Python API
10121028
program depends on patching the module level variable to capture the debug
10131029
output, you will need to update it to capture sys.stderr instead.
10141030

1015-
10161031
Changes in the C API
10171032
--------------------
10181033

Include/modsupport.h

Lines changed: 39 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -12,13 +12,13 @@ extern "C" {
1212
/* If PY_SSIZE_T_CLEAN is defined, each functions treats #-specifier
1313
to mean Py_ssize_t */
1414
#ifdef PY_SSIZE_T_CLEAN
15-
#define PyArg_Parse _PyArg_Parse_SizeT
16-
#define PyArg_ParseTuple _PyArg_ParseTuple_SizeT
17-
#define PyArg_ParseTupleAndKeywords _PyArg_ParseTupleAndKeywords_SizeT
18-
#define PyArg_VaParse _PyArg_VaParse_SizeT
19-
#define PyArg_VaParseTupleAndKeywords _PyArg_VaParseTupleAndKeywords_SizeT
20-
#define Py_BuildValue _Py_BuildValue_SizeT
21-
#define Py_VaBuildValue _Py_VaBuildValue_SizeT
15+
#define PyArg_Parse _PyArg_Parse_SizeT
16+
#define PyArg_ParseTuple _PyArg_ParseTuple_SizeT
17+
#define PyArg_ParseTupleAndKeywords _PyArg_ParseTupleAndKeywords_SizeT
18+
#define PyArg_VaParse _PyArg_VaParse_SizeT
19+
#define PyArg_VaParseTupleAndKeywords _PyArg_VaParseTupleAndKeywords_SizeT
20+
#define Py_BuildValue _Py_BuildValue_SizeT
21+
#define Py_VaBuildValue _Py_VaBuildValue_SizeT
2222
#else
2323
PyAPI_FUNC(PyObject *) _Py_VaBuildValue_SizeT(const char *, va_list);
2424
#endif
@@ -49,6 +49,9 @@ PyAPI_FUNC(int) PyModule_AddIntConstant(PyObject *, const char *, long);
4949
PyAPI_FUNC(int) PyModule_AddStringConstant(PyObject *, const char *, const char *);
5050
#define PyModule_AddIntMacro(m, c) PyModule_AddIntConstant(m, #c, c)
5151
#define PyModule_AddStringMacro(m, c) PyModule_AddStringConstant(m, #c, c)
52+
PyAPI_FUNC(int) PyModule_SetDocString(PyObject *, const char *);
53+
PyAPI_FUNC(int) PyModule_AddFunctions(PyObject *, PyMethodDef *);
54+
PyAPI_FUNC(int) PyModule_ExecDef(PyObject *module, PyModuleDef *def);
5255

5356
#define Py_CLEANUP_SUPPORTED 0x20000
5457

@@ -67,35 +70,35 @@ PyAPI_FUNC(int) PyModule_AddStringConstant(PyObject *, const char *, const char
6770
Please add a line or two to the top of this log for each API
6871
version change:
6972
70-
22-Feb-2006 MvL 1013 PEP 353 - long indices for sequence lengths
73+
22-Feb-2006 MvL 1013 PEP 353 - long indices for sequence lengths
7174
72-
19-Aug-2002 GvR 1012 Changes to string object struct for
73-
interning changes, saving 3 bytes.
75+
19-Aug-2002 GvR 1012 Changes to string object struct for
76+
interning changes, saving 3 bytes.
7477
75-
17-Jul-2001 GvR 1011 Descr-branch, just to be on the safe side
78+
17-Jul-2001 GvR 1011 Descr-branch, just to be on the safe side
7679
7780
25-Jan-2001 FLD 1010 Parameters added to PyCode_New() and
7881
PyFrame_New(); Python 2.1a2
7982
8083
14-Mar-2000 GvR 1009 Unicode API added
8184
82-
3-Jan-1999 GvR 1007 Decided to change back! (Don't reuse 1008!)
85+
3-Jan-1999 GvR 1007 Decided to change back! (Don't reuse 1008!)
8386
84-
3-Dec-1998 GvR 1008 Python 1.5.2b1
87+
3-Dec-1998 GvR 1008 Python 1.5.2b1
8588
86-
18-Jan-1997 GvR 1007 string interning and other speedups
89+
18-Jan-1997 GvR 1007 string interning and other speedups
8790
88-
11-Oct-1996 GvR renamed Py_Ellipses to Py_Ellipsis :-(
91+
11-Oct-1996 GvR renamed Py_Ellipses to Py_Ellipsis :-(
8992
90-
30-Jul-1996 GvR Slice and ellipses syntax added
93+
30-Jul-1996 GvR Slice and ellipses syntax added
9194
92-
23-Jul-1996 GvR For 1.4 -- better safe than sorry this time :-)
95+
23-Jul-1996 GvR For 1.4 -- better safe than sorry this time :-)
9396
94-
7-Nov-1995 GvR Keyword arguments (should've been done at 1.3 :-( )
97+
7-Nov-1995 GvR Keyword arguments (should've been done at 1.3 :-( )
9598
96-
10-Jan-1995 GvR Renamed globals to new naming scheme
99+
10-Jan-1995 GvR Renamed globals to new naming scheme
97100
98-
9-Jan-1995 GvR Initial version (incompatible with older API)
101+
9-Jan-1995 GvR Initial version (incompatible with older API)
99102
*/
100103

101104
/* The PYTHON_ABI_VERSION is introduced in PEP 384. For the lifetime of
@@ -105,21 +108,34 @@ PyAPI_FUNC(int) PyModule_AddStringConstant(PyObject *, const char *, const char
105108
#define PYTHON_ABI_STRING "3"
106109

107110
#ifdef Py_TRACE_REFS
108-
/* When we are tracing reference counts, rename PyModule_Create2 so
111+
/* When we are tracing reference counts, rename module creation functions so
109112
modules compiled with incompatible settings will generate a
110113
link-time error. */
111114
#define PyModule_Create2 PyModule_Create2TraceRefs
115+
#define PyModule_FromDefAndSpec2 PyModule_FromDefAndSpec2TraceRefs
112116
#endif
113117

114118
PyAPI_FUNC(PyObject *) PyModule_Create2(struct PyModuleDef*,
115119
int apiver);
116120

117121
#ifdef Py_LIMITED_API
118122
#define PyModule_Create(module) \
119-
PyModule_Create2(module, PYTHON_ABI_VERSION)
123+
PyModule_Create2(module, PYTHON_ABI_VERSION)
120124
#else
121125
#define PyModule_Create(module) \
122-
PyModule_Create2(module, PYTHON_API_VERSION)
126+
PyModule_Create2(module, PYTHON_API_VERSION)
127+
#endif
128+
129+
PyAPI_FUNC(PyObject *) PyModule_FromDefAndSpec2(PyModuleDef *def,
130+
PyObject *spec,
131+
int module_api_version);
132+
133+
#ifdef Py_LIMITED_API
134+
#define PyModule_FromDefAndSpec(module, spec) \
135+
PyModule_FromDefAndSpec2(module, spec, PYTHON_ABI_VERSION)
136+
#else
137+
#define PyModule_FromDefAndSpec(module, spec) \
138+
PyModule_FromDefAndSpec2(module, spec, PYTHON_API_VERSION)
123139
#endif
124140

125141
#ifndef Py_LIMITED_API

Include/moduleobject.h

Lines changed: 15 additions & 1 deletion

0 commit comments

Comments
 (0)