[3.8] bpo-38234: Backport init path config changes from master (GH-16… · python/cpython@96c8475 · GitHub
Skip to content

Commit 96c8475

Browse files
vstinnervstinner
authored
[3.8] bpo-38234: Backport init path config changes from master (GH-16423)
* bpo-38234: Py_SetPath() uses the program full path (GH-16357) Py_SetPath() now sets sys.executable to the program full path (Py_GetProgramFullPath()), rather than to the program name (Py_GetProgramName()). Fix also memory leaks in pathconfig_set_from_config(). (cherry picked from commit 1ce152a) * bpo-38234: Add tests for Python init path config (GH-16358) (cherry picked from commit bb6bf7d) * bpo-38234: test_embed: test pyvenv.cfg and pybuilddir.txt (GH-16366) Add test_init_pybuilddir() and test_init_pyvenv_cfg() to test_embed to test pyvenv.cfg and pybuilddir.txt configuration files. Fix sysconfig._generate_posix_vars(): pybuilddir.txt uses UTF-8 encoding, not ASCII. (cherry picked from commit 52ad33a) * bpo-38234: Cleanup getpath.c (GH-16367) * search_for_prefix() directly calls reduce() if found is greater than 0. * Add calculate_pybuilddir() subfunction. * search_for_prefix(): add path string buffer for readability. * Fix some error handling code paths: release resources on error. * calculate_read_pyenv(): rename tmpbuffer to filename. * test.pythoninfo now also logs windows.dll_path (cherry picked from commit 221fd84) * bpo-38234: Fix test_embed pathconfig tests (GH-16390) bpo-38234: On macOS and FreeBSD, the temporary directory can be symbolic link. For example, /tmp can be a symbolic link to /var/tmp. Call realpath() to resolve all symbolic links. (cherry picked from commit 00508a7) * bpo-38234: Add test_init_setpath_config() to test_embed (GH-16402) * Add test_embed.test_init_setpath_config(): test Py_SetPath() with PyConfig. * test_init_setpath() and test_init_setpythonhome() no longer call Py_SetProgramName(), but use the default program name. * _PyPathConfig: isolated, site_import and base_executable fields are now only available on Windows. * If executable is set explicitly in the configuration, ignore calculated base_executable: _PyConfig_InitPathConfig() copies executable to base_executable. * Complete path config documentation. (cherry picked from commit 8bf39b6) * bpo-38234: Complete init config documentation (GH-16404) (cherry picked from commit 88feaec) * bpo-38234: Fix test_embed.test_init_setpath_config() on FreeBSD (GH-16406) Explicitly preinitializes with a Python preconfiguration to avoid Py_SetPath() implicit preinitialization with a compat preconfiguration. Fix also test_init_setpath() and test_init_setpythonhome() on macOS: use self.test_exe as the executable (and base_executable), rather than shutil.which('python3'). (cherry picked from commit 49d99f0) * bpo-38234: Py_Initialize() sets global path configuration (GH-16421) * Py_InitializeFromConfig() now writes PyConfig path configuration to the global path configuration (_Py_path_config). * Add test_embed.test_get_pathconfig(). * Fix typo in _PyWideStringList_Join(). (cherry picked from commit 12f2f17)
1 parent 68040ed commit 96c8475

13 files changed

Lines changed: 795 additions & 208 deletions

File tree

‎Doc/c-api/init.rst‎

Lines changed: 6 additions & 2 deletions

‎Doc/c-api/init_config.rst‎

Lines changed: 51 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -241,6 +241,7 @@ PyPreConfig
241241
locale to decide if it should be coerced.
242242
243243
.. c:member:: int coerce_c_locale_warn
244+
244245
If non-zero, emit a warning if the C locale is coerced.
245246
246247
.. c:member:: int dev_mode
@@ -300,7 +301,7 @@ For :ref:`Python Configuration <init-python-config>`
300301
(:c:func:`PyPreConfig_InitPythonConfig`), if Python is initialized with
301302
command line arguments, the command line arguments must also be passed to
302303
preinitialize Python, since they have an effect on the pre-configuration
303-
like encodings. For example, the :option:`-X` ``utf8`` command line option
304+
like encodings. For example, the :option:`-X utf8 <-X>` command line option
304305
enables the UTF-8 Mode.
305306
306307
``PyMem_SetAllocator()`` can be called after :c:func:`Py_PreInitialize` and
@@ -464,7 +465,7 @@ PyConfig
464465
465466
.. c:member:: int dev_mode
466467
467-
Development mode: see :option:`-X` ``dev``.
468+
Development mode: see :option:`-X dev <-X>`.
468469
469470
.. c:member:: int dump_refs
470471
@@ -482,7 +483,7 @@ PyConfig
482483
483484
.. c:member:: int faulthandler
484485
485-
If non-zero, call :func:`faulthandler.enable`.
486+
If non-zero, call :func:`faulthandler.enable` at startup.
486487
487488
.. c:member:: wchar_t* filesystem_encoding
488489
@@ -504,6 +505,9 @@ PyConfig
504505
505506
Python home directory.
506507
508+
Initialized from :envvar:`PYTHONHOME` environment variable value by
509+
default.
510+
507511
.. c:member:: int import_time
508512
509513
If non-zero, profile import time.
@@ -561,7 +565,7 @@ PyConfig
561565
562566
:data:`sys.path`. If :c:member:`~PyConfig.module_search_paths_set` is
563567
equal to 0, the :c:member:`~PyConfig.module_search_paths` is overridden
564-
by the function computing the :ref:`Path Configuration
568+
by the function calculating the :ref:`Path Configuration
565569
<init-path-config>`.
566570
567571
.. c:member:: int optimization_level
@@ -586,49 +590,56 @@ PyConfig
586590
587591
.. c:member:: int pathconfig_warnings
588592
589-
If equal to 0, suppress warnings when computing the path configuration
590-
(Unix only, Windows does not log any warning). Otherwise, warnings are
591-
written into ``stderr``.
593+
If equal to 0, suppress warnings when calculating the :ref:`Path
594+
Configuration <init-path-config>` (Unix only, Windows does not log any
595+
warning). Otherwise, warnings are written into ``stderr``.
592596
593597
.. c:member:: wchar_t* prefix
594598
595599
:data:`sys.prefix`.
596600
597601
.. c:member:: wchar_t* program_name
598602
599-
Program name.
603+
Program name. Used to initialize :c:member:`~PyConfig.executable`, and in
604+
early error messages.
600605
601606
.. c:member:: wchar_t* pycache_prefix
602607
603-
``.pyc`` cache prefix.
608+
:data:`sys.pycache_prefix`: ``.pyc`` cache prefix.
609+
610+
If NULL, :data:`sys.pycache_prefix` is set to ``None``.
604611
605612
.. c:member:: int quiet
606613
607614
Quiet mode. For example, don't display the copyright and version messages
608-
even in interactive mode.
615+
in interactive mode.
609616
610617
.. c:member:: wchar_t* run_command
611618
612-
``python3 -c COMMAND`` argument.
619+
``python3 -c COMMAND`` argument. Used by :c:func:`Py_RunMain`.
613620
614621
.. c:member:: wchar_t* run_filename
615622
616-
``python3 FILENAME`` argument.
623+
``python3 FILENAME`` argument. Used by :c:func:`Py_RunMain`.
617624
618625
.. c:member:: wchar_t* run_module
619626
620-
``python3 -m MODULE`` argument.
627+
``python3 -m MODULE`` argument. Used by :c:func:`Py_RunMain`.
621628
622629
.. c:member:: int show_alloc_count
623630
624631
Show allocation counts at exit?
625632
633+
Set to 1 by :option:`-X showalloccount <-X>` command line option.
634+
626635
Need a special Python build with ``COUNT_ALLOCS`` macro defined.
627636
628637
.. c:member:: int show_ref_count
629638
630639
Show total reference count at exit?
631640
641+
Set to 1 by :option:`-X showrefcount <-X>` command line option.
642+
632643
Need a debug build of Python (``Py_REF_DEBUG`` macro must be defined).
633644
634645
.. c:member:: int site_import
@@ -647,7 +658,7 @@ PyConfig
647658
648659
.. c:member:: int tracemalloc
649660
650-
If non-zero, call :func:`tracemalloc.start`.
661+
If non-zero, call :func:`tracemalloc.start` at startup.
651662
652663
.. c:member:: int use_environment
653664
@@ -669,6 +680,9 @@ PyConfig
669680
670681
If non-zero, write ``.pyc`` files.
671682
683+
:data:`sys.dont_write_bytecode` is initialized to the inverted value of
684+
:c:member:`~PyConfig.write_bytecode`.
685+
672686
.. c:member:: PyWideStringList xoptions
673687
674688
:data:`sys._xoptions`.
@@ -694,8 +708,8 @@ Function to initialize Python:
694708
The caller is responsible to handle exceptions (error or exit) using
695709
:c:func:`PyStatus_Exception` and :c:func:`Py_ExitStatusException`.
696710
697-
``PyImport_FrozenModules``, ``PyImport_AppendInittab()`` or
698-
``PyImport_ExtendInittab()`` is used: they must be set or called after Python
711+
If ``PyImport_FrozenModules``, ``PyImport_AppendInittab()`` or
712+
``PyImport_ExtendInittab()`` are used, they must be set or called after Python
699713
preinitialization and before the Python initialization.
700714
701715
Example setting the program name::
@@ -760,7 +774,7 @@ configuration, and then override some parameters::
760774
761775
/* Append our custom search path to sys.path */
762776
status = PyWideStringList_Append(&config.module_search_paths,
763-
L"/path/to/more/modules");
777+
L"/path/to/more/modules");
764778
if (PyStatus_Exception(status)) {
765779
goto done;
766780
}
@@ -791,9 +805,9 @@ isolate Python from the system. For example, to embed Python into an
791805
application.
792806
793807
This configuration ignores global configuration variables, environments
794-
variables and command line arguments (:c:member:`PyConfig.argv` is not parsed).
795-
The C standard streams (ex: ``stdout``) and the LC_CTYPE locale are left
796-
unchanged by default.
808+
variables, command line arguments (:c:member:`PyConfig.argv` is not parsed)
809+
and user site directory. The C standard streams (ex: ``stdout``) and the
810+
LC_CTYPE locale are left unchanged. Signal handlers are not installed.
797811
798812
Configuration files are still used with this configuration. Set the
799813
:ref:`Path Configuration <init-path-config>` ("output fields") to ignore these
@@ -864,37 +878,46 @@ Path Configuration
864878
865879
:c:type:`PyConfig` contains multiple fields for the path configuration:
866880
867-
* Path configuration input fields:
881+
* Path configuration inputs:
868882
869883
* :c:member:`PyConfig.home`
870884
* :c:member:`PyConfig.pathconfig_warnings`
871885
* :c:member:`PyConfig.program_name`
872886
* :c:member:`PyConfig.pythonpath_env`
887+
* current working directory: to get absolute paths
888+
* ``PATH`` environment variable to get the program full path
889+
(from :c:member:`PyConfig.program_name`)
890+
* ``__PYVENV_LAUNCHER__`` environment variable
891+
* (Windows only) Application paths in the registry under
892+
"Software\Python\PythonCore\X.Y\PythonPath" of HKEY_CURRENT_USER and
893+
HKEY_LOCAL_MACHINE (where X.Y is the Python version).
873894
874895
* Path configuration output fields:
875896
897+
* :c:member:`PyConfig.base_exec_prefix`
876898
* :c:member:`PyConfig.base_executable`
899+
* :c:member:`PyConfig.base_prefix`
877900
* :c:member:`PyConfig.exec_prefix`
878901
* :c:member:`PyConfig.executable`
879-
* :c:member:`PyConfig.prefix`
880902
* :c:member:`PyConfig.module_search_paths_set`,
881903
:c:member:`PyConfig.module_search_paths`
904+
* :c:member:`PyConfig.prefix`
882905
883-
If at least one "output field" is not set, Python computes the path
906+
If at least one "output field" is not set, Python calculates the path
884907
configuration to fill unset fields. If
885908
:c:member:`~PyConfig.module_search_paths_set` is equal to 0,
886909
:c:member:`~PyConfig.module_search_paths` is overridden and
887910
:c:member:`~PyConfig.module_search_paths_set` is set to 1.
888911
889-
It is possible to completely ignore the function computing the default
912+
It is possible to completely ignore the function calculating the default
890913
path configuration by setting explicitly all path configuration output
891914
fields listed above. A string is considered as set even if it is non-empty.
892915
``module_search_paths`` is considered as set if
893916
``module_search_paths_set`` is set to 1. In this case, path
894917
configuration input fields are ignored as well.
895918
896919
Set :c:member:`~PyConfig.pathconfig_warnings` to 0 to suppress warnings when
897-
computing the path configuration (Unix only, Windows does not log any warning).
920+
calculating the path configuration (Unix only, Windows does not log any warning).
898921
899922
If :c:member:`~PyConfig.base_prefix` or :c:member:`~PyConfig.base_exec_prefix`
900923
fields are not set, they inherit their value from :c:member:`~PyConfig.prefix`
@@ -961,7 +984,7 @@ initialization, the core feature of the :pep:`432`:
961984
* Builtin exceptions;
962985
* Builtin and frozen modules;
963986
* The :mod:`sys` module is only partially initialized
964-
(ex: :data:`sys.path` doesn't exist yet);
987+
(ex: :data:`sys.path` doesn't exist yet).
965988
966989
* "Main" initialization phase, Python is fully initialized:
967990
@@ -987,9 +1010,9 @@ No module is imported during the "Core" phase and the ``importlib`` module is
9871010
not configured: the :ref:`Path Configuration <init-path-config>` is only
9881011
applied during the "Main" phase. It may allow to customize Python in Python to
9891012
override or tune the :ref:`Path Configuration <init-path-config>`, maybe
990-
install a custom sys.meta_path importer or an import hook, etc.
1013+
install a custom :data:`sys.meta_path` importer or an import hook, etc.
9911014
992-
It may become possible to compute the :ref:`Path Configuration
1015+
It may become possible to calculatin the :ref:`Path Configuration
9931016
<init-path-config>` in Python, after the Core phase and before the Main phase,
9941017
which is one of the :pep:`432` motivation.
9951018

‎Doc/whatsnew/3.8.rst‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1347,6 +1347,11 @@ Build and C API Changes
13471347
parameter for indicating the number of positional-only arguments.
13481348
(Contributed by Pablo Galindo in :issue:`37221`.)
13491349

1350+
* :c:func:`Py_SetPath` now sets :data:`sys.executable` to the program full
1351+
path (:c:func:`Py_GetProgramFullPath`) rather than to the program name
1352+
(:c:func:`Py_GetProgramName`).
1353+
(Contributed by Victor Stinner in :issue:`38234`.)
1354+
13501355

13511356
Deprecated
13521357
==========

‎Include/internal/pycore_pathconfig.h‎

Lines changed: 12 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -19,19 +19,26 @@ typedef struct _PyPathConfig {
1919
wchar_t *program_name;
2020
/* Set by Py_SetPythonHome() or PYTHONHOME environment variable */
2121
wchar_t *home;
22+
#ifdef MS_WINDOWS
2223
/* isolated and site_import are used to set Py_IsolatedFlag and
2324
Py_NoSiteFlag flags on Windows in read_pth_file(). These fields
2425
are ignored when their value are equal to -1 (unset). */
2526
int isolated;
2627
int site_import;
2728
/* Set when a venv is detected */
2829
wchar_t *base_executable;
30+
#endif
2931
} _PyPathConfig;
3032

31-
#define _PyPathConfig_INIT \
32-
{.module_search_path = NULL, \
33-
.isolated = -1, \
34-
.site_import = -1}
33+
#ifdef MS_WINDOWS
34+
# define _PyPathConfig_INIT \
35+
{.module_search_path = NULL, \
36+
.isolated = -1, \
37+
.site_import = -1}
38+
#else
39+
# define _PyPathConfig_INIT \
40+
{.module_search_path = NULL}
41+
#endif
3542
/* Note: _PyPathConfig_INIT sets other fields to 0/NULL */
3643

3744
PyAPI_DATA(_PyPathConfig) _Py_path_config;
@@ -59,7 +66,7 @@ extern int _Py_FindEnvConfigValue(
5966
extern wchar_t* _Py_GetDLLPath(void);
6067
#endif
6168

62-
extern PyStatus _PyPathConfig_Init(void);
69+
extern PyStatus _PyConfig_WritePathConfig(const PyConfig *config);
6370
extern void _Py_DumpPathConfig(PyThreadState *tstate);
6471

6572
#ifdef __cplusplus

‎Lib/sysconfig.py‎

Lines changed: 1 addition & 1 deletion

0 commit comments

Comments
 (0)