Regression in pip 26.2: UTF-8 pip.ini fails despite Python UTF-8 mode on Windows

Description

With CP936 simulated on Windows, pip 26.2 and 26.2.1 fail to read a UTF-8 pip.ini even when Python UTF-8 mode is enabled. The same file works with pip 26.1.2. Chinese text in a comment is sufficient to trigger the failure:

[global]
# 中文注释
timeout = 15

The file is UTF-8 without a BOM. Results with the same simulated CP936:

Python 3.15.0 already has sys.flags.utf8_mode == 1 by default; explicitly passing -X utf8 gives the same result.

Expected behavior

A UTF-8 configuration readable with Python UTF-8 mode in pip 26.1.2 should remain readable after upgrading pip.

pip version

26.2 and 26.2.1; compared with 26.1.2 using official PyPI wheels.

Python version

3.14.5 and 3.15.0. The latter was tested using the official Windows 64-bit embeddable distribution with each pip package added to its import path.

OS

Windows, build 26200.

Reproduction limitation: The host's actual ANSI code page is CP65001. CP936 is simulated by replacing only locale.getencoding() in the test process; Python's getpreferredencoding(False), UTF-8 mode, and pip code are unchanged. This has not been tested on a machine whose actual ANSI code page is CP936. With the host's native CP65001, all three pip versions read the file successfully.

How to Reproduce

Save this as issue-repro.py and run python -X utf8 issue-repro.py with each pip version. On Python 3.15.0, python issue-repro.py also reproduces. The script uses a temporary configuration file and leaves existing configuration files untouched.

import locale
import os
from pathlib import Path
import runpy
import sys
import tempfile

assert sys.flags.utf8_mode == 1, "Run with python -X utf8 issue-repro.py"

# Simulate the ANSI code page returned on a CP936 Windows installation.
# Remove this line when testing on an actual CP936 Windows installation.
locale.getencoding = lambda: "cp936"

import pip

print(f"pip={pip.__version__}, utf8_mode={sys.flags.utf8_mode}", flush=True)
print(f"getencoding={locale.getencoding()}, "
      f"getpreferredencoding={locale.getpreferredencoding(False)}", flush=True)

with tempfile.TemporaryDirectory() as directory:
    config = Path(directory) / "pip.ini"
    config.write_bytes(
        "[global]\n# \u4e2d\u6587\u6ce8\u91ca\ntimeout = 15\n".encode("utf-8")
    )
    os.environ["PIP_CONFIG_FILE"] = str(config)
    os.environ["PIP_DISABLE_PIP_VERSION_CHECK"] = "1"
    sys.argv = ["pip", "config", "list"]
    runpy.run_module("pip", run_name="__main__")

On an actual CP936 Windows installation, remove the marked override to test the native behavior.

Output

pip 26.2.1 exits with code 2 (temporary path shortened):

pip=26.2.1, utf8_mode=1
getencoding=cp936, getpreferredencoding=utf-8
Configuration file contains invalid cp936 characters in ...\pip.ini.

pip 26.2 fails the same way. pip 26.1.2 exits with code 0 and reports global.timeout='15'.

On Python 3.14.5, the failure also affects pip list and offline pip install --dry-run, so it blocks commands beyond configuration inspection.

Suspected cause

#13960 changed configuration loading from locale.getpreferredencoding(False) to get_locale_encoding(). On Python 3.11+, that helper calls locale.getencoding(), which ignores UTF-8 mode. In this reproduction, pip therefore selects CP936 instead of UTF-8.

Restoring only the old encoding-selection function in the test process makes pip list and the installation dry-run succeed on pip 26.2/26.2.1 with Python 3.14.5. Disabling UTF-8 mode makes all three pip versions fail on the UTF-8 fixture under simulated CP936; the regression is specifically the loss of working UTF-8-mode behavior.

Code of Conduct

I agree to follow the PSF Code of Conduct.

Respect Python UTF-8 Mode when determining locale encoding

I would like to take this on.

Root Cause Analysis

In src/pip/_internal/utils/compat.py, get_locale_encoding() was updated to call locale.getencoding() on Python 3.11+ to prevent EncodingWarning when PYTHONWARNDEFAULTENCODING=1 is set:

def get_locale_encoding() -> str:
    # TODO: Remove the 3.10 fallback once pip drops Python 3.10 support.
    if sys.version_info >= (3, 11):
        return locale.getencoding()
    return locale.getpreferredencoding(False)

Per PEP 540 and Python's standard library documentation, locale.getencoding() explicitly queries the system's ANSI/locale code page (e.g. CP936 or CP1252) and ignores Python UTF-8 Mode. In contrast, Python UTF-8 Mode (sys.flags.utf8_mode == 1, enabled via -X utf8, PYTHONUTF8=1, or by default in Python 3.15+) specifies that text encoding should be UTF-8.

Because Configuration._construct_parser() in src/pip/_internal/configuration.py uses get_locale_encoding(), configuration files containing valid UTF-8 non-ASCII characters (such as comments) fail decoding with ConfigurationFileCouldNotBeLoaded: contains invalid cp936 characters even when Python is run in UTF-8 Mode.

Proposed Fix

In src/pip/_internal/utils/compat.py, inspect sys.flags.utf8_mode:

def get_locale_encoding() -> str:
    """Return the locale encoding.

    Uses ``locale.getencoding()`` when available (Python 3.11+) to avoid the
    ``EncodingWarning`` that ``locale.getpreferredencoding(False)`` raises
    under UTF-8 Mode. Under UTF-8 Mode, returns ``"utf-8"``.
    """
    if getattr(sys.flags, "utf8_mode", 0):
        return "utf-8"
    # TODO: Remove the 3.10 fallback once pip drops Python 3.10 support.
    if sys.version_info >= (3, 11):
        return locale.getencoding()
    return locale.getpreferredencoding(False)

When UTF-8 Mode is enabled, returning "utf-8" directly correctly respects UTF-8 Mode without calling locale.getpreferredencoding(False) (so no EncodingWarning is emitted under PYTHONWARNDEFAULTENCODING=1).
When UTF-8 Mode is not enabled, behavior is completely unchanged (delegating to locale.getencoding() on 3.11+).

Verification & Tests

Unit Tests:
- Added test_get_locale_encoding_under_utf8_mode and test_get_locale_encoding_without_utf8_mode in tests/unit/test_compat.py.
- Added test_utf8_config_loading_under_utf8_mode in tests/unit/test_configuration.py asserting that non-ASCII UTF-8 config files load cleanly under UTF-8 Mode even when locale.getencoding() is simulated as cp936.
- Added news fragment news/14367.bugfix.rst.
Test Suite:
- pytest tests/unit/test_compat.py tests/unit/test_configuration.py tests/unit/test_command_debug.py tests/unit/test_req_file.py: all 112 tests passing.
- ruff check: 100% clean.

Tested and armed on fork branch: duongynhi000005-oss:fix/14367-utf8-mode-locale-encoding (commit dfc9b3b). I can open a PR once assigned.

Python / startup options	pip 26.1.2	pip 26.2	pip 26.2.1
3.14.5, `-X utf8`	Success	Exit 2	Exit 2
3.15.0, default startup	Success	Exit 2	Exit 2

Sunbelt Computer Software

PL/B Language Development and Support

Navigation Menu

Repository navigation

Regression in pip 26.2: UTF-8 pip.ini fails despite Python UTF-8 mode on Windows #14367

Description

Expected behavior

pip version

Python version

OS

How to Reproduce

Output

Suspected cause

Code of Conduct

Root Cause Analysis

Proposed Fix

Verification & Tests

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Sunbelt Computer Software

PL/B Language Development and Support

Regression in pip 26.2: UTF-8 pip.ini fails despite Python UTF-8 mode on Windows #14367

Description

Description

Expected behavior

pip version

Python version

OS

How to Reproduce

Output

Suspected cause

Code of Conduct

Activity

duongynhi000005-oss commented on Oct 27, 2026

Root Cause Analysis

Proposed Fix

Verification & Tests

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions