Issue 42236: os.device_encoding() doesn't respect the UTF-8 Mode - Python tracker

This issue tracker has been migrated to GitHub, and is currently read-only.
For more information, see the GitHub FAQs in the Python's Developer Guide.

classification
process
Status: closed Resolution: fixed
Dependencies: Superseder:
Assigned To: Nosy List: methane, vstinner
Priority: normal Keywords: patch

Created on 2020-11-01 18:12 by vstinner, last changed 2022-04-11 14:59 by admin. This issue is now closed.

Pull Requests
URL Status Linked Edit
PR 23083 merged vstinner, 2020-11-01 18:47
PR 23086 merged vstinner, 2020-11-01 21:40
PR 23109 merged vstinner, 2020-11-02 15:06
PR 23119 merged vstinner, 2020-11-02 22:03
Messages (8)
msg380156 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-01 18:12
When the UTF-8 Mode is enabled, sys.stdout.encoding is always UTF-8, whereas os.devide_encoding(sys.stdout.fileno()) returns the locale encoding. os.devide_encoding() must return UTF-8 when the UTF-8 Mode is used.
msg380158 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-01 18:45
In bpo-42208, I added C _Py_GetLocaleEncoding() function and Python _locale._get_locale_encoding() function.
msg380159 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-01 19:59
New changeset 82458b6cdbae3b849dc11d0d7dc2ab06ef0451c4 by Victor Stinner in branch 'master':
bpo-42236: Enhance _locale._get_locale_encoding() (GH-23083)
https://github.com/python/cpython/commit/82458b6cdbae3b849dc11d0d7dc2ab06ef0451c4
msg380166 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-01 22:07
New changeset e662c398d87f136497f8ec672e83657ae3a599e0 by Victor Stinner in branch 'master':
bpo-42236: Use UTF-8 encoding if nl_langinfo(CODESET) fails (GH-23086)
https://github.com/python/cpython/commit/e662c398d87f136497f8ec672e83657ae3a599e0
msg380234 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-02 15:50
New changeset 4b9aad49992a825d8c76e428ed1aca81dd3878b2 by Victor Stinner in branch 'master':
bpo-42236: Enhance init and encoding documentation (GH-23109)
https://github.com/python/cpython/commit/4b9aad49992a825d8c76e428ed1aca81dd3878b2
msg380261 - (view) Author: Inada Naoki (methane) * (Python committer) Date: 2020-11-03 05:15
I don't think UTF-8 mode should override os.device_encoding() on Windows.

UTF-8 mode can be used to ignore legacy locale encoding, and os.device_encoding() uses the locale encoding on Unix. So overriding it make sense.

But locale encoding and console cp are different on Windows. Users may want to know console cp even when they want to use UTF-8 by default for reading/writing text files.
msg380313 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-04 10:20
New changeset 3529718925f40d14ed48d281d809187bc7314a14 by Victor Stinner in branch 'master':
bpo-42236: os.device_encoding() respects UTF-8 Mode (GH-23119)
https://github.com/python/cpython/commit/3529718925f40d14ed48d281d809187bc7314a14
msg380314 - (view) Author: STINNER Victor (vstinner) * (Python committer) Date: 2020-11-04 10:26
> But locale encoding and console cp are different on Windows. Users may want to know console cp even when they want to use UTF-8 by default for reading/writing text files.

You're right. My final change leaves Windows unchanged.