Latin-1 source code was not being properly decoded when passed through · pythoncapi/cpython@da78043 · GitHub
Skip to content

Commit da78043

Browse files
committed
Latin-1 source code was not being properly decoded when passed through
compile(). This was due to left-over special-casing before UTF-8 became the default source encoding. Closes issue python#3574. Thanks to Victor Stinner for help with the patch.
1 parent 9e9dcd6 commit da78043

5 files changed

Lines changed: 24 additions & 10 deletions

File tree

Lib/test/test_pep3120.py

Lines changed: 17 additions & 1 deletion

Misc/NEWS

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@ What's New in Python 3.0 beta 5
1515
Core and Builtins
1616
-----------------
1717

18+
- Issue #3574: compile() incorrectly handled source code encoded as Latin-1.
19+
1820
- Issues #2384 and #3975: Tracebacks were not correctly printed when the
1921
source file contains a ``coding:`` header: the wrong line was displayed, and
2022
the encoding was not respected.

Parser/tokenizer.c

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -135,6 +135,7 @@ tok_new(void)
135135
tok->decoding_state = STATE_INIT;
136136
tok->decoding_erred = 0;
137137
tok->read_coding_spec = 0;
138+
tok->enc = NULL;
138139
tok->encoding = NULL;
139140
tok->cont_line = 0;
140141
#ifndef PGEN
@@ -274,8 +275,7 @@ check_coding_spec(const char* line, Py_ssize_t size, struct tok_state *tok,
274275
tok->read_coding_spec = 1;
275276
if (tok->encoding == NULL) {
276277
assert(tok->decoding_state == STATE_RAW);
277-
if (strcmp(cs, "utf-8") == 0 ||
278-
strcmp(cs, "iso-8859-1") == 0) {
278+
if (strcmp(cs, "utf-8") == 0) {
279279
tok->encoding = cs;
280280
} else {
281281
r = set_readline(tok, cs);

Parser/tokenizer.h

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -49,14 +49,14 @@ struct tok_state {
4949
enum decoding_state decoding_state;
5050
int decoding_erred; /* whether erred in decoding */
5151
int read_coding_spec; /* whether 'coding:...' has been read */
52-
char *encoding;
52+
char *encoding; /* Source encoding. */
5353
int cont_line; /* whether we are in a continuation line. */
5454
const char* line_start; /* pointer to start of current line */
5555
#ifndef PGEN
5656
PyObject *decoding_readline; /* codecs.open(...).readline */
5757
PyObject *decoding_buffer;
5858
#endif
59-
const char* enc;
59+
const char* enc; /* Encoding for the current str. */
6060
const char* str;
6161
};
6262

Python/ast.c

Lines changed: 1 addition & 5 deletions

0 commit comments

Comments
 (0)