Ad

Python 2 Assumes Different Source Code Encodings

I noticed that without source code encoding declaration, the Python 2 interpreter assumes the source code is encoded in ASCII with scripts and standard input:

$ python test.py  # where test.py holds the line: print u'é'
  File "test.py", line 1
SyntaxError: Non-ASCII character '\xc3' in file test.py on line 1, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details

$ echo "print u'é'" | python
  File "/dev/fd/63", line 1
SyntaxError: Non-ASCII character '\xc3' in file /dev/fd/63 on line 1, but no encoding declared; see http://python.org/dev/peps/pep-0263/ for details

and it is encoded in ISO-8859-1 with the -mmodule and -ccommand flags:

$ python -m test  # where test.py holds the line: print u'é'
é

$ python -c "print u'é'"
é

Where is it documented?

Contrast this to Python 3 which always assumes the source code is encoded in UTF-8 and thus prints é in the four cases.

Note. – I tested this on CPython 2.7.14 on both macOS 10.13 and Ubuntu Linux 17.10 with the console encoding set to UTF-8.

Ad

Answer

The -c and -m switches, ultimately(*) run the code supplied with the exec statement or the compile() function, both of which take Latin-1 source code:

The first expression should evaluate to either a Unicode string, a Latin-1 encoded string, an open file object, a code object, or a tuple.

This is not documented, it's an implementation detail, that may or may not be considered a bug.

I don't think it is something that is worth fixing however, and Latin-1 is a superset of ASCII so little is lost. How code from -c and -m is handled has been cleaned up in Python 3 and is much more consistent there; code passed in with -c is decoded using the current locale, and modules loaded with the -m switch default to UTF-8, as usual.


(*) If you want to know the exact implementations used, start at the Py_Main() function in Modules/main.c, which handles both -c and -m as:

if (command) {
    sts = PyRun_SimpleStringFlags(command, &cf) != 0;
    free(command);
} else if (module) {
    sts = RunModule(module, 1);
    free(module);
}
Ad
source: stackoverflow.com
Ad