Python 3 throws UnicodeEncodeError with environment variable LANG=C

Question

I have the following Python script:

#!/usr/bin/env python3
# -*- coding: utf-8 -*-

print('☺')

When I run it on my Debian system, it produces the following output, as expected:

$ ./test.py 
☺
$

However, when I change locale to "C", by setting the LANG environment variable, the script throws a UnicodeEncodeError:

$ LANG=C ./test.py
Traceback (most recent call last):
  File "./test.py", line 4, in <module>
    print('\u263a')
UnicodeEncodeError: 'ascii' codec can't encode character '\u263a' in position 0: ordinal not in range(128)
$

This problem prevents this script from being executed in minimal environments, such as during boot or in embedded systems. Also, I suspect that many existing Python programs can be broken by executing them with LANG=C. Here's an example on Stackoverflow of a program that presumably broke because it's executed in the "C"-locale.

Is this a bug in Python? What's the best way to prevent this?

"Is this a bug in Python?" — You're telling Python to print WHITE SMILING FACE in a locale that doesn't have it. I don't think that's Python's fault. — jwodder
– jwodder, Commented Aug 29, 2016 at 12:49

Antti Haapala · Accepted Answer · 2016-08-29 12:49:40Z

3

This is because Python 3 uses the locale settings to deduce the output character encoding; that is, Python will use the locale that would be displayed for LC_CTYPE when you execute the locale command:

% locale 
...
LC_CTYPE="en_US.UTF-8"
...

If you force LC_CTYPE to C, then Python will assume that ASCII should be used as the output encoding. And ASCII doesn't have a mapping for U+263A.

If you want Python to know how to encode Unicode properly, set the LC_CTYPE to an appropriate value, or write binary to fd 1.

answered Aug 29, 2016 at 12:49

Antti Haapala

135k23 gold badges297 silver badges349 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Jaap Joris Vens Over a year ago

Does this mean that, in order to make a script portable, I should always test what the current locale setting is before I try to print Unicode strings?

bobince Over a year ago

@hedgie You can't make a truly portable script that prints non-ASCII characters. There are too many terminals that don't support it.

Collectives™ on Stack Overflow

Python 3 throws UnicodeEncodeError with environment variable LANG=C

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related