UnicodeEncodeError in python3

Question

Some of my application's libraries are depending on being able to print UTF-8 characters to stdout and stderr. Therefore this must not fail:

print('\u2122')

On my local machine it works, but on my remote server it raises UnicodeEncodeError: 'ascii' codec can't encode character '\u2122' in position 0: ordinal not in range(128)

I tried $ PYTHONIOENCODING=utf8 with no apparent effect.

sys.stdout = codecs.getwriter("utf-8")(sys.stdout.detach())

works for a while, then stalls and finally fails with ValueError: underlying buffer has been detached

sys.getdefaultencoding() returns 'utf-8', and sys.stdout.encoding returns 'ANSI_X3.4-1968'

What can I do? I don't want to edit third-party libraries.

Side-note: Even if everything else worked, PYTHONIOENCODING=utf8 won't work unless you export it (or prefix the Python launch with it). Otherwise, it's a local variable in bash that isn't inherited in the environment of child processes. export PYTHONIOENCODING=utf-8 would both set and export it in bash. — ShadowRanger
– ShadowRanger, Commented Oct 18, 2016 at 17:08
@SamuelPS: The suggestion of the top answer there is... suboptimal. Forcibly reloading sys to regain access to setdefaultencoding can cause problems, and in any event, the correct solution on modern Python (>=3.3) is to make sure your system is using a broadly useful full Unicode supporting default encoding globally. Anything else means you're using hacks to output characters the OS officially doesn't even recognize, and dependent on it playing along despite it claiming it won't work. — ShadowRanger
– ShadowRanger, Commented Oct 18, 2016 at 17:22

Mirac7 · Accepted Answer · 2016-10-18 17:50:34Z

6

From @ShadowRanger's comment on my question,

PYTHONIOENCODING=utf8 won't work unless you export it (or prefix the Python launch with it). Otherwise, it's a local variable in bash that isn't inherited in the environment of child processes. export PYTHONIOENCODING=utf-8 would both set and export it in bash.

export PYTHONIOENCODING=utf-8 did the trick, UTF-8 characters no longer raise UnicodeEncodeError

answered Oct 18, 2016 at 17:50

Mirac7

1,6567 gold badges28 silver badges45 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Vic Nicethemer Over a year ago

Where to put this line export PYTHONIOENCODING=utf-8 ? I read several similar questions and still cannot understand what file or script should i modify

Mirac7 Over a year ago

@VicNicethemer run export PYTHONIOENCODING=utf-8 in bash before running python your_script.py

Vic Nicethemer Over a year ago

i have done that already, unfortunatelly this doesnot solve my case

Satyaaditya Over a year ago

it only works for stdout/in but not for fileIO, you need to set LC_CTYPE to utf-8 to handle everything

ShadowRanger · Accepted Answer · 2016-10-18 16:59:36Z

0

I'm guessing you're on a UNIX-like system, and your environment set LANG (or LC_ALL or whatever) to C.

Try editing your default shell's startup file to set LANG to something like en_US.utf-8 (or whatever locale makes sense for you)? For example, in bash, edit ~/.bash_profile (or ~/.profile if you're using that instead for sh compatibility) and add:

export LANG="en_US.utf-8"

For (t)csh, edit ~/.cshrc (or ~/.tcshrc if that's what you're using) to add:

setenv LANG "en_US.utf-8"

Making the changes "live" doesn't work, because your shell is likely hosted in a terminal that has configured itself solely for ASCII display, based on the LANG=C in effect when it was launched (and many terminals do session coalescence, so even if you changed LANG and then launched a new terminal, it would coalesce with the shared terminal process with the out-of-date LANG). So after you change ~/.bash_profile, log out and then log back in so your root shell will set LANG correctly for every other process (since they all ultimately fork from the root shell).

answered Oct 18, 2016 at 16:59

ShadowRanger

158k12 gold badges221 silver badges314 bronze badges

2 Comments

Mirac7 Over a year ago

echo $LANG is returning en_US.UTF-8 by default

ShadowRanger Over a year ago

@Mirac7: That doesn't necessarily mean it was set that way at login. If you set it in your .bashrc for instance (and per normal setup, .bashrc isn't run for login shells), you'd see it in your terminals, but the terminals themselves wouldn't see it.

Collectives™ on Stack Overflow

UnicodeEncodeError in python3

2 Answers 2

4 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related