Problems with decoding from command prompt [python]

Question

I have the following problem. I have a german text saved in .txt UTF-8 format, and I'd like to print it out with python. Here's my code:

txt = open(filename, 'r').read()
print txt.decode('utf-8-sig')

It works perfectly in IDLE, but when I save my code and run it from the command prompt, it raises error, specifically:

UnicodeEncodeError: 'charmap' codec can't encode characters in position 3-4: cha
racter maps to <undefined>

In my particular case, the text is "gemäßigt", and in the beginning of .py code I put something like

# -*- coding: utf-8-sig -*-

By the way, my OS is Windows, in Russian. Does anybody have an idea what is my problem?

Best, Alex

What do you get when you import sys and then sys.stdout.encoding in the console? — Tim Pietzcker
– Tim Pietzcker, Commented May 25, 2014 at 12:39
That's your problem, that codepage doesn't support German special characters. What do you get when you try the above command in IDLE? — Tim Pietzcker
– Tim Pietzcker, Commented May 25, 2014 at 12:50

Tim Pietzcker · Accepted Answer · 2014-05-25 12:48:19Z

1

Your console uses the DOS codepage 866 which doesn't have the character symbols for ä or ß, causing the error.

You could try .encoding('cp866', errors='replace') your string before output, replacing all the characters not supported by your terminal by ?s.

answered May 25, 2014 at 12:48

Tim Pietzcker

337k59 gold badges520 silver badges572 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Alekz112 Over a year ago

Somehow, it does not work. I'll goolge later on again, and write down solution if I'll find some. Many thanks for your suggestions how to solve it!

Tim Pietzcker Over a year ago

Maybe I was unclear; you first need to decode it from UTF-8, then encode that to cp866: print txt.decode('utf-8-sig').encode('cp866', errors='replace'). Did you try it that way?

Tim Pietzcker Over a year ago

@Alekz112: if you don't need to output both latin and cyrillic characters, you can try changing the codepage (chcp 850 at the DOS prompt, then python.exe myscript.py). That at least allows you to output words like "gemäßigt".

Walle Cyril · Accepted Answer · 2014-05-25 12:55:01Z

0

Is your text in UTF-8 or utf-8-sig ? It's not exaclty the same. Here you can learn the difference. https://docs.python.org/3/library/codecs.html#encodings-and-unicode

You can also open text file already decoded with

import codecs
txt = codecs.open(filename,'r',"utf-8-sig").read()

I think Tim is correct about the console problem.

answered May 25, 2014 at 12:55

Walle Cyril

3,2674 gold badges28 silver badges60 bronze badges

1 Comment

Alekz112 Over a year ago

Yeah, I suspected console problem, I just can't figure out how to fix it

Collectives™ on Stack Overflow

Problems with decoding from command prompt [python]

2 Answers 2

3 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related