exceptions.UnicodeDecodeError - 'ascii' codec can't decode byte

Question

I keep getting this error:

<type 'exceptions.UnicodeDecodeError'>: 'ascii' codec can't decode byte 0xe2 in position 0: ordinal not in range(128)
      args = ('ascii', '\xe2\x9d\xb6 Senn =)', 0, 1, 'ordinal not in range(128)')
      encoding = 'ascii'
      end = 1
      message = ''
      object = '\xe2\x9d\xb6 Senn =)'
      reason = 'ordinal not in range(128)'
      start = 0

Using this code:

    steamFriend = data['response']['players'][i]
    n = steamUser(steamFriend['personaname'].encode("utf-8"), steamFriend['steamid'], steamFriend['avatarfull'], steamFriend['profileurl'], steamFriend['personastate'], False)

Some things to note here:

steamFriend is a JSON object
I get this error only sometimes, because the steamFriend['personaname'] contains some weird symbols (for example ❶), and I don't know how to parse this correctly so I don't get errors.

Any help is greatly appreciated. Also, \xe2\x9d\xb6 Senn =) is supposed to represent ❶ Senn =), if that helps.

Had a similar error once in my web scraping. The resolution was first decoding the string as ascii then encoding as utf-8. Or perhaps the other way around... — Jacob Bridges
– Jacob Bridges, Commented Dec 11, 2014 at 4:31

Matthias C. M. Troffaes · Accepted Answer · 2014-12-11 09:53:43Z

1

Without seeing the full code it is hard to tell, but it seems that steamUser expects ascii input. If that is the problem, you can solve it by:

streamFriend['personaname'].encode("ascii", errors="ignore")

or

streamFriend['personaname'].encode("ascii", errors="replace")

Obviously you will lose unicode characters in the process.

answered Dec 11, 2014 at 9:53

Matthias C. M. Troffaes

6054 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Varun Iyer Over a year ago

I don't want to lose unicode characters, so what's an alternative?

Matthias C. M. Troffaes Over a year ago

You don't have an alternative if the steamUser function expects ascii data, except perhaps to use errors="xmlcharrefreplace" which will convert the unicode characters to an ascii xml representation. You can then use stackoverflow.com/questions/17341601/… to convert it back to utf8 when need be. A simpler and cleaner solution, if you have control over the full code base, would be to fix the steamUser function so it can accept unicode.

bobince · Accepted Answer · 2014-12-11 10:16:36Z

1

If the quoted error is occurring on the n=... line, the implication is that steamFriend['personaname'] is a byte string, not a Unicode string.

Consequently when you ask to .encode it, Python has to decode the string to Unicode in order to be able to encode it back to bytes. An implicit decoding happens using the default encoding, which is ASCII, so because the byte string does not contain only ASCII you get a failure.

Are you sure you didn't mean to do:

steamFriend['personaname'].decode("utf-8")

decoding the byte string '\xe2\x9d\xb6 Senn =)' using UTF-8 would give you the Unicode string u'\u2776 Senn =)', where U+2776=❶ so that would seem more like what you are after.

(Normally, however, JSON strings are explicitly Unicode, so it's not clear where you would have got the byte string from. How are you loading the JSON content?)

answered Dec 11, 2014 at 10:16

bobince

538k111 gold badges675 silver badges846 bronze badges

2 Comments

Varun Iyer Over a year ago

So if I decode it then I get this error

<type 'exceptions.UnicodeEncodeError'>: 'ascii' codec can't encode character u'\u2776' in position 2954: ordinal not in range(128)

. What do I do now? Do I have to encode back into ascii to get the ❶?

Varun Iyer Over a year ago

I took out the .encode("utf-8") part because you're right- the JSON string is already unicode- it has a value of u'\u2776 Senn =)'

Collectives™ on Stack Overflow

exceptions.UnicodeDecodeError - 'ascii' codec can't decode byte

2 Answers 2

2 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related