Difficulties with character-encoding in Python

Question

I am receiving data via GET request parameters. Some of theese parameters are strings, and I'm having a tough time being able to display them correctly due to encoding issues I guess.

This is an example of what I receive:

{'id_origen': u'9', 'apellidos': u'\xd1\xe9rez', 'nombre': u'Pimp\xe1m'}

You can see that the value for the key 'apellidos' isn't being received properly. It appears

u'\xd1\xe9rez'

instead of

Núñez.

I tried to solve this issues in a very primitive way, replaceing each appearance of a character like "\xe1" with "á", for example. But it is giving me problems also. This is the code I came up with:

tabla = {'\xE1':'á', '\xE9':'é', '\xED': 'í', '\xF3':'ó', '\xFA':'ú'}

logger.info ("Valor del argumento antes del bucle de urldecode: %s" % valor)
for k, v in tabla.iteritems():
    if k in valor:
        valor.replace(k, v)

Of course, it doesn't work as I had expected.

What would be the appropiate treatment for theese type of character encoding that I'm receiving?

hamstergene · Accepted Answer · 2013-10-08 15:21:08Z

2

The values are received correctly (that particular value is “Ñérez” by the way, not “Núñez”).

When Python dumps contents of list or dict to console, every string item is displayed as representation (the result or repr() function), not the original string. For example:

>>> print [0, u"é", 0]
[0, u'\xe9', 0]

I believe the main point of this is to make values directly reusable by copy-pasting them back into the code. Because strings can contain all kinds of quotes and backslashes, and because terminals/webpages/etc may not be capable of displaying non-ascii, printing unchanged string contents would not do the job.

The real text inside those strings is okay:

>>> print u'\xd1\xe9rez'
Ñérez
>>>

answered Oct 8, 2013 at 15:21

hamstergene

24.5k5 gold badges60 silver badges73 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

remram · Accepted Answer · 2013-10-08 15:19:34Z

0

u'\xd1\xe9rez' doesn't seem to be the string Núñez. but rather Ñérez. Are you sure about what your data is?

Other than that, your data is unicode. There is no encoding with unicode because it's already characters; whatever fix you think is necessary should happen upstream. Is your web framework giving you these values?

answered Oct 8, 2013 at 15:19

remram

5,2771 gold badge32 silver badges45 bronze badges

Collectives™ on Stack Overflow

Difficulties with character-encoding in Python

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related