19

I'm trying to decode HTML entries from here NYTimes.com and I cannot figure out what I am doing wrong.

Take for example: 

"U.S. Adviser’s Blunt Memo on Iraq: Time ‘to Go Home’"

I've tried BeautifulSoup, decode('iso-8859-1'), and django.utils.encoding's smart_str without any success.

Improve this question
3
  • This questions seems to come up a lot with no good solution. Makes me want to write something of my own... Commented Jul 30, 2009 at 19:49
  • Ha I think that's the best solution I've found thus far. I might actually try to do that myself. If I do, I'll post my solution. Commented Jul 30, 2009 at 20:01
  • @Triptych: there is unescape(). Commented Oct 7, 2014 at 19:15

4 Answers 4

Reset to default
22 
>>> from HTMLParser import HTMLParser
>>> print HTMLParser().unescape('U.S. Adviser’s Blunt Memo on Iraq: '
...                             'Time ‘to Go Home’')
U.S. Adviser’s Blunt Memo on Iraq: Time ‘to Go Home’

The function is undocumented in Python 2. It is fixed in Python 3.4+: it is exposed as html.unescape() there.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

For future users, this answer appears to have so few upvotes simply because it came 4 years later than the existing answers. It seems to be at least as good an answer. This answer has the advantage that it is simple (unlike writing your own function to to interpret HTML standards using a regex) and uses a standard library (unlike BeautifulSoup). It has the disadvantage that is is using an undocumented function.
20

Actually what you have are not HTML entities. There are THREE varieties of those &.....; thingies -- for example       all mean U+00A0 NO-BREAK SPACE.

  (the type you have) is a "numeric character reference" (decimal).
  is a "numeric character reference" (hexadecimal).
  is an entity.

Further reading: http://htmlhelp.com/reference/html40/entities/

Here you will find code for Python2.x that does all three in one scan through the input: http://effbot.org/zone/re-sub.htm#unescape-html

Improve this answer

Comments

18

This does work:

from BeautifulSoup import BeautifulStoneSoup
s = "U.S. Adviser’s Blunt Memo on Iraq: Time ‘to Go Home’"
decoded = BeautifulStoneSoup(s, convertEntities=BeautifulStoneSoup.HTML_ENTITIES)

If you want a string instead of a Unicode object, you'll need to decode it to an encoding that supports the characters being used; ISO-8859-1 doesn't:

result = decoded.encode("UTF-8")

It's unfortunate that you need an external module for something like this; simple HTML/XML entity decoding should be in the standard library, and not require me to use a library with meaningless class names like "BeautifulStoneSoup". (Class and function names should not be "creative", they should be meaningful.)

Improve this answer

5 Comments

lxml, alas also not in the standard library, also provides a Beautiful Soup parser (and lots more) with somewhat less "creative" names.
Support for entity decoding is in the standard library (module htmlentitydefs). What the OP has are (decimal) numeric character references, not entities.
Works as well with BeautifulSoup instead of BeautifulStoneSoup - one step less "creative" :)
' names should not be "creative" ' is that a stone cold rule, or just personal choice?
@TankorSmash: There's no authority--beyond the compiler--forcing you to follow any coding standards at all, but this seems like common sense to me.
6

Try this:

import re

def _callback(matches):
    id = matches.group(1)
    try:
        return unichr(int(id))
    except:
        return id

def decode_unicode_references(data):
    return re.sub("&#(\d+)(;|(?=\s))", _callback, data)

data = "U.S. Adviser’s Blunt Memo on Iraq: Time ‘to Go Home’"
print decode_unicode_references(data)
Improve this answer

5 Comments

UnicodeEncodeError: 'charmap' codec can't encode character u'\u2019' in position 12: character maps to <undefined> This seems to be the error I keep getting regardless of what I try.
Could you provide more code, then? I just tried it with the function I wrote and the character 2019 works fine. It shows up as: ߣ
A few questions on your regexp: (1) Shouldn't it be \d instead of \w? The regexp will match &#xa0; and &nbsp; but then it will crash in int() (2) Allowing the character reference (it's NOT an entity) to end in a whitespace instead of ';' seems very tolerant -- shouldn't you mention this? (3) Wouldn't the last part be better written as [;\s]?
John, you were correct on point one partially. It won't match &nbsp; since that doesn't start with &#, but yes it should have been \d. Regarding point two to allowing it to end with whitespace, it should be noted that even though it isn't pretty, it's still supported. I've updated the code in the following way: (1) Changed it to \d, (2) made the callback a bit stronger, and (3) used a lookahead assertion for ending whitespace instead of absorbing it like it was.
Evan, thanks for the enlightenment, especially about the tolerance of whitespace, which I didn't know about. I got some more clues by looking in the HTML 4.01 and 2.0 specs. They referred to the SGML standard (ISO 8879). Cost = CHF 238(!) so I didn't read it, but HTML 2.0 commented that ';' is only needed when the character following the reference would otherwise be part of the name. Experiments with FF, IE and Opera using space - / X A and & instead of ; all gave the same result: they terminate the reference and are not swallowed. I'm looking forward to your updated solution ;-)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.