Extracting javascript enabled data from page

Question

I am trying to extract coloured text in this link. I am using BeautifulSoup in python. The code is as follows:

import time, urllib2, re
from bs4 import BeautifulSoup
url='http://de.vroniplag.wikia.com/wiki/Aaf/008'
def gethtml(link):
    time.sleep(2)
    req = urllib2.Request(link, headers={'User-Agent': "Magic Browser"})
    con = urllib2.urlopen(req)
    html = con.read()
    return html

soup=BeautifulSoup(gethtml(url),'html.parser')
print soup.findAll('span', attrs={"class": re.compile('fragmark')})

But the returned result is empty. How can I change it to make it work?

UPDATE:

I am using chromedriver, in the code as follows:

from selenium import webdriver
import os

chromedriver = "./chromedriver"
os.environ["webdriver.chrome.driver"] = chromedriver
driver = webdriver.Chrome(chromedriver)

driver.get('http://de.vroniplag.wikia.com/wiki/Aaf/008')
for tag in driver.find_elements_by_css_selector('[class^=fragmark]'):
    print(tag.text)
driver.quit()

But the browser opens. The result doesn't appear. I close the browser and then an error occurs.

falsetru · Accepted Answer · 2016-07-02 11:13:55Z

1

You need to use library that can interpret javascript. For example using selenium because those elements with fragmark1, fragmark2, ... are created by the javascript.

from selenium.webdriver import Chrome as Driver
# Replace with `Chrome` with your system browser

driver = Driver()
driver.get('http://de.vroniplag.wikia.com/wiki/Aaf/008')
for tag in driver.find_elements_by_css_selector('[class^=fragmark]'):
    print(tag.text)

answered Jul 2, 2016 at 11:13

falsetru

371k69 gold badges769 silver badges659 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Echchama Nayak Over a year ago

Thank you so very much for the answer. I don't have chrome. I am using chromedriver, can you help me? I have posted the new code as an UPDATE. Please do check

falsetru Over a year ago

You can change Chrome with FireFox if you have Firefox installed. (PhantomJS, Edge, Opera, Ie, Safari, ... are also possible). import selenium.webdriver; print(dir(selenium.webdriver))

Collectives™ on Stack Overflow

Extracting javascript enabled data from page

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related