Get page generated with Javascript in Python

Question

I'd like to download web page generated by Javascript and store it to string variable in Python code. The page is generated when you click on button.

If I would know the resulting URL I would use urllib2 but this is not the case.

thank you

Is this generated completly in js or just built from an ajax call ? — Bite code
– Bite code, Commented Jan 22, 2012 at 10:14
Then I'd got with J.F solution, or with python webkit. Just keep in mind they require a display server to be running so if you plan to make it run on a headless server, you'll need to hack a little bit. — Bite code
– Bite code, Commented Jan 22, 2012 at 11:14

jfs · Accepted Answer · 2012-01-22 10:16:04Z

39

You could use Selenium Webdriver:

#!/usr/bin/env python
from contextlib import closing
from selenium.webdriver import Firefox # pip install selenium
from selenium.webdriver.support.ui import WebDriverWait

# use firefox to get page with javascript generated content
with closing(Firefox()) as browser:
     browser.get(url)
     button = browser.find_element_by_name('button')
     button.click()
     # wait for the page to load
     WebDriverWait(browser, timeout=10).until(
         lambda x: x.find_element_by_id('someId_that_must_be_on_new_page'))
     # store it to string variable
     page_source = browser.page_source
print(page_source)

answered Jan 22, 2012 at 10:16

jfs

417k210 gold badges1k silver badges1.7k bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

xralf Over a year ago

is the WebDriverWait with someId_that_must_be_on_new_page neccessary? Could it be done only with some sleep or delay function? And is it possible to set the user-agent string?

xralf Over a year ago

There is one problem yet. On the web page is select element and something have to be selected. If nothing is selected the button won't work. And is neccessary to open and close firefox? Without guit this won't work?

jfs Over a year ago

you could use any condition you like e.g., x.title == 'New Title'. You probably could modify user-agent by using appropriate firefox profile.

jfs Over a year ago

here's an example on how to select option. .quit() is not necessary.

xralf Over a year ago

The method select_option(self, selector, value) takes selector parameter. I'm not sure what this parameter should be. Let's say I want to click on option with value = 100 of select with id = 'sel_id' and name = 'sel_name'. Could this be expressed in CSS?

|

Collectives™ on Stack Overflow

Get page generated with Javascript in Python

1 Answer 1

9 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

9 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related