I want to fetch few data/values from a website. I have used beautifulsoup for this and the fields are blank when I try to fetch them from my Python script, whereas when I am inspecting elements of the webpage I can clearly see the values are available in the table row data.
When i saw the HTML Source I noticed its blank there too.
I came up with a reason, the website is using Javascript to populate the values in their corresponding fields from its own database. If so then how can i fetch them using Python?
-
1your python script is not a browser. You need a browser (or emulate the Javascript interpreter) to run the javascript on the fetched page.kasper Taeymans– kasper Taeymans2015-02-27 12:47:35 +00:00Commented Feb 27, 2015 at 12:47
-
Might be worth looking at this question too.elParaguayo– elParaguayo2015-02-27 12:48:52 +00:00Commented Feb 27, 2015 at 12:48
-
I think this is better suited for Selenium. There are python bindings for Selenium as well: selenium-python.readthedocs.orgJoe Young– Joe Young2015-02-27 12:54:30 +00:00Commented Feb 27, 2015 at 12:54
-
You cannot scrape content which is manipulated by JS (AJAX response etc .... I had faced same issue also ...hence I had to use SeliniumUmair Ayub– Umair Ayub2015-03-04 08:00:51 +00:00Commented Mar 4, 2015 at 8:00
Add a comment
|
2 Answers
The Python binding for Selenium and phantomjs (if you want to use a headless browser as backend) are the appropriate tools for this job.
2 Comments
Nima Derakhshanjan
that's more useful if you put a sample code in your answer.
Biswarup Dass
a brief explanation of the task you mentioned would be of great help @Thierry
Yes, you can scrape JS data, it just takes a bit more hacking. Anything a browser can do, python can do.
If you're using firebug, look at the network tab to see from which particular request your data is coming from. In chrome element inspection, you can find this information in a tab named network, too. Just hit ctrl-F to search the response content of the requests.
If you found the right request, the data might be embedded in JS code, in which case you'll have some regex parsing to do. If you're lucky, the format is xml or json, in which case you can just use the associated builtin parser.
