extract data from website using python

Question

I recently started learning python and one of the first projects I did was to scrap updates from my son's classroom web page and send me notifications that they updated the site. This turned out to be an easy project so I wanted to expand on this and create a script that would automatically check if any of our lotto numbers hit. Unfortunately I haven't been able to figure out how to get the data from the website. Here is one of my attempts from last night.

from bs4 import BeautifulSoup
import urllib.request

webpage = "http://www.masslottery.com/games/lottery/large-winningnumbers.html"

websource = urllib.request.urlopen(webpage)
soup = BeautifulSoup(websource.read(), "html.parser")

span = soup.find("span", {"id": "winning_num_0"})
print (span)

Output is here...
<span id="winning_num_0"></span>

The output listed above is also what I see if I "view source" with a web browser. When I "inspect Element" with the web browser I can see the winning numbers in the inspect element panel. Unfortunately I'm not even sure how/where the web browser is getting the data. is it loading from another page or a script in the background? I thought the following tutorial was going to help me but I wasn't able to get the data using similar commands.

http://zevross.com/blog/2014/05/16/using-the-python-library-beautifulsoup-to-extract-data-from-a-webpage-applied-to-world-cup-rankings/

Any help is appreciated. Thanks

if the content is dynamic, you might need an approach based on, e.g., Selenium - selenium-python.readthedocs.io/api.html — ewcz
– ewcz, Commented Sep 15, 2016 at 12:21
Possible duplicate of Reading dynamically generated web pages using python — Sandeep
– Sandeep, Commented Sep 15, 2016 at 12:24
Checking from the developer console what that page does, it loads the data dynamically from here: masslottery.com/data/json/games/lottery/recent.json So you could just write a script that loads that json-formatted data and checks the numbers from there. A lot easier than scraping html ;) — lari
– lari, Commented Sep 15, 2016 at 12:25
Selenium is definitely the approach that I would recommend in most cases, but you're lucky here - the static approach is actually even easier than what you were trying to do in the first place :) — Wayne Werner
– Wayne Werner, Commented Sep 15, 2016 at 12:36
Thanks for the quick replies. I will try both the static and dynamic approach since this is more of a learning project. — gameoverman
– gameoverman, Commented Sep 15, 2016 at 12:55

Wayne Werner · Accepted Answer · 2016-09-15 12:34:02Z

2

If you look closely at the source of the page (I just used curl) you can see this block

<script type="text/javascript">
    // <![CDATA[
    var dataPath = '../../';
    var json_filename = 'data/json/games/lottery/recent.json';
    var games = new Array();
    var sessions = new Array();
    // ]]>
</script>

That recent.json stuck out like a sore thumb (I actually missed the dataPath part at first).

After giving that a try, I came up with this:

curl http://www.masslottery.com/data/json/games/lottery/recent.json

Which, as lari points out in the comments, is way easier than scraping HTML. This easy, in fact:

import json
import urllib.request
from pprint import pprint

websource = urllib.request.urlopen('http://www.masslottery.com/data/json/games/lottery/recent.json')
data = json.loads(websource.read().decode())
pprint(data)

data is now a dict, and you can do whatever kind of dict-like things you'd like to do with it. And good luck ;)

answered Sep 15, 2016 at 12:34

Wayne Werner

52.3k35 gold badges213 silver badges304 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

gameoverman Over a year ago

Thank you. i will try this tonight!

Wayne Werner Over a year ago

For added fun, you could always use python's random module to guess lotto numbers and see how much money it would make you.

gameoverman Over a year ago

Your solution worked. Now I need to figure out how to easily extract the information from the dictionary since it is multi-level.

Wayne Werner Over a year ago

If it worked, then you should mark this as accepted by clicking the green checkmark to the left <---. For multi-level dictionaries you can simply chain []s, e.g. data['foo']['bar']

Collectives™ on Stack Overflow

extract data from website using python

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related