How to properly scrape data from a javascript website in Python?

Question

I am a newby in python programming and I wanted to practice webscraping. My original aim was to list top 10 players in some countries according their points. But it seems, I cannot get the whole source code of the page, hence I am not able to create my database. So when I click in Chrome on "inspect\elements", I am able to see the necessary data, but with my code below, I cannot get all of it.

from selenium import webdriver
url = 'https://aoe2.net/#aoe2de-leaderboard-rm-1v1'
driver = webdriver.Chrome()
code = driver.get(url)
source = driver.page_source
print(source)

Could you tell me please, what do I do wrong?

Aryan Garg · Accepted Answer · 2021-02-28 17:28:03Z

2

tried out this code and should work:

from selenium import webdriver
from time import sleep
driver = webdriver.Firefox()
code = driver.get('https://aoe2.net/#aoe2de-leaderboard-rm-1v1')
sleep(5) #Since my internet is slow, if yours is fast enough then you can comment this line
d1 = {}
for i in range(1,11):
    if i % 2 == 1:
        val = 'odd'
    else:
        val = 'even'
    a = driver.find_element_by_css_selector(f'tr.{val}:nth-child({i}) > td:nth-child(3) > a:nth-child(3)').text
    d1[i] = a
print(d1)

Just replace firefox with chrome, since I didn't have the Chrome Drivers Installed

answered Feb 28, 2021 at 17:28

Aryan Garg

11711 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Syzygy Over a year ago

Thank you! That really works. However I have some trouble to understand this part of your code: "> td:nth-child(3) > a:nth-child(3)" What the ">" and the (3) stands for exactly? Could you please explain it to me?!

Aryan Garg Over a year ago

Hey There Thats the css selector script, Doing inspect element on the name, and copying for the name column's CSS selector code. tr.{val}:nth-child({i}) > td:nth-child(3) > a:nth-child(3) can be split up into parts: tr - table row (html) val refers to odd and even, this was a repeating pattern in the CSS Selector (odd for odd place and vice versa) nthchild{i}- refers to the actual ranking number (1,2,3,4) the td:nth-child(3) as per my understanding is the location of the data inside the cell. Since the cell which contained name also had blank spacing as well as hyperlink and...

Aryan Garg Over a year ago

avatar information, we took only selective data

Collectives™ on Stack Overflow

How to properly scrape data from a javascript website in Python?

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related