Webscraping a dynamic website with selenium on python

Question

I am trying to automate a task and web scrape from a dynamic website.

This is the site: https://www.kaijinet.com/jpExpress/Default.aspx?f=company&cf=summary&cc=7203

It loads a java script upon retrieval and I've tried, among many other things:

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome()
driver.implicitly_wait(20)

link = "https://www.kaijinet.com/jpExpress/Default.aspx?f=company&cf=summary&cc=7203"

driver.get(link)
selector ="#numberOfIssuedShares > div:nth-child(2) > div.stockDetail > table > tbody > tr > td:nth-child(2)"
shares = driver.find_element_by_css_selector(selector)

as well as adding some

import time
time.sleep(25)

Neither worked. The error is

selenium.common.exceptions.NoSuchElementException: Message: Unable to locate element: #numberOfIssuedShares > div:nth-child(2) > div.stockDetail > table > tbody > tr > td:nth-child(2)

If I print the page source, using

print(driver.page_source)

I get a completely different source code compared to what the browser renders each time. Is there a way to get selenium to execute the page and then work with the dynamically generated html? I googled and none of the solutions that came up lead to a fix for this problem. Thanks!

Maybe use .implicit_wait after getting the link?

phyominh
– phyominh

2021-02-22 05:07:24 +00:00
Commented Feb 22, 2021 at 5:07 — phyominh
– phyominh, Commented Feb 22, 2021 at 5:07

Arundeep Chohan · Accepted Answer · 2021-02-22 05:15:20Z

It was in a double nested iframe.

<iframe src="doc.aspx?f=company&amp;cf=summary&amp;cc=7203" id="main" name="main" frameborder="0" width="750px" style="height: 1721px;"></iframe> 

<iframe src="contents/contents/contents.aspx?path=\7203\7203_20201231_202102101325_summary_d.html" id="ifrmContents" style="width: 685px; height: 1485px;" scrolling="no" frameborder="0"></iframe>

To go through it wait till frame is available and switch to it.

wait = WebDriverWait(driver, 5)
driver.get('https://www.kaijinet.com/jpExpress/Default.aspx?f=company&cf=summary&cc=7203')
wait.until(EC.frame_to_be_available_and_switch_to_it("main"))
wait.until(EC.frame_to_be_available_and_switch_to_it("ifrmContents"))
elem=wait.until(EC.presence_of_element_located((By.CSS_SELECTOR,"#numberOfIssuedShares > div:nth-child(2) > div.stockDetail > table > tbody > tr > td:nth-child(2)")))
print(elem.text)

Outputs

3,262,997,492 shares

Import

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC

Collectives™ on Stack Overflow

Webscraping a dynamic website with selenium on python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related