Webscrape data from webpage -Python/Selenium

Question

My code executes and gets into the page I want to scrape. Once I am there, im having a hard time printing any elements, in this case just the Names.

The page log in through the code so you can replace the "ExampleUsername" with any email / fake account if you are skeptical.

Here is the code:

import time

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from webdriver_manager.chrome import ChromeDriverManager
productlinks=[]
test1=[]

options = Options()

driver = webdriver.Chrome(ChromeDriverManager().install())


url = "https://www.linkedin.com/uas/login?session_redirect=https%3A%2F%2Fwww%2Elinkedin%2Ecom%2Fsearch%2Fresults%2Fpeople%2F%3FcurrentCompany%3D%255B%25221252860%2522%255D%26geoUrn%3D%255B%2522103644278%2522%255D%26keywords%3Dsales%26origin%3DFACETED_SEARCH%26page%3D2&fromSignIn=true&trk=cold_join_sign_in"
driver.get(url)
time.sleep(2)

username = driver.find_element_by_id('username')
username.send_keys('[email protected]')

password = driver.find_element_by_id('password')
password.send_keys('ExamplePassword')
password.submit()

element1 = driver.find_elements_by_class_name("name actor-name")
title=[t.text for t in element1]
print(title)

KunduK · Accepted Answer · 2020-11-17 21:34:58Z

4

find_elements_by_class_name() doesn't accepts multiple class name. Instead you can use css selector.

To avoid synchronization issue Induce WebDriverWait() and wait for visibility_of_all_elements_located() and following css selector.

element1 =WebDriverWait(driver,10).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,".name.actor-name")))
title=[t.text for t in element1]
print(title)

you need to import below libraries.

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait

edited Nov 17, 2020 at 21:34

answered Nov 17, 2020 at 21:24

KunduK

33.4k5 gold badges19 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Void S Over a year ago

Thanks, this works. Why the periods . in the ".name.actor-name"))) If i wanted to get the location of these users, uses class--> subline-level-2 t-12 t-black--light t-normal search-result__truncate how would this be done using visibility_of_all_elements?

Void S Over a year ago

Nvm correct me if im wrong but you replace spaces with a period? Nonetheless it works, thank you so much. Great explanation as well :)

Collectives™ on Stack Overflow

Webscrape data from webpage -Python/Selenium

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related