Unable to scrape data using selenium find_element_by_class_name() in python

Question

I am new to webscraping & not a developer nor have any html exp. and was trying to pull some details from my account after logging in into the website but getting errors in find_element_by_class_name()

this is the code I have tried:

from selenium import webdriver

driver = webdriver.Chrome('path/chromedriver.exe')
driver.get("https://www.URL.COM") 

# logged into account manually & maneuvered to the page manually

driver.find_element_by_class_name('css-901oao css-cens5h r-1khnkhu r-13awgt0 r-1oke55r r-1enofrn r-1wzrnnt')

Error

---------------------------------------------------------------------------
NoSuchElementException                    Traceback (most recent call last)
~\AppData\Local\Temp/ipykernel_14516/1270707966.py in <module>
----> 1 driver.find_element_by_class_name('css-901oao css-cens5h r-1khnkhu r-13awgt0 r-1oke55r r-1enofrn r-1wzrnnt')

C:\ProgramData\Miniconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py in find_element_by_class_name(self, name)
    562             element = driver.find_element_by_class_name('foo')
    563         """
--> 564         return self.find_element(by=By.CLASS_NAME, value=name)
    565 
    566     def find_elements_by_class_name(self, name):

Also tried

driver.find_element_by_css_selector('css-901oao css-cens5h r-1khnkhu r-13awgt0 r-1oke55r r-1enofrn r-1wzrnnt')

From inspect I was able to view this & tried to extract the one highlighted in the image:

cruisepandey · Accepted Answer · 2021-11-21 11:50:03Z

1

class name expect a single class name. where as you are passing multiple class name here

css-901oao css-cens5h r-1khnkhu r-13awgt0 r-1oke55r r-1enofrn r-1wzrnnt

It won't work.

Instead

remove the spaces and make a CSS selector out of it.

driver.find_element(By.CSS_SELECTOR, ".css-901oao.css-cens5h.r-1khnkhu.r-13awgt0.r-1oke55r.r-1enofrn.r-1wzrnnt")

Also, Please remember find_element_by_class_name have been deprecated in newest selenium. You should use this instead

find_element(By.CLASS_NAME, "class name")

having said this, the locator that you've right now looks brittle in nature. Please use static attribute values.

You could try this xpath

//div[starts-with(@class,'css')]//div[@dir='auto' and contains(@style,'-webkit-line-clamp')]

Please check in the dev tools (Google chrome) if we have unique entry in HTML DOM or not.

xpath that you should check :

//div[starts-with(@class,'css')]//div[@dir='auto' and contains(@style,'-webkit-line-clamp')]

Steps to check:

Press F12 in Chrome -> go to element section -> do a CTRL + F -> then paste the xpath and see, if your desired element is getting highlighted with 1/1 matching node.

edited Nov 21, 2021 at 11:50

answered Nov 21, 2021 at 11:27

cruisepandey

29.5k6 gold badges23 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

ViSa Over a year ago

When I try driver.find_element_by( I get an error AttributeError: 'WebDriver' object has no attribute 'find_element_by'. So I tried driver.find_element(By.CSS_SELECTOR, ".css-901oao.css-cens5h.r-1khnkhu.r-13awgt0.r-1oke55r.r-1enofrn.r-1wzrnnt") but then I get error: NameError: name 'By' is not defined

cruisepandey Over a year ago

My bad man ! there was a typo, I corrected it. Please try now.

cruisepandey Over a year ago

Please import from selenium.webdriver.common.by import By

cruisepandey Over a year ago

In case of iteration like you are doing there for tag in table:, it should be find_elements not find_element

ViSa Over a year ago

by using find_elements it worked for me. thanks alot for staying here & helping me out through this. Really really Appreciate that :)

|

Collectives™ on Stack Overflow

Unable to scrape data using selenium find_element_by_class_name() in python

1 Answer 1

8 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

8 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related