Trouble finding element in Selenium with Python

Question

I have been trying to collect a list of live channels/viewers on Youtube Gaming. I am using selenium with Python to force the website to scroll down the page so it loads more that 11 channels. For reference, this is the webpage I am working on.

I have found the location of the data I want, but I am struggling with getting selenium to go there. The part I am having trouble with looks like this:

<div class="style-scope ytg-gaming-video-renderer" id="video-metadata"><span class="title ellipsis-2 style-scope ytg-gaming-video-renderer"><ytg-nav-endpoint class="style-scope ytg-gaming-video-renderer x-scope ytg-nav-endpoint-2"><a href="/watch?v=FFKSD1HHrdA" tabindex="0" class="style-scope ytg-nav-endpoint" target="_blank">
              Live met Bo3
            </a></ytg-nav-endpoint></span>
    <div class="channel-info small layout horizontal center style-scope ytg-gaming-video-renderer">
        <ytg-owner-badges class="style-scope ytg-gaming-video-renderer x-scope ytg-owner-badges-0">
            <template class="style-scope ytg-owner-badges" is="dom-repeat"></template>
        </ytg-owner-badges>
        <ytg-formatted-string class="style-scope ytg-gaming-video-renderer">
            <ytg-nav-endpoint class="style-scope ytg-formatted-string x-scope ytg-nav-endpoint-2"><a href="/channel/UCD8Q9V5wgo8o0XGfUqsRrDQ" tabindex="0" class="style-scope ytg-nav-endpoint" target="_blank">Rico Eeman</a>
            </ytg-nav-endpoint>
        </ytg-formatted-string>
    </div><span class="ellipsis-1 small style-scope ytg-gaming-video-renderer" id="video-viewership-info" hidden=""></span>
    <div id="metadata-badges" class="small style-scope ytg-gaming-video-renderer">
        <ytg-live-badge-renderer class="style-scope ytg-gaming-video-renderer x-scope ytg-live-badge-renderer-1">
            <template class="style-scope ytg-live-badge-renderer" is="dom-if"></template>

            <span aria-label="" class="text layout horizontal center style-scope ytg-live-badge-renderer">4 watching</span>
            <template class="style-scope ytg-live-badge-renderer" is="dom-if"></template>
        </ytg-live-badge-renderer>
    </div>
</div>

Currently, I am trying:

#This part works fine. I can use the unique ID
meta_data = driver.find_element_by_id('video-metadata')

#This part is also fine. Once again, it has an ID.
viewers = meta_data.find_element_by_id('metadata-badges')
print(viewers.text)

However, I am have been having trouble getting to the channel name (in this example 'Rico Eeman', and it is under the first nested div tag). Because its a compound class name, I cannot find the element by class name, and trying the following xpaths doesnt work:

name = meta_data.find_element_by_xpath('/div[@class="channel-info small layout horizontal center style-scope ytg-gaming-video-renderer"]/ytg-formatted-string'

name = meta_data.find_element_by_xpath('/div[1])

They both raise the element not found error. I am not really sure what to do here. Does anyone have a working solution?

Guy · Accepted Answer · 2016-04-25 08:28:27Z

1

The name id not in the <ytg-formatted-string> tag, its in one of it descendants. Try

meta_data.find_element_by_css_selector('.style-scope.ytg-formatted-string.x-scope.ytg-nav-endpoint-2 > a')

Or with xpath

meta_data.find_element_by_xpath('//ytg-nav-endpoint[@class="style-scope ytg-formatted-string x-scope ytg-nav-endpoint-2"]/a')

answered Apr 25, 2016 at 8:28

Guy

51.2k10 gold badges49 silver badges96 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Pieter Over a year ago

Thank you, with css selector worked perfectly! However, if I print the result of the xpath with result.text it prints empty strings. EDIT: Which is no problem ofcourse, as it works with the css selector! :)

Padraic Cunningham · Accepted Answer · 2016-04-25 09:18:14Z

0

This will get all the names, even if your xpath worked using video-metadata would not get all the names, the id is repeated per div for each user so you would need find_elements and to iterate over the returned elements:

names = dr.find_elements_by_css_selector("a.style-scope.ytg-nav-endpoint[href^='/channel/']")
print([name.get_attribute("text") for name in names])

Which gives you:

['NinjaNation Gaming', 'DURX DANIEL', 'DEMON', 'Perfection', 'The one and only jd', 'Violator Games', 'KingLuii718', 'NinjaNation Gaming', 'DURX DANIEL', 'DEMON', 'Perfection']

edited Apr 25, 2016 at 9:18

answered Apr 25, 2016 at 8:35

Padraic Cunningham

181k30 gold badges264 silver badges327 bronze badges

2 Comments

Pieter Over a year ago

I think there might be something missing on the end in your find_elements_by_css_selector()

Padraic Cunningham Over a year ago

@Pieter. yep, fixed.

Collectives™ on Stack Overflow

Trouble finding element in Selenium with Python

2 Answers 2

1 Comment

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related