Selenium Python extract text between Span

Question

I am trying to extract the text "Margaret Osbon" from HTML below via Python with Selenium. But I keep getting blank values when I print. I have tried get_attribute Still getting blank values when I print

<div class="author-info hidden-md">
                By (author)&nbsp;

                    <span itemprop="author" itemtype="http://schema.org/Person" itemscope="Margareta Osborn">
                        <a href="/author/Margareta-Osborn" itemprop="url">
                            <span itemprop="name">
                                Margareta Osborn</span>
                        </a>
                    </span>
                </div>

Below is my code for Python

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time"

PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)

driver.get("https://www.bookdepository.com/")

keyword = "9781925324402"

Search = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.XPATH, '//*[@id="book-search-form"]/div[1]/input[1]'))
        )
Search.clear()
Search.send_keys(keyword)
Search.send_keys(Keys.RETURN)

try:
    authors = driver.find_element_by_xpath("//div[@class='author-info hidden-md']/span/a/span").text
    print(authors)
    driver.quit()

except:
    authors = "Not Available"
    print(authors)
    driver.quit()

cruisepandey · Accepted Answer · 2021-05-27 09:59:57Z

1

You need to call the .text method which is present in the Selenium Python binding.

.text is present for web element

authors = driver.find_element_by_xpath("//div[@class='author-info hidden-md']/span/a/span").text
print(authors)

or

authors = driver.find_element_by_xpath("//a[contains(@href,'/author/Margareta-Osborn')]").get_attribute('innerHTML')
print(authors)

Update 1 :

driver.maximize_window()
wait = WebDriverWait(driver, 30)
driver.get("https://www.bookdepository.com/Rose-River-Margareta-Osborn/9781925324402")
authors = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.author-info.hidden-md span[itemprop='author'] span"))).text
print(authors)

edited May 27, 2021 at 9:59

answered May 27, 2021 at 6:00

cruisepandey

29.5k6 gold badges23 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

Ben Over a year ago

I tried that cruise, but its still showing as blank

cruisepandey Over a year ago

Is URL public ? Can you try the second solution ?

Ben Over a year ago

i tried the 2nd solution and still not giving me the text. This is the URL - bookdepository.com/Rose-River-Margareta-Osborn/9781925324402

cruisepandey Over a year ago

` Margareta Osborn` this is present 5 times in UI. which one you want to extract ?

Ben Over a year ago

that did it. thanks heaps. So just have to add more wait time? and search for elements via CSS instead of XPATH?

|

Adarsh Kumar GM · Accepted Answer · 2021-05-27 06:04:12Z

0

You are missing ".text" to get the value and maybe because of that you are getting some junk value. I am thinking that you are receiving just a reference ID for that.

Using .text -

#Get Element using Xpath
element = //span[@itemprop='name']

#Fetch using the driver findElement
author = driver.find_element_by_xpath(element).text

#Print the text
print(author)

Using JavaScriptExecutor -

driver.execute_script('return arguments[0].innerText;', element)

Using Get Attribute -

driver.find_element_by_xpath(element).get_attribute('innerText')

answered May 27, 2021 at 6:04

Adarsh Kumar GM

1841 gold badge5 silver badges18 bronze badges

Comments

KunduK · Accepted Answer · 2021-05-27 09:58:38Z

To get the value from span. Use WebDriverWait() and wait for visibility_of_element_located() and following css selector.

and use either .text or .get_attribute("textContent"))

driver.get('https://www.bookdepository.com/Rose-River-Margareta-Osborn/9781925324402')
print(WebDriverWait(driver,5).until(EC.visibility_of_element_located((By.CSS_SELECTOR, '.author-info.hidden-md [ itemprop="author"]'))).text)
print(WebDriverWait(driver,5).until(EC.visibility_of_element_located((By.CSS_SELECTOR, '.author-info.hidden-md [ itemprop="author"]'))).get_attribute("textContent"))

you need to import below libraries.

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By

Collectives™ on Stack Overflow

Selenium Python extract text between Span

3 Answers 3

10 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

10 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related