Webscraping issue with Python and Selenium

Question

I am attempting to extract scroll-down menu data from here

I run the following script LINE-BY-LINE in a console and it runs with this error - there is other work of this online but I tried to use those resources with no avail:

import os
import time
import tempfile
from time import sleep
from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

###
# downloadDir = tempfile.mkdtemp()

###
# prefs = { 'download.default_directory' : downloadDir }
# chromeOptions = webdriver.ChromeOptions()
# chromeOptions.add_experimental_option("prefs",prefs)

###
chrome_path = r'C:\Users\Nate\PycharmProjects\testing_1\chromedriver.exe'
cdriver_1  = webdriver.Chrome(chrome_path)
cdriver_1.get(url = 'http://webapps.rrc.texas.gov/eds/eds_searchUic.xhtml')

###

cdriver_1.find_element_by_xpath("//*[@id='SearchUicForm:district_1']/..").click()
# cdriver_1.find_element_by_xpath("//*[@id='SearchUicForm:county_1']").click()
# cdriver_1.find_element_by_xpath("//*[@id='ui-datepicker-div']/table/tbody/tr[1]/td[1]/a").click()
# cdriver_1.find_element_by_xpath("//*[@id='ui-datepicker-div']/table/tbody/tr[2]/td[2]").click()

ERROR:

selenium.common.exceptions.ElementNotVisibleException: Message: element not interactable
  (Session info: chrome=76.0.3809.132)
  (Driver info: chromedriver=72.0.3626.7 (efcef9a3ecda02b2132af215116a03852d08b9cb),platform=Mac OS X 10.14.3 x86_64)

Not that I want to grab any option from the scroll-down menu options like District, Permit Number, etc. How do I fix this?

JeffC · Accepted Answer · 2019-09-10 14:12:18Z

1

The issue is that you are clicking an LI element that is not currently visible. You must first click the dropdown to open it, then click the desired LI.

cdriver_1.find_element_by_id("SearchUicForm:district_label").click() # open the District dropdown
cdriver_1.find_element_by_xpath("//ul[@id='SearchUicForm:district_items']/li[.='01']").click() # select district "01"

I would suggest that you wrap this up in a function, set_district(district), and put the parameter into the locator so that it's reusable.

cdriver_1.find_element_by_xpath("//ul[@id='SearchUicForm:district_items']/li[.='{0}']".format(district)).click()

See How do I put a variable inside a string? for more info, examples.

answered Sep 10, 2019 at 14:12

JeffC

26.4k5 gold badges35 silver badges56 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Webscraping issue with Python and Selenium

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related