0

How would one go about scraping a list from a < ul> tag if the list is only loaded in the div after a click is given on the original website?

I have tried using Selenium to automate the click, but because it doesn't change the https address of the website I am unsure how to get at this loaded-in data that otherwise returns < ul>< /ul> with Beautiful soup. The end goal is to get all of the data inside around 400 < li> tags inside of this dynamically loaded list, which is not seen in the inspect tool until after a button is clicked.

The website is: https://www.ibiscycles.com/test-ride/find-a-dealer

After manually clicking the search button the list of all applicable locations is loaded under this < div> name: This screenshot shows the breakdown using inspect

Thank you in advance!

Improve this question
2
  • 1
    What website? What code did you run? What output are you looking for (include the html element in question)? Minimal reproducible example is crucial here. Commented Jun 24, 2022 at 23:02
  • transfer the webdriver page_source to bs4. BeautifulSoup(browser.page_source, 'lxml') Commented Jun 25, 2022 at 3:52

1 Answer 1

Reset to default
0

Why can't you click the button? Wait for the ul to be clickable before selecting it. This should allow you to grab the ul after you've clicked on the button that loads it in

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.