1

i want to put the table and its headings i scraped into excel. i tried multiple things, but i can't seem to figure out how to display it properly in excel. there is also an image below to show how i'd like it displayed ideally. thank you in advance.

from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome("drivers/chromedriver")
driver.get("https://web3.ncaa.org/hsportal/exec/hsAction")
Select(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.ID, "state")))).select_by_visible_text("New Hampshire")
driver.find_element_by_xpath("//input[@id='city']").send_keys("Moultonborough")
driver.find_element_by_xpath("//input[@id='name']").send_keys("Moultonborough Academy")
driver.find_element_by_xpath("//input[@value='Search']").click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@name='hsCode']"))).click()
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@id='approvedCourseTable_1']//th[@class='header']")))])

table = ([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "table#approvedCourseTable_1.tablesorter")))])

with open('out.csv', 'w', newline='') as csvfile:
    writer = csv.writer(csvfile)
    writer.writerow(table)



for some reason scraping the table to excel when using table#approvedCourseTable_1.tablesorter only shows "Course" and that's it. when i separate the headings and table content, i can scrape them to excel separately, but not together. also, the table contents are not lined up properly when i do manage to scrape it to excel.

x = ([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "table#approvedCourseTable_1 th.header")))])
y = ([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "table#approvedCourseTable_1 td")))])

i would like it to be displayed like this if possible: enter image description here

4
  • I think the easier way to achieve it is to create a template in Excel that has row 1 to 4 as you need it and pass all your data into a pandas dataframe in that way you will be able to easily achieve what you need. Commented Aug 3, 2020 at 15:15
  • so would i have to web scrape everything again using pandas and dataframe? Commented Aug 3, 2020 at 17:58
  • No, you just need to parse your table variable into a pandas dataframe , you can check some information here: parsing a list into a pandas dataframe Commented Aug 3, 2020 at 19:50
  • i'm looking into pandas, but i did manage to get into excel. however, it is not aligning properly. maybe you know? stackoverflow.com/questions/63235754/… Commented Aug 3, 2020 at 20:02

1 Answer 1

2

I have this working using Selenium/Python. Try the below code sample,

from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import csv
csvFile = open('out.csv', 'w')
writer = csv.writer(csvFile)


driver = webdriver.Chrome("drivers/chromedriver")
driver.get("https://web3.ncaa.org/hsportal/exec/hsAction")
Select(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.ID, "state")))).select_by_visible_text("New Hampshire")
driver.find_element_by_xpath("//input[@id='city']").send_keys("Moultonborough")
driver.find_element_by_xpath("//input[@id='name']").send_keys("Moultonborough Academy")
driver.find_element_by_xpath("//input[@value='Search']").click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//input[@name='hsCode']"))).click()
print([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.XPATH, "//table[@id='approvedCourseTable_1']//th[@class='header']")))])

#table = ([my_elem.text for my_elem in WebDriverWait(driver, 20).until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, "table#approvedCourseTable_1.tablesorter")))])

table_header = driver.find_element_by_xpath("(//table[@id='NcaaCrs_ApprovedCategory_All']//td[@class='hs_tableHeader'])[1]")
print(table_header.text)
writer.writerow(table_header.text)

#Find All Approved Categories
approved_Categories = driver.find_elements_by_xpath("//div[contains(@id,'NcaaCrs_ApprovedCategory_')]")

for i in range(len(approved_Categories)):
    cateogry_header = driver.find_element_by_xpath("//div[contains(@id,'NcaaCrs_ApprovedCategory_"+str(i+1)+"')]//td[@class='hs_tableHeader']")
    print(cateogry_header.text)
    writer.writerow(cateogry_header.text)
    #Find Course table header and rows
    course_headers = driver.find_elements_by_xpath("//table[contains(@id,'approvedCourseTable_"+str(i+1)+"')]/thead//th")
    header_val = []
    for headers in course_headers:
        header_val.append(headers.text)
    print(header_val)
    writer.writerow(header_val)
    course_rows = driver.find_elements_by_xpath("//table[@id='approvedCourseTable_"+str(i+1)+"']//tbody/tr")
    for j in range(len(course_rows)):
        row_values = driver.find_elements_by_xpath("//table[@id='approvedCourseTable_"+str(i+1)+"']//tbody/tr["+str(j+1)+"]/td")
        row_val = []
        for row in row_values:
            row_val.append(row.text)
        print(row_val)
        writer.writerow(row_val)


csvFile.close()
driver.quit()

The CSV output will be somthing like this,

['Course\nWeight', 'Title', 'Notes', 'Max\nCredits', 'OK\nThrough', 'Disability\nCourse']
Approved Courses
English
['Course\nWeight', 'Title', 'Notes', 'Max\nCredits', 'OK\nThrough', 'Disability\nCourse']
['', 'AFRICAN LITERATURE', '', '', '', 'No']
['', 'AMERICAN LITERATURE', '', '', '', 'No']
['', 'AP ENGLISH LANGUAGE & COMPOSITION', '', '', '', 'No']
['', 'AP ENGLISH LITERATURE & COMPOSITION', '', '', '', 'No']
['', 'COLLEGE COMPOSITION', '', '', '', 'No']
['', 'ENGLISH 9 (ENG 091/092/093)', '', '', '', 'No']
['', 'ENGLISH 9/H', '', '', '', 'No']
['', 'PUBLIC SPEAKING', '', '', '', 'No']
['', 'WORLD STUDIES', '', '', '', 'No']
['', 'WORLD STUDIES HBC', '', '', '', 'No']
Social Science
['Course\nWeight', 'Title', 'Notes', 'Max\nCredits', 'OK\nThrough', 'Disability\nCourse']
['', 'AP WORLD HISTORY', '', '', '', 'No']
['', 'ECONOMICS', '', '', '', 'No']
['', 'GOVERNMENT', '', '', '', 'No']
['', 'PSYCHOLOGY', '', '', '', 'No']
['', 'US HISTORY', '', '', '', 'No']
['', 'US HISTORY/AP', '', '', '', 'No']
['', 'WORLD STUDIES', '', '', '', 'No']
['', 'WORLD STUDIES HBC', '', '', '', 'No']
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.