How to write a text list to a defined csv file? Python selenium scraping

Question

I am new to Python, so please forgive me if this is a simple issue.

I am web scraping an entire experience section from Linkedin using Selenium. Below is my relevant code:

from time import sleep
from selenium import webdriver

ChromeOptions = webdriver.ChromeOptions()
driver = webdriver.Chrome('/Users/jones/Downloads/chromedriver')

driver.get('https://www.linkedin.com/in/pauljgarner/')

##writing 'Name' to excel
writer = csv.writer(open(parameters.file_name, 'w', encoding='utf8'))
writer.writerow(['Name'])

name = sel.xpath('normalize-space(//li[@class="inline t-24 t-black t-normal break-words"])').extract_first()

writer.writerow([name])

##scraping the entire work experience section:

experience = driver.find_elements_by_xpath('//section[@id = "experience-section"]/ul//li')
for item in experience:
    print(item.text)
    print("")

The output I get from the experience section is a text list that looks like the below:

Freelance Python Developer
Company Name
Depop
Dates Employed
Jun 2015 – Present
Employment Duration
4 yrs 11 mos
Location
London, United Kingdom
Python development using: Django, PostgreSQL, ElasticSearch, TensorFlow, Redis, gevent, Mongodb, Django REST Framework

I want to write this output into the same excel sheet I used to capture the 'Name'.

The excel format I'm looking for would look like:

Name  Title   CompanyName  DatesEmployed  EmploymentDuration  Location  Description
Paul  Freel.. Depop        Jun 2015 – P.. 4 yrs 11 mos        London    Python Dev..

The issue is that I do not know how to convert the text list I scraped from the experience section into the same Excel sheet that I defined earlier with a specific element (with 'Names').

Shubham Sharma · Accepted Answer · 2020-04-02 03:56:48Z

1

Try this:

from selenium import webdriver

ChromeOptions = webdriver.ChromeOptions()
driver = webdriver.Chrome('/home/shubham/Downloads/chromedriver')
driver.get('https://www.linkedin.com/in/pauljgarner/')


rows = []

name = sel.xpath('normalize-space(//li[@class="inline t-24 t-black t-normal break-words"])').extract_first()
experience = driver.find_elements_by_xpath('//section[@id = "experience-section"]/ul//li')

rows.append([name])
for item in experience:
    rows[0].append(item)
    print(item.text)
    print("")

with open(parameters.file_name, 'w', encoding='utf8') as file:
    writer = csv.writer(file)
    writer.writerows(rows)

answered Apr 2, 2020 at 3:56

Shubham Sharma

71.8k6 gold badges26 silver badges58 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Yu Na Over a year ago

Hi Shubham, Thanks for your answer. The code ran without error, but the excel file still doesn't seem quite right. Instead of the observations in the experience section printing, each cell has a string that looks like:

<selenium.webdriver.remote.webelement.WebElement (session="4a06b20b8e051c532fbe7a333a4d6aec", element="beb29e54-aab5-4da6-9026-1fb5dd1fe3c0")>

The cell for xpath 'Name', however, printed correctly

Yu Na Over a year ago

Update: I edited the code to be rows[0].append(item.text) but this Excel output only prints one row. Each item in experience prints as one cell, instead of multiple column variables (Company, Years of Experience, etc.) with observations (such as in the ideal output I'd described above)

Yu Na Over a year ago

I suspect the issue is that a delimiter will need to be inserted in the text list for the .csv to put them in separate columns, but am not sure how to do that in this case

Collectives™ on Stack Overflow

How to write a text list to a defined csv file? Python selenium scraping

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related