Problem parsing a table from html using BeautifulSoup and saving it as a csv

Question

import requests
import csv
import requests
from bs4 import BeautifulSoup

r = requests.get('https://pqt.cbp.gov/report/YYZ_1/12-01-2017')
soup = BeautifulSoup(r)
table = soup.find('table', attrs={ "class" : "table-horizontal-line"})
headers = [header.text for header in table.find_all('th')]
rows = []
for row in table.find_all('tr'):
    rows.append([val.text.encode('utf8') for val in row.find_all('td')])

with open('output_file.csv', 'wb') as f:
    writer = csv.writer(f)
    writer.writerow(headers)
    writer.writerows(row for row in rows if row)

I am trying to parse all table data in this particular webpage: https://pqt.cbp.gov/report/YYZ_1/12-01-2017

I am getting an error in the line soup = BeautifulSoup(r). I get an error TypeError: object of type 'Response' has no len(). I am also not sure if my logic is correct. Plz help me pasing the table data.

QHarr · Accepted Answer · 2018-12-01 11:07:51Z

1

I would do it this way

import pandas as pd
result = pd.read_html("https://pqt.cbp.gov/report/YYZ_1/12-01-2017")
df = result[0]
# df = df.drop(labels='Unnamed: 8', axis=1)
df.to_csv(r'C:\Users\User\Desktop\Data.csv', sep=',', encoding='utf-8',index = False )

edited Dec 1, 2018 at 11:07

answered Dec 1, 2018 at 10:56

QHarr

84.5k14 gold badges58 silver badges105 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Dawn17 Over a year ago

After u made the edit, the output file does not appear without an error. You were dropping the last row, right?

QHarr Over a year ago

The last column

QHarr Over a year ago

But simply exclude the 4th line above

Uku Loskit · Accepted Answer · 2018-12-01 10:31:39Z

0

try:

r = requests.get('https://pqt.cbp.gov/report/YYZ_1/12-01-2017')
soup = BeautifulSoup(r.content)

answered Dec 1, 2018 at 10:31

Uku Loskit

42.2k9 gold badges97 silver badges98 bronze badges

1 Comment

Dawn17 Over a year ago

Now I get an error on headers = [header.text for header in table.find_all('th')] AttributeError: 'NoneType' object has no attribute 'find_all''

ewwink · Accepted Answer · 2018-12-01 10:51:30Z

0

variable r is type Response not str, use r.text or r.content and there are no table with class table-horizontal-line, do you mean results ?

soup = BeautifulSoup(r.text)
table = soup.find('table', attrs={"class" : "results"})

answered Dec 1, 2018 at 10:51

ewwink

19.3k2 gold badges49 silver badges56 bronze badges

Collectives™ on Stack Overflow

Problem parsing a table from html using BeautifulSoup and saving it as a csv

3 Answers 3

3 Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related