How to scrape from a Jquery table using Python

Question

I am trying to scrape the first ten items from this website. I am using Python Selenium/BeautifulSoup. It seems the table is loading using some jquery script. I am honestly stumped where to start as the tutorials and guides aren't matching up with this website.

EX: A lot of them are saying check the Network tab in inspect element to find the XHR data. This website however doesn't have anything worth value load in the XHR tab but rather in the JS tab. I found the request URl https://www.anime-planet.com/dist/3p/jquery.min.js?t=1657108207 but it doesn't seem to do me any justice.

Am I overthinking things and should scrape from the html directly? Any advice would be very appreciated.

Barry the Platipus · Accepted Answer · 2022-09-13 18:48:07Z

Here is a solution based on pandas & requests:

import requests
import pandas as pd

headers = {
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.5112.79 Safari/537.36'
}

url = 'https://www.anime-planet.com/manga/top-manga/week'

r = requests.get(url, headers=headers)
df = pd.read_html(r.text)[0]
print(df[:10])

Result printed in terminal:

	Rank	Title	Year	StatusEpsRating
0	1	The Beginning After the End	2018	Unread Reading Want to Read Stalled Dropped Won't Read
1	2	Damn Reincarnation	2022	Unread Reading Want to Read Stalled Dropped Won't Read
2	3	One Piece	1997	Unread Reading Want to Read Stalled Dropped Won't Read
3	4	Omniscient Reader	2020	Unread Reading Want to Read Stalled Dropped Won't Read
4	5	The Swordmaster's Son	2022	Unread Reading Want to Read Stalled Dropped Won't Read
5	6	My School Life Pretending To Be a Worthless Person	2022	Unread Reading Want to Read Stalled Dropped Won't Read
6	7	Player Who Returned 10,000 Years Later	2022	Unread Reading Want to Read Stalled Dropped Won't Read
7	8	Jujutsu Kaisen	2018	Unread Reading Want to Read Stalled Dropped Won't Read
8	9	Grim Reaper's Floating Moon	2022	Unread Reading Want to Read Stalled Dropped Won't Read
9	10	Villains Are Destined to Die	2020	Unread Reading Want to Read Stalled Dropped Won't Read

Relevant pandas docs: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_html.html

And for requests: https://requests.readthedocs.io/en/latest/

code · Accepted Answer · 2022-09-13 19:25:20Z

0

This table is NOT loaded from jQuery. It is server-rendered and easily scrapable. You only need requests and beautifulsoup; Selenium is unnecessary.

With some quick DOM inspection, this should be pretty simple. You can do something like this:

import requests
from bs4 import BeautifulSoup

# send HTTP request to page and parse
page = requests.get('https://www.anime-planet.com/manga/top-manga/week')
soup = BeautifulSoup(page.text, 'html.parser')

top_ten = []
count = 0

for link in soup.select('td.tableTitle > a.tooltip'):
  if count == 10:
    break
  top_ten.append(link.getText())
  count += 1

print(top_ten)

edited Sep 13, 2022 at 19:25

answered Sep 13, 2022 at 18:44

code

6,3994 gold badges22 silver badges47 bronze badges

2 Comments

moosepowa Over a year ago

It works like a charm! To confirm, does the 'td.tableTitle > a.tooltip' phrase look for a td with the tableTitle class and then the a with the class tooltip? If so how would I expand upon this to get the href? I would like to include a description for the manga so I would assume going to the url/href would be the best bet?

code Over a year ago

@moosepowa td.tableTitle > a.tooltip looks for every td with the tableTitle class who and finds its children of tag a with class tooltip, so yes. Getting the href would be easy from this point: in the loop you can access it with link.get('href').

Collectives™ on Stack Overflow

How to scrape from a Jquery table using Python

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related