Problem with creating a web scraping script with python

Question

i have a problem. I want to create a web scraping script that finds specific html tags from a website and give as a result the text inside the tags. The problem is that i cannot do that. It keeps giving me some html code and i cannot find specific html tags from the code i found earlier. Here is the code i have written:

from bs4 import BeautifulSoup as bs
import requests

url = "https://ec.europa.eu/info/strategy/priorities-2019-2024/european-green-deal_en"
page = requests.get(url).text
doc = bs(page, "html.parser")

h2 = doc.find_all("h2", id="latest")

h3 = h2.next_siblings

print(h3)

# print(list(h2[0].next_siblings))

I want to take the latest news from the europa.eu. What can i do in order to improve my code?

Anand Gautam · Accepted Answer · 2022-01-07 18:52:57Z

1

Something like this. For one thing: In order to take the latest news, I had to click on the latest link (as you can see it appended in the url). I hope this is what you are looking for - or a similar kind.

from bs4 import BeautifulSoup as bs
import requests

url = "https://ec.europa.eu/info/strategy/priorities-2019-2024/european-green-deal/delivering-european-green-deal_en#latest"
page = requests.get(url).text
doc = bs(page, "html.parser")
# print(doc.prettify())
h2 = doc.find_all("h3")
for i in h2:
    print(i.text)

Output:
European Green Deal: Commission proposes transformation of EU economy and society to meet climate ambitions
Statement by President von der Leyen on delivering the European Green Deal
Statement by Executive Vice-President Timmermans on delivering the European Green Deal
Statement by Commissioner Simson on delivering the European Green Deal
Statement by Commissioner Sinkevičius on delivering the European Green Deal
Delivering the European Green Deal

Process finished with exit code 0

answered Jan 7, 2022 at 18:52

Anand Gautam

2,1011 gold badge6 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Theologos Diamand Over a year ago

Thank you very much for your help. I want to take the link from the article too, not just the title. Can i just add print(i.href) on the loop in order to achieve that?

Collectives™ on Stack Overflow

Problem with creating a web scraping script with python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related