How to access the web with Python?

Question

I want to access websites without using their API. Would i do this by using something like Mechanize?

dwich · Accepted Answer · 2022-04-28 11:15:37Z

9

Two suggested answers:

http://docs.python.org/library/httplib.html

http://docs.python.org/library/urllib.html

Good introduction is also a chapter from Dive into Python3 Chapter 13. HTTP Web Services

edited Apr 28, 2022 at 11:15

answered Jul 21, 2010 at 18:21

dwich

1,7141 gold badge15 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

user216171 Over a year ago

Ofcourse i've used Google, but i thought there were frameworks that would do all of the mundane tasks by itself. Thanks anyway.

Sadık Over a year ago

Why I hate the anwer "ask google": the answers I find on SO after asking google always want me to ask google.

Kenneth Eggering Over a year ago

@dwich, the "Chapter 11..." link in your answer is no longer available.

Cheery · Accepted Answer · 2010-07-21 18:18:48Z

8

You can access websites by HTTP protocol client: httplib

Though maybe you'd like urllib2, in specific the urllib2.urlopen

Here's little example about using urllib2:

import urllib2
page = urllib2.urlopen("http://example.com/").read()
print page

answered Jul 21, 2010 at 18:18

Cheery

25.6k16 gold badges62 silver badges83 bronze badges

Comments

Aniket · Accepted Answer · 2016-12-13 19:41:46Z

4

#for Python 3.2
import urllib.request
page = urllib.request.urlopen("http://www.google.com")
print (page.read())

answered Dec 13, 2016 at 19:41

Aniket

513 bronze badges

Comments

John Howard · Accepted Answer · 2010-07-21 18:22:42Z

3

Use urllib.request in python 3.x, and urllib2 in python 2.x

answered Jul 21, 2010 at 18:22

John Howard

64.9k23 gold badges53 silver badges67 bronze badges

Collectives™ on Stack Overflow

How to access the web with Python?

4 Answers 4

3 Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

3 Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related