Python HTTP Request Binary Data using Requests

Question

The following code only works in curl. It would be nice if you could tell me why it isnt working in Python using Requests

curl 'http://cdcnepal.com/Modules/HOmeMoviesLists/WebService2.asmx/GetShowsByDate' \
 -H 'Content-Type: application/json; charset=UTF-8' \
 -d '{"portalId":"1","showDate":"26/05/2014","flag":0,"size":9}'

However in Python with the following code

import requests
import json
url = """http://cdcnepal.com/Modules/HOmeMoviesLists/WebService2.asmx/GetShowsByDate"""
headers = {"content-type":["application/json", "charset=UTF-8"]}
payload = {"portalId":"1","showDate":"26/05/2014","flag":0,"size":9}
r = requests.get(url, headers=headers, data=payload)
print r.text

Originally the curl request had other content, below, however I realised I could remove several. I'm not sure that is causing the error because the curl request is working. I'm not getting the same response from both the code.

This might be useful. A Curl Requests extracted from Chrome Dev Tools

curl 'http://cdcnepal.com/Modules/HOmeMoviesLists/WebService2.asmx/GetShowsByDate'
 -H 'Cookie: OriginalReferrer=https://www.google.com/;
     OriginalURL=http://cdcnepal.com/;
     ASP.NET_SessionId=i5lbnql5hpp0wm1ywyqbywtj;
     VisitCount=4' 
 -H 'Origin: http://cdcnepal.com' 
 -H 'Accept-Encoding: gzip,deflate,sdch' 
 -H 'Accept-Language: en-US,en;q=0.8,hi;q=0.6' 
 -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3)
     AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.114 Safari/537.36' 
 -H 'Content-Type: application/json; charset=UTF-8' 
 -H 'Accept: application/json, text/javascript, */*; q=0.01' 
 -H 'Referer:http://cdcnepal.com/Home.aspx' 
 -H 'X-Requested-With: XMLHttpRequest' -H 'Connection: keep-alive' 
 -H 'DNT: 1' 
 --data-binary '{"portalId":"1","showDate":"27/05/2014","flag":0,"size":9}' --compressed

Martijn Pieters · Accepted Answer · 2014-12-07 02:43:05Z

15

The curl -d switch sends a POST request, but you are using requests.get() instead, sending a GET request (whose body is ignored).

Make it a POST instead, by using request.post():

import requests
import json

url = "http://cdcnepal.com/Modules/HOmeMoviesLists/WebService2.asmx/GetShowsByDate"
headers = {"content-type": "application/json; charset=UTF-8"}
payload = {"portalId":"1","showDate":"26/05/2014","flag":0,"size":9}
r = requests.post(url, headers=headers, data=json.dumps(payload))
print r.text

You also need to:

not use a list for the content-type header, there is no support for paramaters being specified separately.
Encode your JSON data to a JSON string; requests doesn't do this for you. Instead, a dictionary passed to data is encoded as application/x-www-form-urlencoded data instead.

You can compare the curl command with requests more easily using http://httpbin.org/post:

$ curl http://httpbin.org/post \
>  -H 'Content-Type: application/json; charset=UTF-8' \
>  -d '{"portalId":"1","showDate":"26/05/2014","flag":0,"size":9}'

{
  "args": {},
  "data": "{\"portalId\":\"1\",\"showDate\":\"26/05/2014\",\"flag\":0,\"size\":9}",
  "files": {},
  "form": {},
  "headers": {
    "Accept": "*/*",
    "Connection": "close",
    "Content-Length": "58",
    "Content-Type": "application/json; charset=UTF-8",
    "Host": "httpbin.org",
    "User-Agent": "curl/7.30.0",
    "X-Request-Id": "78d7bb7d-e29b-482b-908a-48d2395a050f"
  },
  "json": {
    "flag": 0,
    "portalId": "1",
    "showDate": "26/05/2014",
    "size": 9
  },
  "origin": "84.92.98.170",
  "url": "http://httpbin.org/post"
}

and

>>> import requests
>>> import json
>>> from pprint import pprint
>>> url = 'http://httpbin.org/post'
>>> headers = {"content-type":"application/json; charset=UTF-8"}
>>> payload = {"portalId":"1","showDate":"26/05/2014","flag":0,"size":9}
>>> r = requests.post(url, headers=headers, data=json.dumps(payload))
>>> pprint(r.json())
{u'args': {},
 u'data': u'{"portalId": "1", "flag": 0, "size": 9, "showDate": "26/05/2014"}',
 u'files': {},
 u'form': {},
 u'headers': {u'Accept': u'*/*',
              u'Accept-Encoding': u'gzip, deflate, compress',
              u'Connection': u'close',
              u'Content-Length': u'65',
              u'Content-Type': u'application/json; charset=UTF-8',
              u'Host': u'httpbin.org',
              u'User-Agent': u'python-requests/2.2.1 CPython/2.7.6 Darwin/13.1.0',
              u'X-Request-Id': u'06d6b542-c279-4898-8701-2c0d502aa36e'},
 u'json': {u'flag': 0,
           u'portalId': u'1',
           u'showDate': u'26/05/2014',
           u'size': 9},
 u'origin': u'84.92.98.170',
 u'url': u'http://httpbin.org/post'}

Both cases show the same json dictionary being returned.

If you are using requests version 2.4.2 or newer, you can also leave the JSON encoding to the library; it'll set the correct Content-Type header too, if you pass in the data to send as the json keyword argument:

import requests

url = "http://cdcnepal.com/Modules/HOmeMoviesLists/WebService2.asmx/GetShowsByDate"
payload = {"portalId":"1","showDate":"26/05/2014","flag":0,"size":9}
r = requests.post(url, json=payload)
print r.text

edited Dec 7, 2014 at 2:43

answered May 26, 2014 at 15:45

Martijn Pieters

1.1m326 gold badges4.2k silver badges3.4k bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

ShapNepal Over a year ago

I tried what you said but I'm getting a error [{"Message":"There was an error processing the request.","StackTrace":"","ExceptionType":""}] Even tried adding cUrl's user agent didnt work.. anything else I can try out?

Martijn Pieters Over a year ago

@ShapNepal: I get the exact same JSON response for your URL as the CURL response. It works for me.

Martijn Pieters Over a year ago

@ShapNepal: make sure your code matches mine in my answer; if I copy and paste the code at the start of my answer into a Python interactive interpreter it prints out a full JSON response.

ShapNepal Over a year ago

Yes!!, it's working in iPython, funny, I was using Sublime Text 3 and it gave me a error. Thanks guys!

Collectives™ on Stack Overflow

Python HTTP Request Binary Data using Requests

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related