Python requests send text with binary data

Question

I'm trying to send both binary data and a text string (or any other normal variable) between servers using python's requests library.

I have two servers talking to each other. Server A sends a GET request via python's requests library. It looks something like this:

color, file_data = requests.get('https://www.serverb.com/testing', params = {...})

Server B then does something like:

var1 = request.args.get('var1')
var2 = request.args.get('var2')

# This function creates a binary file, saves it to disk, and returns the file name.
res = some_function(var1, var2)

file_data = codecs.open(res, 'rb').read()

return 'Success!', file_data

However, when I attempt this, Server B throws a 500 error.

Is what I'm attempting to do impossible? If I can only send back the binary data, but with custom headers, that would work as well, and help on how to set those up would be appreciated.

Thanks in advance.

xyres · Accepted Answer · 2018-07-02 07:46:31Z

3

An HTTP response looks like this:

HTTP/1.1 200 OK
<headers>
<blank line>
<response body>

The <blank line> generally contains these characters - \r\n, also known as CRLF.

So, when a server sends a response, the client (or browser) can differentiate from the headers and the response body by looking at the <blank line>. So, anything after <blank line> is treated as the response body.

What that means is, there can be at most 1 response body only. So, even if you think you're sending color and file_data separately like you're doing, they're still going in one response and the client will treat them as a single body.

What you can do to get around this?

You'll need a way to know which part of the response body is the color and which is file_data. You can do that by sending a custom header along with the response.

Let's say, your response looks like this:

HTTP/1.1 200 OK
<headers>
<blank line>
<color><file_data>

Now, you can set a custom header on your response which will tell you how many bytes are there of the color variable.

For example, if the value of color is yellow and it's 6 characters, that means it has 6 bytes. So, you can set a custom header that would tell you the total length of color:

HTTP/1.1 200 OK
<headers>
X-Color-Length: 6
<blank line>
<color><file_data>

Then, in your client side, you can get the total bytes from the X-Color-Length header and read that number of bytes from the body to get the value of the color. Example:

response = requests.get(...)

color_len = response.headers['X-Color-Length']

color = response.content[:color_len]
file_data = response.content[color_len:]

UPDATE: Now that I think about it, I've made this a little more complex than it should be. You can just send the color's value directly in a header like X-Color: <color>, instead of the number of bytes it has.

edited Jul 2, 2018 at 7:46

answered Dec 7, 2017 at 0:06

xyres

22k5 gold badges64 silver badges89 bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

elPastor Over a year ago

xyres, thanks for the quick and detailed response. That said, the length of "color" is 5 characters, is it not? More importantly, isn't it the length of "red" that we need to know? And if so, the color might be "blue", which is 4 characters - how would the client know how many characters the return vale is?

xyres Over a year ago

@pshep123 Yes, it can be of any length. You'll have to set the header value by calculating the length of the color variable on the server.

elPastor Over a year ago

Ok, I guess the bigger question is that I'm simply using the line: return color, file_data, not building out the whole HTTP/1.1 200 OK... bit. How do I include those headers in a simple return?

xyres Over a year ago

@pshep123 Are you using any framework to generate the response?

heemayl Over a year ago

+1. But note that, while sending 6 bytes in the header makes sense, but sending too much will incur a HEAD latency -- which is not desirable.

|

Collectives™ on Stack Overflow

Python requests send text with binary data

1 Answer 1

10 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

10 Comments

Your Answer

Sign up or log in

Post as a guest

Related