python requests upload large file with additional data

Question

I've been looking around for ways to upload large file with additional data, but there doesn't seem to be any solution. To upload file, I've been using this code and it's been working fine with small file:

with open("my_file.csv", "rb") as f:
    files = {"documents": ("my_file.csv", f, "application/octet-stream")}
    data = {"composite": "NONE"}
    headers = {"Prefer": "respond-async"}
    resp = session.post("my/url", headers=headers, data=data, files=files)

The problem is that the code loads the whole file up before sending, and I would run into MemoryError when uploading large files. I've looked around, and the way to stream data is to set

resp = session.post("my/url", headers=headers, data=f)

but I need to add {"composite": "NONE"} to the data. If not, the server wouldn't recognize the file.

Looking at the code in models.py, PreparedRequest class, prepare_body method, there's no way to use requests to stream upload large file with additional data. — Thien Nguyen
– Thien Nguyen, Commented Mar 3, 2016 at 20:28

MRSharff · Accepted Answer · 2018-09-01 08:51:54Z

18

You can use the requests-toolbelt to do this:

import requests
from requests_toolbelt.multipart import encoder

session = requests.Session()
with open('my_file.csv', 'rb') as f:
    form = encoder.MultipartEncoder({
        "documents": ("my_file.csv", f, "application/octet-stream"),
        "composite": "NONE",
    })
    headers = {"Prefer": "respond-async", "Content-Type": form.content_type}
    resp = session.post(url, headers=headers, data=form)
session.close()

This will cause requests to stream the multipart/form-data upload for you.

edited Sep 1, 2018 at 8:51

MRSharff

3823 silver badges7 bronze badges

answered Mar 3, 2016 at 22:10

Ian Stapleton Cordasco

29.2k4 gold badges73 silver badges73 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Thien Nguyen Over a year ago

Thanks so much! I remember coming across requests-toolbelt before, but didn't think of putting Content-Type into the header!

Ian Stapleton Cordasco Over a year ago

Yeah the documentation for the toolbelt pretty explicitly says you need that. :)

Collectives™ on Stack Overflow

python requests upload large file with additional data

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related