Python script to use data from Azure Storage Blob by stream, and update blob by stream without local file reading and uploading

Question

I have a python code for data processing , i want to use azure block blob as the data input for the code, to be specify, a csv file from block blob. its all good to download the csv file from azure blob to local path, and upload other way around for the python code if running locally, but the problem is my code running on azure virtual machine because its quite heavy for my Apple Air , pandas read_csv from local path does not work in this case, therefore i have to download and upload and update csv files to azure storage by stream without local saving. both download and upload csv are quite small volume, much less than the blob block limits

there wasn't that much tutorial to explain how to do this step by step, MS Docs are generally suck to explain as well, my minimal code are as following:

for downloading from azure blob storage

from azure.storage.blob import BlockBlobService
storage = BlockBlobService(account_name='myname', account_key = 'mykey')
#here i don't know how to make a csv stream that could could be used in next steps#
file = storage.get_blob_to_stream('accountname','blobname','stream')
df = pd.read_csv(file)
#df for later steps#

for uploading and updating a blob by a dataframe from code by stream

df #dataframe generated by code 
'i don't know how to do the preparation steps for df and the final fire up operation'
storage.put_blob_to_list _by_stream('accountname','blobname','stream')

can you please make a step by step tutorial for me, for ppl has experience to azure blob , this should be not very difficult.

or if you have better solution other than use blob for my case , please drop some hits. Thanks.

Thomas · Accepted Answer · 2018-03-26 00:03:19Z

6

So the document is still in progress, I think it is getting better and better... Useful link:

To download a file as a stream from blob storage, you can use BytesIO:

from azure.storage.blob import BlockBlobService
from io import BytesIO
from shutil import copyfileobj 
with BytesIO() as input_blob:
    with BytesIO() as output_blob:
        block_blob_service = BlockBlobService(account_name='my_account_name', account_key='my_account_key')
        # Download as a stream
        block_blob_service.get_blob_to_stream('mycontainer', 'myinputfilename', input_blob)

        # Do whatever you want to do - here I am just copying the input stream to the output stream
        copyfileobj(input_blob, output_blob)
        ...

        # Create the a new blob
        block_blob_service.create_blob_from_stream('mycontainer', 'myoutputfilename', output_blob)

        # Or update the same blob
        block_blob_service.create_blob_from_stream('mycontainer', 'myinputfilename', output_blob)

answered Mar 26, 2018 at 0:03

Thomas

30.5k6 gold badges99 silver badges143 bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

Pepin Peng Over a year ago

HI , i want to ask the Creat_blob_from Stream method , if repeatly executed without change filename, will it replace the blob with same name in the azure storage, or it will append the changes to the exist blob ? also what is the best method to solve the header lost problems when download a blob ?

Thomas Over a year ago

the create_blob_from_stream will replace the file. what do you mean by header lost problems ?

Pepin Peng Over a year ago

Hi thanks again, as far as i understand, azure blob storage does not have column headers, thats why when i upload csv to blob storage, then download, i found tthe csv lost its header, all the column names are gone.

Pepin Peng Over a year ago

also i tried use the method you provided to download a csv to string stream, i tried use read_csv method and failed because empty value error, use the copyfileobject create a new blob, its also empty, that means there is no data download into the stream , why is that happen.. why this has to be so complicate to use..

Thomas Over a year ago

It is not related to the blob storage. Blob storage stores files, any type. are you just copying the file ? My guess is that you need to copy the header also when you read the file. Is the example I gave you work ? Check the read_csv method. you may need to copy the header also

|

Collectives™ on Stack Overflow

Python script to use data from Azure Storage Blob by stream, and update blob by stream without local file reading and uploading

1 Answer 1

10 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

10 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related