Loading numpy files into memory to append arrays causing MemoryError

Question

When trying to run this:

import numpy as np
import glob

npfiles = glob.glob("*.npy")
npfiles.sort()

for file in npfiles:
    with open(file, 'rb') as f:
        array = np.load(f, allow_pickle=True)
    try:
        test
    except NameError:
        test = array
    else:
        test = np.append(test, array)
array.shape

This is the error I receive:

MemoryError: Unable to allocate 1.28 GiB for an array with shape (24115,) and data type <U14262

I'm trying to simply load several .npy files into memory and append them to one another. Their total size is only 1.58GB and I have 64GB of physical RAM.

I found that stackoverflow posts with the same problem were due to under-allocated virtual memory, and to overcome this users increased alloted virtual memory by increasing the page size maximum. Since I am using Windows 10, I increased virtual memory by doing this.

I've rebooted my PC and still experience the issue. I am baffled.

Also can you check if you have the 64bit Python and not 32bit? Another small optimization may be to check datatypes and decide if you can downcast data without losing useful information (numpy.org/devdocs/user/basics.types.html) — Equinox
– Equinox, Commented Aug 13, 2020 at 5:50
When you use np.append iteratively, you need space for at least 2 full copies. np.concatenate makes a new array, copying values from all the arguments. List append with one final np.array(...) is more efficient, though probably won't help with overall memory use in your case. — hpaulj
– hpaulj, Commented Aug 13, 2020 at 6:20
@juanpa.arrivillaga I thought for certain I had the 64 bit version installed, but it was the 32 bit version. I reinstalled the 64 bit version and it worked perfectly! — Nicholas Taylor
– Nicholas Taylor, Commented Aug 14, 2020 at 6:10

Nicholas Taylor · Accepted Answer · 2020-08-14 18:10:27Z

1

I reinstalled the 64 bit version of python because I have a 64 bit operating system and machine. This fixed the issue!

answered Aug 14, 2020 at 18:10

Nicholas Taylor

7510 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

frazb · Accepted Answer · 2020-08-13 07:33:20Z

My experience of this error has always been when I've been cavalier with the data I'm using, and reading in too much and trying to store it all at once. I'm not 100% sure from your question whether you are able to process the data per file as you load it in, or whether the files are independent.

import numpy as np
import glob

def process_my_data(data_array):
    . . . 
    # do data processing here
    return processed_data

def do_something_with_all_processed_data(my_processed_data):
    combined_arr = np.array([])
    for k in my_processed_data:
        combined_arr = np.append(processed_data[k], combined_arr)
    print(combined_arr.shape)
    
def load_and_process_files():
    npfiles = sorted(glob.glob("*.npy"))
    
    my_processed_data = dict()
    for file in npfiles:
        with open(file, 'rb') as f:
            array = np.load(f, allow_pickle=True)
        my_processed_data[file] = process_my_data(array)
    do_something_with_all_processed_data(my_processed_data)

if __name__ == "__main__":
    load_and_process_files()

In cases of large data sets, it's always nice as well to save the output of any processes in to a file which you can read again later so you don't always have to crunch through gigabytes of data.

Collectives™ on Stack Overflow

Loading numpy files into memory to append arrays causing MemoryError

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related