Iterating a numpy array in chunks

Question

How can I arrange list_ into groups of numbers using the numpy.where() or the numpy.select(). I want to break the data into groups of 3 and then calculate the standard deviation std() of those functions. So the program will take in 457.334015,424.440002,394.795990 for the first values of standard dev to be calculated and the will take 424.440002,394.795990, 408.903992 and calculate the standard deviation etc. It will keep on going like this till the end it reaches the end of the list. I want the first chunk 457.334015,424.440002,394.795990 to be deleted before it calculates the second chunk 424.440002,394.795990, 408.903992. I want to delete chunks from the memory so i do not have a memory error. Would this be possible with numpy and without using a for loop.

number = 3
list_= np.array([457.334015,424.440002,394.795990,408.903992,398.821014,402.152008,435.790985,423.204987,411.574005,
404.424988,399.519989,377.181000,375.467010,386.944000,383.614990,375.071991,359.511993,328.865997,
320.510010,330.079010,336.187012,352.940002,365.026001,361.562012,362.299011,378.549011,390.414001,
400.869995,394.773010,382.556000])

Are you guaranteed that the number of values in the array will be in multiples of number=3? — Sudheesh Singanamalla
– Sudheesh Singanamalla, Commented Jan 22, 2021 at 0:19

Georgi Ganchev · Accepted Answer · 2021-01-22 00:38:45Z

1

From this post:

from numpy.lib.stride_tricks import as_strided
def strided_app(a, L, S ):  # Window len = L, Stride len/stepsize = S
    nrows = ((a.size-L)//S)+1
    n = a.strides[0]
    return np.lib.stride_tricks.as_strided(a, shape=(nrows,L), strides=(S*n,n))
list_= np.array([457.334015,424.440002,394.795990,408.903992,398.821014,402.152008,435.790985,423.204987,411.574005,
404.424988,399.519989,377.181000,375.467010,386.944000,383.614990,375.071991,359.511993,328.865997,
320.510010,330.079010,336.187012,352.940002,365.026001,361.562012,362.299011,378.549011,390.414001,
400.869995,394.773010,382.556000])
np.std(strided_app(list_, 3, 1), axis=1)

However, this code does not delete any elements from the array. Also, keep in mind that the function used here comes with a warning from the numpy docs!

answered Jan 22, 2021 at 0:38

Georgi Ganchev

1165 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Sudheesh Singanamalla Over a year ago

While this is correct, where does the rolling window approach you presented here remove the items from memory as the OP asked?

tony selcuk Over a year ago

I want to separate the list into chunks and then calculate the standard deviation one by one

Collectives™ on Stack Overflow

Iterating a numpy array in chunks

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest