Iterate over files in a folder to create numpy array

Question

this is my first posting and I am really new to programming - I have a folder with some files that I want to process and then create a numpy array with the values I need I do:

listing = os.listdir(datapath)
my_array=np.zeros(shape=(0,5))
for infile in listing:
    dataset = open(infile).readlines()[1:]
    data = np.genfromtxt(dataset, usecols=(1,6,7,8,9))
    new_array = np.vstack((my_array, data))

and although I have 2 files in listing (datapath folder) the new_array array overwrites the data and gives me only the values of the second file any ideas? thanks,

HyperCube · Accepted Answer · 2012-11-06 23:23:21Z

3

If I understand you correctly, the solution to your problem is simply that you need to vstack it to "my_array" not to a new one.

Just replace the last line with this one and it should work:

my_array = np.vstack((my_array, data))

However, I do not think this is the most efficient way to do it. Since you know how many files are in that folder, just predefine the size of the array and fill its content.

answered Nov 6, 2012 at 23:23

HyperCube

3,98010 gold badges45 silver badges54 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Dimitris Over a year ago

thanks HyperCube - that works fine (and makes me feel idiot on how simple it was) -

Dimitris Over a year ago

I actually tried to predefine an array (np.zeros) but then I failed to fill it in properly - I was either appending/adding rows, or replacing the whole array with a new one...

drmaa · Accepted Answer · 2019-10-09 10:33:10Z

2

Here is what you need to do to read all files in a numpy array from a specific folder. I have a folder test containing only .txt files. My following file.py is in the same test folder along with all .txt files. Each .txt file contains a 4x4 matrix/array. After running the script the obtained matrices will be a numpy array of [Nx4x4].

import numpy as np
from glob import glob

def read_all_files():
   file_names = glob('test/*')
   arrays = [np.loadtxt(f) for f in file_names]
   matrices = np.concatenate(arrays)

edited Oct 9, 2019 at 10:33

answered Feb 27, 2019 at 10:23

drmaa

3,74411 gold badges53 silver badges78 bronze badges

1 Comment

LayneSadler Over a year ago

if you have a Python list of ndarrays, then you can also call np.array(arrays) instead of concat

Collectives™ on Stack Overflow

Iterate over files in a folder to create numpy array

2 Answers 2

2 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related