0

I have a numpy array of pandas Dataframes which I need to convert into a 3D numpy array of the form (samples, rows, columns) in order to feed into a Keras model for training. I have 46 samples in my dataset and each sample is 1101 rows by 64 columns.

Here is the code for my 1D numpy array of Dataframes:

static_dfs = []
#read in static csvs as pandas df
#static files is my np array of csv files
for x in range(0, static_files.size):
  df = pd.read_csv(static_files[x], sep='\t', skiprows=skip_rows, header=(0))
  #append df to list  
  static_dfs.append(df)

#convert list to np array
static_dfs = np.asarray(static_dfs)

Indeed the shape of the array is (46,) [the number of samples]. If I look at one of the Dataframes in the array (static_dfs[0] for instance) the shape is (1101, 64).

I then try to convert this to 3D numpy array:

static_nps = []

for x in range(0, static_dfs.size):
  static_nps.append(static_dfs[x].to_numpy())

#convert to numpy array
static_nps = np.asarray(static_nps)

However it gives me this error:

could not broadcast input array from shape (1101,64) into shape (1101)

for the line of code:

    #convert to numpy array
    static_nps = np.asarray(static_nps)

Worst part is I had it working before, but a collaborator of mine went through my code and edited it after we found a bug in one of our data files. Now I can't seem to get it back to working like before and am stuck :(

The desired shape of my 3D array would look like (46, 1101, 64). If anyone could solve this you would be a huge help! Thanks

4
  • 1
    Does this answer your question? How to convert a list of pandas dataframes into a 3d numpy array? Commented Jun 17, 2020 at 14:23
  • 1
    What's the shape of the arrays in static_nps. I suspect there's a mismatch. Commented Jun 17, 2020 at 15:01
  • @hpaulj I went back and printed the shapes of all of the Dataframe files and noticed after my collaborator edited the code, it cause one of the files to drop a column, leaving with the shape (1101, 63). I went back and inserted the missing columns so that the shapes match and got the code working. Thanks! Commented Jun 17, 2020 at 16:12
  • 1
    Sometimes when the shapes differ, it creates an object dtype array, but with this shape difference (in columns), it throws an error. But both cases are wrong if unintended. Commented Jun 17, 2020 at 16:33

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.